System

Abstract

To provide a system for providing a user with consistent storytelling experiences.SOLUTION: Provided is a system comprising: means for recognizing a user's emotion; means for generating a story using prompt sentences to instruct generation of the story according to the user's emotion and a generative AI; means for generating sound effects and music according to the generated story; means for reading the generated story with text reading function; and means for reproducing the generated sound effects and music according to the story being read.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional picture book creation systems are unable to generate stories, sound effects, music, and storytelling in a unified manner, and they also lack the ability to print them on paper, take photos of the storytelling, and save them as memories.Furthermore, they lack the ability to combine existing stories to create new stories. [Means for solving the problem]

[0005] The system of this invention uses generative AI-based AI to automatically generate elements such as sound effects, music, and storytelling, and then creates a story based on these. It also has the ability to print the created story on paper, take photos of the storytelling, and save them as memories. It also has the ability to combine existing stories to create new stories, thereby solving the problems of the past. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13]FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the disclosed technique will be described below with reference to the accompanying drawings.

[0008] do.

[0009] First, the terms used in the following description will be explained.

[0010] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Also, the processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit),

[0011] Examples include a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment]

[0017] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0018] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0019] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0020] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0021] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0022] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0023] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0024] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0025] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0026] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0027] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0028] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0029] "Example 1"

[0030] The system of the present invention uses generative AI-based AI to automatically generate elements such as sound effects, music, and storytelling. Specifically, the AI ​​generates a story based on user input,

[0031] The AI ​​will generate sound effects and music to go along with the story, and will also generate a reading of the story, all of which will be combined to create a picture book.

[0032] "Example 2"

[0033] The system of this invention has the function of printing the created picture book on paper. Specifically, it digitally creates a picture book by combining an AI-generated story, sound effects, music, and storytelling, and then prints the picture book on paper. This allows users to obtain a physical picture book.

[0034] "Example 3"

[0035] The system of the present invention has the function of recording the reading aloud and saving it as a memory. Specifically, the system records the user reading a picture book on video and saves the video as digital data. This allows the user to play back the reading aloud later and keep it as a memory.

[0036] "Example 4"

[0037] The system of the present invention has the function of combining existing stories to create new stories. Specifically, the AI ​​combines existing stories, such as "Momotaro" and "Kintaro," to create new stories, such as "Momotaro vs. Kintaro." This allows users to enjoy new stories based on existing stories.

[0038] The processing flow of each embodiment will be described below.

[0039] "Example 1"

[0040] Step 1: The user inputs the story's theme, characters, etc. into the system.

[0041] Step 2: Generative AI-based AI generates a story based on user input.

[0042] Step 3: Generate sound effects and music to match the AI-generated story.

[0043] Step 4: The AI ​​generates a story reading.

[0044] Step 5: Combine them all to create a digital picture book.

[0045] "Example 2"

[0046] Step 1: Create a digital picture book by combining an AI-generated story, sound effects, music, and storytelling.

[0047] Step 2: The system prints the digitally created picture book onto paper.

[0048] "Example 3"

[0049] Step 1: The user reads the book aloud.

[0050] Step 2: The system records a video of the reading.

[0051] Step 3: Save the captured video as digital data.

[0052] "Example 4"

[0053] Step 1: The AI ​​selects an existing story, such as "Momotaro" or "Kintaro."

[0054] Step 2: Combine the selected stories to create a new story, for example, "Momotaro vs. Kintaro."

[0055] Example 1

[0056] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0057] Conventional story creation systems require users to manually create stories and add sound effects, music, and audio recordings individually, which requires a great deal of time and effort. Furthermore, the process of integrating these elements into a single picture book is cumbersome, making it difficult to automatically generate picture books, especially for children.

[0058] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0059] In this invention, the server includes means for a user to input a prompt sentence, means for generating a story based on the prompt sentence using a generative AI model, means for generating sound effects based on the generated story, means for generating music based on the generated story, means for generating a storytelling audio based on the generated story, means for creating a picture book by combining the story, sound effects, music, and storytelling audio, and means for a user to view the picture book on a terminal. This allows a user to easily create a story, automatically add sound effects, music, and storytelling audio, and integrate them into a single picture book.

[0060] A "prompt sentence" is text that the user inputs to the system, including instructions about the theme, characters, etc. of the story.

[0061] A "generative AI model" is an artificial intelligence algorithm that automatically generates a story based on a prompt from the user.

[0062] A "story" is text data generated by a generative AI model based on a prompt sentence.

[0063] "Sound effects" are sound effects generated to match the scenes in the story.

[0064] "Music" is background music generated to match the atmosphere of the story.

[0065] "Reading audio" is audio data generated from the story text using speech synthesis technology.

[0066] A "picture book" is an e-book format content created by combining a story, sound effects, music, and audio reading.

[0067] A "terminal" is a device through which a user accesses the system, inputs prompt sentences, and views the generated picture book.

[0068] This invention relates to a system that uses a generative AI model to automatically generate a story, sound effects, music, and reading audio, and combines them to create a picture book. Specific embodiments of this system are described below.

[0069] A user accesses the system using a terminal and inputs a prompt statement such as the theme of the story, characters, etc. For example, the user might input, "Create an adventure story about a brave boy who fights a dragon in a magical forest."

[0070] The server receives a prompt sentence from the user and generates a story based on the prompt sentence using a generative AI model (e.g., OpenAI's GPT-4®). The generated story is stored as text data on the server.

[0071] The server then generates sound effects based on the story, using audio libraries such as the Freesound API to select sound effects that match each scene in the story. For example, for a scene featuring a dragon, the server selects the sound of a dragon roaring.

[0072] In addition, the server generates music that matches the atmosphere of the story. For music generation, it uses music generation AI such as Magenta. For example, exciting music is generated for adventure scenes, and calm music is generated for quiet scenes.

[0073] The server also uses speech synthesis technology, such as the Google® Text-to-Speech API, to convert the story text into audio data, which generates an audio reading of the story.

[0074] Finally, the server combines the generated story, sound effects, music, and audio reading to create a picture book, which is saved as a PDF or e-book and made available for download by users.

[0075] Users can use their devices to download and view picture books provided by the server. While reading the picture books, users can enjoy sound effects, music, and audio recordings of the story being told.

[0076] The system allows users to easily create stories, automatically add sound effects, music, and audio readings, and integrate them into a single picture book.

[0077] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0078] Step 1:

[0079] The user uses a terminal to input prompts into the system, such as the theme and characters of the story. For example, the user might input "Please create an adventure story about a brave boy battling a dragon in a magical forest." The input prompt is then sent to the server.

[0080] Step 2:

[0081] The server receives the prompt sentence and sends it as input to a generative AI model (e.g., OpenAI's GPT-4). The generative AI model generates a story based on the prompt sentence and returns the text data to the server. The server saves the generated story as text data.

[0082] Step 3:

[0083] The server analyzes the generated story text and generates sound effects appropriate for each scene. For example, in a scene where a dragon appears in the story, the server uses the Freesound API to obtain the dragon's roar. The input is the story text data, and the output is the audio data of the sound effects.

[0084] Step 4:

[0085] The server generates music that matches the atmosphere of the story. For music generation, it uses music generation AI such as Magenta. For example, exciting music is generated for adventure scenes, and calm music is generated for quiet scenes. The input is the text data of the story, and the output is the audio data of the music.

[0086] Step 5:

[0087] The server uses speech synthesis technology such as the Google Text-to-Speech API to convert the story text into audio data, which then generates a reading of the story. The input is the text data of the story, and the output is the audio data of the reading.

[0088] Step 6:

[0089] The server combines the generated story, sound effects, music, and reading audio to create a picture book. The picture book is saved in PDF or e-book format and can be downloaded by users. The input is the text data of the story, audio data of the sound effects, audio data of the music, and audio data of the reading audio, and the output is the integrated picture book data.

[0090] Step 7:

[0091] Users can use their devices to download and view picture books provided by the server. While reading the picture book, users can enjoy sound effects, music, and audio recordings. The input is the picture book data from the server, and the output is the user's viewing experience.

[0092] (Application example 1)

[0093] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] Conventional story generation systems can generate stories based on a theme entered by the user, but they lack the functionality to automatically generate sound effects, music, and storytelling audio and combine them to create a digital picture book. Furthermore, they lack the means to make the generated digital picture book viewable on a smartphone, limiting the user experience. This poses a challenge for users, making it difficult to easily create and enjoy high-quality digital picture books.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0096] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on them, means for digitally saving the created story, means for generating a story based on user input, means for generating sound effects and music to match the generated story, means for generating audio for storytelling, means for creating a digital picture book by combining the generated story, sound effects, music, and storytelling audio, and means for making the created digital picture book viewable on a smartphone. This allows high-quality digital picture books to be automatically generated based on a theme entered by the user, and can be easily viewed and shared on a smartphone.

[0097] "Generative AI-based AI" is an artificial intelligence technology that automatically generates stories, sound effects, music, reading audio, and more based on user input.

[0098] "Sound effects" are sound effects generated to match the scenes and situations in the story.

[0099] "Music" is music generated to enhance the atmosphere and emotion of the story.

[0100] "Read-aloud audio" is audio data for reading the generated story aloud.

[0101] A "narrative" is a story that is generated based on input from the user.

[0102] "Digital format" means a data format that is stored and displayed electronically.

[0103] "User input" refers to information such as keywords and themes that a user provides to the system.

[0104] A "digital picture book" is an e-book format picture book created by combining generated stories, sound effects, music, and audio readings.

[0105] A "smartphone" is a portable information terminal that can connect to the Internet and run applications.

[0106] "Viewable" means that the user can visually check the digital picture book.

[0107] The system for implementing this invention uses generative AI-based AI to automatically generate stories, sound effects, music, and storytelling audio based on user input, and combines them to create a digital picture book. Specific embodiments of this system are described below.

[0108] System configuration

[0109] 1. Hardware Configuration

[0110] Server: A high-performance computer (e.g., a server equipped with a GPU) for running generative AI models.

[0111] User device: A device such as a smartphone or tablet that allows users to input and view the generated digital picture book.

[0112] 2. Software Configuration

[0113] Generative AI models: AI models for generating stories, sound effects, music, and storytelling audio (e.g., OpenAI's GPT-4).

[0114] Database: A database for storing the generated digital storybooks and user input.

[0115] Application: A smartphone application that allows users to enter input and view the generated digital picture book.

[0116] Processing flow

[0117] 1. User Input

[0118] Users enter the story's theme and keywords through a smartphone application.

[0119] Example: "A story of adventure and friendship"

[0120] 2. Narrative Generation

[0121] The server uses a generative AI model to generate a story based on user input.

[0122] Example prompt: "Generate a story of adventure and friendship."

[0123] 3. Sound Effects and Music Generation

[0124] The server generates sound effects and music to accompany the generated story.

[0125] Example of a sound effect generation prompt: "Generate a sound effect that matches the following story:\n\n(generated story)"

[0126] Example of a music generation prompt: "Generate music to fit the following story: (generated story)"

[0127] 4. Generation of reading audio

[0128] The server generates audio to read the generated story aloud.

[0129] Example prompt for storytelling audio generation: "Generate audio to read the following story: (generated story)"

[0130] 5. Creating a digital picture book

[0131] The server combines the generated story, sound effects, music, and reading audio to create a digital picture book.

[0132] 6. Viewing digital picture books

[0133] Users can view the generated digital picture book through a smartphone application.

[0134] Specific examples

[0135] When a user inputs the theme "A Story of Adventure and Friendship," the server uses a generative AI model to generate a story. It then generates sound effects and music to match the story, and generates audio to read the story. These elements are combined to create a digital picture book, which users can then view on their smartphones.

[0136] In this way, users can easily create and enjoy high-quality digital picture books.

[0137] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0138] Step 1:

[0139] The user starts the smartphone application and inputs the theme and keywords of the story. The input theme and keywords are sent to the server. The input data is used as prompts for generating the story.

[0140] Step 2:

[0141] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate a story based on themes and keywords submitted by the user. Specifically, the server inputs the prompt "Generate a story of adventure and friendship" into the generative AI model and outputs the story text.

[0142] Step 3:

[0143] The server generates sound effects based on the generated story text. It inputs the prompt "Generate sound effects that fit the following story:\n\n(generated story)" into the generative AI model and outputs a description of the sound effects. The description of the sound effects is used to retrieve appropriate sound effect files from the sound effect database.

[0144] Step 4:

[0145] The server generates music based on the generated story text. It inputs the prompt "Generate music that matches the following story: (generated story)" into the generative AI model and outputs a music description. The music description is used to retrieve appropriate music files from a music database.

[0146] Step 5:

[0147] The server generates audio to read the generated story text aloud. The prompt "Please generate audio to read the following story aloud:\n\n(generated story)" is input into the generative AI model, and the reading audio data is output.

[0148] Step 6:

[0149] The server combines the generated story text, sound effects, music, and reading audio to create a digital picture book, which includes generating a script to play each element at the appropriate timing and creating a digital picture book file.

[0150] Step 7:

[0151] The server sends the created digital picture book file to the user's smartphone. The user can view the created digital picture book through a smartphone application. The digital picture book is displayed in an integrated format, including text, sound effects, music, and audio recordings.

[0152] Example 2

[0153] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0154] Conventional picture book creation systems require users to manually create stories, sound effects, music, and audio for storytelling, which takes time and effort. Furthermore, the content and quality of the stories are not consistent, which can lead to low user satisfaction. Furthermore, the time and cost required to print the stories on paper is also an issue.

[0155] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0156] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative artificial intelligence and creating a story based on these, means for generating a story based on a prompt sentence entered by the user, means for generating sound effects, music, and storytelling audio based on the generated story, means for creating a digital picture book by combining these elements, means for sending the created digital picture book to a terminal, means for the terminal to display the digital picture book and for the user to check it, means for the user to instruct printing, means for the terminal to print the picture book on paper using a printer, and means for capturing footage of the storytelling and saving it as a memory. This allows users to easily create picture books containing high-quality stories and obtain them in physical form.

[0157] "Generative AI" is an AI technology that automatically generates elements such as stories, sound effects, music, and reading audio based on prompts entered by the user.

[0158] A "prompt" is an instruction that a user inputs to a generative AI, including details about the story's theme, characters, etc.

[0159] A "narrative" is a story or narrative that a generative artificial intelligence generates based on a prompt.

[0160] "Sound effects" are sound effects that are generated according to the scenes and situations in the story.

[0161] "Music" refers to musical compositions and melodies generated to enhance the atmosphere and emotion of the story.

[0162] "Read-aloud audio" refers to audio data generated to read aloud the generated story.

[0163] A "digital picture book" is an electronic picture book created by combining generated stories, sound effects, music, and audio readings.

[0164] A "terminal" is an electronic device that a user uses to check a digital picture book and instruct it to be printed.

[0165] A "printer" is a printing device for printing digital picture books on paper.

[0166] "Reading situation" refers to the situation or scene when the user reads the generated story aloud.

[0167] "Means for preserving memories" refers to methods or devices for recording the reading aloud and saving it for later playback.

[0168] This invention is a system that uses generative artificial intelligence to automatically generate stories, sound effects, music, and audio for storytelling, combines them to create a digital picture book, and finally prints it on paper. Specific embodiments of this system are described below.

[0169] The server uses generative AI to generate a story based on the prompt entered by the user. For example, a model using natural language processing technology (e.g., GPT-4) is used as the generative AI. When the user enters a prompt, the server analyzes it and generates a story.

[0170] The server then generates sound effects, music, and audio based on the story. Sound effects and music are generated using text-to-speech software (e.g., Google Cloud Text-to-Speech). Audio for the story is also generated.

[0171] The server combines the generated story, sound effects, music, and audio reading to create a digital picture book, which is then saved in a format that can be viewed by the user.

[0172] The server then sends the created digital picture book to the user's terminal. The terminal receives the digital picture book sent from the server and displays it for the user to review. The user can review the displayed digital picture book and make corrections as necessary.

[0173] If the user is satisfied with the content of the digital picture book, they can request printing. The request is made through the terminal interface. Upon receiving the user's request, the terminal uses a printer (e.g., an inkjet printer) to print the digital picture book onto paper. The printed picture book is then provided to the user in physical form.

[0174] As a concrete example, consider a scenario where a user inputs a prompt such as "Please create a story about animals having an adventure in the forest. The main characters are a rabbit, a fox, and a bear." The server receives this prompt and uses generative artificial intelligence to generate the following story:

[0175] "One day, Lily the rabbit, Fred the fox, and Ben the bear embark on a great adventure in the forest. They make new friends, overcome obstacles, and uncover the secrets of the forest."

[0176] The server then generates sound effects and music to accompany the story, and creates audio for the story being read aloud. These elements are then combined to create a digital picture book, which is then sent to the device. When the user reviews the digital book and requests printing, the device uses a printer to print the book onto paper, giving the user a physical copy.

[0177] In this way, users can easily create picture books containing high-quality stories and have them available in physical form.

[0178] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0179] Step 1:

[0180] The user enters a prompt statement.

[0181] The user accesses the system's interface and inputs a prompt sentence including details such as the theme and characters of the story. For example, the user might input, "Please create a story about animals having an adventure in the forest. The main characters are a rabbit, a fox, and a bear." The input prompt sentence is then sent to the server.

[0182] Step 2:

[0183] The server receives the prompt and generates a story using a generative AI model.

[0184] The server receives the prompt sent by the user. Then, it uses a generative AI model (e.g., a model using natural language processing technology) to generate a story based on the prompt. The generated story is saved as text data. The input is the prompt, and the output is the generated story.

[0185] Step 3:

[0186] The server generates sound effects, music, and storytelling audio based on the generated story.

[0187] The server uses speech synthesis software (e.g., text-to-speech technology) to analyze the generated story and generate appropriate sound effects and music, as well as audio for reading the story aloud. The input is the generated story, and the output is sound effects, music, and audio for reading the story aloud.

[0188] Step 4:

[0189] The server combines these elements to create a digital picture book.

[0190] The server combines the generated story, sound effects, music, and audio reading to create a digital picture book. This digital picture book is saved in a format that can be viewed by the user. The input is the story, sound effects, music, and audio reading, and the output is the digital picture book.

[0191] Step 5:

[0192] The server sends the digital picture book to the terminal.

[0193] The server sends the created digital picture book to the user's terminal. The sent digital picture book is displayed on the terminal so that the user can check it. The input is the digital picture book, and the output is the digital picture book displayed on the terminal.

[0194] Step 6:

[0195] The terminal displays the digital picture book for the user to check.

[0196] The terminal receives the digital picture book sent from the server and displays it for the user to check. The user can check the displayed digital picture book and instruct corrections as necessary. The input is the digital picture book, and the output is the user's confirmation and correction instructions.

[0197] Step 7:

[0198] The user issues a print instruction.

[0199] When the user is satisfied with the content of the digital picture book, he or she instructs the book to be printed. The printing instruction is given through the terminal interface. The input is the user's print instruction, and the output is the transmission of the print instruction.

[0200] Step 8:

[0201] The device uses a printer to print the picture book onto paper.

[0202] The terminal receives a print instruction from the user and prints the digital picture book on paper using a printer (e.g., an inkjet printer). The printed picture book is provided to the user in a physical form. The input is the print instruction, and the output is the picture book printed on paper.

[0203] (Application example 2)

[0204] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0205] In conventional picture book creation systems, users could freely select story themes and characters, combine the generated story with sound effects, music, and storytelling to create a digital picture book, and then print the book on paper. Furthermore, there was a lack of a way for users to easily save the picture books they created and obtain them in physical form. This prevented users from fully expressing their creativity and enjoying the fun of creating picture books.

[0206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0207] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on them, means for printing the created story on paper, means for filming the storytelling and saving it as a memory, means for providing an interface for the user to select the theme and characters of the story, means for creating a digital picture book by combining the generated story with the sound effects, music, and storytelling, and means for saving the created digital picture book in PDF format so that the user can print it on paper using a printer.This allows the user to freely select the theme and characters of the story, create a digital picture book by combining the generated story with the sound effects, music, and storytelling, and print it on paper to obtain it in physical form.

[0208] "Generative AI-based AI" is artificial intelligence that automatically generates specific data or content using generative models.

[0209] "Sound effects" are sound effects used to emphasize particular scenes or actions in a story or film.

[0210] "Music" is an art form of sound created by combining elements such as melody, harmony, and rhythm.

[0211] "Reading aloud" is the act of reading a story or text aloud, especially for the education or entertainment of children.

[0212] A "narrative" is a form of writing or image that describes a series of events composed of characters, events, and settings.

[0213] A "means for printing on paper" is a device or method for transferring digital data onto a physical paper medium.

[0214] "Means for filming the reading aloud" refers to devices or methods for recording the act of reading aloud as video or audio.

[0215] A "memory storage device" is a device or method for recording and storing specific events or actions in a form that can be later reproduced.

[0216] An "interface that allows the user to select the story theme and characters" is an operation screen or input device that allows the user to select the story content and characters.

[0217] A "digital picture book" is a picture book created in digital format that includes elements such as text, images, and audio.

[0218] "Means for saving in PDF format" refers to a device or method for saving digital data in Portable Document Format (PDF).

[0219] A "printer" is a device for printing digital data onto a physical medium such as paper.

[0220] A system for implementing this invention includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on these elements; means for printing the created story on paper; means for filming the storytelling and saving it as a memory; means for providing an interface for users to select the theme and characters of the story; means for combining the generated story with the sound effects, music, and storytelling to create a digital picture book; and means for saving the created digital picture book in PDF format so that the user can print it on paper using a printer.

[0221] Hardware and software used

[0222] Hardware: Smartphone, tablet, printer, internet connection

[0223] Software: Python, TENSORFLOW (registered trademark) (implementation of generative AI models), Flask (web application framework), Firebase (database and authentication), PIL (Python Imaging Library, for image processing)

[0224] Program processing

[0225] Building the user interface

[0226] Users use the touchscreen of their smartphone or tablet to navigate an interface that allows them to select themes and characters for their stories, such as "fairy tales," "animal adventures," and "space exploration."

[0227] Running generative AI models

[0228] The server inputs prompts into the generative AI model based on the theme selected by the user, and generates a story, sound effects, music, and storytelling. For example, if the prompt is "Generate a story about a fairy tale theme in which a brave prince defeats a dragon," the AI ​​will generate a story.

[0229] Creating a digital picture book

[0230] The server combines the generated story with sound effects, music, and storytelling to create a digital storybook. It uses PIL to generate pages that combine images and text.

[0231] Implementing the printing function

[0232] The server saves the digital book in PDF format so that users can print it on paper using a printer, and uses Flask to provide a web interface where users can download the PDF.

[0233] Specific examples

[0234] Prompt Sentence Examples

[0235] "Generate a fairy tale story about a brave prince slaying a dragon."

[0236] Example of a generated story

[0237] "Once upon a time, there was a brave prince. He set out on a journey to save his village from a dragon. With his wisdom and courage, the prince defeated the dragon and saved his village."

[0238] Digital picture book page example

[0239] Page 1: "Once upon a time, there was a brave prince." (Illustration of the prince)

[0240] Page 2: "He set out on a journey to save a village that had been attacked by a dragon." (Illustration of a travel scene)

[0241] Page 3: "The prince used his wisdom and courage to defeat the dragon and save the village." (Illustration of a dragon battle scene)

[0242] In this way, users can easily create their own original picture books and enjoy them by saving them digitally or printing them out on paper.

[0243] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0244] Step 1:

[0245] The user operates the interface of a smartphone or tablet to select the story's theme and characters.

[0246] Input: User-selected theme and character information

[0247] Output: Data on selected themes and characters

[0248] Specific operation: The user operates the touchscreen to select a theme such as "Fairy Tales," "Animal Adventures," or "Space Exploration." The selected information is sent from the device to the server.

[0249] Step 2:

[0250] Based on the theme and character information received by the server, prompts are input into the generative AI model, which then generates a story, sound effects, music, and reading aloud.

[0251] Input: User-selected theme, character data, prompt text

[0252] Output: Generated story, sound effects, music, and storytelling data

[0253] How it works: The server uses TensorFlow to generate a story using the prompt "Generate a fairy tale-themed story about a brave prince slaying a dragon" and outputs the generated story along with associated sound effects, music, and storytelling data.

[0254] Step 3:

[0255] The server combines the generated story with sound effects, music, and storytelling to create a digital picture book.

[0256] Input: Generated story, sound effects, music, and storytelling data

[0257] Output: Digital picture book data

[0258] What it does: The server uses PIL to create pages that combine the generated story text with corresponding images, with each page containing story text and related illustrations.

[0259] Step 4:

[0260] The server saves the created digital picture book in PDF format so that users can print it on paper using a printer.

[0261] Input: Digital picture book data

[0262] Output: Digital picture book in PDF format

[0263] What it does: The server uses Flask to provide a web interface that allows users to download PDFs. Users download the PDFs through this interface and print them on a printer.

[0264] Step 5:

[0265] Users can take photos of the reading aloud and save them as memories.

[0266] Input: Video and audio data of a user reading aloud

[0267] Output: Stored video and audio data

[0268] How it works: Users use the camera on their smartphone or tablet to record themselves reading aloud, and the recorded video and audio are saved on the device and saved in a format that can be played back later.

[0269] Example 3

[0270] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0271] Conventional systems for recording reading aloud involve the cumbersome process of saving and playing back video data, making it difficult for users to easily save and play back their memories. Furthermore, they lack the functionality to generate new stories by combining existing stories, limiting users' opportunities to enjoy new stories. Furthermore, there is no way to print the generated stories on paper, making it impossible to physically store the stories. To address these issues, the present invention aims to provide a system that can easily save and play back reading aloud, and a function to generate new stories by combining existing stories.

[0272] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0273] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using a generative AI-based AI and creating a story based on them, means for printing the created story on paper, means for recording the storytelling and saving it as a memory, means for uploading the recorded video data to cloud storage, means for receiving the video data from the cloud storage and saving it in a database, means for playing the saved video data, means for inputting an existing story and inputting a prompt sentence to the generative AI model, means for the generative AI model to generate a new story, and means for saving the generated story and providing it to the user. This allows users to easily save and play back storytelling, as well as to combine existing stories to create and enjoy new stories.

[0274] "Generative AI-based AI" is an artificial intelligence technology that automatically generates new data and content based on existing data.

[0275] "Sound effects" are sound effects used to give a sense of realism to a story or image.

[0276] "Music" is the art of sound created by combining elements such as melody, harmony, and rhythm.

[0277] "Reading aloud" is the act of reading books or stories aloud, often to children.

[0278] A "narrative" is a story that unfolds through events and the actions of characters.

[0279] "Printing to paper" is the process of transferring digital data onto a physical paper medium.

[0280] "Filming" is the act of recording footage using a camera or video camera.

[0281] "Saving as a memory" means recording a particular event or experience so that it can be replayed later.

[0282] "Cloud storage" is a collection of remote servers that store and access data over the Internet.

[0283] A "database" is a system for efficiently storing, managing, and retrieving data.

[0284] "Playback" refers to the act of displaying or playing back stored audio or video data.

[0285] An "existing narrative" is a story that has already been created and is widely known.

[0286] A "generative AI model" is an algorithm or model that uses artificial intelligence to generate new data or content.

[0287] A "prompt sentence" is an input sentence that gives specific instructions or questions to a generative AI model.

[0288] A "new story" is a new story generated based on existing stories and elements.

[0289] "Providing to users" means making the generated data and content available to users.

[0290] This invention provides a system that records the reading session and saves it as a memory, and a system that combines existing stories to create a new story. Specific embodiments of this system are described below.

[0291] First, the user uses a device such as a smartphone or tablet to record a video of the picture book being read aloud. Specifically, the user launches the smartphone's camera app and presses the record button to record the reading. Once the recording is complete, the device uploads the video data to a cloud storage service (e.g., Google Drive or Dropbox). The user then presses the share button on the camera app to upload the video data to cloud storage.

[0292] Next, the server receives the uploaded video data from the cloud storage. Specifically, it downloads the video data using the cloud storage's API and stores it in a database management system (e.g., MySQL (registered trademark) or MongoDB). When storing the video data, it saves it in an appropriate format (e.g., MP4 format). This allows the user to play the saved video data later.

[0293] Furthermore, users can input existing stories through the system interface. Specifically, they input the title and content of the story into a text box. The server inputs a prompt into the generative AI model (e.g., GPT-4) based on the story input by the user. An example of a prompt is, "Please combine the stories of Momotaro and Kintaro to create a new story."

[0294] The generative AI model generates a new story based on the prompt. Specifically, the AI ​​combines elements of "Momotaro" and "Kintaro" to generate a new story such as "Momotaro vs. Kintaro." The generated story is stored in a database by the server and made available to users for later reading.

[0295] For example, if a user wants to create a new story by combining the stories of "Momotaro" and "Kintaro," the user inputs the stories of "Momotaro" and "Kintaro" into the system. The server receives this and inputs the following prompt sentence into the generative AI model:

[0296] "Combine the stories of Momotaro and Kintaro to create a new story."

[0297] Based on this prompt, the generative AI model generates a new story, such as "Momotaro vs. Kintaro." The generated story is saved as text data and can be read by the user later.

[0298] In this way, the system of the present invention not only allows you to easily save and play back the story-telling, but also allows you to combine existing stories to create new stories and enjoy them. The flow of the specific processing in the third embodiment will be described with reference to Figure 15.

[0299] Step 1:

[0300] The user takes a photo of the reading aloud

[0301] The user uses a device such as a smartphone or tablet to record a video of the picture book being read aloud. Specifically, the user starts the smartphone's camera app and presses the record button to record the reading. The input is the user's operation, and the output is video data.

[0302] Step 2:

[0303] The device uploads the video data to cloud storage.

[0304] The device uploads the captured video data to a cloud storage service (e.g., Google Drive or Dropbox). Specifically, the user selects the video data in the camera app and presses the share button to upload it to cloud storage. The input is the video data, and the output is the video data saved in the cloud storage.

[0305] Step 3:

[0306] The server receives the video data and stores it in a database.

[0307] The server receives the video data uploaded from the cloud storage. Specifically, it downloads the video data using the cloud storage's API and stores it in a database management system (e.g., MySQL or MongoDB). The input is the video data from the cloud storage, and the output is the video data stored in the database.

[0308] Step 4:

[0309] User plays a saved video

[0310] Users can play stored videos through the system's interface. Specifically, they use the system's web app or mobile app to select the video they want to play from a list of stored videos and press the play button. The input is the video data stored in the database, and the output is the video being played.

[0311] Step 5:

[0312] User enters an existing story

[0313] The user inputs an existing story (e.g., "Momotaro" and "Kintaro") through the system interface. Specifically, the user inputs the title and content of the story into a text box. The input is the text data of the story entered by the user, and the output is the text data sent to the server.

[0314] Step 6:

[0315] The server inputs a prompt to the generative AI model.

[0316] The server inputs a prompt to the generative AI model (e.g., GPT-4) based on the story input by the user. Specifically, it sends the prompt, "Please combine the stories of Momotaro and Kintaro to create a new story," to the generative AI model. The input is the text data of the story from the user, and the output is the prompt sent to the generative AI model.

[0317] Step 7:

[0318] Generative AI models generate new stories

[0319] The generative AI model generates a new story based on the prompt. Specifically, the AI ​​combines elements of "Momotaro" and "Kintaro" to generate a new story such as "Momotaro vs. Kintaro." The input is the prompt, and the output is the text data of the generated story.

[0320] Step 8:

[0321] The server stores the generated stories and serves them to the user.

[0322] The server receives the generated stories and stores them in a database. The stored stories are then made available to users for later reading. The input is the text data of the generated stories, and the output is the stories stored in the database.

[0323] (Application example 3)

[0324] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0325] Traditional methods for recording storytimes have limited the means to preserve storytime memories and have made it impossible to combine existing stories to create new stories. Furthermore, there is a lack of easy ways to share saved videos or play them back at any time. This makes it difficult for users to effectively preserve storytime memories and enjoy new stories.

[0326] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0327] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on them, means for printing the created story on paper, means for recording the storytelling and saving it as a memory, means for uploading the recorded video to the cloud and making it playable at any time, and means for combining existing stories to create a new story. This allows users to effectively save the storytelling and play it back at any time, and further allows them to enjoy new stories by combining existing stories.

[0328] "Generative AI-based AI" is an artificial intelligence technology that generates new data based on existing data.

[0329] "Sound effects" are sound effects used to enhance the sense of realism in stories and images.

[0330] "Music" is the art of sound created by combining elements such as melody, harmony, and rhythm.

[0331] "Reading aloud" is the act of reading a book or story aloud to others.

[0332] A "narrative" is a written or visual form that describes events and the actions of characters.

[0333] "Printing on paper" is the act of transferring digital data onto a physical paper medium.

[0334] "Filming the reading aloud" means recording the act of reading aloud with a video camera or other filming equipment.

[0335] "Saving as a memory" means recording a particular event or action in a form that can be played back later.

[0336] "Uploading to the cloud" is the act of storing digital data on a remote server over the internet.

[0337] "Making it playable at any time" means that stored digital data can be played back at any time.

[0338] "Generating a new story by combining existing stories" means creating a new story based on multiple stories that already exist.

[0339] A system for implementing this invention is configured as follows: First, a user uses a smartphone to record the reading aloud. The smartphone is equipped with a video capture function that uses OpenCV, and can record the reading aloud as a high-quality video.

[0340] The captured video is uploaded to cloud storage (e.g., Google Drive) using rclone, allowing users to play the video anytime, anywhere.

[0341] Next, a generative AI-based AI is used to combine existing stories to generate a new story. Specifically, the Transformers library is used to input a prompt into the GPT-3 (registered trademark) model to generate a new story. An example of this prompt is "What would happen if Momotaro and Kintaro met?"

[0342] The new story is then displayed on the smartphone for the user to enjoy, and can also be printed on paper if desired.

[0343] The specific process of this system is as follows: First, the reading is filmed using the smartphone camera and saved in AVI format. Next, the saved video is uploaded to cloud storage using rclone. Next, a prompt sentence is input into the GPT-3 model to generate a new story. Finally, the generated story is displayed on the smartphone and, if necessary, printed on paper.

[0344] This system allows users to effectively save the storytelling and play it back at any time, and also allows users to combine existing stories to create new stories.

[0345] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0346] Step 1:

[0347] The user activates the camera on their smartphone and takes a picture of the reading aloud.

[0348] Input: Smartphone camera image

[0349] Output: AVI format video file

[0350] Specific operation: The user launches the camera app on their smartphone and presses the record button to record the reading. Once the recording is complete, the video file is saved in the smartphone's storage.

[0351] Step 2:

[0352] The device uploads the captured video file to cloud storage.

[0353] Input: AVI format video file

[0354] Output: Video files on cloud storage

[0355] Specific operation: The device uses rclone to upload the video file to the specified cloud storage (e.g., Google Drive). Once the upload is complete, the video file is saved in the cloud.

[0356] Step 3:

[0357] Users input prompts into a generative AI-based AI to generate new stories.

[0358] Input: Prompt sentence (e.g. "What would happen if Momotaro and Kintaro met?")

[0359] Output: Text data of the new story

[0360] How it works: A user launches a smartphone application and enters a prompt. The device then uses the transformers library to send the prompt to the GPT-3 model, which then receives the text data for the new story.

[0361] Step 4:

[0362] The terminal displays the new story that has been generated.

[0363] Input: Text data of a new story

[0364] Output: A story displayed on a smartphone screen

[0365] Specific operation: The device displays the received text data of the new story on the smartphone screen, and the user can read the generated story.

[0366] Step 5:

[0367] The user prints the generated story on paper if desired.

[0368] Input: Text data of a new story

[0369] Output: A printed version of the story

[0370] Specific operation: When a user presses the print button on the smartphone application, the device sends text data to the printer and the story is printed on paper.

[0371] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0372] "Example 1"

[0373] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes a user's emotions and adjusts the generation of a story according to the user's emotions. Specifically, when the user is feeling happy, the system generates a happy story, and when the user is sad, the system generates a comforting story. The emotion engine can also use facial recognition technology or voice recognition technology to recognize the user's emotions.

[0374] "Example 2"

[0375] Another embodiment of the present invention provides a system incorporating an emotion engine. This system recognizes the user's emotion and adjusts the generation of storytelling according to the emotion. Specifically, when the user feels surprised, the system generates storytelling that includes elements of surprise, and when the user feels relieved, the system generates storytelling that gives a sense of relief.

[0376] "Example 3"

[0377] Furthermore, as another embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes the emotion of a user and adjusts the generation of sound effects and music according to the emotion. Specifically, when the user is excited, fast-paced music and sound effects are generated, and when the user is relaxed, slower music and sound effects are generated.

[0378] "Example 4"

[0379] Another embodiment of the present invention provides a system incorporating an emotion engine that recognizes the user's emotions and adjusts the printing of the story accordingly, specifically printing bright colors when the user is happy and muted colors when the user is sad.

[0380] The processing flow of each embodiment will be described below.

[0381] "Example 1"

[0382] Step 1: Recognize the user's emotions with the emotion engine.

[0383] Step 2: Adjust your narrative production according to the emotions you recognize.

[0384] Step 3: Serve the adjusted story to the user.

[0385] "Example 2"

[0386] Step 1: Recognize the user's emotions with the emotion engine.

[0387] Step 2: Adjust your storytelling production according to the emotions you recognize.

[0388] Step 3: Provide the adapted reading to the user.

[0389] "Example 3"

[0390] Step 1: Recognize the user's emotions with the emotion engine.

[0391] Step 2: Adjust the sound effects and music generation according to the recognized emotion.

[0392] Step 3: Provide the adjusted sound effects and music to users.

[0393] "Example 4"

[0394] Step 1: Recognize the user's emotions with the emotion engine.

[0395] Step 2: Adjust the print of the story according to the emotions you recognize.

[0396] Step 3: Print the adjusted story and provide it to the user.

[0397] Example 1

[0398] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0399] Conventional story generation systems have had difficulty generating stories that respond to the user's emotions or creating digital picture books that combine sound effects, music, and audio reading. Furthermore, they lacked the functionality to recognize the user's emotions and adjust the content of the story, making it impossible to provide a story that was in tune with the user's emotions. Furthermore, the process of sending the generated story to the device and displaying it to the user was not automated, limiting the user experience.

[0400] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0401] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using a generative AI-based AI and creating a story based on them, means for receiving input from a user and generating a story by inputting a prompt sentence into a generative AI model, means for generating sound effects and music based on the generated story, means for generating audio for storytelling, means for creating a digital picture book by combining the generated story, sound effects, music, and storytelling audio, means for transmitting the created digital picture book to a terminal, means for the terminal to display the digital picture book to the user, means for recognizing the user's emotions and adjusting the generation of the story in accordance with the emotions, and means for recognizing the user's emotions using facial recognition technology and voice recognition technology. This makes it possible to generate a story in accordance with the user's emotions and automatically create a digital picture book by combining sound effects, music, and storytelling audio.

[0402] "Generative AI-based AI" is an artificial intelligence technology that automatically generates stories, sound effects, music, reading audio, and more based on user input.

[0403] A "prompt" is a text sentence that gives a generative AI model instructions for generating a story or other content.

[0404] "Narrative" refers to stories or narratives generated by generative AI-based AI.

[0405] "Sound effects" are sound effects generated to match the scenes in the story.

[0406] "Music" refers to songs and melodies that are generated to match the atmosphere and scenes of the story.

[0407] "Reading audio" refers to audio data in which the generated story is read aloud using speech synthesis technology.

[0408] A "digital picture book" is an electronic picture book created by combining generated stories, sound effects, music, and audio readings.

[0409] "User emotion" refers to the emotional state that the user is feeling, such as joy, sadness, surprise, etc.

[0410] "Facial recognition technology" is a technology that captures a user's face with a camera or other device and recognizes emotions from their facial expressions.

[0411] "Voice recognition technology" is a technology that records a user's voice using a microphone or other device and recognizes emotions from that voice.

[0412] A "terminal" refers to an electronic device used by a user, such as a smartphone, tablet, or PC.

[0413] This invention is a system that uses generative AI to automatically generate elements such as sound effects, music, and storytelling, and then creates a story based on these elements. Specific embodiments of this system are described below.

[0414] System configuration

[0415] This system is mainly composed of three elements: a server, a terminal, and a user. The server is responsible for generating stories, sound effects, music, and reading audio using a generative AI model. The terminal is a device through which the user inputs and displays the generated digital picture book. The user inputs the theme and keywords of the story and enjoys the generated digital picture book.

[0416] Hardware and software used

[0417] The server is a high-performance computer equipped with software for running generative AI models (e.g., OpenAI's GPT-4). It also uses music generation AI (e.g., Jukedeck or Amper Music) and speech synthesis software (e.g., Google Text-to-Speech or Amazon Polly). For facial recognition, a common facial recognition API (e.g., Microsoft® Azure® Face API) is used, and for speech recognition, a common speech recognition API (e.g., Google Speech-to-Text) is used.

[0418] Program processing

[0419] The server receives input from the user and generates a story by inputting prompt sentences into the generative AI model. For example, if the user inputs the theme "Pirate Adventure," the server inputs the following prompt sentences into the generative AI model:

[0420] Example prompt sentence:

[0421] "Create a story about a pirate adventure. The story includes brave pirates, treasure maps, and dangerous voyages."

[0422] The generative AI model generates a story based on this prompt. The server then analyzes the content of the generated story and uses the music generation AI to generate sound effects and music that match the scene. For example, the sound of waves and wind is generated for a scene where a pirate ship is sailing.

[0423] The server then inputs the story text into speech synthesis software to generate a reading voice, such as a voice that emotively recites the lines of a brave pirate.

[0424] Specific examples

[0425] The user opens the smartphone application and enters the theme "Pirate Adventure." The device sends this input data to the server. The server generates prompt text and inputs it into a generative AI model to generate a story. Based on the generated story, it generates sound effects and music, as well as a reading audio. These elements are combined to create a digital picture book, which is then sent to the device. The device then displays the received digital picture book to the user, who can enjoy the sound effects, music, and reading audio as they read the story.

[0426] In this way, it becomes possible to generate stories that correspond to the user's emotions and automatically generate digital picture books that combine sound effects, music, and storytelling audio.

[0427] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0428] Step 1:

[0429] The user inputs a theme or keyword.

[0430] The user uses the input interface of the terminal to input the theme and keywords of the story they want to create. For example, they input the theme "Pirate Adventure." The input data is "Pirate Adventure."

[0431] Step 2:

[0432] The terminal sends the input data to the server.

[0433] The terminal sends the theme and keywords entered by the user to the server. This data is used to create prompts for story generation. The input data is "Pirate Adventure," and the output is the theme data sent to the server.

[0434] Step 3:

[0435] The server inputs prompt sentences into the generative AI model to generate a story.

[0436] The server creates a prompt based on the received theme and keywords and inputs it into the generative AI model. For example, a prompt such as "Create a story about a pirate adventure. The story includes brave pirates, a treasure map, and a dangerous voyage" is input into the generative AI model. The generative AI model generates a story based on this prompt. The input data is theme data, and the output is the generated story.

[0437] Step 4:

[0438] The server generates sound effects and music based on the generated story.

[0439] The server analyzes the content of the generated story and uses music generation AI to generate sound effects and music that match the scene. For example, the sound of waves and wind is generated for a scene where a pirate ship is sailing. The input data is the generated story, and the output is sound effects and music.

[0440] Step 5:

[0441] The server generates audio for reading the story.

[0442] The server inputs the story text into speech synthesis software and generates a reading voice, for example, a voice that reads the lines of a brave pirate with great emotion. The input data is the story text, and the output is the reading voice.

[0443] Step 6:

[0444] The server combines the generated story, sound effects, music, and reading audio to create a digital picture book.

[0445] The server combines the generated story, sound effects, music, and audio reading into a single file to create a digital picture book. This picture book is designed to be enjoyed both visually and aurally by the user. The input data are the story, sound effects, music, and audio reading, and the output is a digital picture book.

[0446] Step 7:

[0447] The server sends the completed digital picture book to the terminal.

[0448] The server sends the completed digital picture book to the terminal, which saves the picture book in a format that the user can access. The input data is the digital picture book, and the output is the digital picture book sent to the terminal.

[0449] Step 8:

[0450] The terminal displays the digital picture book to the user.

[0451] The device displays the received digital picture book to the user. The user can read the picture book through the device and enjoy sound effects, music, and audio storytelling. For example, a user can pick up a tablet, tap the screen to open the picture book, read the story, and listen to sound effects and music. The input data is the digital picture book, and the output is the digital picture book displayed to the user.

[0452] (Application example 1)

[0453] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0454] Conventional story generation systems have difficulty generating stories that respond to the user's emotions, making it impossible to provide a personalized experience for each user. Furthermore, they are unable to combine sound effects or music with the generated stories, making it difficult to enhance the sense of realism of the story. Furthermore, they are unable to read the generated stories aloud using a text-to-speech function, making it impossible to provide a consistent storytelling experience for users.

[0455] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0456] In this invention, the server includes: means for recognizing a user's emotions; means for generating a story using a prompt sentence that instructs the generation of a story according to the user's emotions and a generative AI; means for generating sound effects and music according to the generated story; means for reading the generated story using a text-to-speech function; and means for playing the generated sound effects and music in sync with the read story. This makes it possible to generate a personalized story according to the user's emotions and provide an immersive storytelling experience that combines sound effects and music. The server further includes means for recording the user's behavior while the story is being read aloud and means for saving the recorded video.

[0457] "Generative AI-based AI" is an artificial intelligence technology that automatically generates stories, sound effects, music, etc. based on user input.

[0458] "Sound effects" are sound effects generated to match the scenes and situations in the story.

[0459] "Music" is music generated to enhance the atmosphere and emotion of the story.

[0460] "Reading aloud" is the act of reading the generated story aloud.

[0461] A "narrative" is a story created by an AI based on generative AI.

[0462] "Means of printing on paper" refers to the technology used to print the generated story onto physical paper media.

[0463] "Means to preserve memories" refers to a technique for recording the reading aloud and saving it so that it can be played back later.

[0464] "Means for recognizing user emotions" refers to technology that detects user emotions using facial recognition technology or voice recognition technology.

[0465] The "means for adjusting story generation according to emotions" is a technology that changes the content of a story based on the recognized emotions of a user.

[0466] "Facial recognition technology" is a technology that detects a user's face from camera footage and analyzes their facial expressions.

[0467] "Voice recognition technology" is a technology that analyzes the user's voice and understands their emotions and intentions.

[0468] "Text-to-speech" is a technology that reads the generated story aloud.

[0469] "Means for generating sound effects and music and playing them in accordance with the story" refers to technology that automatically generates sound effects and music according to the scenes in the story and plays them in accordance with the progress of the story.

[0470] The system for implementing this invention uses generative AI-based AI to automatically generate elements such as sound effects, music, and storytelling, and then creates a story based on these elements. The system has the ability to recognize the user's emotions and adjust the story generation accordingly. Specific embodiments of this system are described below.

[0471] Hardware and software used

[0472] Hardware: Uses the camera and microphone on your smartphone or head-mounted display.

[0473] software:

[0474] OpenCV: Used for camera image acquisition and face recognition.

[0475] EmotionRecognition: Used as an emotion recognition model.

[0476] GPT-4: Used as a generative AI to generate stories.

[0477] Librosa: Used to generate sound effects and music.

[0478] TextToSpeech: Used as a text-to-speech function.

[0479] System Operation

[0480] 1. Emotion recognition:

[0481] When a user activates the system through a smartphone or head-mounted display, it uses a camera and microphone to capture the user's face and voice.

[0482] Facial recognition is performed using OpenCV, and the EmotionRecognition model is used to recognize user emotions.

[0483] 2. Narrative Generation:

[0484] Based on the recognized emotion, a prompt sentence for story generation is generated. For example, if the user indicates the emotion "joy", the following prompt sentence is generated:

[0485] "The user's emotion is 'joy.' Please generate a fun story that matches this emotion."

[0486] This prompt sentence is input into a generative AI to generate a story.

[0487] 3. Sound effects and music generation:

[0488] Librosa is used to generate sound effects and music to match the generated story, providing sound effects that correspond to the scenes in the story.

[0489] 4. Reading aloud:

[0490] The generated story is then read aloud using the TextToSpeech feature, providing users with a consistent storytelling experience.

[0491] Specific examples

[0492] For example, if a user activates the system through their smartphone camera and the emotion is recognized as "joy," the following prompt will be generated:

[0493] "The user's emotion is 'joy.' Please generate a fun story that matches this emotion."

[0494] When this prompt is input into the generative AI, an entertaining story is generated. Sound effects and music are automatically generated to match the generated story, and the story is read aloud using the Text-to-Speech function. This allows users to enjoy a personalized story that responds to their emotions and an immersive storytelling experience.

[0495] In this way, the system is able to consistently generate stories that respond to the user's emotions, as well as generate accompanying sound effects and music and read aloud.

[0496] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0497] Step 1:

[0498] The user activates the system through a smartphone or head-mounted display. The system captures the user's face and voice using a camera and microphone. The input is the camera image and audio data, and the output is the captured image and audio.

[0499] Step 2:

[0500] The server recognizes faces from the captured video using OpenCV. The result of the face recognition is the position and feature points of the user's face. The input is the captured video, and the output is the position and feature points of the face.

[0501] Step 3:

[0502] The server uses the EmotionRecognition model to recognize the user's emotions from facial feature points and voice data. The input is facial feature points and voice data, and the output is the recognized emotion (e.g., joy, sadness, anger).

[0503] Step 4:

[0504] The server generates a prompt sentence for story generation based on the recognized emotion. For example, if the user expresses the emotion of "joy," the server generates the prompt sentence: "The user's emotion is 'joy.' Please generate a fun story that matches this emotion." The input is the recognized emotion, and the output is the prompt sentence.

[0505] Step 5:

[0506] The server inputs the generated prompt sentences into the generative AI to generate a story. The input is the prompt sentence, and the output is the text of the generated story.

[0507] Step 6:

[0508] The server uses Librosa to generate sound effects and music to match the generated story. The input is the text of the generated story, and the output is the sound effects and music data.

[0509] Step 7:

[0510] The server reads the generated story aloud using the TextToSpeech function: the input is the text of the generated story, and the output is the spoken audio.

[0511] Step 8:

[0512] The device plays the generated sound effects and music in accordance with the story progression. The input is sound effect and music data and spoken voice, and the output is a consistent storytelling experience for the user.

[0513] Step 9:

[0514] The device uses a camera to capture the user's behavior while reading a story and saves the resulting video as a memory. The input is the video of the reading, and the output is the saved video data.

[0515] It is also possible to repeatedly generate a story and sound effects and music. For example, the user's emotions while the story is being read aloud are recognized, and a continuation of the story is generated using a prompt sentence that instructs the generation of a story according to the user's emotions and the generative AI, and sound effects and music are generated according to the generated continuation of the story. The generated story is read aloud using a text-to-speech function, and the generated sound effects and music are played in accordance with the story being read aloud. These processes are repeated until the end of the story.

[0516] Example 2

[0517] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0518] Conventional picture book creation systems require users to manually create stories and add sound effects and music, which takes time and effort. Furthermore, it is difficult to provide an optimal experience for each individual user because the reading cannot be adjusted based on the user's emotions. Furthermore, the time and effort required to print digital picture books on paper is also a problem.

[0519] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using a generative AI-based AI and creating a story based on these elements, means for printing the created story on paper, means for recognizing the user's emotions and adjusting the generation of the storytelling in accordance with the emotions, means for transmitting digital picture book data to the terminal, and means for the terminal to receive the digital picture book data and print it on paper using a printer. This allows users to easily create the optimal picture book that suits their individual emotions and obtain a physical picture book.

[0520] "Generative AI-based AI" is a system that uses artificial intelligence technology to automatically generate elements such as stories, sound effects, music, and storytelling.

[0521] The "means for creating a story" is a function that uses generative AI-based AI to generate a new story based on a prompt sentence entered by the user.

[0522] "Means for printing on paper" refers to a device such as a printer for printing the generated digital picture book data on physical paper.

[0523] "Means for recognizing user emotions" refers to technology that analyzes emotions from the user's facial expressions, voice, etc., and identifies those emotions.

[0524] The "means for adjusting the generation of storytelling" is a function for changing the content and tone of storytelling according to the user's emotions.

[0525] The "means for transmitting digital picture book data to a terminal" is a function for transmitting the generated digital picture book data to a user's terminal via a network.

[0526] "Means for the terminal to receive digital picture book data" refers to the function by which the user's terminal receives digital picture book data sent from the server.

[0527] "Means for printing onto paper using a printer" refers to a device for printing onto physical paper the digital picture book data received by the terminal.

[0528] This invention is a system that uses generative AI to automatically generate stories, sound effects, music, and storytelling, and then creates a digital picture book based on these and prints it on paper. It also has the ability to recognize the user's emotions and adjust the storytelling generation accordingly.

[0529] The server uses a generative AI model (e.g., a natural language processing model) to generate a story based on the prompt entered by the user. It also uses speech synthesis software (e.g., a speech synthesis engine) to add sound effects and music to the generated story. It also uses an emotion engine (e.g., emotion analysis software) to recognize the user's emotions and generate a reading voice that corresponds to those emotions.

[0530] The device receives the digital picture book data sent from the server and prints it on paper using a printer (e.g., an inkjet printer), allowing the user to obtain a physical picture book.

[0531] As a concrete example, if the user inputs a prompt such as "Please write an adventure story for a 5-year-old child," the following processing will be performed.

[0532] 1. The user inputs through the system interface, "I want you to create an adventure story for a 5-year-old child."

[0533] 2. The server uses the generative AI model to generate an "adventure story for 5-year-olds."

[0534] 3. The server uses speech synthesis software to add sound effects and music to the generated story.

[0535] 4. The server uses an emotion engine to recognize the user's emotions and generates a reading audio that includes an element of surprise.

[0536] 5. The server combines the generated story, sound effects, music, and reading audio to create digital picture book data and send it to the terminal.

[0537] 6. The device receives the digital picture book data and prints it on paper using an inkjet printer.

[0538] 7. The user receives a physical printed picture book.

[0539] In this way, users can obtain a physical picture book containing a story tailored to their needs.

[0540] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0541] Step 1:

[0542] The user inputs a prompt sentence. The user inputs "I want you to create an adventure story for a 5-year-old child" through the system interface. The input prompt sentence is sent to the server.

[0543] Step 2:

[0544] The server receives the prompt sentence and generates a story using a generative AI model. The server uses the received prompt sentence as input and generates a story using a generative AI model (e.g., a natural language processing model). The generated story is output as text data.

[0545] Step 3:

[0546] The server adds sound effects and music to the generated story. The server uses the text data of the generated story as input and generates sound effects and music using speech synthesis software (e.g., a speech synthesis engine). The generated sound effects and music are output as audio data.

[0547] Step 4:

[0548] The server uses an emotion engine to recognize the user's emotions and generates a reading voice. The server uses an emotion engine (e.g., emotion analysis software) to recognize the user's emotions using the user's facial expressions and voice data as input. The server generates a reading voice based on the recognized emotion data. The generated reading voice is output as audio data.

[0549] Step 5:

[0550] The server sends the digital picture book data to the terminal. The server then combines the generated story, sound effects, music, and reading audio to create digital picture book data. This digital picture book data is then sent to the terminal via the network.

[0551] Step 6:

[0552] The terminal receives the digital picture book data and prints it on paper using a printer. The terminal receives the digital picture book data sent from the server. Using the received digital picture book data as input, it prints it on paper using a printer (e.g., inkjet printer). The printed picture book is output in a physical form.

[0553] Step 7:

[0554] The user receives the physical picture book. The user receives the physical picture book printed from the terminal. This allows the user to obtain a picture book containing the story that suits their preferences.

[0555] (Application example 2)

[0556] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0557] Previous picture book creation systems were unable to adjust the reading experience based on the user's emotions, and it was difficult to save the book digitally or display it on a smartphone. This limited the user experience and made it difficult to customize it to meet individual needs.

[0558] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0559] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on these, means for printing the created story on paper, means for taking pictures of the storytelling and saving them as memories, means for recognizing the user's emotions and adjusting the content of the storytelling according to those emotions, and means for saving the created story in digital format and displaying it on a smartphone. This enables customized storytelling according to the user's emotions, and also enables it to be saved in digital format and displayed on a smartphone.

[0560] "Generative AI-based AI" is an artificial intelligence technology that automatically generates elements such as stories, sound effects, music, and storytelling.

[0561] "Sound effects" are sounds used to enhance the sense of realism and emotion in a story or video.

[0562] "Music" refers to the melodies and rhythms used to create emotion and atmosphere in a story or film.

[0563] "Reading aloud" is the act of reading a story aloud, and is used especially in picture books for children.

[0564] A "narrative" is a series of stories that unfold through characters and events.

[0565] "Paper media" refers to the technology and devices used to transfer digitally created stories onto physical paper.

[0566] "Means for filming the reading aloud" refers to techniques and devices for recording the act of reading aloud on video.

[0567] "Means of preserving memories" refers to technologies and devices that allow for the long-term preservation of filmed footage and created stories.

[0568] "Means for recognizing user emotions" refers to technology or devices for analyzing and recognizing emotions from the user's facial expressions, voice, etc.

[0569] "Means for adjusting the content of reading aloud according to emotions" refers to techniques or devices for changing the content or tone of reading aloud based on the user's recognized emotions.

[0570] "Digital preservation means" refers to the technology or devices used to preserve the generated narratives and associated data in electronic form.

[0571] "Devices for displaying on smartphones" refers to the technology and devices for displaying digitally stored stories and related data on a smartphone screen.

[0572] A system for implementing this invention includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on these elements, means for printing the created story on paper, means for filming the storytelling and saving it as a memory, means for recognizing the user's emotions and adjusting the content of the storytelling in accordance with those emotions, and means for saving the created story in a digital format and displaying it on a smartphone.

[0573] Program processing explanation

[0574] Hardware

[0575] Smartphone: A device through which users can enter prompts and view the generated narrative.

[0576] Camera: Used to recognize the user's emotions.

[0577] Printer: Used to print the created story on paper.

[0578] software

[0579] Generative AI models: Used to generate stories, sound effects, music, and storytelling, such as OpenAI's GPT-3.

[0580] Emotion Recognition Software: Analyzes user emotions using OpenCV and Google Cloud Natural Language API.

[0581] PDF Generation Software: FPDF was used to convert the generated stories into PDF format.

[0582] Data processing and calculation

[0583] 1. Prompt Input: The user uses their smartphone to input a story prompt.

[0584] 2. Emotion Recognition: Capture the user's face using the smartphone camera and analyze emotions using OpenCV and Google Cloud Natural Language API.

[0585] 3. Narrative generation: Use a generative AI model (e.g., GPT-3) to generate a narrative based on user prompts and perceived emotions.

[0586] 4. Digital preservation: The generated stories are saved in digital format and displayed on a smartphone.

[0587] 5. Print to paper: If necessary, convert the generated story into PDF format and print it on paper using a printer.

[0588] Specific examples

[0589] Example prompt: "Create a story about a brave child fighting a dragon."

[0590] Emotion recognition example: If the user is feeling surprised, add an element of surprise to the story.

[0591] The system allows users to generate customized stories based on their emotions and save them digitally or print them on paper, enriching the user experience and enabling picture books to be created to suit individual needs.

[0592] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0593] Step 1:

[0594] A user inputs a story prompt using a smartphone, which is then sent to a generative AI model. The input data is the prompt in text format, and the output data is the prompt passed to the generative AI model.

[0595] Step 2:

[0596] The device's camera is used to capture the user's face. The captured image is sent to emotion recognition software. The input data is image data, and the output data is image data passed to the emotion recognition software.

[0597] Step 3:

[0598] The server uses emotion recognition software (OpenCV and Google Cloud Natural Language API) to analyze the user's emotions from the captured image. The input data is image data, and the output data is analyzed emotional information. Specifically, it analyzes facial expressions and identifies emotions such as surprise, joy, and sadness.

[0599] Step 4:

[0600] The server uses a generative AI model (e.g., GPT-3) to generate a story based on the user's prompt and the analyzed emotional information. The input data is the prompt and the emotional information, and the output data is the generated story. Specifically, the server combines the emotional information with the prompt to adjust the content and tone of the story.

[0601] Step 5:

[0602] The server saves the generated story in digital format and displays it on the smartphone. The input data is the generated story, and the output data is a digital story file. Specifically, the story is saved in text file or PDF format and displayed on the smartphone screen.

[0603] Step 6:

[0604] The user prints the generated story on paper as desired. The input data is a digital story file, and the output data is the printed story. Specifically, the digital story is sent to a printer, which prints it on paper.

[0605] Step 7:

[0606] The device records the reading and saves it as a memory. The input data is the video of the reading, and the output data is the saved video file. Specifically, the device records the reading and saves it as a video file.

[0607] Example 3

[0608] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0609] Conventional storytelling and story generation systems were unable to generate sound effects and music based on the user's emotions or adjust the printing of stories, making it difficult to provide an optimal experience for each individual user. Furthermore, they lacked the ability to combine existing stories to create new stories, limiting the opportunities for users to enjoy new stories. Furthermore, there were also insufficient ways to save the storytelling experience as a memory.

[0610] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0611] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI, means for printing the created story on paper, means for filming the storytelling and saving it as a memory, means for recognizing the user's emotions and adjusting the generation of sound effects and music according to those emotions, means for adjusting the printing of the story according to the user's emotions, and means for creating a new story by combining existing stories. This makes it possible to provide an optimal experience according to the user's emotions, increase opportunities to enjoy new stories, and save the storytelling as a memory.

[0612] "Generative AI-based AI" is an artificial intelligence technology that automatically generates new data and content based on existing data.

[0613] "Sound effects" are sound effects used to emphasize particular scenes or emotions in a story or film.

[0614] "Music" is an art form of sound that combines elements such as melody, harmony, and rhythm.

[0615] "Reading aloud" is the act of reading a book or story aloud to others.

[0616] A "narrative" is a series of events and a story that unfolds through the actions of characters.

[0617] "Printing to paper" is the process of transferring digital data onto a physical paper medium.

[0618] "Photography" is the act of recording video or images using a camera.

[0619] "Saving as a memory" means recording and storing a specific event or experience in a way that can be replayed later.

[0620] "Recognizing the user's emotions" means analyzing and determining the user's emotional state from their facial expressions, voice, etc.

[0621] "Adjusting the generation of sound effects and music" means selecting and playing appropriate sound effects and music according to the user's emotional state.

[0622] "Adjusting the print of a story" means changing the color tone and design of the print depending on the user's emotional state.

[0623] "Creating a new story by combining existing stories" means generating a new story based on multiple stories that already exist.

[0624] MODE FOR CARRYING OUT THE INVENTION

[0625] This system allows users to record themselves reading a picture book aloud and save the video as digital data. It also has the ability to create new stories by combining existing stories, and adjust sound effects, music, and story printing according to the user's emotions.

[0626] Hardware and software used

[0627] Hardware: Cameras, microphones, printers, computers

[0628] Software: Generative AI models, emotion recognition engines, video editing software

[0629] Specific Embodiments of the System

[0630] 1. Filming and saving story time

[0631] The user prepares to read a picture book aloud.

[0632] The device activates the camera and microphone.

[0633] The user begins reading aloud.

[0634] The device takes pictures of the reading and records the audio.

[0635] The device stores the video it captures as digital data.

[0636] The user plays the saved video.

[0637] Examples:

[0638] The user will film and save the footage of themselves reading "Momotaro" aloud.

[0639] Example prompt: "Please record and save the reading of Momotaro."

[0640] 2. Creating a new story

[0641] The user selects an existing story.

[0642] The server receives the selected story.

[0643] The server generates new stories using generative AI models.

[0644] The server provides the generated stories to the user.

[0645] Users enjoy new stories.

[0646] Examples:

[0647] The user selects "Momotaro" and "Kintaro," and the generative AI model creates a new story called "Momotaro vs. Kintaro."

[0648] Example prompt: "Create a new story by combining Momotaro and Kintaro."

[0649] 3. Emotion-based sound effects and music generation

[0650] The user boots the system.

[0651] The device analyzes the user's emotions using an emotion recognition engine.

[0652] The server receives the user's emotion data.

[0653] The server generates sound effects and music according to the emotion.

[0654] The device plays the generated sound effects and music.

[0655] Examples:

[0656] The user starts the system and speaks into the camera.

[0657] The device analyzes the camera footage using an emotion recognition engine (e.g., Affectiva) to detect whether the user is excited.

[0658] The server receives the user's emotional data and generates fast-paced music.

[0659] The server transmits the generated music data to the terminal.

[0660] The device plays fast-paced music to further heighten the user's excitement.

[0661] 4. Printing emotional stories

[0662] A user requests a printout of a story.

[0663] The device analyzes the user's emotions using an emotion recognition engine.

[0664] The server receives the user's emotion data.

[0665] The server sends instructions to the terminal to print the story in a color tone corresponding to the emotion.

[0666] The terminal prints the story in the specified color tone.

[0667] Examples:

[0668] A user requests printing of "Momotaro" through an application.

[0669] The device analyzes the camera footage using an emotion recognition engine to detect when the user is feeling joy.

[0670] The server receives the user's emotional data and sends an instruction to the terminal to print in bright colors.

[0671] The terminal instructs the printer to print "Momotaro" in bright colors.

[0672] The device will provide users with "Momotaro" printed in bright colors.

[0673] The above is a specific embodiment of the system of the present invention. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0674] Filming and saving story time

[0675] Step 1:

[0676] The user picks up a picture book and gets ready to start reading aloud.

[0677] Input: Picture book, reading intention

[0678] Output: Ready to read

[0679] Step 2:

[0680] The device will activate the built-in camera and microphone and set them to recording mode.

[0681] Input: User actions

[0682] Output: Camera and microphone activation

[0683] Step 3:

[0684] The user begins reading a picture book.

[0685] Input: Picture book, start reading

[0686] Output: Reading execution

[0687] Step 4:

[0688] The device captures high-resolution images of the user reading aloud and records the audio clearly.

[0689] Input: Camera video, microphone audio

[0690] Data processing: video and audio synchronization, noise removal

[0691] Output: High quality video data

[0692] Step 5:

[0693] The video recorded by the device will be saved in MP4 format and the file name will be "Reading_2023-10-01.mp4".

[0694] Input: High-quality video data

[0695] Data processing: file format conversion, file name setting

[0696] Output: Saved video file

[0697] Step 6:

[0698] Users can play saved videos and enjoy them with their families.

[0699] Input: Saved video file

[0700] Output: Video playback

[0701] Creating a new story

[0702] Step 1:

[0703] The user opens the application and selects an existing story.

[0704] Input: Application, selection of existing stories

[0705] Output: Selected stories

[0706] Step 2:

[0707] The server receives the user's selection and retrieves the selected story data from the database.

[0708] Input: Selected story

[0709] Data processing: Retrieving data from the database

[0710] Output: Story data

[0711] Step 3:

[0712] The server uses the generative AI model to generate new stories.

[0713] Input: Story data

[0714] Data Computation: Story Generation with Generative AI Models

[0715] Output: A new story

[0716] Step 4:

[0717] The server sends the generated story in text format to the user's device.

[0718] Input: A new story

[0719] Data processing: conversion to text format, data transmission

[0720] Output: Sending the story to the user's device

[0721] Step 5:

[0722] Users read and enjoy new stories.

[0723] Input: Submitted story

[0724] Output: Read and enjoy the story

[0725] Generating sound effects and music according to emotions

[0726] Step 1:

[0727] The user starts the system and speaks into the camera.

[0728] Input: System startup, camera footage

[0729] Output: Camera image acquisition

[0730] Step 2:

[0731] The device analyzes the camera footage using an emotion recognition engine to detect the user's emotions.

[0732] Input: Camera image

[0733] Data Calculation: Analysis by Emotion Recognition Engine

[0734] Output: Emotion data

[0735] Step 3:

[0736] The server receives the user's emotional data and generates fast-paced music.

[0737] Input: Emotion data

[0738] Data Computation: Applying Music Generation Algorithms

[0739] Output: Music data

[0740] Step 4:

[0741] The server transmits the generated music data to the terminal.

[0742] Input: Music data

[0743] Data processing: Data transmission

[0744] Output: Send music data to your device

[0745] Step 5:

[0746] The device plays fast-paced music to further heighten the user's excitement.

[0747] Input: Music data sent

[0748] Output: Music playback

[0749] Emotionally responsive story printing

[0750] Step 1:

[0751] A user requests a printout of a story through the application.

[0752] Input: Print request

[0753] Output: Receiving a print request

[0754] Step 2:

[0755] The device analyzes the camera footage using an emotion recognition engine to detect the user's emotions.

[0756] Input: Camera image

[0757] Data Calculation: Analysis by Emotion Recognition Engine

[0758] Output: Emotion data

[0759] Step 3:

[0760] The server receives the user's emotional data and sends an instruction to the terminal to print in bright colors.

[0761] Input: Emotion data

[0762] Data processing: generating print instructions, sending data

[0763] Output: Sending print instructions

[0764] Step 4:

[0765] The terminal instructs the printer to print the story in bright colors.

[0766] Input: Print instructions

[0767] Output: Instructions to the printer

[0768] Step 5:

[0769] The device presents the user with a story printed in bright colors.

[0770] Input: Printed story

[0771] Output: Provide printed materials

[0772] (Application example 3)

[0773] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0774] Conventional storytelling systems lack the ability to record the storytelling process, or to generate new stories by combining existing stories. They also lack the ability to adjust sound effects and music according to the user's emotions, or to adjust the printing of the story, making it difficult to enrich the user experience.

[0775] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0776] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on those elements, means for printing the created story on paper, means for filming the storytelling and saving it as a memory, means for recognizing the user's emotions and adjusting the sound effects and music according to those emotions, means for generating a new story by combining existing stories, and means for adjusting the printing of the story according to the user's emotions. This allows the user to record the storytelling, generate a new story, and enjoy sound effects, music, and printing according to their emotions.

[0777] "Generative AI-based AI" is an artificial intelligence technology that automatically generates new data and content based on existing data.

[0778] "Sound effects" are sound effects used to emphasize specific scenes or situations in content such as stories or videos.

[0779] "Music" is an art form of sound created by combining elements such as melody, harmony, and rhythm.

[0780] "Reading aloud" is the act of reading a book or story aloud to others.

[0781] A "narrative" is a form of writing or image that depicts a series of events, including characters, events, and settings.

[0782] "Printing to paper" is the process of transferring digital data onto a physical paper medium.

[0783] "Filming the reading aloud" means recording the act of reading aloud on video using a camera or other filming device.

[0784] "Saving as a memory" means preserving a specific event or action as digital data so that it can be replayed at a later date.

[0785] "Recognizing user emotions" refers to technology that determines a user's emotional state from their facial expressions, tone of voice, etc.

[0786] "Adjusting sound effects and music according to emotions" means changing the type and tempo of the sound effects and music to be played based on the emotional state of the user.

[0787] "Generating a new story by combining existing stories" means creating a new story by combining elements of multiple existing stories.

[0788] "Adjusting story printing according to emotions" means changing the color tone and design of the story when it is printed based on the user's emotional state.

[0789] A system for carrying out this invention has the following configuration: The server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on these, means for printing the created story on paper, means for recording the storytelling and saving it as a memory, means for recognizing the user's emotions and adjusting the sound effects and music according to those emotions, means for generating a new story by combining existing stories, and means for adjusting the printing of the story according to the user's emotions.

[0790] Program processing explanation

[0791] Recording and saving storytimes

[0792] The server uses a camera to record the user reading a picture book and saves the video as digital data. Specifically, it uses the OpenCV library to save the video data acquired from the camera in real time.

[0793] New story generation

[0794] The server uses a generative AI model to combine existing stories to generate new ones. For example, it combines "Momotaro" and "Kintaro" to generate a new story such as "Momotaro vs. Kintaro." This process uses a generative AI model that uses natural language processing technology.

[0795] Emotion recognition and music / sound effect adjustment

[0796] The server uses emotion recognition software to recognize the user's emotions. Specifically, it analyzes the user's facial expressions from video data to determine their emotional state. It also uses music generation software to adjust sound effects and music according to the user's emotional state. For example, it generates fast-paced music when the user is excited and slow-paced music when the user is relaxed.

[0797] Emotionally responsive story printing

[0798] The server adjusts the printing of the story based on the user's emotions. Specifically, it uses emotion recognition software to determine the user's emotions and changes the color tone and design of the printout based on that emotion. For example, if the user is feeling happy, it will print in bright colors, and if they are sad, it will print in muted colors.

[0799] Examples and prompts

[0800] As a concrete example, the following prompt sentence can be input to a generative AI model:

[0801] Prompt Sentence Examples

[0802] Combine "Momotaro" and "Kintaro" to create a new story. The title of the new story is "Momotaro vs Kintaro."

[0803] This prompt can then be used to instruct a generative AI model to generate a new story.

[0804] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0805] Step 1:

[0806] The server uses a camera to record the user reading a picture book. As input, it receives the video data from the camera and saves it in real time using the OpenCV library. As output, the reading experience is saved as digital data.

[0807] Step 2:

[0808] The server uses a generative AI model to combine existing stories to generate a new story. As input, the generative AI model receives data from existing stories and a prompt. For example, a prompt that combines "Momotaro" and "Kintaro" to create a new story is used. As output, a new story is generated.

[0809] Step 3:

[0810] The server uses emotion recognition software to recognize the user's emotions. As input, it uses the video data saved in step 1 and analyzes the user's facial expressions. As output, it determines the user's emotional state.

[0811] Step 4:

[0812] The server adjusts the sound effects and music based on the user's emotional state. As input, it uses the emotional state data obtained in step 3 and generates appropriate sound effects and music using music generation software. For example, if the user is excited, it generates fast-paced music. As output, sound effects and music corresponding to the user's emotion are generated.

[0813] Step 5:

[0814] The server adjusts the printing of the story according to the user's emotions. As input, it uses the emotional state data obtained in step 3 and the new story data generated in step 2. It uses emotion recognition software to change the color tone and design of the printout; for example, it prints in bright colors when the user is feeling happy. As output, it prints a story according to the emotion.

[0815] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0816] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is also input. The data generation model 58 converts the input inference data into the instruction indicated by the prompt.

[0817] Therefore, inference is performed and the inference results are output in the form of data such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0818] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0819] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0820] [Second embodiment]

[0821] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0822] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0823] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0824] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0825] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0826] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0827] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0828] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0829] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0830] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0831] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0832] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0833] "Example 1"

[0834] The system of this invention uses generative AI-based AI to automatically generate elements such as sound effects, music, and storytelling. Specifically, the AI ​​generates a story based on user input, then generates sound effects and music to match the story. The AI ​​also generates a story read aloud, and combines all of these to create a picture book.

[0835] "Example 2"

[0836] The system of this invention has the function of printing the created picture book on paper. Specifically, it digitally creates a picture book by combining an AI-generated story, sound effects, music, and storytelling, and then prints the picture book on paper. This allows users to obtain a physical picture book.

[0837] "Example 3"

[0838] The system of the present invention has the function of recording the reading aloud and saving it as a memory. Specifically, the system records the user reading a picture book on video and saves the video as digital data. This allows the user to play back the reading aloud later and keep it as a memory.

[0839] "Example 4"

[0840] The system of the present invention has the function of combining existing stories to create new stories. Specifically, the AI ​​combines existing stories, such as "Momotaro" and "Kintaro," to create new stories, such as "Momotaro vs. Kintaro." This allows users to enjoy new stories based on existing stories.

[0841] The processing flow of each embodiment will be described below.

[0842] "Example 1"

[0843] Step 1: The user inputs the story's theme, characters, etc. into the system.

[0844] Step 2: Generative AI-based AI generates a story based on user input.

[0845] Step 3: Generate sound effects and music to match the AI-generated story.

[0846] Step 4: The AI ​​generates a story reading.

[0847] Step 5: Combine them all to create a digital picture book.

[0848] "Example 2"

[0849] Step 1: Create a digital picture book by combining an AI-generated story, sound effects, music, and storytelling.

[0850] Step 2: The system prints the digitally created picture book onto paper.

[0851] "Example 3"

[0852] Step 1: The user reads the book aloud.

[0853] Step 2: The system records a video of the reading.

[0854] Step 3: Save the captured video as digital data.

[0855] "Example 4"

[0856] Step 1: The AI ​​selects an existing story, such as "Momotaro" or "Kintaro."

[0857] Step 2: Combine the selected stories to create a new story, for example, "Momotaro vs. Kintaro."

[0858] Example 1

[0859] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0860] Conventional story creation systems require users to manually create stories and add sound effects, music, and audio recordings individually, which requires a great deal of time and effort. Furthermore, the process of integrating these elements into a single picture book is cumbersome, making it difficult to automatically generate picture books, especially for children.

[0861] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0862] In this invention, the server includes means for a user to input a prompt sentence, means for generating a story based on the prompt sentence using a generative AI model, means for generating sound effects based on the generated story, means for generating music based on the generated story, means for generating a storytelling audio based on the generated story, means for creating a picture book by combining the story, sound effects, music, and storytelling audio, and means for a user to view the picture book on a terminal. This allows a user to easily create a story, automatically add sound effects, music, and storytelling audio, and integrate them into a single picture book.

[0863] A "prompt sentence" is text that the user inputs to the system, including instructions about the theme, characters, etc. of the story.

[0864] A "generative AI model" is an artificial intelligence algorithm that automatically generates a story based on a prompt from the user.

[0865] A "story" is text data generated by a generative AI model based on a prompt sentence.

[0866] "Sound effects" are sound effects generated to match the scenes in the story.

[0867] "Music" is background music generated to match the atmosphere of the story.

[0868] "Reading audio" is audio data generated from the story text using speech synthesis technology.

[0869] A "picture book" is an e-book format content created by combining a story, sound effects, music, and audio reading.

[0870] A "terminal" is a device through which a user accesses the system, inputs prompt sentences, and views the generated picture book.

[0871] This invention relates to a system that uses a generative AI model to automatically generate a story, sound effects, music, and reading audio, and combines them to create a picture book. Specific embodiments of this system are described below.

[0872] A user accesses the system using a terminal and inputs a prompt statement such as the theme of the story, characters, etc. For example, the user might input, "Create an adventure story about a brave boy who fights a dragon in a magical forest."

[0873] The server receives a prompt from the user and generates a story based on the prompt using a generative AI model (e.g., OpenAI's GPT-4). The generated story is stored as text data on the server.

[0874] The server then generates sound effects based on the story, using audio libraries such as the Freesound API to select sound effects that match each scene in the story. For example, for a scene featuring a dragon, the server selects the sound of a dragon roaring.

[0875] In addition, the server generates music that matches the atmosphere of the story. For music generation, it uses music generation AI such as Magenta. For example, exciting music is generated for adventure scenes, and calm music is generated for quiet scenes.

[0876] The server also uses speech synthesis technology, such as the Google Text-to-Speech API, to convert the story text into audio data, which then generates an audio recording of the story being read aloud.

[0877] Finally, the server combines the generated story, sound effects, music, and audio reading to create a picture book, which is saved as a PDF or e-book and made available for download by users.

[0878] Users can use their devices to download and view picture books provided by the server. While reading the picture books, users can enjoy sound effects, music, and audio recordings of the story being told.

[0879] The system allows users to easily create stories, automatically add sound effects, music, and audio readings, and integrate them into a complete picture book.

[0880] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0881] Step 1:

[0882] The user uses a terminal to input prompts into the system, such as the theme and characters of the story. For example, the user might input "Please create an adventure story about a brave boy battling a dragon in a magical forest." The input prompt is then sent to the server.

[0883] Step 2:

[0884] The server receives the prompt sentence and sends it as input to a generative AI model (e.g., OpenAI's GPT-4). The generative AI model generates a story based on the prompt sentence and returns the text data to the server. The server saves the generated story as text data.

[0885] Step 3:

[0886] The server analyzes the generated story text and generates sound effects appropriate for each scene. For example, in a scene where a dragon appears in the story, the server uses the Freesound API to obtain the dragon's roar sound. The input is the story text data, and the output is the audio data of the sound effects.

[0887] Step 4:

[0888] The server generates music that matches the atmosphere of the story. For music generation, it uses music generation AI such as Magenta. For example, exciting music is generated for adventure scenes, and calm music is generated for quiet scenes. The input is the text data of the story, and the output is the audio data of the music.

[0889] Step 5:

[0890] The server uses speech synthesis technology such as the Google Text-to-Speech API to convert the story text into audio data, which then generates a story read aloud. The input is the story text data, and the output is the audio data of the story read aloud.

[0891] Step 6:

[0892] The server combines the generated story, sound effects, music, and reading audio to create a picture book. The picture book is saved in PDF or e-book format and can be downloaded by users. The input is the text data of the story, audio data of the sound effects, audio data of the music, and audio data of the reading audio, and the output is the integrated picture book data.

[0893] Step 7:

[0894] Users can use their devices to download and view picture books provided by the server. While reading the picture book, users can enjoy sound effects, music, and audio recordings. The input is the picture book data from the server, and the output is the user's viewing experience.

[0895] (Application example 1)

[0896] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0897] Conventional story generation systems can generate stories based on a theme entered by the user, but they lack the functionality to automatically generate sound effects, music, and storytelling audio and combine them to create a digital picture book. Furthermore, they lack the means to make the generated digital picture book viewable on a smartphone, limiting the user experience. This poses a challenge for users, making it difficult to easily create and enjoy high-quality digital picture books.

[0898] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0899] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on them, means for digitally saving the created story, means for generating a story based on user input, means for generating sound effects and music to match the generated story, means for generating audio for storytelling, means for creating a digital picture book by combining the generated story, sound effects, music, and storytelling audio, and means for making the created digital picture book viewable on a smartphone. This allows high-quality digital picture books to be automatically generated based on a theme entered by the user, and can be easily viewed and shared on a smartphone.

[0900] "Generative AI-based AI" is an artificial intelligence technology that automatically generates stories, sound effects, music, reading audio, and more based on user input.

[0901] "Sound effects" are sound effects generated to match the scenes and situations in the story.

[0902] "Music" is music generated to enhance the atmosphere and emotion of the story.

[0903] "Read-aloud audio" is audio data for reading the generated story aloud.

[0904] A "narrative" is a story that is generated based on input from the user.

[0905] "Digital format" means a data format that is stored and displayed electronically.

[0906] "User input" refers to information such as keywords and themes that a user provides to the system.

[0907] A "digital picture book" is an e-book format picture book created by combining generated stories, sound effects, music, and audio readings.

[0908] A "smartphone" is a portable information terminal that can connect to the Internet and run applications.

[0909] "Viewable" means that the user can visually check the digital picture book.

[0910] The system for implementing this invention uses generative AI-based AI to automatically generate stories, sound effects, music, and storytelling audio based on user input, and combines them to create a digital picture book. Specific embodiments of this system are described below.

[0911] System configuration

[0912] 1. Hardware Configuration

[0913] Server: A high-performance computer (e.g., a server equipped with a GPU) for running generative AI models.

[0914] User device: A device such as a smartphone or tablet that allows users to input and view the generated digital picture book.

[0915] 2. Software Configuration

[0916] Generative AI models: AI models for generating stories, sound effects, music, and storytelling audio (e.g., OpenAI's GPT-4).

[0917] Database: A database for storing the generated digital storybooks and user input.

[0918] Application: A smartphone application that allows users to enter input and view the generated digital picture book.

[0919] Processing flow

[0920] 1. User Input

[0921] Users enter the story's theme and keywords through a smartphone application.

[0922] Example: "A story of adventure and friendship"

[0923] 2. Narrative Generation

[0924] The server uses a generative AI model to generate a story based on user input.

[0925] Example prompt: "Generate a story of adventure and friendship."

[0926] 3. Sound Effects and Music Generation

[0927] The server generates sound effects and music to accompany the generated story.

[0928] Example of a sound effect generation prompt: "Generate a sound effect that matches the following story:\n\n(generated story)"

[0929] Example of a music generation prompt: "Generate music to fit the following story: (generated story)"

[0930] 4. Generation of reading audio

[0931] The server generates audio to read the generated story aloud.

[0932] Example prompt for storytelling audio generation: "Generate audio to read the following story: (generated story)"

[0933] 5. Creating a digital picture book

[0934] The server combines the generated story, sound effects, music, and reading audio to create a digital picture book.

[0935] 6. Viewing digital picture books

[0936] Users can view the generated digital picture book through a smartphone application.

[0937] Specific examples

[0938] When a user inputs the theme "A Story of Adventure and Friendship," the server uses a generative AI model to generate a story. It then generates sound effects and music to match the story, and generates audio to read the story. These elements are combined to create a digital picture book, which users can then view on their smartphones.

[0939] In this way, users can easily create and enjoy high-quality digital picture books.

[0940] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0941] Step 1:

[0942] The user starts the smartphone application and inputs the theme and keywords of the story. The input theme and keywords are sent to the server. The input data is used as prompts for generating the story.

[0943] Step 2:

[0944] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate a story based on themes and keywords submitted by the user. Specifically, the server inputs the prompt "Generate a story of adventure and friendship" into the generative AI model and outputs the story text.

[0945] Step 3:

[0946] The server generates sound effects based on the generated story text. It inputs the prompt "Generate sound effects that fit the following story:\n\n(generated story)" into the generative AI model and outputs a description of the sound effects. The description of the sound effects is used to retrieve appropriate sound effect files from the sound effect database.

[0947] Step 4:

[0948] The server generates music based on the generated story text. It inputs the prompt "Generate music that matches the following story: (generated story)" into the generative AI model and outputs a music description. The music description is used to retrieve appropriate music files from a music database.

[0949] Step 5:

[0950] The server generates audio to read the generated story text aloud. The prompt "Please generate audio to read the following story aloud:\n\n(generated story)" is input into the generative AI model, and the reading audio data is output.

[0951] Step 6:

[0952] The server combines the generated story text, sound effects, music, and reading audio to create a digital picture book, which includes generating a script to play each element at the appropriate timing and creating a digital picture book file.

[0953] Step 7:

[0954] The server sends the created digital picture book file to the user's smartphone. The user can view the created digital picture book through a smartphone application. The digital picture book is displayed in an integrated format, including text, sound effects, music, and audio recordings.

[0955] Example 2

[0956] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0957] Conventional picture book creation systems require users to manually create stories, sound effects, music, and audio for storytelling, which takes time and effort. Furthermore, the content and quality of the stories are not consistent, which can lead to low user satisfaction. Furthermore, the time and cost required to print the stories on paper is also an issue.

[0958] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0959] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative artificial intelligence and creating a story based on these, means for generating a story based on a prompt sentence entered by the user, means for generating sound effects, music, and storytelling audio based on the generated story, means for creating a digital picture book by combining these elements, means for sending the created digital picture book to a terminal, means for the terminal to display the digital picture book and for the user to check it, means for the user to instruct printing, means for the terminal to print the picture book on paper using a printer, and means for capturing footage of the storytelling and saving it as a memory. This allows users to easily create picture books containing high-quality stories and obtain them in physical form.

[0960] "Generative AI" is an AI technology that automatically generates elements such as stories, sound effects, music, and reading audio based on prompts entered by the user.

[0961] A "prompt" is an instruction that a user inputs to a generative AI, including details about the story's theme, characters, etc.

[0962] A "narrative" is a story or narrative that a generative artificial intelligence generates based on a prompt.

[0963] "Sound effects" are sound effects that are generated according to the scenes and situations in the story.

[0964] "Music" refers to musical compositions and melodies generated to enhance the atmosphere and emotion of the story.

[0965] "Read-aloud audio" refers to audio data generated to read aloud the generated story.

[0966] A "digital picture book" is an electronic picture book created by combining generated stories, sound effects, music, and audio readings.

[0967] A "terminal" is an electronic device that a user uses to check a digital picture book and instruct it to be printed.

[0968] A "printer" is a printing device for printing digital picture books on paper.

[0969] "Reading situation" refers to the situation or scene when the user reads the generated story aloud.

[0970] "Means for preserving memories" refers to methods or devices for recording the reading aloud and saving it for later playback.

[0971] This invention is a system that uses generative artificial intelligence to automatically generate stories, sound effects, music, and audio for storytelling, combines them to create a digital picture book, and finally prints it on paper. Specific embodiments of this system are described below.

[0972] The server uses generative AI to generate a story based on the prompt entered by the user. For example, a model using natural language processing technology (e.g., GPT-4) is used as the generative AI. When the user enters a prompt, the server analyzes it and generates a story.

[0973] The server then generates sound effects, music, and audio based on the story. Sound effects and music are generated using text-to-speech software (e.g., Google Cloud Text-to-Speech). Audio for the story is also generated.

[0974] The server combines the generated story, sound effects, music, and audio reading to create a digital picture book, which is then saved in a format that can be viewed by the user.

[0975] The server then sends the created digital picture book to the user's terminal. The terminal receives the digital picture book sent from the server and displays it for the user to review. The user can review the displayed digital picture book and make corrections as necessary.

[0976] If the user is satisfied with the content of the digital picture book, they can request printing. The request is made through the terminal interface. Upon receiving the user's request, the terminal uses a printer (e.g., an inkjet printer) to print the digital picture book onto paper. The printed picture book is then provided to the user in physical form.

[0977] As a concrete example, consider a scenario where a user inputs a prompt such as "Please create a story about animals having an adventure in the forest. The main characters are a rabbit, a fox, and a bear." The server receives this prompt and uses generative artificial intelligence to generate the following story:

[0978] "One day, Lily the rabbit, Fred the fox, and Ben the bear embark on a great adventure in the forest. They make new friends, overcome obstacles, and uncover the secrets of the forest."

[0979] The server then generates sound effects and music to accompany the story, and creates audio for the story being read aloud. These elements are then combined to create a digital picture book, which is then sent to the device. When the user reviews the digital book and requests printing, the device uses a printer to print the book onto paper, giving the user a physical copy.

[0980] In this way, users can easily create picture books containing high-quality stories and have them available in physical form.

[0981] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0982] Step 1:

[0983] The user enters a prompt statement.

[0984] The user accesses the system's interface and inputs a prompt sentence including details such as the theme and characters of the story. For example, the user might input, "Please create a story about animals having an adventure in the forest. The main characters are a rabbit, a fox, and a bear." The input prompt sentence is then sent to the server.

[0985] Step 2:

[0986] The server receives the prompt and generates a story using a generative AI model.

[0987] The server receives the prompt sent by the user. Then, it uses a generative AI model (e.g., a model using natural language processing technology) to generate a story based on the prompt. The generated story is saved as text data. The input is the prompt, and the output is the generated story.

[0988] Step 3:

[0989] The server generates sound effects, music, and storytelling audio based on the generated story.

[0990] The server uses speech synthesis software (e.g., text-to-speech technology) to analyze the generated story and generate appropriate sound effects and music, as well as audio for reading the story aloud. The input is the generated story, and the output is sound effects, music, and audio for reading the story aloud.

[0991] Step 4:

[0992] The server combines these elements to create a digital picture book.

[0993] The server combines the generated story, sound effects, music, and audio reading to create a digital picture book. This digital picture book is saved in a format that can be viewed by the user. The input is the story, sound effects, music, and audio reading, and the output is the digital picture book.

[0994] Step 5:

[0995] The server sends the digital picture book to the terminal.

[0996] The server sends the created digital picture book to the user's terminal. The sent digital picture book is displayed on the terminal so that the user can check it. The input is the digital picture book, and the output is the digital picture book displayed on the terminal.

[0997] Step 6:

[0998] The terminal displays the digital picture book for the user to check.

[0999] The terminal receives the digital picture book sent from the server and displays it for the user to check. The user can check the displayed digital picture book and instruct corrections as necessary. The input is the digital picture book, and the output is the user's confirmation and correction instructions.

[1000] Step 7:

[1001] The user issues a print instruction.

[1002] When the user is satisfied with the content of the digital picture book, he or she instructs the book to be printed. The printing instruction is given through the terminal interface. The input is the user's print instruction, and the output is the transmission of the print instruction.

[1003] Step 8:

[1004] The device uses a printer to print the picture book onto paper.

[1005] The terminal receives a print instruction from the user and prints the digital picture book on paper using a printer (e.g., an inkjet printer). The printed picture book is provided to the user in a physical form. The input is the print instruction, and the output is the picture book printed on paper.

[1006] (Application example 2)

[1007] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1008] In conventional picture book creation systems, users could freely select story themes and characters, combine the generated story with sound effects, music, and storytelling to create a digital picture book, and then print the book on paper. Furthermore, there was a lack of a way for users to easily save the picture books they created and obtain them in physical form. This prevented users from fully expressing their creativity and enjoying the fun of creating picture books.

[1009] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1010] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on them, means for printing the created story on paper, means for filming the storytelling and saving it as a memory, means for providing an interface for the user to select the theme and characters of the story, means for creating a digital picture book by combining the generated story with the sound effects, music, and storytelling, and means for saving the created digital picture book in PDF format so that the user can print it on paper using a printer.This allows the user to freely select the theme and characters of the story, create a digital picture book by combining the generated story with the sound effects, music, and storytelling, and print it on paper to obtain it in physical form.

[1011] "Generative AI-based AI" is artificial intelligence that automatically generates specific data or content using generative models.

[1012] "Sound effects" are sound effects used to emphasize particular scenes or actions in a story or film.

[1013] "Music" is an art form of sound created by combining elements such as melody, harmony, and rhythm.

[1014] "Reading aloud" is the act of reading a story or text aloud, especially for the education or entertainment of children.

[1015] A "narrative" is a form of writing or image that describes a series of events composed of characters, events, and settings.

[1016] A "means for printing on paper" is a device or method for transferring digital data onto a physical paper medium.

[1017] "Means for filming the reading aloud" refers to devices or methods for recording the act of reading aloud as video or audio.

[1018] A "memory storage device" is a device or method for recording and storing specific events or actions in a form that can be later reproduced.

[1019] An "interface that allows the user to select the story theme and characters" is an operation screen or input device that allows the user to select the story content and characters.

[1020] A "digital picture book" is a picture book created in digital format that includes elements such as text, images, and audio.

[1021] "Means for saving in PDF format" refers to a device or method for saving digital data in Portable Document Format (PDF).

[1022] A "printer" is a device for printing digital data onto a physical medium such as paper.

[1023] A system for implementing this invention includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on these elements, means for printing the created story on paper, means for filming the storytelling and saving it as a memory, means for providing an interface for users to select the theme and characters of the story, means for combining the generated story with the sound effects, music, and storytelling to create a digital picture book, and means for saving the created digital picture book in PDF format so that the user can print it on paper using a printer.

[1024] Hardware and software used

[1025] Hardware: Smartphone, tablet, printer, internet connection

[1026] Software: Python, TensorFlow (implementation of generative AI models), Flask (web application framework), Firebase (database and authentication), PIL (Python Imaging Library, for image processing)

[1027] Program processing

[1028] Building the user interface

[1029] Users use the touchscreen of their smartphone or tablet to navigate an interface that allows them to select themes and characters for their stories, such as "fairy tales," "animal adventures," and "space exploration."

[1030] Running generative AI models

[1031] The server inputs prompts into the generative AI model based on the theme selected by the user, and generates a story, sound effects, music, and storytelling. For example, if the prompt is "Generate a story about a fairy tale theme in which a brave prince defeats a dragon," the AI ​​will generate a story.

[1032] Creating a digital picture book

[1033] The server combines the generated story with sound effects, music, and storytelling to create a digital storybook. It uses PIL to generate pages that combine images and text.

[1034] Implementing the printing function

[1035] The server saves the digital book in PDF format so that users can print it on paper using a printer, and uses Flask to provide a web interface where users can download the PDF.

[1036] Specific examples

[1037] Prompt Sentence Examples

[1038] "Generate a fairy tale story about a brave prince slaying a dragon."

[1039] Example of a generated story

[1040] "Once upon a time, there was a brave prince. He set out on a journey to save his village from a dragon. With his wisdom and courage, the prince defeated the dragon and saved his village."

[1041] Digital picture book page example

[1042] Page 1: "Once upon a time, there was a brave prince." (Illustration of the prince)

[1043] Page 2: "He set out on a journey to save a village that had been attacked by a dragon." (Illustration of a travel scene)

[1044] Page 3: "The prince used his wisdom and courage to defeat the dragon and save the village." (Illustration of a dragon battle scene)

[1045] In this way, users can easily create their own original picture books and enjoy them by saving them digitally or printing them out on paper.

[1046] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1047] Step 1:

[1048] The user operates the interface of a smartphone or tablet to select the story's theme and characters.

[1049] Input: User-selected theme and character information

[1050] Output: Data on selected themes and characters

[1051] Specific operation: The user operates the touchscreen to select a theme such as "Fairy Tales," "Animal Adventures," or "Space Exploration." The selected information is sent from the device to the server.

[1052] Step 2:

[1053] Based on the theme and character information received by the server, prompts are input into the generative AI model, which then generates a story, sound effects, music, and reading aloud.

[1054] Input: User-selected theme, character data, prompt text

[1055] Output: Generated story, sound effects, music, and storytelling data

[1056] How it works: The server uses TensorFlow to generate a story using the prompt "Generate a fairy tale-themed story about a brave prince slaying a dragon" and outputs the generated story along with associated sound effects, music, and storytelling data.

[1057] Step 3:

[1058] The server combines the generated story with sound effects, music, and storytelling to create a digital picture book.

[1059] Input: Generated story, sound effects, music, and storytelling data

[1060] Output: Digital picture book data

[1061] What it does: The server uses PIL to create pages that combine the generated story text with corresponding images, with each page containing story text and related illustrations.

[1062] Step 4:

[1063] The server saves the created digital picture book in PDF format so that users can print it on paper using a printer.

[1064] Input: Digital picture book data

[1065] Output: Digital picture book in PDF format

[1066] What it does: The server uses Flask to provide a web interface that allows users to download PDFs. Users download the PDFs through this interface and print them on a printer.

[1067] Step 5:

[1068] Users can take photos of the reading aloud and save them as memories.

[1069] Input: Video and audio data of a user reading aloud

[1070] Output: Stored video and audio data

[1071] How it works: Users use the camera on their smartphone or tablet to record themselves reading aloud, and the recorded video and audio are saved on the device and saved in a format that can be played back later.

[1072] Example 3

[1073] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1074] Conventional systems for recording reading aloud involve the cumbersome process of saving and playing back video data, making it difficult for users to easily save and play back their memories. Furthermore, they lack the functionality to generate new stories by combining existing stories, limiting users' opportunities to enjoy new stories. Furthermore, there is no way to print the generated stories on paper, making it impossible to physically store the stories. To address these issues, the present invention aims to provide a system that can easily save and play back reading aloud, and a function to generate new stories by combining existing stories.

[1075] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1076] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using a generative AI-based AI and creating a story based on them, means for printing the created story on paper, means for recording the storytelling and saving it as a memory, means for uploading the recorded video data to cloud storage, means for receiving the video data from the cloud storage and saving it in a database, means for playing the saved video data, means for inputting an existing story and inputting a prompt sentence to the generative AI model, means for the generative AI model to generate a new story, and means for saving the generated story and providing it to the user. This allows users to easily save and play back storytelling, as well as to combine existing stories to create and enjoy new stories.

[1077] "Generative AI-based AI" is an artificial intelligence technology that automatically generates new data and content based on existing data.

[1078] "Sound effects" are sound effects used to give a sense of realism to a story or image.

[1079] "Music" is the art of sound created by combining elements such as melody, harmony, and rhythm.

[1080] "Reading aloud" is the act of reading books or stories aloud, often to children.

[1081] A "narrative" is a story that unfolds through events and the actions of characters.

[1082] "Printing to paper" is the process of transferring digital data onto a physical paper medium.

[1083] "Filming" is the act of recording footage using a camera or video camera.

[1084] "Saving as a memory" means recording a particular event or experience so that it can be replayed later.

[1085] "Cloud storage" is a collection of remote servers that store and access data over the Internet.

[1086] A "database" is a system for efficiently storing, managing, and retrieving data.

[1087] "Playback" refers to the act of displaying or playing back stored audio or video data.

[1088] An "existing narrative" is a story that has already been created and is widely known.

[1089] A "generative AI model" is an algorithm or model that uses artificial intelligence to generate new data or content.

[1090] A "prompt sentence" is an input sentence that gives specific instructions or questions to a generative AI model.

[1091] A "new story" is a new story generated based on existing stories and elements.

[1092] "Providing to users" means making the generated data and content available to users.

[1093] This invention provides a system that records the reading session and saves it as a memory, and a system that combines existing stories to create a new story. Specific embodiments of this system are described below.

[1094] First, the user uses a device such as a smartphone or tablet to record a video of the picture book being read aloud. Specifically, the user launches the smartphone's camera app and presses the record button to record the reading. Once the recording is complete, the device uploads the video data to a cloud storage service (e.g., Google Drive or Dropbox). The user then presses the share button on the camera app to upload the video data to cloud storage.

[1095] Next, the server receives the uploaded video data from the cloud storage. Specifically, it downloads the video data using the cloud storage's API and stores it in a database management system (e.g., MySQL or MongoDB). When storing the video data, it saves it in an appropriate format (e.g., MP4 format). This allows users to play the saved video data later.

[1096] Furthermore, users can input existing stories through the system interface. Specifically, they input the title and content of the story into a text box. The server inputs a prompt into the generative AI model (e.g., GPT-4) based on the story input by the user. An example of a prompt is, "Please combine the stories of Momotaro and Kintaro to create a new story."

[1097] The generative AI model generates a new story based on the prompt. Specifically, the AI ​​combines elements of "Momotaro" and "Kintaro" to generate a new story such as "Momotaro vs. Kintaro." The generated story is stored in a database by the server and made available to users for later reading.

[1098] For example, if a user wants to create a new story by combining the stories of "Momotaro" and "Kintaro," the user inputs the stories of "Momotaro" and "Kintaro" into the system. The server receives this and inputs the following prompt sentence into the generative AI model:

[1099] "Combine the stories of Momotaro and Kintaro to create a new story."

[1100] Based on this prompt, the generative AI model generates a new story, such as "Momotaro vs. Kintaro." The generated story is saved as text data and can be read by the user later.

[1101] In this way, the system of the present invention not only allows you to easily save and play back the story-telling, but also allows you to combine existing stories to create new stories and enjoy them. The flow of the specific processing in the third embodiment will be described with reference to Figure 15.

[1102] Step 1:

[1103] The user takes a photo of the reading aloud

[1104] The user uses a device such as a smartphone or tablet to record a video of the picture book being read aloud. Specifically, the user starts the smartphone's camera app and presses the record button to record the reading. The input is the user's operation, and the output is video data.

[1105] Step 2:

[1106] The device uploads the video data to cloud storage.

[1107] The device uploads the captured video data to a cloud storage service (e.g., Google Drive or Dropbox). Specifically, the user selects the video data in the camera app and presses the share button to upload it to cloud storage. The input is the video data, and the output is the video data saved in the cloud storage.

[1108] Step 3:

[1109] The server receives the video data and stores it in a database.

[1110] The server receives the video data uploaded from the cloud storage. Specifically, it downloads the video data using the cloud storage's API and stores it in a database management system (e.g., MySQL or MongoDB). The input is the video data from the cloud storage, and the output is the video data stored in the database.

[1111] Step 4:

[1112] User plays a saved video

[1113] Users can play stored videos through the system's interface. Specifically, they use the system's web app or mobile app to select the video they want to play from a list of stored videos and press the play button. The input is the video data stored in the database, and the output is the video being played.

[1114] Step 5:

[1115] User enters an existing story

[1116] The user inputs an existing story (e.g., "Momotaro" and "Kintaro") through the system interface. Specifically, the user inputs the title and content of the story into a text box. The input is the text data of the story entered by the user, and the output is the text data sent to the server.

[1117] Step 6:

[1118] The server inputs a prompt to the generative AI model.

[1119] The server inputs a prompt to the generative AI model (e.g., GPT-4) based on the story input by the user. Specifically, it sends the prompt, "Please combine the stories of Momotaro and Kintaro to create a new story," to the generative AI model. The input is the text data of the story from the user, and the output is the prompt sent to the generative AI model.

[1120] Step 7:

[1121] Generative AI models generate new stories

[1122] The generative AI model generates a new story based on the prompt. Specifically, the AI ​​combines elements of "Momotaro" and "Kintaro" to generate a new story such as "Momotaro vs. Kintaro." The input is the prompt, and the output is the text data of the generated story.

[1123] Step 8:

[1124] The server stores the generated stories and serves them to the user.

[1125] The server receives the generated stories and stores them in a database. The stored stories are then made available to users for later reading. The input is the text data of the generated stories, and the output is the stories stored in the database.

[1126] (Application example 3)

[1127] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1128] Traditional methods for recording storytimes have limited the means to preserve storytime memories and have made it impossible to combine existing stories to create new stories. Furthermore, there is a lack of easy ways to share saved videos or play them back at any time. This makes it difficult for users to effectively preserve storytime memories and enjoy new stories.

[1129] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1130] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on them, means for printing the created story on paper, means for recording the storytelling and saving it as a memory, means for uploading the recorded video to the cloud and making it playable at any time, and means for combining existing stories to create a new story. This allows users to effectively save the storytelling and play it back at any time, and further allows them to enjoy new stories by combining existing stories.

[1131] "Generative AI-based AI" is an artificial intelligence technology that generates new data based on existing data.

[1132] "Sound effects" are sound effects used to enhance the sense of realism in stories and images.

[1133] "Music" is the art of sound created by combining elements such as melody, harmony, and rhythm.

[1134] "Reading aloud" is the act of reading a book or story aloud to others.

[1135] A "narrative" is a written or visual form that describes events and the actions of characters.

[1136] "Printing on paper" is the act of transferring digital data onto a physical paper medium.

[1137] "Filming the reading aloud" means recording the act of reading aloud with a video camera or other filming equipment.

[1138] "Saving as a memory" means recording a particular event or action in a form that can be played back later.

[1139] "Uploading to the cloud" is the act of storing digital data on a remote server over the internet.

[1140] "Making it playable at any time" means that stored digital data can be played back at any time.

[1141] "Generating a new story by combining existing stories" means creating a new story based on multiple stories that already exist.

[1142] A system for implementing this invention is configured as follows: First, a user uses a smartphone to record the reading aloud. The smartphone is equipped with a video capture function that uses OpenCV, and can record the reading aloud as a high-quality video.

[1143] The captured video is uploaded to cloud storage (e.g., Google Drive) using rclone, allowing users to play the video anytime, anywhere.

[1144] Next, a generative AI-based AI is used to combine existing stories to generate new stories. Specifically, the transformers library is used to input a prompt into the GPT-3 model to generate a new story. An example of this prompt is "What would happen if Momotaro and Kintaro met?"

[1145] The new story is then displayed on the smartphone for the user to enjoy, and can also be printed on paper if desired.

[1146] The specific process of this system is as follows: First, the reading is filmed using the smartphone camera and saved in AVI format. Next, the saved video is uploaded to cloud storage using rclone. Next, a prompt sentence is input into the GPT-3 model to generate a new story. Finally, the generated story is displayed on the smartphone and, if necessary, printed on paper.

[1147] This system allows users to effectively save the storytelling and play it back at any time, and also allows users to combine existing stories to create new stories.

[1148] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1149] Step 1:

[1150] The user activates the camera on their smartphone and takes a picture of the reading aloud.

[1151] Input: Smartphone camera image

[1152] Output: AVI format video file

[1153] Specific operation: The user launches the camera app on their smartphone and presses the record button to record the reading. Once the recording is complete, the video file is saved in the smartphone's storage.

[1154] Step 2:

[1155] The device uploads the captured video file to cloud storage.

[1156] Input: AVI format video file

[1157] Output: Video files on cloud storage

[1158] Specific operation: The device uses rclone to upload the video file to the specified cloud storage (e.g., Google Drive). Once the upload is complete, the video file is saved in the cloud.

[1159] Step 3:

[1160] Users input prompts into a generative AI-based AI to generate new stories.

[1161] Input: Prompt sentence (e.g. "What would happen if Momotaro and Kintaro met?")

[1162] Output: Text data of the new story

[1163] How it works: A user launches a smartphone application and enters a prompt. The device then uses the transformers library to send the prompt to the GPT-3 model, which then receives the text data for the new story.

[1164] Step 4:

[1165] The terminal displays the new story that has been generated.

[1166] Input: Text data of a new story

[1167] Output: A story displayed on a smartphone screen

[1168] Specific operation: The device displays the received text data of the new story on the smartphone screen, and the user can read the generated story.

[1169] Step 5:

[1170] The user prints the generated story on paper if desired.

[1171] Input: Text data of a new story

[1172] Output: A printed version of the story

[1173] Specific operation: When a user presses the print button on the smartphone application, the device sends text data to the printer and the story is printed on paper.

[1174] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1175] "Example 1"

[1176] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes a user's emotions and adjusts the generation of a story according to the user's emotions. Specifically, when the user is feeling happy, the system generates a happy story, and when the user is sad, the system generates a comforting story. The emotion engine can also use facial recognition technology or voice recognition technology to recognize the user's emotions.

[1177] "Example 2"

[1178] Another embodiment of the present invention provides a system incorporating an emotion engine. This system recognizes the user's emotion and adjusts the generation of storytelling according to the emotion. Specifically, when the user feels surprised, the system generates storytelling that includes elements of surprise, and when the user feels relieved, the system generates storytelling that gives a sense of relief.

[1179] "Example 3"

[1180] Furthermore, as another embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes the emotion of a user and adjusts the generation of sound effects and music according to the emotion. Specifically, when the user is excited, fast-paced music and sound effects are generated, and when the user is relaxed, slower music and sound effects are generated.

[1181] "Example 4"

[1182] Another embodiment of the present invention provides a system incorporating an emotion engine that recognizes the user's emotions and adjusts the printing of the story accordingly, specifically printing bright colors when the user is happy and muted colors when the user is sad.

[1183] The processing flow of each embodiment will be described below.

[1184] "Example 1"

[1185] Step 1: Recognize the user's emotions with the emotion engine.

[1186] Step 2: Adjust your narrative production according to the emotions you recognize.

[1187] Step 3: Serve the adjusted story to the user.

[1188] "Example 2"

[1189] Step 1: Recognize the user's emotions with the emotion engine.

[1190] Step 2: Adjust your storytelling production according to the emotions you recognize.

[1191] Step 3: Provide the adapted reading to the user.

[1192] "Example 3"

[1193] Step 1: Recognize the user's emotions with the emotion engine.

[1194] Step 2: Adjust the sound effects and music generation according to the recognized emotion.

[1195] Step 3: Provide the adjusted sound effects and music to users.

[1196] "Example 4"

[1197] Step 1: Recognize the user's emotions with the emotion engine.

[1198] Step 2: Adjust the print of the story according to the emotions you recognize.

[1199] Step 3: Print the adjusted story and provide it to the user.

[1200] Example 1

[1201] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1202] Conventional story generation systems have had difficulty generating stories that respond to the user's emotions or creating digital picture books that combine sound effects, music, and audio reading. Furthermore, they lacked the functionality to recognize the user's emotions and adjust the content of the story, making it impossible to provide a story that was in tune with the user's emotions. Furthermore, the process of sending the generated story to the device and displaying it to the user was not automated, limiting the user experience.

[1203] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1204] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using a generative AI-based AI and creating a story based on them, means for receiving input from a user and generating a story by inputting a prompt sentence into a generative AI model, means for generating sound effects and music based on the generated story, means for generating audio for storytelling, means for creating a digital picture book by combining the generated story, sound effects, music, and storytelling audio, means for transmitting the created digital picture book to a terminal, means for the terminal to display the digital picture book to the user, means for recognizing the user's emotions and adjusting the generation of the story in accordance with the emotions, and means for recognizing the user's emotions using facial recognition technology and voice recognition technology. This makes it possible to generate a story in accordance with the user's emotions and automatically create a digital picture book by combining sound effects, music, and storytelling audio.

[1205] "Generative AI-based AI" is an artificial intelligence technology that automatically generates stories, sound effects, music, reading audio, and more based on user input.

[1206] A "prompt" is a text sentence that gives a generative AI model instructions for generating a story or other content.

[1207] "Narrative" refers to stories or narratives generated by generative AI-based AI.

[1208] "Sound effects" are sound effects generated to match the scenes in the story.

[1209] "Music" refers to songs and melodies that are generated to match the atmosphere and scenes of the story.

[1210] "Reading audio" refers to audio data in which the generated story is read aloud using speech synthesis technology.

[1211] A "digital picture book" is an electronic picture book created by combining generated stories, sound effects, music, and audio readings.

[1212] "User emotion" refers to the emotional state that the user is feeling, such as joy, sadness, surprise, etc.

[1213] "Facial recognition technology" is a technology that captures a user's face with a camera or other device and recognizes emotions from their facial expressions.

[1214] "Voice recognition technology" is a technology that records a user's voice using a microphone or other device and recognizes emotions from that voice.

[1215] A "terminal" refers to an electronic device used by a user, such as a smartphone, tablet, or PC.

[1216] This invention is a system that uses generative AI to automatically generate elements such as sound effects, music, and storytelling, and then creates a story based on these elements. Specific embodiments of this system are described below.

[1217] System configuration

[1218] This system is mainly composed of three elements: a server, a terminal, and a user. The server is responsible for generating stories, sound effects, music, and reading audio using a generative AI model. The terminal is a device through which the user inputs and displays the generated digital picture book. The user inputs the theme and keywords of the story and enjoys the generated digital picture book.

[1219] Hardware and software used

[1220] The server is a high-performance computer equipped with software for running generative AI models (e.g., OpenAI's GPT-4). It also uses music generation AI (e.g., Jukedeck or Amper Music) and speech synthesis software (e.g., Google Text-to-Speech or Amazon Polly). For facial recognition, a common facial recognition API (e.g., Microsoft Azure Face API) is used, and for speech recognition, a common speech recognition API (e.g., Google Speech-to-Text) is used.

[1221] Program processing

[1222] The server receives input from the user and generates a story by inputting prompt sentences into the generative AI model. For example, if the user inputs the theme "Pirate Adventure," the server inputs the following prompt sentences into the generative AI model:

[1223] Example prompt sentence:

[1224] "Create a story about a pirate adventure. The story includes brave pirates, treasure maps, and dangerous voyages."

[1225] The generative AI model generates a story based on this prompt. The server then analyzes the content of the generated story and uses the music generation AI to generate sound effects and music that match the scene. For example, the sound of waves and wind is generated for a scene where a pirate ship is sailing.

[1226] The server then inputs the story text into speech synthesis software to generate a reading voice, such as a voice that emotively recites the lines of a brave pirate.

[1227] Specific examples

[1228] The user opens the smartphone application and enters the theme "Pirate Adventure." The device sends this input data to the server. The server generates prompt text and inputs it into a generative AI model to generate a story. Based on the generated story, it generates sound effects and music, as well as a reading audio. These elements are combined to create a digital picture book, which is then sent to the device. The device then displays the received digital picture book to the user, who can enjoy the sound effects, music, and reading audio as they read the story.

[1229] In this way, it becomes possible to generate stories that correspond to the user's emotions and automatically generate digital picture books that combine sound effects, music, and storytelling audio.

[1230] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1231] Step 1:

[1232] The user inputs a theme or keyword.

[1233] The user uses the input interface of the terminal to input the theme and keywords of the story to be generated. For example, the theme "Pirate Adventure" is input. The input data is "Pirate Adventure."

[1234] Step 2:

[1235] The terminal sends the input data to the server.

[1236] The terminal sends the theme and keywords entered by the user to the server. This data is used to create prompts for story generation. The input data is "Pirate Adventure," and the output is the theme data sent to the server.

[1237] Step 3:

[1238] The server inputs prompt sentences into the generative AI model to generate a story.

[1239] The server creates a prompt based on the received theme and keywords and inputs it into the generative AI model. For example, a prompt such as "Create a story about a pirate adventure. The story includes brave pirates, a treasure map, and a dangerous voyage" is input into the generative AI model. The generative AI model generates a story based on this prompt. The input data is theme data, and the output is the generated story.

[1240] Step 4:

[1241] The server generates sound effects and music based on the generated story.

[1242] The server analyzes the content of the generated story and uses music generation AI to generate sound effects and music that match the scene. For example, the sound of waves and wind is generated for a scene where a pirate ship is sailing. The input data is the generated story, and the output is sound effects and music.

[1243] Step 5:

[1244] The server generates audio for reading the story.

[1245] The server inputs the story text into speech synthesis software and generates a reading voice, for example, a voice that reads the lines of a brave pirate with great emotion. The input data is the story text, and the output is the reading voice.

[1246] Step 6:

[1247] The server combines the generated story, sound effects, music, and reading audio to create a digital picture book.

[1248] The server combines the generated story, sound effects, music, and audio reading into a single file to create a digital picture book. This picture book is designed to be enjoyed both visually and aurally by the user. The input data are the story, sound effects, music, and audio reading, and the output is a digital picture book.

[1249] Step 7:

[1250] The server sends the completed digital picture book to the terminal.

[1251] The server sends the completed digital picture book to the terminal, which saves the picture book in a format that the user can access. The input data is the digital picture book, and the output is the digital picture book sent to the terminal.

[1252] Step 8:

[1253] The terminal displays the digital picture book to the user.

[1254] The device displays the received digital picture book to the user. The user can read the picture book through the device and enjoy sound effects, music, and audio storytelling. For example, a user can pick up a tablet, tap the screen to open the picture book, read the story, and listen to sound effects and music. The input data is the digital picture book, and the output is the digital picture book displayed to the user.

[1255] (Application example 1)

[1256] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1257] Conventional story generation systems have difficulty generating stories that respond to the user's emotions, making it impossible to provide a personalized experience for each user. Furthermore, they are unable to combine sound effects or music with the generated stories, making it difficult to enhance the sense of realism of the story. Furthermore, they are unable to read the generated stories aloud using a text-to-speech function, making it impossible to provide a consistent storytelling experience for users.

[1258] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1259] In this invention, the server includes: means for recognizing a user's emotions; means for generating a story using a prompt sentence that instructs the generation of a story according to the user's emotions and a generative AI; means for generating sound effects and music according to the generated story; means for reading the generated story using a text-to-speech function; and means for playing the generated sound effects and music in sync with the read story. This makes it possible to generate a personalized story according to the user's emotions and provide an immersive storytelling experience that combines sound effects and music. The server further includes means for recording the user's behavior while the story is being read aloud and means for saving the recorded video.

[1260] "Generative AI-based AI" is an artificial intelligence technology that automatically generates stories, sound effects, music, etc. based on user input.

[1261] "Sound effects" are sound effects generated to match the scenes and situations in the story.

[1262] "Music" is music generated to enhance the atmosphere and emotion of the story.

[1263] "Reading aloud" is the act of reading the generated story aloud.

[1264] A "narrative" is a story created by an AI based on generative AI.

[1265] "Means of printing on paper" refers to the technology used to print the generated story onto physical paper media.

[1266] "Means to preserve memories" refers to a technique for recording the reading aloud and saving it so that it can be played back later.

[1267] "Means for recognizing user emotions" refers to technology that detects user emotions using facial recognition technology or voice recognition technology.

[1268] The "means for adjusting story generation according to emotions" is a technology that changes the content of a story based on the recognized emotions of a user.

[1269] "Facial recognition technology" is a technology that detects a user's face from camera footage and analyzes their facial expressions.

[1270] "Voice recognition technology" is a technology that analyzes the user's voice and understands their emotions and intentions.

[1271] "Text-to-speech" is a technology that reads the generated story aloud.

[1272] "Means for generating sound effects and music and playing them in accordance with the story" refers to technology that automatically generates sound effects and music according to the scenes in the story and plays them in accordance with the progress of the story.

[1273] The system for implementing this invention uses generative AI-based AI to automatically generate elements such as sound effects, music, and storytelling, and then creates a story based on these elements. The system has the ability to recognize the user's emotions and adjust the story generation accordingly. Specific embodiments of this system are described below.

[1274] Hardware and software used

[1275] Hardware: Uses the camera and microphone on your smartphone or head-mounted display.

[1276] software:

[1277] OpenCV: Used for camera image acquisition and face recognition.

[1278] EmotionRecognition: Used as an emotion recognition model.

[1279] GPT-4: Used as a generative AI to generate stories.

[1280] Librosa: Used to generate sound effects and music.

[1281] TextToSpeech: Used as a text-to-speech function.

[1282] System Operation

[1283] 1. Emotion recognition:

[1284] When a user activates the system through a smartphone or head-mounted display, it uses a camera and microphone to capture the user's face and voice.

[1285] Facial recognition is performed using OpenCV, and the EmotionRecognition model is used to recognize user emotions.

[1286] 2. Narrative Generation:

[1287] Based on the recognized emotion, a prompt sentence for story generation is generated. For example, if the user indicates the emotion "joy", the following prompt sentence is generated:

[1288] "The user's emotion is 'joy.' Please generate a fun story that matches this emotion."

[1289] This prompt sentence is input into a generative AI to generate a story.

[1290] 3. Sound effects and music generation:

[1291] Librosa is used to generate sound effects and music to match the generated story, providing sound effects that correspond to the scenes in the story.

[1292] 4. Reading aloud:

[1293] The generated story is then read aloud using the TextToSpeech feature, providing users with a consistent storytelling experience.

[1294] Specific examples

[1295] For example, if a user activates the system through their smartphone camera and the emotion is recognized as "joy," the following prompt will be generated:

[1296] "The user's emotion is 'joy.' Please generate a fun story that matches this emotion."

[1297] When this prompt is input into the generative AI, an entertaining story is generated. Sound effects and music are automatically generated to match the generated story, and the story is read aloud using the Text-to-Speech function. This allows users to enjoy a personalized story that responds to their emotions and an immersive storytelling experience.

[1298] In this way, the system is able to consistently generate stories that respond to the user's emotions, as well as generate accompanying sound effects and music and read aloud.

[1299] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1300] Step 1:

[1301] The user activates the system through a smartphone or head-mounted display. The system captures the user's face and voice using a camera and microphone. The input is the camera image and audio data, and the output is the captured image and audio.

[1302] Step 2:

[1303] The server recognizes faces from the captured video using OpenCV. The result of the face recognition is the position and feature points of the user's face. The input is the captured video, and the output is the position and feature points of the face.

[1304] Step 3:

[1305] The server uses the EmotionRecognition model to recognize the user's emotions from facial feature points and voice data. The input is facial feature points and voice data, and the output is the recognized emotion (e.g., joy, sadness, anger).

[1306] Step 4:

[1307] The server generates a prompt sentence for story generation based on the recognized emotion. For example, if the user expresses the emotion of "joy," the server generates the prompt sentence: "The user's emotion is 'joy.' Please generate a fun story that matches this emotion." The input is the recognized emotion, and the output is the prompt sentence.

[1308] Step 5:

[1309] The server inputs the generated prompt sentences into the generative AI to generate a story. The input is the prompt sentence, and the output is the text of the generated story.

[1310] Step 6:

[1311] The server uses Librosa to generate sound effects and music to match the generated story. The input is the text of the generated story, and the output is the sound effects and music data.

[1312] Step 7:

[1313] The server reads the generated story aloud using the TextToSpeech function: the input is the text of the generated story, and the output is the spoken audio.

[1314] Step 8:

[1315] The device plays the generated sound effects and music in accordance with the story progression. The input is sound effect and music data and spoken voice, and the output is a consistent storytelling experience for the user.

[1316] Step 9:

[1317] The device uses a camera to capture the user's behavior while reading a story and saves the resulting video as a memory. The input is the video of the reading, and the output is the saved video data.

[1318] It is also possible to repeatedly generate a story and sound effects and music. For example, the user's emotions while the story is being read aloud are recognized, and a continuation of the story is generated using a prompt sentence that instructs the generation of a story according to the user's emotions and the generative AI, and sound effects and music are generated according to the generated continuation of the story. The generated story is read aloud using a text-to-speech function, and the generated sound effects and music are played in accordance with the story being read aloud. These processes are repeated until the end of the story.

[1319] Example 2

[1320] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1321] Conventional picture book creation systems require users to manually create stories and add sound effects and music, which takes time and effort. Furthermore, it is difficult to provide an optimal experience for each individual user because the reading cannot be adjusted based on the user's emotions. Furthermore, the time and effort required to print digital picture books on paper is also a problem.

[1322] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using a generative AI-based AI and creating a story based on these elements, means for printing the created story on paper, means for recognizing the user's emotions and adjusting the generation of the storytelling in accordance with the emotions, means for transmitting digital picture book data to the terminal, and means for the terminal to receive the digital picture book data and print it on paper using a printer. This allows users to easily create the optimal picture book that suits their individual emotions and obtain a physical picture book.

[1323] "Generative AI-based AI" is a system that uses artificial intelligence technology to automatically generate elements such as stories, sound effects, music, and storytelling.

[1324] The "means for creating a story" is a function that uses generative AI-based AI to generate a new story based on a prompt sentence entered by the user.

[1325] "Means for printing on paper" refers to a device such as a printer for printing the generated digital picture book data on physical paper.

[1326] "Means for recognizing user emotions" refers to technology that analyzes emotions from the user's facial expressions, voice, etc., and identifies those emotions.

[1327] The "means for adjusting the generation of storytelling" is a function for changing the content and tone of storytelling according to the user's emotions.

[1328] The "means for transmitting digital picture book data to a terminal" is a function for transmitting the generated digital picture book data to a user's terminal via a network.

[1329] "Means for the terminal to receive digital picture book data" refers to the function by which the user's terminal receives digital picture book data sent from the server.

[1330] "Means for printing onto paper using a printer" refers to a device for printing onto physical paper the digital picture book data received by the terminal.

[1331] This invention is a system that uses generative AI to automatically generate stories, sound effects, music, and storytelling, and then creates a digital picture book based on these and prints it on paper. It also has the ability to recognize the user's emotions and adjust the storytelling generation accordingly.

[1332] The server uses a generative AI model (e.g., a natural language processing model) to generate a story based on the prompt entered by the user. It also uses speech synthesis software (e.g., a speech synthesis engine) to add sound effects and music to the generated story. It also uses an emotion engine (e.g., emotion analysis software) to recognize the user's emotions and generate a reading voice that corresponds to those emotions.

[1333] The device receives the digital picture book data sent from the server and prints it on paper using a printer (e.g., an inkjet printer), allowing the user to obtain a physical picture book.

[1334] As a concrete example, if the user inputs a prompt such as "Please write an adventure story for a 5-year-old child," the following processing will be performed.

[1335] 1. The user inputs through the system interface, "I want you to create an adventure story for a 5-year-old child."

[1336] 2. The server uses the generative AI model to generate an "adventure story for 5-year-olds."

[1337] 3. The server uses speech synthesis software to add sound effects and music to the generated story.

[1338] 4. The server uses an emotion engine to recognize the user's emotions and generates a reading audio that includes an element of surprise.

[1339] 5. The server combines the generated story, sound effects, music, and reading audio to create digital picture book data and send it to the terminal.

[1340] 6. The device receives the digital picture book data and prints it on paper using an inkjet printer.

[1341] 7. The user receives a physical printed picture book.

[1342] In this way, users can obtain a physical picture book containing a story tailored to their needs.

[1343] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1344] Step 1:

[1345] The user inputs a prompt sentence. The user inputs "I want you to create an adventure story for a 5-year-old child" through the system interface. The input prompt sentence is sent to the server.

[1346] Step 2:

[1347] The server receives the prompt sentence and generates a story using a generative AI model. The server uses the received prompt sentence as input and generates a story using a generative AI model (e.g., a natural language processing model). The generated story is output as text data.

[1348] Step 3:

[1349] The server adds sound effects and music to the generated story. The server uses the text data of the generated story as input and generates sound effects and music using speech synthesis software (e.g., a speech synthesis engine). The generated sound effects and music are output as audio data.

[1350] Step 4:

[1351] The server uses an emotion engine to recognize the user's emotions and generates a reading voice. The server uses an emotion engine (e.g., emotion analysis software) to recognize the user's emotions using the user's facial expressions and voice data as input. The server generates a reading voice based on the recognized emotion data. The generated reading voice is output as audio data.

[1352] Step 5:

[1353] The server sends the digital picture book data to the terminal. The server then combines the generated story, sound effects, music, and reading audio to create digital picture book data. This digital picture book data is then sent to the terminal via the network.

[1354] Step 6:

[1355] The terminal receives the digital picture book data and prints it on paper using a printer. The terminal receives the digital picture book data sent from the server. Using the received digital picture book data as input, it prints it on paper using a printer (e.g., inkjet printer). The printed picture book is output in a physical form.

[1356] Step 7:

[1357] The user receives the physical picture book. The user receives the physical picture book printed from the terminal. This allows the user to obtain a picture book containing the story that suits their preferences.

[1358] (Application example 2)

[1359] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1360] Previous picture book creation systems were unable to adjust the reading experience based on the user's emotions, and it was difficult to save the book digitally or display it on a smartphone. This limited the user experience and made it difficult to customize it to meet individual needs.

[1361] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1362] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on these, means for printing the created story on paper, means for taking pictures of the storytelling and saving them as memories, means for recognizing the user's emotions and adjusting the content of the storytelling according to those emotions, and means for saving the created story in digital format and displaying it on a smartphone. This enables customized storytelling according to the user's emotions, and also enables it to be saved in digital format and displayed on a smartphone.

[1363] "Generative AI-based AI" is an artificial intelligence technology that automatically generates elements such as stories, sound effects, music, and storytelling.

[1364] "Sound effects" are sounds used to enhance the sense of realism and emotion in a story or video.

[1365] "Music" refers to the melodies and rhythms used to create emotion and atmosphere in a story or film.

[1366] "Reading aloud" is the act of reading a story aloud, and is used especially in picture books for children.

[1367] A "narrative" is a series of stories that unfold through characters and events.

[1368] "Paper media" refers to the technology and devices used to transfer digitally created stories onto physical paper.

[1369] "Means for filming the reading aloud" refers to techniques and devices for recording the act of reading aloud on video.

[1370] "Means of preserving memories" refers to technologies and devices that allow for the long-term preservation of filmed footage and created stories.

[1371] "Means for recognizing user emotions" refers to technology or devices for analyzing and recognizing emotions from the user's facial expressions, voice, etc.

[1372] "Means for adjusting the content of reading aloud according to emotions" refers to techniques or devices for changing the content or tone of reading aloud based on the user's recognized emotions.

[1373] "Digital preservation means" refers to the technology or devices used to preserve the generated narratives and associated data in electronic form.

[1374] "Devices for displaying on smartphones" refers to technologies and devices for displaying digitally stored stories and related data on a smartphone screen.

[1375] A system for implementing this invention includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on these elements, means for printing the created story on paper, means for filming the storytelling and saving it as a memory, means for recognizing the user's emotions and adjusting the content of the storytelling in accordance with those emotions, and means for saving the created story in a digital format and displaying it on a smartphone.

[1376] Program processing explanation

[1377] Hardware

[1378] Smartphone: A device through which users can enter prompts and view the generated narrative.

[1379] Camera: Used to recognize the user's emotions.

[1380] Printer: Used to print the created story on paper.

[1381] software

[1382] Generative AI models: Used to generate stories, sound effects, music, and storytelling, such as OpenAI's GPT-3.

[1383] Emotion Recognition Software: Analyzes user emotions using OpenCV and Google Cloud Natural Language API.

[1384] PDF Generation Software: FPDF was used to convert the generated stories into PDF format.

[1385] Data processing and calculation

[1386] 1. Prompt Input: The user uses their smartphone to input a story prompt.

[1387] 2. Emotion Recognition: Capture the user's face using the smartphone camera and analyze emotions using OpenCV and Google Cloud Natural Language API.

[1388] 3. Narrative generation: Use a generative AI model (e.g., GPT-3) to generate a narrative based on user prompts and perceived emotions.

[1389] 4. Digital preservation: The generated stories are saved in digital format and displayed on a smartphone.

[1390] 5. Print to paper: If necessary, convert the generated story into PDF format and print it on paper using a printer.

[1391] Specific examples

[1392] Example prompt: "Create a story about a brave child fighting a dragon."

[1393] Emotion recognition example: If the user is feeling surprised, add an element of surprise to the story.

[1394] The system allows users to generate customized stories based on their emotions and save them digitally or print them on paper, enriching the user experience and enabling picture books to be created to suit individual needs.

[1395] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1396] Step 1:

[1397] A user inputs a story prompt using a smartphone, which is then sent to a generative AI model. The input data is the prompt in text format, and the output data is the prompt passed to the generative AI model.

[1398] Step 2:

[1399] The device's camera is used to capture the user's face. The captured image is sent to emotion recognition software. The input data is image data, and the output data is image data passed to the emotion recognition software.

[1400] Step 3:

[1401] The server uses emotion recognition software (OpenCV and Google Cloud Natural Language API) to analyze the user's emotions from the captured image. The input data is image data, and the output data is analyzed emotional information. Specifically, it analyzes facial expressions and identifies emotions such as surprise, joy, and sadness.

[1402] Step 4:

[1403] The server uses a generative AI model (e.g., GPT-3) to generate a story based on the user's prompt and the analyzed emotional information. The input data is the prompt and the emotional information, and the output data is the generated story. Specifically, the server combines the emotional information with the prompt to adjust the content and tone of the story.

[1404] Step 5:

[1405] The server saves the generated story in digital format and displays it on the smartphone. The input data is the generated story, and the output data is a digital story file. Specifically, the story is saved in text file or PDF format and displayed on the smartphone screen.

[1406] Step 6:

[1407] The user prints the generated story on paper as desired. The input data is a digital story file, and the output data is the printed story. Specifically, the digital story is sent to a printer, which prints it on paper.

[1408] Step 7:

[1409] The device records the reading and saves it as a memory. The input data is the video of the reading, and the output data is the saved video file. Specifically, the device records the reading and saves it as a video file.

[1410] Example 3

[1411] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1412] Conventional storytelling and story generation systems were unable to generate sound effects and music based on the user's emotions or adjust the printing of stories, making it difficult to provide an optimal experience for each individual user. Furthermore, they lacked the ability to combine existing stories to create new stories, limiting the opportunities for users to enjoy new stories. Furthermore, there were also insufficient ways to save the storytelling experience as a memory.

[1413] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1414] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI, means for printing the created story on paper, means for filming the storytelling and saving it as a memory, means for recognizing the user's emotions and adjusting the generation of sound effects and music according to those emotions, means for adjusting the printing of the story according to the user's emotions, and means for creating a new story by combining existing stories. This makes it possible to provide an optimal experience according to the user's emotions, increase opportunities to enjoy new stories, and save the storytelling as a memory.

[1415] "Generative AI-based AI" is an artificial intelligence technology that automatically generates new data and content based on existing data.

[1416] "Sound effects" are sound effects used to emphasize particular scenes or emotions in a story or film.

[1417] "Music" is an art form of sound that combines elements such as melody, harmony, and rhythm.

[1418] "Reading aloud" is the act of reading a book or story aloud to others.

[1419] A "narrative" is a series of events and a story that unfolds through the actions of characters.

[1420] "Printing to paper" is the process of transferring digital data onto a physical paper medium.

[1421] "Photography" is the act of recording video or images using a camera.

[1422] "Saving as a memory" means recording and storing a specific event or experience in a way that can be replayed later.

[1423] "Recognizing the user's emotions" means analyzing and determining the user's emotional state from their facial expressions, voice, etc.

[1424] "Adjusting the generation of sound effects and music" means selecting and playing appropriate sound effects and music according to the user's emotional state.

[1425] "Adjusting the print of a story" means changing the color tone and design of the print depending on the user's emotional state.

[1426] "Creating a new story by combining existing stories" means generating a new story based on multiple stories that already exist.

[1427] MODE FOR CARRYING OUT THE INVENTION

[1428] This system allows users to record themselves reading a picture book aloud and save the video as digital data. It also has the ability to create new stories by combining existing stories, and adjust sound effects, music, and story printing according to the user's emotions.

[1429] Hardware and software used

[1430] Hardware: Cameras, microphones, printers, computers

[1431] Software: Generative AI models, emotion recognition engines, video editing software

[1432] Specific Embodiments of the System

[1433] 1. Filming and saving story time

[1434] The user prepares to read a picture book aloud.

[1435] The device activates the camera and microphone.

[1436] The user begins reading aloud.

[1437] The device takes pictures of the reading and records the audio.

[1438] The device stores the video it captures as digital data.

[1439] The user plays the saved video.

[1440] Examples:

[1441] The user will film and save the footage of themselves reading "Momotaro" aloud.

[1442] Example prompt: "Please record and save the reading of Momotaro."

[1443] 2. Creating a new story

[1444] The user selects an existing story.

[1445] The server receives the selected story.

[1446] The server generates new stories using generative AI models.

[1447] The server provides the generated stories to the user.

[1448] Users enjoy new stories.

[1449] Examples:

[1450] The user selects "Momotaro" and "Kintaro," and the generative AI model creates a new story called "Momotaro vs. Kintaro."

[1451] Example prompt: "Create a new story by combining Momotaro and Kintaro."

[1452] 3. Emotion-based sound effects and music generation

[1453] The user boots the system.

[1454] The device analyzes the user's emotions using an emotion recognition engine.

[1455] The server receives the user's emotion data.

[1456] The server generates sound effects and music according to the emotion.

[1457] The device plays the generated sound effects and music.

[1458] Examples:

[1459] The user starts the system and speaks into the camera.

[1460] The device analyzes the camera footage using an emotion recognition engine (e.g., Affectiva) to detect whether the user is excited.

[1461] The server receives the user's emotional data and generates fast-paced music.

[1462] The server transmits the generated music data to the terminal.

[1463] The device plays fast-paced music to further heighten the user's excitement.

[1464] 4. Printing emotional stories

[1465] A user requests a printout of a story.

[1466] The device analyzes the user's emotions using an emotion recognition engine.

[1467] The server receives the user's emotion data.

[1468] The server sends instructions to the terminal to print the story in a color tone corresponding to the emotion.

[1469] The terminal prints the story in the specified color tone.

[1470] Examples:

[1471] A user requests printing of "Momotaro" through an application.

[1472] The device analyzes the camera footage using an emotion recognition engine to detect when the user is feeling joy.

[1473] The server receives the user's emotional data and sends an instruction to the terminal to print in bright colors.

[1474] The terminal instructs the printer to print "Momotaro" in bright colors.

[1475] The device will provide users with "Momotaro" printed in bright colors.

[1476] The above is a specific embodiment of the system of the present invention. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1477] Filming and saving story time

[1478] Step 1:

[1479] The user picks up a picture book and gets ready to start reading aloud.

[1480] Input: Picture book, reading intention

[1481] Output: Ready to read

[1482] Step 2:

[1483] The device will activate the built-in camera and microphone and set them to recording mode.

[1484] Input: User actions

[1485] Output: Camera and microphone activation

[1486] Step 3:

[1487] The user begins reading a picture book.

[1488] Input: Picture book, start reading

[1489] Output: Reading execution

[1490] Step 4:

[1491] The device captures high-resolution images of the user reading aloud and records the audio clearly.

[1492] Input: Camera video, microphone audio

[1493] Data processing: video and audio synchronization, noise removal

[1494] Output: High quality video data

[1495] Step 5:

[1496] The video recorded by the device will be saved in MP4 format and the file name will be "Reading_2023-10-01.mp4".

[1497] Input: High-quality video data

[1498] Data processing: file format conversion, file name setting

[1499] Output: Saved video file

[1500] Step 6:

[1501] Users can play saved videos and enjoy them with their families.

[1502] Input: Saved video file

[1503] Output: Video playback

[1504] Creating a new story

[1505] Step 1:

[1506] The user opens the application and selects an existing story.

[1507] Input: Application, selection of existing stories

[1508] Output: Selected stories

[1509] Step 2:

[1510] The server receives the user's selection and retrieves the selected story data from the database.

[1511] Input: Selected story

[1512] Data processing: Retrieving data from the database

[1513] Output: Story data

[1514] Step 3:

[1515] The server uses the generative AI model to generate new stories.

[1516] Input: Story data

[1517] Data Computation: Story Generation with Generative AI Models

[1518] Output: A new story

[1519] Step 4:

[1520] The server sends the generated story in text format to the user's device.

[1521] Input: A new story

[1522] Data processing: conversion to text format, data transmission

[1523] Output: Sending the story to the user's device

[1524] Step 5:

[1525] Users read and enjoy new stories.

[1526] Input: Submitted story

[1527] Output: Read and enjoy the story

[1528] Generating sound effects and music according to emotions

[1529] Step 1:

[1530] The user starts the system and speaks into the camera.

[1531] Input: System startup, camera footage

[1532] Output: Camera image acquisition

[1533] Step 2:

[1534] The device analyzes the camera footage using an emotion recognition engine to detect the user's emotions.

[1535] Input: Camera image

[1536] Data Calculation: Analysis by Emotion Recognition Engine

[1537] Output: Emotion data

[1538] Step 3:

[1539] The server receives the user's emotional data and generates fast-paced music.

[1540] Input: Emotion data

[1541] Data Computation: Applying Music Generation Algorithms

[1542] Output: Music data

[1543] Step 4:

[1544] The server transmits the generated music data to the terminal.

[1545] Input: Music data

[1546] Data processing: Data transmission

[1547] Output: Send music data to your device

[1548] Step 5:

[1549] The device plays fast-paced music to further heighten the user's excitement.

[1550] Input: Music data sent

[1551] Output: Music playback

[1552] Printing emotional stories

[1553] Step 1:

[1554] A user requests a printout of a story through the application.

[1555] Input: Print request

[1556] Output: Receiving a print request

[1557] Step 2:

[1558] The device analyzes the camera footage using an emotion recognition engine to detect the user's emotions.

[1559] Input: Camera image

[1560] Data Calculation: Analysis by Emotion Recognition Engine

[1561] Output: Emotion data

[1562] Step 3:

[1563] The server receives the user's emotional data and sends an instruction to the terminal to print in bright colors.

[1564] Input: Emotion data

[1565] Data processing: generating print instructions, sending data

[1566] Output: Sending print instructions

[1567] Step 4:

[1568] The terminal instructs the printer to print the story in bright colors.

[1569] Input: Print instructions

[1570] Output: Instructions to the printer

[1571] Step 5:

[1572] The device presents the user with a story printed in bright colors.

[1573] Input: Printed story

[1574] Output: Provide printed materials

[1575] (Application example 3)

[1576] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1577] Conventional storytelling systems lack the ability to record the storytelling process, or to generate new stories by combining existing stories. They also lack the ability to adjust sound effects and music according to the user's emotions, or to adjust the printing of the story, making it difficult to enrich the user experience.

[1578] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1579] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on those elements, means for printing the created story on paper, means for filming the storytelling and saving it as a memory, means for recognizing the user's emotions and adjusting the sound effects and music according to those emotions, means for generating a new story by combining existing stories, and means for adjusting the printing of the story according to the user's emotions. This allows the user to record the storytelling, create a new story, and enjoy sound effects, music, and printing according to their emotions.

[1580] "Generative AI-based AI" is an artificial intelligence technology that automatically generates new data and content based on existing data.

[1581] "Sound effects" are sound effects used to emphasize specific scenes or situations in content such as stories or videos.

[1582] "Music" is an art form of sound created by combining elements such as melody, harmony, and rhythm.

[1583] "Reading aloud" is the act of reading a book or story aloud to others.

[1584] A "narrative" is a form of writing or image that depicts a series of events, including characters, events, and settings.

[1585] "Printing to paper" is the process of transferring digital data onto a physical paper medium.

[1586] "Filming the reading aloud" means recording the act of reading aloud on video using a camera or other filming device.

[1587] "Saving as a memory" means preserving a specific event or action as digital data so that it can be replayed at a later date.

[1588] "Recognizing user emotions" refers to technology that determines a user's emotional state from their facial expressions, tone of voice, etc.

[1589] "Adjusting sound effects and music according to emotions" means changing the type and tempo of the sound effects and music to be played based on the emotional state of the user.

[1590] "Generating a new story by combining existing stories" means creating a new story by combining elements of multiple existing stories.

[1591] "Adjusting story printing according to emotions" means changing the color tone and design of the story when it is printed based on the user's emotional state.

[1592] A system for carrying out this invention has the following configuration: The server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on these, means for printing the created story on paper, means for recording the storytelling and saving it as a memory, means for recognizing the user's emotions and adjusting the sound effects and music according to those emotions, means for generating a new story by combining existing stories, and means for adjusting the printing of the story according to the user's emotions.

[1593] Program processing explanation

[1594] Recording and saving storytimes

[1595] The server uses a camera to record the user reading a picture book and saves the video as digital data. Specifically, it uses the OpenCV library to save the video data acquired from the camera in real time.

[1596] New story generation

[1597] The server uses a generative AI model to combine existing stories to generate new ones. For example, it combines "Momotaro" and "Kintaro" to generate a new story such as "Momotaro vs. Kintaro." This process uses a generative AI model that uses natural language processing technology.

[1598] Emotion recognition and music / sound effect adjustment

[1599] The server uses emotion recognition software to recognize the user's emotions. Specifically, it analyzes the user's facial expressions from video data to determine their emotional state. It also uses music generation software to adjust sound effects and music according to the user's emotional state. For example, it generates fast-paced music when the user is excited and slow-paced music when the user is relaxed.

[1600] Printing emotional stories

[1601] The server adjusts the printing of the story based on the user's emotions. Specifically, it uses emotion recognition software to determine the user's emotions and changes the color tone and design of the printout based on that emotion. For example, if the user is feeling happy, it will print in bright colors, and if they are sad, it will print in muted colors.

[1602] Examples and prompts

[1603] As a concrete example, the following prompt sentence can be input to a generative AI model:

[1604] Prompt Sentence Examples

[1605] Combine "Momotaro" and "Kintaro" to create a new story. The title of the new story is "Momotaro vs Kintaro."

[1606] This prompt can then be used to instruct a generative AI model to generate a new story.

[1607] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1608] Step 1:

[1609] The server uses a camera to record the user reading a picture book. As input, it receives the video data from the camera and saves it in real time using the OpenCV library. As output, the reading experience is saved as digital data.

[1610] Step 2:

[1611] The server uses a generative AI model to combine existing stories to generate a new story. As input, the generative AI model receives data from existing stories and a prompt. For example, a prompt that combines "Momotaro" and "Kintaro" to create a new story is used. As output, a new story is generated.

[1612] Step 3:

[1613] The server uses emotion recognition software to recognize the user's emotions. As input, it uses the video data saved in step 1 and analyzes the user's facial expressions. As output, it determines the user's emotional state.

[1614] Step 4:

[1615] The server adjusts the sound effects and music based on the user's emotional state. As input, it uses the emotional state data obtained in step 3 and generates appropriate sound effects and music using music generation software. For example, if the user is excited, it generates fast-paced music. As output, sound effects and music corresponding to the user's emotion are generated.

[1616] Step 5:

[1617] The server adjusts the printing of the story according to the user's emotions. As input, it uses the emotional state data obtained in step 3 and the new story data generated in step 2. It uses emotion recognition software to change the color tone and design of the printout; for example, it prints in bright colors when the user is feeling happy. As output, it prints a story according to the emotion.

[1618] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1619] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1620] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1621] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1622] [Third embodiment]

[1623] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1624] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1625] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1626] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1627] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1628] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1629] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1630] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1631] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1632] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1633] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1634] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1635] "Example 1"

[1636] The system of this invention uses generative AI-based AI to automatically generate elements such as sound effects, music, and storytelling. Specifically, the AI ​​generates a story based on user input, then generates sound effects and music to match the story. The AI ​​also generates a story read aloud, and combines all of these to create a picture book.

[1637] "Example 2"

[1638] The system of this invention has the function of printing the created picture book on paper. Specifically, it digitally creates a picture book by combining the AI-generated story, sound effects, music, and storytelling, and then prints the picture book on paper. This allows users to obtain a physical picture book.

[1639] "Example 3"

[1640] The system of the present invention has the function of recording the reading aloud and saving it as a memory. Specifically, the system records the user reading a picture book on video and saves the video as digital data. This allows the user to play back the reading aloud later and keep it as a memory.

[1641] "Example 4"

[1642] The system of the present invention has the function of combining existing stories to create new stories. Specifically, the AI ​​combines existing stories, such as "Momotaro" and "Kintaro," to create new stories, such as "Momotaro vs. Kintaro." This allows users to enjoy new stories based on existing stories.

[1643] The processing flow of each embodiment will be described below.

[1644] "Example 1"

[1645] Step 1: The user inputs the story's theme, characters, etc. into the system.

[1646] Step 2: Generative AI-based AI generates a story based on user input.

[1647] Step 3: Generate sound effects and music to match the AI-generated story.

[1648] Step 4: The AI ​​generates a story reading.

[1649] Step 5: Combine them all to create a digital picture book.

[1650] "Example 2"

[1651] Step 1: Create a digital picture book by combining an AI-generated story, sound effects, music, and storytelling.

[1652] Step 2: The system prints the digitally created picture book onto paper.

[1653] "Example 3"

[1654] Step 1: The user reads the book aloud.

[1655] Step 2: The system records a video of the reading.

[1656] Step 3: Save the captured video as digital data.

[1657] "Example 4"

[1658] Step 1: The AI ​​selects an existing story, such as "Momotaro" or "Kintaro."

[1659] Step 2: Combine the selected stories to create a new story, for example, "Momotaro vs. Kintaro."

[1660] Example 1

[1661] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1662] Conventional story creation systems require users to manually create stories and add sound effects, music, and audio recordings individually, which requires a great deal of time and effort. Furthermore, the process of integrating these elements into a single picture book is cumbersome, making it difficult to automatically generate picture books, especially for children.

[1663] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1664] In this invention, the server includes means for a user to input a prompt sentence, means for generating a story based on the prompt sentence using a generative AI model, means for generating sound effects based on the generated story, means for generating music based on the generated story, means for generating a storytelling audio based on the generated story, means for creating a picture book by combining the story, sound effects, music, and storytelling audio, and means for a user to view the picture book on a terminal. This allows a user to easily create a story, automatically add sound effects, music, and storytelling audio, and integrate them into a single picture book.

[1665] A "prompt sentence" is text that the user inputs to the system, including instructions about the theme, characters, etc. of the story.

[1666] A "generative AI model" is an artificial intelligence algorithm that automatically generates a story based on a prompt from the user.

[1667] A "story" is text data generated by a generative AI model based on a prompt sentence.

[1668] "Sound effects" are sound effects generated to match the scenes in the story.

[1669] "Music" is background music generated to match the atmosphere of the story.

[1670] "Reading audio" is audio data generated from the story text using speech synthesis technology.

[1671] A "picture book" is an e-book format content created by combining a story, sound effects, music, and audio reading.

[1672] A "terminal" is a device through which a user accesses the system, inputs prompt sentences, and views the generated picture book.

[1673] This invention relates to a system that uses a generative AI model to automatically generate a story, sound effects, music, and reading audio, and combines them to create a picture book. Specific embodiments of this system are described below.

[1674] A user accesses the system using a terminal and inputs a prompt statement such as the theme of the story, characters, etc. For example, the user might input, "Create an adventure story about a brave boy who fights a dragon in a magical forest."

[1675] The server receives a prompt from the user and generates a story based on the prompt using a generative AI model (e.g., OpenAI's GPT-4). The generated story is stored as text data on the server.

[1676] The server then generates sound effects based on the story, using audio libraries such as the Freesound API to select sound effects that match each scene in the story. For example, for a scene featuring a dragon, the server selects the sound of a dragon roaring.

[1677] In addition, the server generates music that matches the atmosphere of the story. For music generation, it uses music generation AI such as Magenta. For example, exciting music is generated for adventure scenes, and calm music is generated for quiet scenes.

[1678] The server also uses speech synthesis technology, such as the Google Text-to-Speech API, to convert the story text into audio data, which then generates an audio recording of the story being read aloud.

[1679] Finally, the server combines the generated story, sound effects, music, and audio reading to create a picture book, which is saved as a PDF or e-book and made available for download by users.

[1680] Users can use their devices to download and view picture books provided by the server. While reading the picture books, users can enjoy sound effects, music, and audio recordings of the story being told.

[1681] The system allows users to easily create stories, automatically add sound effects, music, and audio readings, and integrate them into a complete picture book.

[1682] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1683] Step 1:

[1684] The user uses a terminal to input prompts into the system, such as the theme and characters of the story. For example, the user might input "Please create an adventure story about a brave boy battling a dragon in a magical forest." The input prompt is then sent to the server.

[1685] Step 2:

[1686] The server receives the prompt sentence and sends it as input to a generative AI model (e.g., OpenAI's GPT-4). The generative AI model generates a story based on the prompt sentence and returns the text data to the server. The server saves the generated story as text data.

[1687] Step 3:

[1688] The server analyzes the generated story text and generates sound effects appropriate for each scene. For example, in a scene where a dragon appears in the story, the server uses the Freesound API to obtain the dragon's roar sound. The input is the story text data, and the output is the audio data of the sound effects.

[1689] Step 4:

[1690] The server generates music that matches the atmosphere of the story. For music generation, it uses music generation AI such as Magenta. For example, exciting music is generated for adventure scenes, and calm music is generated for quiet scenes. The input is the text data of the story, and the output is the audio data of the music.

[1691] Step 5:

[1692] The server uses speech synthesis technology such as the Google Text-to-Speech API to convert the story text into audio data, which then generates a story read aloud. The input is the story text data, and the output is the audio data of the story read aloud.

[1693] Step 6:

[1694] The server combines the generated story, sound effects, music, and reading audio to create a picture book. The picture book is saved in PDF or e-book format and can be downloaded by users. The input is the text data of the story, audio data of the sound effects, audio data of the music, and audio data of the reading audio, and the output is the integrated picture book data.

[1695] Step 7:

[1696] Users can use their devices to download and view picture books provided by the server. While reading the picture book, users can enjoy sound effects, music, and audio recordings. The input is the picture book data from the server, and the output is the user's viewing experience.

[1697] (Application example 1)

[1698] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1699] Conventional story generation systems can generate stories based on a theme entered by the user, but they lack the functionality to automatically generate sound effects, music, and storytelling audio and combine them to create a digital picture book. Furthermore, they lack the means to make the generated digital picture book viewable on a smartphone, limiting the user experience. This poses a challenge for users, making it difficult to easily create and enjoy high-quality digital picture books.

[1700] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1701] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on them, means for digitally saving the created story, means for generating a story based on user input, means for generating sound effects and music to match the generated story, means for generating audio for storytelling, means for creating a digital picture book by combining the generated story, sound effects, music, and storytelling audio, and means for making the created digital picture book viewable on a smartphone. This allows high-quality digital picture books to be automatically generated based on a theme entered by the user, and can be easily viewed and shared on a smartphone.

[1702] "Generative AI-based AI" is an artificial intelligence technology that automatically generates stories, sound effects, music, reading audio, and more based on user input.

[1703] "Sound effects" are sound effects generated to match the scenes and situations in the story.

[1704] "Music" is music generated to enhance the atmosphere and emotion of the story.

[1705] "Read-aloud audio" is audio data for reading the generated story aloud.

[1706] A "narrative" is a story that is generated based on input from the user.

[1707] "Digital format" means a data format that is stored and displayed electronically.

[1708] "User input" refers to information such as keywords and themes that a user provides to the system.

[1709] A "digital picture book" is an e-book format picture book created by combining generated stories, sound effects, music, and audio readings.

[1710] A "smartphone" is a portable information terminal that can connect to the Internet and run applications.

[1711] "Viewable" means that the user can visually check the digital picture book.

[1712] The system for implementing this invention uses generative AI-based AI to automatically generate stories, sound effects, music, and storytelling audio based on user input, and combines them to create a digital picture book. Specific embodiments of this system are described below.

[1713] System configuration

[1714] 1. Hardware Configuration

[1715] Server: A high-performance computer (e.g., a server equipped with a GPU) for running generative AI models.

[1716] User device: A device such as a smartphone or tablet that allows users to input and view the generated digital picture book.

[1717] 2. Software Configuration

[1718] Generative AI models: AI models for generating stories, sound effects, music, and storytelling audio (e.g., OpenAI's GPT-4).

[1719] Database: A database for storing the generated digital storybooks and user input.

[1720] Application: A smartphone application that allows users to enter input and view the generated digital picture book.

[1721] Processing flow

[1722] 1. User Input

[1723] Users enter the story's theme and keywords through a smartphone application.

[1724] Example: "A story of adventure and friendship"

[1725] 2. Narrative Generation

[1726] The server uses a generative AI model to generate a story based on user input.

[1727] Example prompt: "Generate a story of adventure and friendship."

[1728] 3. Sound Effects and Music Generation

[1729] The server generates sound effects and music to accompany the generated story.

[1730] Example of a sound effect generation prompt: "Generate a sound effect that matches the following story:\n\n(generated story)"

[1731] Example of a music generation prompt: "Generate music to fit the following story: (generated story)"

[1732] 4. Generation of reading audio

[1733] The server generates audio to read the generated story aloud.

[1734] Example prompt for storytelling audio generation: "Generate audio to read the following story: (generated story)"

[1735] 5. Creating a digital picture book

[1736] The server combines the generated story, sound effects, music, and reading audio to create a digital picture book.

[1737] 6. Viewing digital picture books

[1738] Users can view the generated digital picture book through a smartphone application.

[1739] Specific examples

[1740] When a user inputs the theme "A Story of Adventure and Friendship," the server uses a generative AI model to generate a story. It then generates sound effects and music to match the story, and generates audio to read the story. These elements are combined to create a digital picture book, which users can then view on their smartphones.

[1741] In this way, users can easily create and enjoy high-quality digital picture books.

[1742] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1743] Step 1:

[1744] The user starts the smartphone application and inputs the theme and keywords of the story. The input theme and keywords are sent to the server. The input data is used as prompts for generating the story.

[1745] Step 2:

[1746] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate a story based on themes and keywords submitted by the user. Specifically, the server inputs the prompt "Generate a story of adventure and friendship" into the generative AI model and outputs the story text.

[1747] Step 3:

[1748] The server generates sound effects based on the generated story text. It inputs the prompt "Generate sound effects that fit the following story:\n\n(generated story)" into the generative AI model and outputs a description of the sound effects. The description of the sound effects is used to retrieve appropriate sound effect files from the sound effect database.

[1749] Step 4:

[1750] The server generates music based on the generated story text. It inputs the prompt "Generate music that matches the following story: (generated story)" into the generative AI model and outputs a music description. The music description is used to retrieve appropriate music files from a music database.

[1751] Step 5:

[1752] The server generates audio to read the generated story text aloud. The prompt "Please generate audio to read the following story aloud:\n\n(generated story)" is input into the generative AI model, and the reading audio data is output.

[1753] Step 6:

[1754] The server combines the generated story text, sound effects, music, and reading audio to create a digital picture book, which includes generating a script to play each element at the appropriate timing and creating a digital picture book file.

[1755] Step 7:

[1756] The server sends the created digital picture book file to the user's smartphone. The user can view the created digital picture book through a smartphone application. The digital picture book is displayed in an integrated format, including text, sound effects, music, and audio recordings.

[1757] Example 2

[1758] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1759] Conventional picture book creation systems require users to manually create stories, sound effects, music, and audio for storytelling, which takes time and effort. Furthermore, the content and quality of the stories are not consistent, which can lead to low user satisfaction. Furthermore, the time and cost required to print the stories on paper is also an issue.

[1760] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1761] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative artificial intelligence and creating a story based on these, means for generating a story based on a prompt sentence entered by the user, means for generating sound effects, music, and storytelling audio based on the generated story, means for creating a digital picture book by combining these elements, means for sending the created digital picture book to a terminal, means for the terminal to display the digital picture book and for the user to check it, means for the user to instruct printing, means for the terminal to print the picture book on paper using a printer, and means for capturing footage of the storytelling and saving it as a memory. This allows users to easily create picture books containing high-quality stories and obtain them in physical form.

[1762] "Generative AI" is an AI technology that automatically generates elements such as stories, sound effects, music, and reading audio based on prompts entered by the user.

[1763] A "prompt" is an instruction that a user inputs to a generative AI, including details about the story's theme, characters, etc.

[1764] A "narrative" is a story or narrative that a generative artificial intelligence generates based on a prompt.

[1765] "Sound effects" are sound effects that are generated according to the scenes and situations in the story.

[1766] "Music" refers to musical compositions and melodies generated to enhance the atmosphere and emotion of the story.

[1767] "Read-aloud audio" refers to audio data generated to read aloud the generated story.

[1768] A "digital picture book" is an electronic picture book created by combining generated stories, sound effects, music, and audio readings.

[1769] A "terminal" is an electronic device that a user uses to check a digital picture book and instruct it to be printed.

[1770] A "printer" is a printing device for printing digital picture books on paper.

[1771] "Reading situation" refers to the situation or scene when the user reads the generated story aloud.

[1772] "Means for preserving memories" refers to methods or devices for recording the reading aloud and saving it for later playback.

[1773] This invention is a system that uses generative artificial intelligence to automatically generate stories, sound effects, music, and audio for storytelling, combines them to create a digital picture book, and finally prints it on paper. Specific embodiments of this system are described below.

[1774] The server uses generative AI to generate a story based on the prompt entered by the user. For example, a model using natural language processing technology (e.g., GPT-4) is used as the generative AI. When the user enters a prompt, the server analyzes it and generates a story.

[1775] The server then generates sound effects, music, and audio based on the story. Sound effects and music are generated using text-to-speech software (e.g., Google Cloud Text-to-Speech). Audio for the story is also generated.

[1776] The server combines the generated story, sound effects, music, and audio reading to create a digital picture book, which is then saved in a format that can be viewed by the user.

[1777] The server then sends the created digital picture book to the user's terminal. The terminal receives the digital picture book sent from the server and displays it for the user to review. The user can review the displayed digital picture book and make corrections as necessary.

[1778] If the user is satisfied with the content of the digital picture book, they can request printing. The request is made through the terminal interface. Upon receiving the user's request, the terminal uses a printer (e.g., an inkjet printer) to print the digital picture book onto paper. The printed picture book is then provided to the user in physical form.

[1779] As a concrete example, consider a scenario where a user inputs a prompt such as "Please create a story about animals having an adventure in the forest. The main characters are a rabbit, a fox, and a bear." The server receives this prompt and uses generative artificial intelligence to generate the following story:

[1780] "One day, Lily the rabbit, Fred the fox, and Ben the bear embark on a great adventure in the forest. They make new friends, overcome obstacles, and uncover the secrets of the forest."

[1781] The server then generates sound effects and music to accompany the story, and creates audio for the story being read aloud. These elements are then combined to create a digital picture book, which is then sent to the device. When the user reviews the digital book and requests printing, the device uses a printer to print the book onto paper, giving the user a physical copy.

[1782] In this way, users can easily create picture books containing high-quality stories and have them available in physical form.

[1783] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1784] Step 1:

[1785] The user enters a prompt statement.

[1786] The user accesses the system's interface and inputs a prompt sentence including details such as the theme and characters of the story. For example, the user might input, "Please create a story about animals having an adventure in the forest. The main characters are a rabbit, a fox, and a bear." The input prompt sentence is then sent to the server.

[1787] Step 2:

[1788] The server receives the prompt and generates a story using a generative AI model.

[1789] The server receives the prompt sent by the user. Then, it uses a generative AI model (e.g., a model using natural language processing technology) to generate a story based on the prompt. The generated story is saved as text data. The input is the prompt, and the output is the generated story.

[1790] Step 3:

[1791] The server generates sound effects, music, and storytelling audio based on the generated story.

[1792] The server uses speech synthesis software (e.g., text-to-speech technology) to analyze the generated story and generate appropriate sound effects and music, as well as audio for reading the story aloud. The input is the generated story, and the output is sound effects, music, and audio for reading the story aloud.

[1793] Step 4:

[1794] The server combines these elements to create a digital picture book.

[1795] The server combines the generated story, sound effects, music, and audio reading to create a digital picture book. This digital picture book is saved in a format that can be viewed by the user. The input is the story, sound effects, music, and audio reading, and the output is the digital picture book.

[1796] Step 5:

[1797] The server sends the digital picture book to the terminal.

[1798] The server sends the created digital picture book to the user's terminal. The sent digital picture book is displayed on the terminal so that the user can check it. The input is the digital picture book, and the output is the digital picture book displayed on the terminal.

[1799] Step 6:

[1800] The terminal displays the digital picture book for the user to check.

[1801] The terminal receives the digital picture book sent from the server and displays it for the user to check. The user can check the displayed digital picture book and instruct corrections as necessary. The input is the digital picture book, and the output is the user's confirmation and correction instructions.

[1802] Step 7:

[1803] The user issues a print instruction.

[1804] When the user is satisfied with the content of the digital picture book, he or she instructs the book to be printed. The printing instruction is given through the terminal interface. The input is the user's print instruction, and the output is the transmission of the print instruction.

[1805] Step 8:

[1806] The device uses a printer to print the picture book onto paper.

[1807] The terminal receives a print instruction from the user and prints the digital picture book on paper using a printer (e.g., an inkjet printer). The printed picture book is provided to the user in a physical form. The input is the print instruction, and the output is the picture book printed on paper.

[1808] (Application example 2)

[1809] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1810] In conventional picture book creation systems, users could freely select story themes and characters, combine the generated story with sound effects, music, and storytelling to create a digital picture book, and then print the book on paper. Furthermore, there was a lack of a way for users to easily save the picture books they created and obtain them in physical form. This prevented users from fully expressing their creativity and enjoying the fun of creating picture books.

[1811] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1812] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on them, means for printing the created story on paper, means for filming the storytelling and saving it as a memory, means for providing an interface for the user to select the theme and characters of the story, means for creating a digital picture book by combining the generated story with the sound effects, music, and storytelling, and means for saving the created digital picture book in PDF format so that the user can print it on paper using a printer.This allows the user to freely select the theme and characters of the story, create a digital picture book by combining the generated story with the sound effects, music, and storytelling, and print it on paper to obtain it in physical form.

[1813] "Generative AI-based AI" is artificial intelligence that automatically generates specific data or content using generative models.

[1814] "Sound effects" are sound effects used to emphasize particular scenes or actions in a story or film.

[1815] "Music" is an art form of sound created by combining elements such as melody, harmony, and rhythm.

[1816] "Reading aloud" is the act of reading a story or text aloud, especially for the education or entertainment of children.

[1817] A "narrative" is a form of writing or image that describes a series of events composed of characters, events, and settings.

[1818] A "means for printing on paper" is a device or method for transferring digital data onto a physical paper medium.

[1819] "Means for filming the reading aloud" refers to devices or methods for recording the act of reading aloud as video or audio.

[1820] A "memory storage device" is a device or method for recording and storing specific events or actions in a form that can be later reproduced.

[1821] An "interface that allows the user to select the story theme and characters" is an operation screen or input device that allows the user to select the story content and characters.

[1822] A "digital picture book" is a picture book created in digital format that includes elements such as text, images, and audio.

[1823] "Means for saving in PDF format" refers to a device or method for saving digital data in Portable Document Format (PDF).

[1824] A "printer" is a device for printing digital data onto a physical medium such as paper.

[1825] A system for implementing this invention includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on these elements, means for printing the created story on paper, means for filming the storytelling and saving it as a memory, means for providing an interface for users to select the theme and characters of the story, means for combining the generated story with the sound effects, music, and storytelling to create a digital picture book, and means for saving the created digital picture book in PDF format so that the user can print it on paper using a printer.

[1826] Hardware and software used

[1827] Hardware: Smartphone, tablet, printer, internet connection

[1828] Software: Python, TensorFlow (implementation of generative AI models), Flask (web application framework), Firebase (database and authentication), PIL (Python Imaging Library, for image processing)

[1829] Program processing

[1830] Building the user interface

[1831] Users use the touchscreen of their smartphone or tablet to navigate an interface that allows them to select themes and characters for their stories, such as "fairy tales," "animal adventures," and "space exploration."

[1832] Running generative AI models

[1833] The server inputs prompts into the generative AI model based on the theme selected by the user, and generates a story, sound effects, music, and storytelling. For example, if the prompt is "Generate a story about a fairy tale theme in which a brave prince defeats a dragon," the AI ​​will generate a story.

[1834] Creating a digital picture book

[1835] The server combines the generated story with sound effects, music, and storytelling to create a digital storybook. It uses PIL to generate pages that combine images and text.

[1836] Implementing the printing function

[1837] The server saves the digital book in PDF format so that users can print it on paper using a printer, and uses Flask to provide a web interface where users can download the PDF.

[1838] Specific examples

[1839] Prompt Sentence Examples

[1840] "Generate a fairy tale story about a brave prince slaying a dragon."

[1841] Example of a generated story

[1842] "Once upon a time, there was a brave prince. He set out on a journey to save his village from a dragon. With his wisdom and courage, the prince defeated the dragon and saved his village."

[1843] Digital picture book page example

[1844] Page 1: "Once upon a time, there was a brave prince." (Illustration of the prince)

[1845] Page 2: "He set out on a journey to save a village that had been attacked by a dragon." (Illustration of a travel scene)

[1846] Page 3: "The prince used his wisdom and courage to defeat the dragon and save the village." (Illustration of a dragon battle scene)

[1847] In this way, users can easily create their own original picture books and enjoy them by saving them digitally or printing them out on paper.

[1848] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1849] Step 1:

[1850] The user operates the interface of a smartphone or tablet to select the story's theme and characters.

[1851] Input: User-selected theme and character information

[1852] Output: Data on selected themes and characters

[1853] Specific operation: The user operates the touchscreen to select a theme such as "Fairy Tales," "Animal Adventures," or "Space Exploration." The selected information is sent from the device to the server.

[1854] Step 2:

[1855] Based on the theme and character information received by the server, prompts are input into the generative AI model, which then generates a story, sound effects, music, and storytelling.

[1856] Input: User-selected theme, character data, prompt text

[1857] Output: Generated story, sound effects, music, and storytelling data

[1858] How it works: The server uses TensorFlow to generate a story using the prompt "Generate a fairy tale-themed story about a brave prince slaying a dragon" and outputs the generated story along with associated sound effects, music, and storytelling data.

[1859] Step 3:

[1860] The server combines the generated story with sound effects, music, and storytelling to create a digital picture book.

[1861] Input: Generated story, sound effects, music, storytelling data

[1862] Output: Digital picture book data

[1863] What it does: The server uses PIL to create pages that combine the generated story text with corresponding images, with each page containing story text and related illustrations.

[1864] Step 4:

[1865] The server saves the created digital picture book in PDF format so that users can print it on paper using a printer.

[1866] Input: Digital picture book data

[1867] Output: Digital picture book in PDF format

[1868] What it does: The server uses Flask to provide a web interface that allows users to download PDFs. Users download the PDFs through this interface and print them on a printer.

[1869] Step 5:

[1870] Users can take photos of the reading aloud and save them as memories.

[1871] Input: Video and audio data of a user reading aloud

[1872] Output: Stored video and audio data

[1873] How it works: Users use the camera on their smartphone or tablet to record themselves reading aloud, and the recorded video and audio are saved on the device and saved in a format that can be played back later.

[1874] Example 3

[1875] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1876] Conventional systems for recording reading aloud involve the cumbersome process of saving and playing back video data, making it difficult for users to easily save and play back their memories. Furthermore, they lack the functionality to generate new stories by combining existing stories, limiting users' opportunities to enjoy new stories. Furthermore, there is no way to print the generated stories on paper, making it impossible to physically store the stories. To address these issues, the present invention aims to provide a system that can easily save and play back reading aloud, and a function to generate new stories by combining existing stories.

[1877] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1878] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using a generative AI-based AI and creating a story based on them, means for printing the created story on paper, means for recording the storytelling and saving it as a memory, means for uploading the recorded video data to cloud storage, means for receiving the video data from the cloud storage and saving it in a database, means for playing the saved video data, means for inputting an existing story and inputting a prompt sentence to the generative AI model, means for the generative AI model to generate a new story, and means for saving the generated story and providing it to the user. This allows users to easily save and play back storytelling, as well as to combine existing stories to create and enjoy new stories.

[1879] "Generative AI-based AI" is an artificial intelligence technology that automatically generates new data and content based on existing data.

[1880] "Sound effects" are sound effects used to give a sense of realism to a story or image.

[1881] "Music" is the art of sound created by combining elements such as melody, harmony, and rhythm.

[1882] "Reading aloud" is the act of reading books or stories aloud, often to children.

[1883] A "narrative" is a story that unfolds through events and the actions of characters.

[1884] "Printing to paper" is the process of transferring digital data onto a physical paper medium.

[1885] "Filming" is the act of recording footage using a camera or video camera.

[1886] "Saving as a memory" means recording a particular event or experience so that it can be replayed later.

[1887] "Cloud storage" is a collection of remote servers that store and access data over the Internet.

[1888] A "database" is a system for efficiently storing, managing, and retrieving data.

[1889] "Playback" refers to the act of displaying or playing back stored audio or video data.

[1890] An "existing narrative" is a story that has already been created and is widely known.

[1891] A "generative AI model" is an algorithm or model that uses artificial intelligence to generate new data or content.

[1892] A "prompt sentence" is an input sentence that gives specific instructions or questions to a generative AI model.

[1893] A "new story" is a new story generated based on existing stories and elements.

[1894] "Providing to users" means making the generated data and content available to users.

[1895] This invention provides a system that records the reading session and saves it as a memory, and a system that combines existing stories to create a new story. Specific embodiments of this system are described below.

[1896] First, the user uses a device such as a smartphone or tablet to record a video of the picture book being read aloud. Specifically, the user launches the smartphone's camera app and presses the record button to record the reading. Once the recording is complete, the device uploads the video data to a cloud storage service (e.g., Google Drive or Dropbox). The user then presses the share button on the camera app to upload the video data to cloud storage.

[1897] Next, the server receives the uploaded video data from the cloud storage. Specifically, it downloads the video data using the cloud storage's API and stores it in a database management system (e.g., MySQL or MongoDB). When storing the video data, it saves it in an appropriate format (e.g., MP4 format). This allows users to play the saved video data later.

[1898] Furthermore, users can input existing stories through the system interface. Specifically, they input the title and content of the story into a text box. The server inputs a prompt into the generative AI model (e.g., GPT-4) based on the story input by the user. An example of a prompt is, "Please combine the stories of Momotaro and Kintaro to create a new story."

[1899] The generative AI model generates a new story based on the prompt. Specifically, the AI ​​combines elements of "Momotaro" and "Kintaro" to generate a new story such as "Momotaro vs. Kintaro." The generated story is stored in a database by the server and made available to users for later reading.

[1900] For example, if a user wants to create a new story by combining the stories of "Momotaro" and "Kintaro," the user inputs the stories of "Momotaro" and "Kintaro" into the system. The server receives this and inputs the following prompt sentence into the generative AI model:

[1901] "Combine the stories of Momotaro and Kintaro to create a new story."

[1902] Based on this prompt, the generative AI model generates a new story, such as "Momotaro vs. Kintaro." The generated story is saved as text data and can be read by the user later.

[1903] In this way, the system of the present invention not only allows you to easily save and play back the story-telling, but also allows you to combine existing stories to create new stories and enjoy them. The flow of the specific processing in the third embodiment will be described with reference to Figure 15.

[1904] Step 1:

[1905] The user takes a photo of the reading aloud

[1906] The user uses a device such as a smartphone or tablet to record a video of the picture book being read aloud. Specifically, the user starts the smartphone's camera app and presses the record button to record the reading. The input is the user's operation, and the output is video data.

[1907] Step 2:

[1908] The device uploads the video data to cloud storage.

[1909] The device uploads the captured video data to a cloud storage service (e.g., Google Drive or Dropbox). Specifically, the user selects the video data in the camera app and presses the share button to upload it to cloud storage. The input is the video data, and the output is the video data saved in the cloud storage.

[1910] Step 3:

[1911] The server receives the video data and stores it in a database.

[1912] The server receives the video data uploaded from the cloud storage. Specifically, it downloads the video data using the cloud storage's API and stores it in a database management system (e.g., MySQL or MongoDB). The input is the video data from the cloud storage, and the output is the video data stored in the database.

[1913] Step 4:

[1914] User plays a saved video

[1915] Users can play stored videos through the system's interface. Specifically, they use the system's web app or mobile app to select the video they want to play from a list of stored videos and press the play button. The input is the video data stored in the database, and the output is the video being played.

[1916] Step 5:

[1917] User enters an existing story

[1918] The user inputs an existing story (e.g., "Momotaro" and "Kintaro") through the system interface. Specifically, the user inputs the title and content of the story into a text box. The input is the text data of the story entered by the user, and the output is the text data sent to the server.

[1919] Step 6:

[1920] The server inputs a prompt to the generative AI model.

[1921] The server inputs a prompt to the generative AI model (e.g., GPT-4) based on the story input by the user. Specifically, it sends the prompt, "Please combine the stories of Momotaro and Kintaro to create a new story," to the generative AI model. The input is the text data of the story from the user, and the output is the prompt sent to the generative AI model.

[1922] Step 7:

[1923] Generative AI models generate new stories

[1924] The generative AI model generates a new story based on the prompt. Specifically, the AI ​​combines elements of "Momotaro" and "Kintaro" to generate a new story such as "Momotaro vs. Kintaro." The input is the prompt, and the output is the text data of the generated story.

[1925] Step 8:

[1926] The server stores the generated stories and serves them to the user.

[1927] The server receives the generated stories and stores them in a database. The stored stories are then made available to users for later reading. The input is the text data of the generated stories, and the output is the stories stored in the database.

[1928] (Application example 3)

[1929] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1930] Traditional methods for recording storytimes have limited the means to preserve storytime memories and have made it impossible to combine existing stories to create new stories. Furthermore, there is a lack of easy ways to share saved videos or play them back at any time. This makes it difficult for users to effectively preserve storytime memories and enjoy new stories.

[1931] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1932] In this invention, the server includes means for automatically generating elements such as sound effects, music, and storytelling using generative AI-based AI and creating a story based on them, means for printing the created story on paper, means for recording the storytelling and saving it as a memory, means for uploading the recorded video to the cloud and making it playable at any time, and means for combining existing stories to create a new story. This allows users to effectively save the storytelling and play it back at any time, and further allows them to enjoy new stories by combining existing stories.

[1933] "Generative AI-based AI" is an artificial intelligence technology that generates new data based on existing data.

[1934] "Sound effects" are sound effects used to enhance the sense of realism in stories and images.

[1935] "Music" is the art of sound created by combining elements such as melody, harmony, and rhythm.

[1936] "Reading aloud" is the act of reading a book or story aloud to others.

[1937] A "narrative" is a written or visual form that describes events and the actions of characters.

[1938] "Printing on paper" is the act of transferring digital data onto a physical paper medium.

[1939] "Filming the reading aloud" means recording the act of reading aloud with a video camera or other filming equipment.

[1940] "Saving as a memory" means recording a particular event or action in a form that can be played back later.

[1941] "Uploading to the cloud" is the act of storing digital data on a remote server over the internet.

[1942] "Making it playable at any time" means that stored digital data can be played back at any time.

[1943] "Generating a new story by combining existing stories" means creating a new story based on multiple stories that already exist.

[1944] A system for implementing this invention is configured as follows: First, a user uses a smartphone to record the reading aloud. The smartphone is equipped with a video capture function that uses OpenCV, and can record the reading aloud as a high-quality video.

[1945] The captured video is uploaded to cloud storage (e.g., Google Drive) using rclone, allowing users to play the video anytime, anywhere.

[1946] Next, a generative AI-based AI is used to combine existing stories to generate new stories. Specifically, the transformers library is used to input a prompt into the GPT-3 model to generate a new story. An example of this prompt is "What would happen if Momotaro and Kintaro met?"

[1947] The new story is then displayed on the smartphone for the user to enjoy, and can also be printed on paper if desired.

[1948] The specific process of this system is as follows: First, the reading is filmed using the smartphone camera and saved in AVI format. Next, the saved video is uploaded to cloud storage using rclone. Next, a prompt sentence is input into the GPT-3 model to generate a new story. Finally, the generated story is displayed on the smartphone and, if necessary, printed on paper.

[1949] This system allows users to effectively save the storytelling and play it back at any time, and also allows users to combine existing stories to create new stories.

[1950] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1951] Step 1:

[1952] The user activates the camera on their smartphone and takes a picture of the reading aloud.

[1953] Input: Smartphone camera image

[1954] Output: AVI format video file

[1955] Specific operation: The user launches the camera app on their smartphone and presses the record button to record the reading. Once the recording is complete, the video file is saved in the smartphone's storage.

[1956] Step 2:

[1957] The device uploads the captured video file to cloud storage.

[1958] Input: AVI format video file

[1959] Output: Video files on cloud storage

[1960] Specific operation: The device uses rclone to upload the video file to the specified cloud storage (e.g., Google Drive). Once the upload is complete, the video file is saved in the cloud.

[1961] Step 3:

[1962] Users input prompts into a generative AI-based AI to generate new stories.

[1963] Input: Prompt sentence (e.g. "What would happen if Momotaro and Kintaro met?")

[1964] Output: Text data of the new story

[1965] How it works: A user launches a smartphone application and enters a prompt. The device then uses the transformers library to send the prompt to the GPT-3 model, which then receives the text data for the new story.

[1966] Step 4:

[1967] The terminal displays the new story that has been generated.

[1968] Input: Text data of a new story

[1969] Output: A story displayed on a smartphone screen

[1970] Specific operation: The device displays the received text data of the new story on the smartphone screen, and the user can read the generated story.

[1971] Step 5:

[1972] The user prints the generated story on paper if desired.

[1973] Input: Text data of a new story

[1974] Output: A printed version of the story

[1975] Specific operation: When a user presses the print button on the smartphone application, the device sends text data to the printer and the story is printed on paper.

[1976] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1977] "Example 1"

[1978] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes a user's emotions and adjusts the generation of a story according to the user's emotions. Specifically, when the user is feeling happy, the system generates a happy story, and when the user is sad, the system generates a comforting story. The emotion engine can also use facial recognition technology or voice recognition technology to recognize the user's emotions.

[1979] "Example 2"

[1980] Another embodiment of the present invention provides a system incorporating an emotion engine. This system recognizes the user's emotion and adjusts the generation of storytelling according to the emotion. Specifically, when the user feels surprised, the system generates storytelling that includes elements of surprise, and when the user feels relieved, the system generates storytelling that gives a sense of relief.

[1981] "Example 3"

[1982] Furthermore, as another embodiment of the present invention, a system incorporating an emotion engine is provided. This system recognizes the emotion of a user and adjusts the generation of sound effects and music according to the emotion. Specifically, when the user is excited, fast-paced music and sound effects are generated, and when the user is relaxed, slower music and sound effects are generated.

[1983] "Example 4"

[1984] Another embodiment of the present invention provides a system incorporating an emotion engine that recognizes the user's emotions and adjusts the printing of the story accordingly, specifically printing bright colors when the user is happy and muted colors when the user is sad.

[1985] The processing flow of each embodiment will be described below.

[1986] "Example 1"

[1987] Step 1: Recognize the user's emotions with the emotion engine.

[1988] Step 2: Adjust your narrative production according to the emotions you recognize.

[1989] Step 3: Serve the adjusted story to the user.

[1990] "Example 2"

[1991] Step 1: Recognize the user's emotions with the emotion engine.

[1992] Step 2: Adjust your storytelling production according to the emotions you recognize.

[1993] Step 3: Provide the adapted reading to the user.

[1994] "Example 3"

[1995] Step 1: Recognize the user's emotions with the emotion engine.

[1996] Step 2: Adjust the sound effects and music generation according to the recognized emotion.

[1997] Step 3: Provide the adjusted sound effects and music to users.

[1998] "Example 4"

[1999] Step 1: Recognize the user's emotions with the emotion engine.

[2000] Step 2: Adjust the print of the story according to the emotions you recognize.

[2001] Step 3: Print the adjusted story and provide it to the user.

[2002] Example 1

[2003] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2004] Conventional story generation systems have had difficulty generating stories that respond to the user's emotions or creating digital picture books that combine sound effects, music, and audio reading. Furthermore, they lacked the functionality to recognize the user's emotions and adjust the content of the story, making it impossible to provide a story that was in tune with the user's emotions. Furthermore, the process of sending the generated story to the device and displaying it to the user was not automated, limiting the user experience.

[2005] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2006] In this invention, the server includes means for automatically generating elements such as sound ef...

Claims

1. A means for recognizing the user's emotions; and A means for generating a story using a prompt sentence that instructs the generation of a story according to the user's emotions and a generation system AI; means for generating sound effects and music in response to the generated story; a means for reading the generated story aloud using a text-to-speech function; means for playing the generated sound effects and music in sync with the story being read; A system including:

2. A means for capturing an image of the user while reading the story; A means for storing the captured image; The system of claim 1 further comprising:

3. The means for recognizing further recognizes an emotion of the user when reading the story; The story generating means further generates a continuation of the story using a prompt sentence that instructs the generation of the story according to the user's emotion and the generation system AI; 2. The system of claim 1, wherein said means for generating sound effects and music further generates sound effects and music in response to said generated story progression.

Images