system

Abstract

Provide a system. [Solution] a means for recording voice data of the child; a voice recognition means for converting recorded voice data into text data; natural language processing means for analyzing text data and generating narrative text; an image generating means for generating picture data based on the generated narrative text; a picture book generating means for generating an electronic picture book by combining a story and picture data; a distribution means for providing the generated electronic picture book to a user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] It shows the "problem that the invention aims to solve" and the "means for solving the problem."

[0005] Conventionally, there have been a wide variety of methods for stimulating and cultivating children's creativity, but no specific method for giving shape to a child's fantasies. Furthermore, there have been limited methods for easily creating memories between parents and children. As a result, there is a need for a system that can turn a child's fantasies into a story, provide visual enjoyment, and save it as a paper medium. Therefore, an objective of the present invention is to provide a system that can easily turn a child's fantasies into a story and provide it as a picture book, thereby cultivating a child's creativity and facilitating the creation of memories between parents and children. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present invention provides the following means: A system comprising: means for recording a child's voice data; speech recognition means for converting the recorded voice data into text data; natural language processing means for analyzing the text data and generating a narrative; image generation means for generating picture data based on the generated narrative; picture book generation means for generating an electronic picture book by combining the narrative and the picture data; and distribution means for providing the generated electronic picture book to users. Furthermore, the system further comprises a function for providing the generated electronic picture book in a printable format, allowing children to put their fantasies into shape and enjoy them on paper.

[0007] "Audio data" refers to audio information recorded by a user and stored in digital format.

[0008] A "recording means" is a device or application that allows a user to record audio data.

[0009] "Speech recognition means" refers to the technology and devices used to convert recorded voice data into text data.

[0010] "Text data" is character string information converted by a voice recognition means, and is a written representation of the contents of the voice data.

[0011] "Natural language processing means" refers to technologies and devices that analyze text data, understand meaning and context, and perform specific tasks (e.g., generating a story).

[0012] A "narrative text" is a sentence generated by natural language processing means, and is text information for explaining a particular story.

[0013] "Image generation means" refers to the technology and device for generating pictorial data based on a narrative text.

[0014] "Picture data" refers to visual information generated by an image generating means and stored in digital format.

[0015] "Picture book generation means" refers to the technology and device for generating an electronic picture book by combining story text and picture data.

[0016] An "electronic picture book" is a digital picture book that combines story text and picture data.

[0017] "Distribution means" refers to the technology and devices for providing the generated electronic picture book to users.

[0018] "Printable format" means a digital format of the narrative and pictorial data suitable for printing. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention relates to a system that creates picture books that can give shape to a child's creativity and be enjoyed by parents and children. This system can automatically generate a story based on a child's imagination and create a picture book based on that story.

[0041] Overall system overview

[0042] The system includes the following major components:

[0043] 1. Audio recording method (used by the user)

[0044] Users use the application to record what their child says, and the audio recordings are stored digitally and uploaded to a server.

[0045] 2. Speech recognition method (runs on the server)

[0046] The server receives the voice data and converts it into text data using a voice recognition engine, which uses an online API (for example, a general voice recognition API).

[0047] 3. Natural language processing means (running on the server)

[0048] The server analyzes the converted text data and generates a narrative using a natural language processing engine.

[0049] 4. Image generation method (runs on the server)

[0050] The server generates pictorial data based on the generated narrative text using an image generation engine (e.g., using common image generation technology).

[0051] 5. Picture book generation method (runs on the server)

[0052] The server combines the story text and image data to generate an electronic picture book, which is saved in PDF or ePub format.

[0053] 6. Delivery method (runs on the server)

[0054] The server distributes the generated digital picture books to users, who can view them within the app and use the printing service if necessary.

[0055] Program processing flow

[0056] The specific flow of program processing in this system will be explained below.

[0057] Recording and uploading

[0058] User

[0059] The user launches the app, records what the child is imagining, and then presses the upload button to send the audio data to the server.

[0060] Audio data processing

[0061] server

[0062] The server sends the received voice data to a speech recognition engine and converts it into text data, which is then temporarily stored in a database.

[0063] Text analytics

[0064] server

[0065] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure and extracts information such as nouns, verbs, subjects, and predicates. The extracted information is then used to reconstruct the story.

[0066] Story Generation

[0067] server

[0068] The server uses a story generation algorithm to create a storyline based on the analysis results, and generates a narrative in simple language. The generated narrative is then stored in a database.

[0069] Picture generation

[0070] server

[0071] The server sends the story to an image generation engine, which generates pictorial data corresponding to the story, and stores the pictorial data in a database.

[0072] Picture book generation

[0073] server

[0074] The server combines the story text and the picture data to generate and store an electronic picture book file. If necessary, it also generates a high-resolution file for printing.

[0075] Picture book distribution

[0076] server

[0077] The server delivers the generated digital picture book file to the user's app, where the user can view the picture book and use the printing service.

[0078] Specific examples

[0079] For example, a child might say, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[0080] 1. Recording and uploading

[0081] The user records this content using the app and uploads it to the server.

[0082] 2. Audio Data Processing

[0083] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[0084] 3. Text Analysis

[0085] The server analyzes this using a natural language processing engine and extracts nouns (me, dragon, rainbow bridge, mysterious forest), verbs (found, crossed, went), etc.

[0086] 4. Narrative Generation

[0087] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[0088] 5. Image Generation

[0089] The server generates picture data for "a flying dragon," "a rainbow bridge," and "a mysterious forest" based on the story text.

[0090] 6. Picture Book Generation

[0091] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[0092] 7. Picture book distribution

[0093] The server delivers the generated e-book to the user's app, allowing the user to view the book within the app. Furthermore, printing services are also available if necessary.

[0094] This process transforms what the child says into a story in real time and provides it as a visually appealing picture book. This system is a groundbreaking way to foster children's creativity and create memories for parents and children.

[0095] The processing flow will be explained below.

[0096] Step 1:

[0097] The user launches the app and uses the recording feature to record the child's imaginary speech. When the recording is finished, the user presses the stop recording button.

[0098] Step 2:

[0099] The user presses the upload button to upload the recorded audio data to the server. The device sends the recorded data to the server.

[0100] Step 3:

[0101] The server stores the received voice data and sends it to a voice recognition engine (e.g., a common online API).

[0102] Step 4:

[0103] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[0104] Step 5:

[0105] The server receives the returned text data and temporarily stores it in a database.

[0106] Step 6:

[0107] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure.

[0108] Step 7:

[0109] The natural language processing engine extracts nouns, verbs, subjects, predicates, etc. from text data and performs analysis to understand the context of the text.

[0110] Step 8:

[0111] The server receives the analysis results and creates a storyline using a narrative generation algorithm.

[0112] Step 9:

[0113] The server generates a story in simple language and stores the generated story in a database.

[0114] Step 10:

[0115] The server sends the stored narrative to an image generation engine (eg, general image generation technology).

[0116] Step 11:

[0117] The image generation engine generates picture data based on the story text and returns the generated picture data to the server.

[0118] Step 12:

[0119] The server receives the returned painting data and stores it in a database.

[0120] Step 13:

[0121] The server combines the stored story text and picture data to generate an electronic picture book file (e.g., PDF or ePub format).

[0122] Step 14:

[0123] The server distributes the generated electronic picture book file to the user's terminal.

[0124] Step 15:

[0125] Users can view the digital picture books delivered within the app, and if necessary, can save them in paper form using the printing service.

[0126] Example 1

[0127] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] Existing picture book creation systems make it difficult to give form to children's creativity, and lack an easy way for parents and children to enjoy creating together. Furthermore, there was no system that efficiently converts recorded audio into text and then automatically generates illustrations that correspond to the story text based on that. This limited parent-child communication and restricted opportunities to develop children's creativity.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0130] In this invention, the server includes means for recording a child's voice data, means for converting the recorded voice data into text data, means for analyzing the text data and generating a story, means for generating image data based on the generated story, means for generating an electronic book by combining the story and image data, and means for providing the generated electronic book to users. This makes it possible for parents and children to easily give shape to their children's creativity and create an electronic picture book that they can enjoy together.

[0131] A "means for recording a child's voice data" is a device or application that digitally records what a child says and stores that voice data.

[0132] "A speech recognition means for converting recorded speech data into text data" is a technology or engine that analyzes recorded speech data and converts it into corresponding text data.

[0133] "Natural language processing means for analyzing text data and generating narrative text" refers to a technology or algorithm that understands the language structure based on text data and reconstructs it as narrative text.

[0134] The "image generation means for generating image data based on the generated narrative text" refers to technology or software for automatically creating visual pictures or images that correspond to the content of the narrative text.

[0135] "Electronic book generation means for combining narrative text and image data to generate an electronic book" refers to a technology or system that combines narrative text and corresponding images into a single electronic book format (such as PDF or ePub).

[0136] The "distribution means for providing the generated electronic book to the user" refers to a technology or service for transmitting the completed electronic book to the user's terminal so that it can be viewed or printed.

[0137] This invention relates to a system that creates e-books that can give shape to children's creativity and be enjoyed by parents and children. This system can automatically generate stories based on children's fantasies and create picture books based on those stories.

[0138] Overall system overview

[0139] The system includes the following major components:

[0140] 1. Audio recording method (used by the device)

[0141] The device uses an application to record what the child says and saves the audio data, which is then uploaded digitally to a server.

[0142] 2. Speech recognition method (runs on the server)

[0143] The server receives the voice data and converts it into text data using a voice recognition engine (e.g., a general voice recognition API).

[0144] 3. Natural language processing means (running on the server)

[0145] The server analyzes the converted text data and generates a narrative using a natural language processing engine (e.g., a generative AI model).

[0146] 4. Image generation method (runs on the server)

[0147] The server generates image data based on the generated narrative text using an image generation engine (e.g., a general image generation technology).

[0148] 5. E-book generation means (running on the server)

[0149] The server combines the narrative text and image data to generate an e-book, which is saved in PDF or ePub format.

[0150] 6. Delivery method (runs on the server)

[0151] The server distributes the generated e-books to users, who can view them within the app and use printing services if necessary.

[0152] Program processing flow

[0153] 1. Recording and uploading

[0154] The user launches the app, records what the child says, and when they're done, they press the upload button to send the audio data to the server.

[0155] 2. Audio Data Processing

[0156] The server sends the received voice data to a voice recognition engine and converts it into text data, which is then temporarily stored in a database.

[0157] 3. Text Analysis

[0158] The server sends the stored text data to a natural language processing engine (generative AI model), which analyzes the grammatical structure and extracts information such as nouns, verbs, subjects, and predicates. The extracted information is then used to reconstruct the story.

[0159] 4. Narrative Generation

[0160] The server uses a story generation algorithm based on the analysis results to create a storyline and generate a narrative in simple language. The generated narrative is then stored in a database.

[0161] 5. Image Generation

[0162] The server sends the story text to an image generation engine, which generates image data corresponding to the story text. The generated image data is stored in a database.

[0163] 6. Picture Book Generation

[0164] The server combines the narrative text and image data to generate an e-book, which is then saved in PDF or ePub format, and a high-resolution print file is generated if necessary.

[0165] 7. Picture book distribution

[0166] The server delivers the generated e-book to the user's device, where the user can view the e-book within the app. In addition, printing services are available if necessary.

[0167] Specific examples

[0168] For example, suppose a child says, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went to a mysterious forest." In this case, the process will proceed as follows:

[0169] 1. Recording and uploading

[0170] The user records this content using the app and uploads it to the server.

[0171] 2. Audio Data Processing

[0172] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[0173] 3. Text Analysis

[0174] The server analyzes this using a natural language processing engine and extracts nouns (me, dragon, rainbow bridge, mysterious forest), verbs (found, crossed, went), etc.

[0175] 4. Narrative Generation

[0176] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[0177] 5. Image Generation

[0178] The server generates image data for "a flying dragon," "a rainbow bridge," and "a mysterious forest" based on the story text.

[0179] 6. Picture Book Generation

[0180] The server combines the narrative text and image data to generate an e-book, which is then saved in PDF or ePub format.

[0181] 7. Picture book distribution

[0182] The server delivers the generated e-book to the user's app, allowing the user to view the picture book within the app. In addition, printing services are available if necessary.

[0183] Examples of prompt statements

[0184] The generative AI model is given a prompt such as, "Create a story about a child finding a flying dragon that crosses the Rainbow Bridge and goes into the forest."

[0185] Throughout this process, what the child says is translated into a story in real time and presented as a visually engaging e-book. This system is a great way to foster children's creativity and create memorable memories for parents and children.

[0186] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0187] Step 1: Record and upload

[0188] User:

[0189] Input: What the child says.

[0190] What it does: The user launches the application on their smartphone or tablet and taps the record button to record what their child says.

[0191] Output: Recorded audio data.

[0192] Once you have finished recording, tap the upload button to send the recording data to the server.

[0193] Step 2: Receiving and converting audio data

[0194] server:

[0195] Input: Recorded audio data.

[0196] Specific operation: The server receives the voice data sent by the user.

[0197] The received voice data is sent to a voice recognition engine (e.g., a general voice recognition API).

[0198] Output: Converted to text by a speech recognition engine.

[0199] A voice recognition engine converts the voice data into text data.

[0200] Step 3: Save the text data

[0201] server:

[0202] Input: Text data.

[0203] Specific operation: The server temporarily stores the converted text data in a database.

[0204] Output: Text data stored in a database.

[0205] The converted text data is stored in a database so that it can be used for further processing.

[0206] Step 4: Text Analysis

[0207] server:

[0208] Input: Saved text data.

[0209] Specific operation: The server sends the stored text data to a natural language processing engine (e.g., a generative AI model).

[0210] A natural language processing engine analyzes the text data and extracts information such as nouns, verbs, subjects, and predicates.

[0211] Output: Parsed information (nouns, verbs, etc.).

[0212] Reconstruct the story based on the extracted information.

[0213] Step 5: Narrative generation

[0214] server:

[0215] Input: Parsed information.

[0216] Specific operation: The server creates a storyline using a narrative generation algorithm based on the analysis results of the natural language processing engine.

[0217] Generate narrative sentences in simple language.

[0218] Output: The generated narrative text.

[0219] The generated narrative text is stored in a database.

[0220] Step 6: Image generation

[0221] server:

[0222] Input: The generated narrative text.

[0223] Specific operation: The server sends the narrative text to an image generation engine (e.g., general image generation technology).

[0224] An image generation engine generates image data based on the narrative text.

[0225] Output: The generated image data.

[0226] The generated image data is stored in a database.

[0227] Step 7: Generate eBook

[0228] server:

[0229] Input: Generated narrative text and image data.

[0230] Specific operation: The server combines the narrative text and image data to generate an e-book.

[0231] Save your e-books in PDF or ePub format.

[0232] Output: The generated e-book.

[0233] High-resolution print files are also generated if needed.

[0234] Step 8: Distributing your picture book

[0235] server:

[0236] Input: The generated e-book.

[0237] Specific operation: The server delivers the generated e-book to the user's device.

[0238] Users will be able to read e-books within the app.

[0239] Output: An e-book that can be read by the user.

[0240] Printing services are also available if required.

[0241] By following these steps in sequence, what the child says is converted into a story in real time and presented as an e-book that can be enjoyed visually.

[0242] (Application example 1)

[0243] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0244] Today's parents and children are busy, with limited time to spend together and opportunities to foster creativity. Therefore, there is a need for a way to give shape to children's creativity and allow parents and children to spend fun time together. There is also a need for a system that can easily convert what a child says into a story and create a digital picture book with beautiful illustrations. Furthermore, there is a need for a way to easily view and distribute the created picture book.

[0245] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0246] In this invention, the server includes means for recording a child's voice data, voice recognition means for converting the recorded voice data into text data, natural language processing means for analyzing the text data and generating a narrative, image generation means for generating picture data based on the generated narrative, picture book generation means for generating an electronic picture book by combining the narrative and picture data, distribution means for providing the generated electronic picture book to a user, and a smartphone application for making the generated electronic picture book viewable. This allows what a child says to be easily converted into a story and picture book that can be enjoyed by parents and children.

[0247] "Children's voice data" refers to data that has been recorded and stored in digital format, containing what a child says.

[0248] A "recording means" is any device or software that records what a child says as a digital audio file.

[0249] "Speech recognition means" refers to technology or tools for analyzing recorded voice data and converting it into text information.

[0250] "Text data" is character information extracted using a voice recognition means.

[0251] "Natural language processing means" refers to technologies and engines for analyzing text data and generating narrative text and sentence structures.

[0252] A "narrative text" is a piece of writing that describes a story or narrative generated by natural language processing means.

[0253] "Image generation means" refers to techniques and tools for generating visual content based on narrative text.

[0254] "Pictorial data" is visual content generated using image generation means.

[0255] "Picture book generation means" refers to techniques and tools for creating an electronic picture book by combining the generated story text and picture data.

[0256] "Distribution means" refers to the technology and platform for providing the generated digital picture book to users.

[0257] A "smartphone application" is an app that runs on a smartphone and allows users to view and operate the generated digital picture book.

[0258] A "generative AI model" is a machine learning model that generates images and other data from text or prompts.

[0259] A "prompt" is an instruction or text used as input to a generative AI model.

[0260] System Overview

[0261] This invention relates to a system that creates picture books that give shape to children's creativity and can be enjoyed by parents and children. The entire system consists of the following components:

[0262] 1. Audio recording means

[0263] The device (smartphone) has a voice recording function to record what the child says. This voice recording function has the function to save the voice data in digital format and upload it to a server. As a concrete example, a voice recording API is used.

[0264] 2. Voice Recognition Method

[0265] The server receives the recorded voice data and converts it into text data using a speech recognition tool, such as an online API like the Google® Cloud Speech-to-Text API.

[0266] 3. Natural Language Processing Methods

[0267] The server uses natural language processing to analyze the converted text data. For example, a natural language processing engine such as spaCy is used as a text analysis tool. Story elements (nouns, verbs, etc.) are extracted from the analyzed text data, and a simple narrative is generated.

[0268] 4. Image Generation Method

[0269] The server generates pictorial data based on the generated narrative text. The image generation method uses a generative AI model such as DALL-E from OpenAI (registered trademark). When a prompt text is entered, an image is generated based on it.

[0270] 5. Picture book generation method

[0271] The server combines the story text and the picture data to generate an electronic picture book. This picture book generation method uses a PDF generation library (e.g., ReportLab) to create an electronic picture book in PDF format.

[0272] 6. Distribution Method

[0273] The server has a distribution means for providing the generated digital picture book to users. Specifically, the server distributes the generated digital picture book through a smartphone application, allowing users to easily view it.

[0274] Hardware and software used

[0275] Hardware: Smartphones, servers

[0276] Software: Voice Recording API, Google Cloud Speech-to-Text API, spaCy, OpenAI DALL-E, ReportLab

[0277] Specific flow of program processing

[0278] For example, consider the example of a child saying, "One day, I became a wizard and defeated the dragon that lived in the castle."

[0279] 1. The device records this content using the voice recording API and uploads it to the server.

[0280] 2. The server converts the audio data into text data using the Google Cloud Speech-to-Text API.

[0281] 3. The server analyzes the converted text data using spaCy and generates a narrative text.

[0282] 4. The server sends the generated story text to OpenAI DALL-E, which generates picture data using the prompt text "Become a wizard and defeat a dragon."

[0283] 5. The server combines the story text and picture data using ReportLab to generate an electronic picture book in PDF format.

[0284] 6. The server distributes the generated e-book to the user via a smartphone application.

[0285] Prompt Sentence Examples

[0286] "One day, I became a wizard and defeated the dragon that lived in the castle."

[0287] Using this system, what a child says can be easily converted into stories and visual content that can be enjoyed by both parents and children.

[0288] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0289] Step 1:

[0290] Recording and uploading

[0291] The user launches the application on their device (smartphone) and uses the voice recording API to record what the child is imagining. The recorded data is saved in digital format. When the user finishes recording, they press the upload button, and the recorded audio data is sent to the server.

[0292] Input: Child's voice

[0293] Output: Recorded audio data (digital format)

[0294] Step 2:

[0295] Audio data conversion

[0296] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API, and the converted text data is stored in a database.

[0297] Input: Recorded audio data

[0298] Output: Text data

[0299] Step 3:

[0300] Text analytics

[0301] The server analyzes the text data using a natural language processing engine (e.g., spaCy) and extracts information such as nouns, verbs, subjects, and predicates. Based on the extracted information, it reconstructs the story and generates a simple narrative sentence. This narrative sentence is then stored in a database.

[0302] Input: Text data

[0303] Output: Analyzed information, narrative text

[0304] Step 4:

[0305] Story Generation

[0306] The server uses a narrative generation algorithm to create a storyline based on the information generated by the natural language processing engine, and generates a narrative in simple language. The generated narrative is then stored in a database.

[0307] Input: Parsed information

[0308] Output: Narrative text

[0309] Step 5:

[0310] Image generation

[0311] The server sends the generated story text to OpenAI's generative AI model (e.g., DALL-E), which generates corresponding picture data. It generates images by providing prompts for story elements (e.g., "wizard," "dragon," "castle," etc.) as input. This picture data is stored in a database.

[0312] Input: Narrative text, prompt text

[0313] Output: Painting data

[0314] Step 6:

[0315] Picture book generation

[0316] The server combines the story text and the picture data and generates an e-book file using a PDF generation library (e.g., ReportLab). The generated e-book is saved in PDF format.

[0317] Input: Narrative text, picture data

[0318] Output: Digital picture book file (PDF format)

[0319] Step 7:

[0320] Picture book distribution

[0321] The server distributes the generated digital picture book file to the user's terminal via a distribution means, and the user can view the digital picture book through a smartphone application.

[0322] Input: Digital picture book file

[0323] Output: Digital picture book delivery to user devices

[0324] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0325] This invention relates to a system that creates picture books that give shape to children's creativity and can be enjoyed by parents and children. This system not only automatically generates stories based on children's fantasies and creates picture books based on them, but also uses an emotion engine to adjust the story text and picture data according to the user's emotional state.

[0326] Overall system overview

[0327] The system includes the following major components:

[0328] 1. Audio recording method (used by the user)

[0329] Users use the application to record what their child says, and the audio recordings are stored digitally and uploaded to a server.

[0330] 2. Speech recognition method (runs on the server)

[0331] The server receives the voice data and converts it into text using a voice recognition engine, which uses an online API.

[0332] 3. Natural language processing means (running on the server)

[0333] The server analyzes the converted text data and generates a narrative using a natural language processing engine.

[0334] 4. Image generation method (runs on the server)

[0335] The server generates pictorial data based on the generated narrative text using an image generation engine.

[0336] 5. Picture book generation method (runs on the server)

[0337] The server combines the story text and image data to generate an electronic picture book, which is saved in PDF or ePub format.

[0338] 6. Delivery method (runs on the server)

[0339] The server distributes the generated digital picture books to users, who can view them within the app and use the printing service if necessary.

[0340] 7. Emotion recognition method (runs on the server)

[0341] The server is equipped with an emotion engine that recognizes emotional information in the recorded voice data, thereby analyzing the user's emotional state and storing it in a database.

[0342] 8. Emotional Adaptation Method (Runs on Server)

[0343] Based on the analysis results of the emotion engine, the server adjusts the content of the narrative text to be generated and the color tone and style of the painting data.

[0344] Program processing flow

[0345] The specific flow of program processing in this system will be explained below.

[0346] 1. Recording and uploading

[0347] The user launches the app and records what the child is imagining. After recording, the user presses the upload button to send the audio data to the server.

[0348] 2. Audio Data Processing

[0349] The server stores the received voice data and sends it to a voice recognition engine (e.g., an online API).

[0350] 3. Voice Recognition

[0351] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[0352] 4. Text Analysis

[0353] The server receives the returned text data and temporarily stores it in a database.

[0354] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure.

[0355] 5. Narrative Generation

[0356] The server uses a story generation algorithm to create a storyline based on the analysis results, and generates a narrative in simple language. The generated narrative is then stored in a database.

[0357] 6. Emotion recognition

[0358] The server sends the recorded voice data to the emotion engine to recognize the user's emotional state, and stores the information on the emotional state in a database.

[0359] 7. Story and Art Coordination

[0360] The server adjusts the content of the narrative sentence to be generated based on the emotional state recognized by the emotion engine.

[0361] The server further sends instructions to the image generation engine to adjust the color tone and style of the generated painting data based on the emotional state.

[0362] 8. Image Generation

[0363] The server generates pictorial data based on the narrative text using an image generation engine, and the pictorial data is adjusted based on the context of the story and the user's emotional state.

[0364] 9. Picture Book Generation

[0365] The server combines the story text and the picture data to generate an electronic picture book file (e.g., PDF or ePub format).

[0366] 10. Distribution

[0367] The server distributes the generated electronic picture book file to the user's terminal.

[0368] Users can view the digital picture books distributed within the app, and if necessary, can use the printing service to save them in paper form.

[0369] Specific examples

[0370] For example, suppose a child says, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went to a mysterious forest." Also, suppose the recording contains the emotions of a child talking happily.

[0371] 1. Recording and uploading

[0372] The user records this content using the app and uploads it to the server.

[0373] 2. Audio Data Processing

[0374] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[0375] 3. Text Analysis

[0376] The server analyzes this using a natural language processing engine and extracts nouns (me, dragon, rainbow bridge, mysterious forest), verbs (found, crossed, went), etc.

[0377] 4. Narrative Generation

[0378] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[0379] 5. Emotion recognition

[0380] The server sends the recorded voice data to the emotion engine, which recognizes that the user is speaking with a happy tone. Based on this emotional information, positive and fun elements are added to the story.

[0381] 6. Story and picture coordination

[0382] Based on the information from the emotion engine, the server adds more excitement and adventure to the story and instructs the image generation engine to draw bright and colorful pictures.

[0383] 7. Image Generation

[0384] The server generates art data for a "flying dragon," a "rainbow bridge," and a "mysterious forest" based on the story text. This art data is adjusted to create a bright and cheerful atmosphere.

[0385] 8. Picture Book Generation

[0386] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[0387] 9. Distribution

[0388] The server delivers the generated digital picture book to the user's app, where the user can view the book within the app. Furthermore, if necessary, the user can use a printing service to save the book on paper.

[0389] Through this process, what the child says is transformed into a story, and a picture book that reflects the child's emotions is generated. This system is a groundbreaking way to foster children's creativity and further enrich memories between parents and children.

[0390] The processing flow will be explained below.

[0391] Step 1:

[0392] The user launches the app and uses the recording feature to record the child's imaginary speech. When the recording is finished, the user presses the stop recording button.

[0393] Step 2:

[0394] The user presses the upload button to upload the recorded audio data to the server. The device sends the recorded data to the server.

[0395] Step 3:

[0396] The server stores the received voice data and sends it to the voice recognition engine.

[0397] Step 4:

[0398] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[0399] Step 5:

[0400] The server receives the returned text data and temporarily stores it in a database.

[0401] Step 6:

[0402] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure.

[0403] Step 7:

[0404] The natural language processing engine extracts nouns, verbs, subjects, predicates, etc. from text data and performs analysis to understand the context of the text.

[0405] Step 8:

[0406] The server receives the analysis results and creates a storyline using a narrative generation algorithm.

[0407] Step 9:

[0408] The server generates a story in simple language and stores the generated story in a database.

[0409] Step 10:

[0410] The server sends the recorded voice data to the emotion engine to recognize the user's emotional state. The emotion engine analyzes the emotional information (e.g., joy, sadness, surprise) from the voice and sends the emotional state back to the server.

[0411] Step 11:

[0412] The server adjusts the content of the story it generates based on the emotional state recognized by the emotion engine. For example, if the user is having fun, it adds positive elements to the story.

[0413] Step 12:

[0414] The server further transmits the narrative text to the image generation engine to adjust the color tone and style of the generated painting data based on the emotional state.

[0415] Step 13:

[0416] The image generation engine generates pictorial data based on the narrative text and returns the generated pictorial data to the server, where it is adjusted based on the emotional information.

[0417] Step 14:

[0418] The server combines the story text and the picture data to generate an electronic picture book file (e.g., PDF or ePub format).

[0419] Step 15:

[0420] The server distributes the generated electronic picture book file to the user's terminal.

[0421] Step 16:

[0422] Users can view the digital picture books delivered within the app, and if necessary, can save them in paper form using the printing service.

[0423] Example 2

[0424] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0425] Conventional picture book generation systems have struggled to automatically generate content that reflects a child's creativity and emotions. Furthermore, there was no technology that could accurately convert a child's speech into text and generate appropriate picture data based on that context, or that could adjust the story and picture data to reflect a child's emotional state. This made it difficult to provide personalized digital picture books that parents and children could enjoy.

[0426] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0427] In this invention, the server includes recording means for recording a child's voice data, voice recognition means for converting the recorded voice data into text data, natural language processing means for analyzing the text data and generating a narrative, image generation means for generating picture data based on the generated narrative, picture book generation means for generating an electronic picture book by combining the narrative and picture data, distribution means for providing the generated electronic picture book to a user, emotion engine means for recognizing an emotional state from the recorded voice data, and emotion adaptation means for adjusting the narrative and picture data based on the recognized emotional state. This allows for the automatic generation of a personalized electronic picture book that reflects the child's creativity and emotions, allowing parents and children to enjoy it together.

[0428] "Recording means" means a device or method for recording a child's voice data in digital form.

[0429] "Speech recognition means" refers to the technology or algorithms used to convert recorded voice data into text data.

[0430] "Natural language processing means" is a technology for analyzing text data generated by speech recognition means and constructing meaningful narrative text.

[0431] "Image generation means" refers to algorithms and techniques for generating pictorial data based on the generated narrative text.

[0432] "Picture book generation means" refers to a technique or method for synthesizing narrative text and picture data to generate a formatted electronic picture book.

[0433] "Distribution means" refers to the technology or system for delivering the generated electronic picture book to users.

[0434] The "emotion engine means" is a technology for recognizing an emotional state from recorded voice data and adjusting narrative text and picture data based on that.

[0435] The "emotion adaptation means" is a technique for dynamically changing the content and style of narrative text and picture data based on the emotional state recognized by the emotion engine means.

[0436] The present invention relates to a system for generating picture books that give shape to children's creativity and allow parents and children to enjoy them together. This system not only automatically generates stories based on children's fantasies and creates picture books based on those stories, but also uses an emotion engine to adjust the story text and picture data according to the user's emotional state. A specific embodiment of this system is described below.

[0437] Hardware and software used

[0438] 1. Audio recording method (used by the user)

[0439] Users use an application on their smartphone or tablet to record what their child says, and the audio data is stored digitally and uploaded to a server.

[0440] 2. Speech recognition method (runs on the server)

[0441] The server receives the voice data and converts it into text data using an online API (e.g., Google Cloud Speech-to-Text) as a voice recognition engine.

[0442] 3. Natural language processing means (running on the server)

[0443] The server analyzes the converted text data and generates a narrative using a natural language processing engine (e.g., OpenAI's GPT-3 (registered trademark)).

[0444] 4. Image generation method (runs on the server)

[0445] The server generates pictorial data based on the generated narrative text using an image generation engine (e.g., DALL-E).

[0446] 5. Picture book generation method (runs on the server)

[0447] The server combines the story text and image data to generate an electronic picture book, which is saved in PDF or ePub format.

[0448] 6. Delivery method (runs on the server)

[0449] The server distributes the generated digital picture books to users, who can view them within the app and use the printing service if necessary.

[0450] 7. Emotion recognition method (runs on the server)

[0451] The server is equipped with an emotion engine that recognizes emotional information in the recorded voice data, thereby analyzing the user's emotional state and storing it in a database.

[0452] 8. Emotional Adaptation Method (Runs on Server)

[0453] Based on the analysis results of the emotion engine, the server adjusts the content of the narrative text to be generated and the color tone and style of the painting data.

[0454] Example of a system

[0455] For example, suppose a child says, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went to a mysterious forest." This recording contains the emotions of a child talking happily.

[0456] 1. Recording and uploading

[0457] The user records this content using the app and uploads it to the server.

[0458] 2. Audio Data Processing

[0459] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[0460] 3. Text Analysis

[0461] The server analyzes this using a natural language processing engine and extracts nouns ("I," "dragon," "Rainbow Bridge," "Mysterious Forest"), verbs ("I found it," "crossed," "I went"), etc.

[0462] 4. Narrative Generation

[0463] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[0464] 5. Emotion recognition

[0465] The server sends the recorded voice data to the emotion engine, which recognizes that the user is speaking with a happy tone. Based on this emotional information, positive and fun elements are added to the story.

[0466] 6. Story and picture coordination

[0467] Based on the information from the emotion engine, the server adds more excitement and adventure to the story and instructs the image generation engine to draw bright and colorful pictures.

[0468] 7. Image Generation

[0469] The server generates art data for a "flying dragon," a "rainbow bridge," and a "mysterious forest" based on the story text. This art data is adjusted to create a bright and cheerful atmosphere.

[0470] 8. Picture Book Generation

[0471] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[0472] 9. Distribution

[0473] The server delivers the generated digital picture book to the user's app, where the user can view the book within the app. Furthermore, if necessary, the user can use a printing service to save the book on paper.

[0474] The following are examples of prompt sentences that can be input to a generative AI model:

[0475] Example prompt sentence:

[0476] Generate a new adventure story based on a story my child told me. My child said with a happy expression, "One day, I found a flying dragon. The dragon crossed the Rainbow Bridge and went to a mysterious forest." Based on this, generate a story and illustrations that contain many positive elements.

[0477] In this way, a system is realized that can generate a story based on what a child says and automatically generate an electronic picture book that reflects the user's emotional state, providing a personalized picture book that can be enjoyed by both parents and children.

[0478] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0479] Step 1:

[0480] Recording and uploading

[0481] The user launches the app, taps the record button to record the child's imaginary speech, and then taps the upload button to send the audio data to the server.

[0482] Input: The audio spoken by the user.

[0483] Output: Audio data uploaded to the server.

[0484] How it works: When the user presses the "Record" button on the smartphone app screen, the built-in microphone activates and records what the child is saying. When the recording is finished, the user taps the "Upload" button, and the app sends the audio data to the server.

[0485] Step 2:

[0486] Audio data processing

[0487] The server temporarily stores the received voice data and sends it to the voice recognition engine.

[0488] Input: The audio data sent to the server.

[0489] Output: The audio data sent to the speech recognition engine.

[0490] Specific operation: When the server receives the voice data, it stores it in temporary storage and then sends it to a voice recognition engine (e.g., Google Cloud Speech-to-Text) via API.

[0491] Step 3:

[0492] Voice Recognition

[0493] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[0494] Input: The audio data sent to the speech recognition engine.

[0495] Output: The text data sent back to the server.

[0496] Specific operation: The speech recognition engine analyzes the voice signal and converts the voice into text. The text data is sent back to the server as an API response.

[0497] Step 4:

[0498] Text analytics

[0499] The server receives the returned text data and temporarily stores it in a database, after which the text data is sent to a natural language processing engine for grammatical analysis.

[0500] Input: The returned text data.

[0501] Output: Data with grammatical structure parsed.

[0502] How it works: The server stores the text data in a database and sends it to a natural language processing engine (e.g., OpenAI's GPT-3). The analysis results are returned and stored on the server.

[0503] Step 5:

[0504] Story Generation

[0505] The server uses a narrative generation algorithm to create a story based on the analysis results, and generates a narrative text, which is then stored in a database.

[0506] Input: Analysis results.

[0507] Output: The generated narrative text.

[0508] Specific operation: The server generates the main plot of the story from the analysis results and converts it into simple sentences. The generated story is then saved in the database again.

[0509] Step 6:

[0510] emotion recognition

[0511] The server sends the recorded voice data to the emotion engine to recognize the user's emotional state, and stores the information on the emotional state in a database.

[0512] Input: Recorded audio data.

[0513] Output: Emotional state information.

[0514] How it works: The server sends the voice data to the emotion engine, which interprets emotions from the tone and expressions of the voice. Once the emotional state is recognized, it is stored in a database.

[0515] Step 7:

[0516] Story and picture coordination

[0517] The server adjusts the content of the narrative text to be generated based on the emotional state recognized by the emotion engine, and also sends instructions to the image generation engine to adjust the color tone and style of the generated painting data.

[0518] Input: Emotional state information.

[0519] Output: Adjusted narrative text and pictorial data instructions.

[0520] Specific actions: If the emotional state is "happy," add positive elements to the story. Send specific instructions to the image generation engine to adjust the color tone and style of the painting data to be bright and cheerful.

[0521] Step 8:

[0522] Picture generation

[0523] The server generates pictorial data based on the narrative text using an image generation engine, and the pictorial data is adjusted based on the context of the story and the user's emotional state.

[0524] Input: Narrative text and tailored instructions.

[0525] Output: The generated painting data.

[0526] How it works: The server sends the story text to the image generation engine, which creates an illustration based on the specified style and emotion. The generated illustration is then sent back to the server.

[0527] Step 9:

[0528] Picture book generation

[0529] The server combines the story text and the picture data to generate an electronic picture book file, which is saved in PDF or ePub format.

[0530] Input: Narrative text and picture data.

[0531] Output: Digital picture book file (PDF or ePub format).

[0532] Specific operation: The server inputs the story text and picture data into the template engine and determines the layout of the digital picture book. The generated picture book is output in PDF or ePub format and saved in storage.

[0533] Step 10:

[0534] delivery

[0535] The server delivers the generated digital picture book file to the user's device. The user can view the delivered digital picture book within the app and can also use the printing service if necessary.

[0536] Input: E-book file.

[0537] Output: An electronic picture book that can be viewed by the user.

[0538] Specific operation: The server sends the user a link to the generated e-book file. The user clicks the link through the app to view the e-book. They can also access the printing service if they wish.

[0539] (Application example 2)

[0540] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0541] Systems that generate picture books that give shape to children's creativity and can be enjoyed by parents and children alike are required to provide a more immersive, interactive experience that reflects the content and emotions of children.However, conventional systems often generate picture books with static content, and are unable to provide an interactive experience where parents and children can actually interact with the characters and story world in a virtual space.

[0542] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for recording a child's voice data; means for converting the recorded voice data into text data; means for analyzing the text data and generating a narrative; means for generating picture data based on the generated narrative; means for adjusting the generated narrative and picture data and displaying them in a virtual space; means for interactively experiencing the generated electronic picture book in the virtual space; means for generating an electronic picture book by combining the narrative and picture data; and means for providing the generated electronic picture book to a user. This enables a dynamic and interactive story experience that reflects a child's creativity.

[0543] "Children's voice data" is digital voice data that records what a child is saying.

[0544] "Speech recognition means" refers to a technical means for converting recorded voice data into text data.

[0545] "Text data" is data that has been converted from audio data into text format.

[0546] "Natural language processing means" refers to technical means for analyzing text data and generating narrative text.

[0547] A "narrative text" is a story-form text generated from analyzed text data.

[0548] "Image generation means" refers to technical means for generating picture data based on the generated narrative text.

[0549] "Picture data" is digital image data generated based on a narrative text.

[0550] "Adjustment means" refers to technical means for appropriately adjusting the generated narrative text and pictorial data for display in a virtual space.

[0551] The "virtual space" is a virtual digital space in which the generated electronic picture book and other digital content are displayed.

[0552] The "experiential means" refers to a technical means that allows a user to interactively experience the generated digital picture book in a virtual space.

[0553] The "picture book generation means" is a technical means for generating an electronic picture book by combining a story and picture data.

[0554] "Distribution means" refers to the technical means for providing the generated electronic picture book to the user.

[0555] "Interactive" refers to the ability of users to directly interact with and respond to generated content.

[0556] This invention relates to a virtual picture book creation system that allows parents and children to express their creativity and have fun together. This system automatically creates a realistic, interactive picture book that reflects the child's words and emotions, allowing parents and children to experience the world of the picture book in a virtual space.

[0557] Overall system overview

[0558] The system includes the following major components:

[0559] 1. Audio recording means (user terminal)

[0560] Users use the application to record what their child says, and the audio recordings are stored digitally and uploaded to a server.

[0561] 2. Speech recognition means (server)

[0562] The server receives the voice data and converts it into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0563] 3. Natural language processing means (server)

[0564] The server analyzes the converted text data and generates a narrative using a natural language processing engine (e.g., GPT-4 (registered trademark)).

[0565] 4. Image generation means (server)

[0566] The server generates pictorial data based on the generated narrative text, using an image generation engine (e.g., DALL-E).

[0567] 5. Coordination means (server)

[0568] The server adjusts the content of the narrative text and the color tone and style of the pictorial data to be generated based on the emotional state recognized by an emotional engine (e.g., IBM Watson (registered trademark) Tone Analyzer).

[0569] 6. Experience Method (User Device)

[0570] Users can interactively experience the generated picture book in a virtual space using smart glasses or a head-mounted display (e.g., Magic Leap, Oculus Rift).

[0571] 7. Picture book generation means (server)

[0572] The server combines the story text and the picture data to generate an electronic picture book, which is saved in PDF or ePub format.

[0573] 8. Delivery Method (Server)

[0574] The server distributes the generated digital picture book to the user, who can then view the picture book within the app.

[0575] Specific examples

[0576] The following is a specific example of the system.

[0577] For example, consider a situation where a system generates a story and pictures based on a child saying, "One day I want to play in a castle with a princess." The recording contains the emotion of the child talking happily.

[0578] 1. Audio recording and uploading

[0579] The user (parent or child) starts the app and records what the child says. After recording is complete, they press the upload button to send the audio data to the server.

[0580] 2. Voice Recognition

[0581] The server sends the received voice data to a speech recognition engine and converts it into text data, for example, "One day I want to play at the castle with the princess."

[0582] 3. Natural Language Processing

[0583] The server sends the converted text data to a natural language processing engine to generate a narrative sentence, such as "One day, the child had fun playing in the castle with the princess."

[0584] 4. Emotion recognition and regulation

[0585] The server sends the recorded voice data to the emotion engine, which recognizes positive emotions. Based on this emotional information, the color tone and style of the story text and painting data are adjusted to be brighter.

[0586] 5. Image Generation

[0587] Based on the story text, the server uses an image generation engine to generate pictures of "princesses" and "castles," resulting in colorful and fun images that fit the context of the story.

[0588] 6. Virtual Space Display and Experience

[0589] Users wear smart glasses or a head-mounted display and experience the picture book generated in a virtual space. In the virtual space, users can interact with the characters in the book by touching them. For example, the characters may walk around the castle or talk to them.

[0590] 7. Picture book generation and distribution

[0591] The server combines the story text and picture data to create an electronic picture book and delivers it to the user's device. Users can view the electronic picture book created within the app, and if necessary, can use a printing service to save it on paper.

[0592] Prompt Sentence Examples

[0593] Prompt: "Based on a child's joyful statement, 'One day I want to play in a castle with a princess,' please generate a story with a positive and fun atmosphere."

[0594] In this way, a dynamic and interactive storytelling experience is created that reflects the child's creativity in real time.

[0595] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0596] Step 1:

[0597] Audio recording and uploading

[0598] The user launches the application and records what the child says.

[0599] Input: User spoken voice data

[0600] What it does: Uses the device's microphone to collect audio and store it digitally.

[0601] Output: Recorded audio data (e.g. audio_file.wav)

[0602] Step 2:

[0603] Sending audio data

[0604] After the user has finished recording, he or she presses the upload button to send the audio data to the server.

[0605] Input: Recorded audio data (audio_file.wav)

[0606] Specific operation: The application sends an HTTP request to the server and uploads the audio data.

[0607] Output: Audio data received on the server side

[0608] Step 3:

[0609] Voice Recognition

[0610] The server sends the received voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data.

[0611] Input: Audio data uploaded to the server

[0612] Specific operation: The speech recognition engine analyzes the voice data and converts it into a string of characters.

[0613] Output: Text data obtained by speech recognition (e.g., "One day I want to play at the castle with the princess.")

[0614] Step 4:

[0615] Natural Language Processing (NLP)

[0616] The server sends the converted text data to a natural language processing engine (e.g., GPT-4) to generate a narrative.

[0617] Input: Text data (e.g., "One day I want to play in a castle with a princess.")

[0618] Specific operation: The NLP engine performs grammatical analysis and generates a narrative storyline.

[0619] Output: Generated story (e.g., "One day, the child had fun playing in the castle with the princess.")

[0620] Step 5:

[0621] Emotion Recognition and Regulation

[0622] The server sends the recorded voice data to an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotional state.

[0623] Input: Recorded audio data

[0624] Specific operation: The emotion engine recognizes emotions in the voice and returns the analysis results.

[0625] Output: Sentiment analysis result (e.g., positive)

[0626] Step 6:

[0627] Story and picture coordination

[0628] Based on the analysis results of the emotion engine, the server adjusts the content of the narrative text to be generated and the color tone and style of the painting data.

[0629] Input: Narrative text, sentiment analysis results

[0630] Specific actions: Add positive elements to the story and send instructions to the image generation engine (e.g., DALL-E) for bright colors and a happy atmosphere.

[0631] Output: Coordinated narrative and image generation instructions

[0632] Step 7:

[0633] Image generation

[0634] The server generates picture data based on the story using an image generation engine.

[0635] Input: tailored narrative text and image generation instructions

[0636] How it works: The image generation engine draws story scenes and characters and outputs them as digital images.

[0637] Output: Generated painting data (e.g., colorful painting)

[0638] Step 8:

[0639] Virtual space display and experience

[0640] Users can experience the picture book generated in a virtual space using smart glasses or a head-mounted display.

[0641] Input: Generated narrative text and picture data

[0642] Specific operation: The application displays narrative text and pictorial data in a virtual space, allowing users to experience it interactively.

[0643] Output: An interactive picture book experience in a virtual space

[0644] Step 9:

[0645] Picture book generation

[0646] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[0647] Input: Generated narrative text and picture data

[0648] Specific actions: Integrate the narrative text and picture data into a format and save it as an electronic file.

[0649] Output: Generated e-book (e.g. ebook.pdf)

[0650] Step 10:

[0651] delivery

[0652] The server distributes the generated electronic picture book to the user terminal.

[0653] Input: Generated e-book

[0654] Specific operation: The server sends the electronic picture book file to the user's application.

[0655] Output: Digital picture book delivered to the user's device

[0656] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0657] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0658] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0659] [Second embodiment]

[0660] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0661] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0662] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0663] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0664] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0665] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0666] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0667] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0668] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0669] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0670] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0671] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0672] This invention relates to a system that creates picture books that can give shape to a child's creativity and be enjoyed by parents and children. This system can automatically generate a story based on a child's imagination and create a picture book based on that story.

[0673] Overall system overview

[0674] The system includes the following major components:

[0675] 1. Audio recording method (used by the user)

[0676] Users use the application to record what their child says, and the audio recordings are stored digitally and uploaded to a server.

[0677] 2. Speech recognition method (runs on the server)

[0678] The server receives the voice data and converts it into text data using a voice recognition engine, which uses an online API (for example, a general voice recognition API).

[0679] 3. Natural language processing means (running on the server)

[0680] The server analyzes the converted text data and generates a narrative using a natural language processing engine.

[0681] 4. Image generation method (runs on the server)

[0682] The server generates pictorial data based on the generated narrative text using an image generation engine (e.g., using common image generation technology).

[0683] 5. Picture book generation method (runs on the server)

[0684] The server combines the story text and image data to generate an electronic picture book, which is saved in PDF or ePub format.

[0685] 6. Delivery method (runs on the server)

[0686] The server distributes the generated digital picture books to users, who can view them within the app and use the printing service if necessary.

[0687] Program processing flow

[0688] The specific flow of program processing in this system will be explained below.

[0689] Recording and uploading

[0690] User

[0691] The user launches the app, records what the child is imagining, and then presses the upload button to send the audio data to the server.

[0692] Audio data processing

[0693] server

[0694] The server sends the received voice data to a speech recognition engine and converts it into text data, which is then temporarily stored in a database.

[0695] Text analytics

[0696] server

[0697] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure and extracts information such as nouns, verbs, subjects, and predicates. The extracted information is then used to reconstruct the story.

[0698] Story Generation

[0699] server

[0700] The server uses a story generation algorithm to create a storyline based on the analysis results, and generates a narrative in simple language. The generated narrative is then stored in a database.

[0701] Picture generation

[0702] server

[0703] The server sends the story to an image generation engine, which generates pictorial data corresponding to the story, and stores the pictorial data in a database.

[0704] Picture book generation

[0705] server

[0706] The server combines the story text and the picture data to generate and store an electronic picture book file. If necessary, it also generates a high-resolution file for printing.

[0707] Picture book distribution

[0708] server

[0709] The server delivers the generated digital picture book file to the user's app, where the user can view the picture book and use the printing service.

[0710] Specific examples

[0711] For example, a child might say, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[0712] 1. Recording and uploading

[0713] The user records this content using the app and uploads it to the server.

[0714] 2. Audio Data Processing

[0715] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[0716] 3. Text Analysis

[0717] The server analyzes this using a natural language processing engine and extracts nouns (me, dragon, rainbow bridge, mysterious forest), verbs (found, crossed, went), etc.

[0718] 4. Narrative Generation

[0719] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[0720] 5. Image Generation

[0721] The server generates picture data for "a flying dragon," "a rainbow bridge," and "a mysterious forest" based on the story text.

[0722] 6. Picture Book Generation

[0723] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[0724] 7. Picture book distribution

[0725] The server delivers the generated e-book to the user's app, allowing the user to view the book within the app. Furthermore, printing services are also available if necessary.

[0726] This process transforms what the child says into a story in real time and provides it as a visually appealing picture book. This system is a groundbreaking way to foster children's creativity and create memories for parents and children.

[0727] The processing flow will be explained below.

[0728] Step 1:

[0729] The user launches the app and uses the recording feature to record the child's imaginary speech. When the recording is finished, the user presses the stop recording button.

[0730] Step 2:

[0731] The user presses the upload button to upload the recorded audio data to the server. The device sends the recorded data to the server.

[0732] Step 3:

[0733] The server stores the received voice data and sends it to a voice recognition engine (e.g., a common online API).

[0734] Step 4:

[0735] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[0736] Step 5:

[0737] The server receives the returned text data and temporarily stores it in a database.

[0738] Step 6:

[0739] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure.

[0740] Step 7:

[0741] The natural language processing engine extracts nouns, verbs, subjects, predicates, etc. from text data and performs analysis to understand the context of the text.

[0742] Step 8:

[0743] The server receives the analysis results and creates a storyline using a narrative generation algorithm.

[0744] Step 9:

[0745] The server generates a story in simple language and stores the generated story in a database.

[0746] Step 10:

[0747] The server sends the stored narrative to an image generation engine (eg, general image generation technology).

[0748] Step 11:

[0749] The image generation engine generates picture data based on the story text and returns the generated picture data to the server.

[0750] Step 12:

[0751] The server receives the returned painting data and stores it in a database.

[0752] Step 13:

[0753] The server combines the stored story text and picture data to generate an electronic picture book file (e.g., PDF or ePub format).

[0754] Step 14:

[0755] The server distributes the generated electronic picture book file to the user's terminal.

[0756] Step 15:

[0757] Users can view the digital picture books delivered within the app, and if necessary, can save them in paper form using the printing service.

[0758] Example 1

[0759] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0760] Existing picture book creation systems make it difficult to give form to children's creativity, and lack an easy way for parents and children to enjoy creating together. Furthermore, there was no system that efficiently converts recorded audio into text and then automatically generates illustrations that correspond to the story text based on that. This limited parent-child communication and restricted opportunities to develop children's creativity.

[0761] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0762] In this invention, the server includes means for recording a child's voice data, means for converting the recorded voice data into text data, means for analyzing the text data and generating a story, means for generating image data based on the generated story, means for generating an electronic book by combining the story and image data, and means for providing the generated electronic book to users. This makes it possible for parents and children to easily give shape to their children's creativity and create an electronic picture book that they can enjoy together.

[0763] A "means for recording a child's voice data" is a device or application that digitally records what a child says and stores that voice data.

[0764] "A speech recognition means for converting recorded speech data into text data" is a technology or engine that analyzes recorded speech data and converts it into corresponding text data.

[0765] "Natural language processing means for analyzing text data and generating narrative text" refers to a technology or algorithm that understands the language structure based on text data and reconstructs it as narrative text.

[0766] The "image generation means for generating image data based on the generated narrative text" refers to technology or software for automatically creating visual pictures or images that correspond to the content of the narrative text.

[0767] "Electronic book generation means for combining narrative text and image data to generate an electronic book" refers to a technology or system that combines narrative text and corresponding images into a single electronic book format (such as PDF or ePub).

[0768] The "distribution means for providing the generated electronic book to the user" refers to a technology or service for transmitting the completed electronic book to the user's terminal so that it can be viewed or printed.

[0769] This invention relates to a system that creates e-books that can give shape to children's creativity and be enjoyed by parents and children. This system can automatically generate stories based on children's fantasies and create picture books based on those stories.

[0770] Overall system overview

[0771] The system includes the following major components:

[0772] 1. Audio recording method (used by the device)

[0773] The device uses an application to record what the child says and saves the audio data, which is then uploaded digitally to a server.

[0774] 2. Speech recognition method (runs on the server)

[0775] The server receives the voice data and converts it into text data using a voice recognition engine (e.g., a general voice recognition API).

[0776] 3. Natural language processing means (running on the server)

[0777] The server analyzes the converted text data and generates a narrative using a natural language processing engine (e.g., a generative AI model).

[0778] 4. Image generation method (runs on the server)

[0779] The server generates image data based on the generated narrative text using an image generation engine (e.g., a general image generation technology).

[0780] 5. E-book generation means (running on the server)

[0781] The server combines the narrative text and image data to generate an e-book, which is saved in PDF or ePub format.

[0782] 6. Delivery method (runs on the server)

[0783] The server distributes the generated e-books to users, who can view them within the app and use printing services if necessary.

[0784] Program processing flow

[0785] 1. Recording and uploading

[0786] The user launches the app, records what the child says, and when they're done, they press the upload button to send the audio data to the server.

[0787] 2. Audio Data Processing

[0788] The server sends the received voice data to a voice recognition engine and converts it into text data, which is then temporarily stored in a database.

[0789] 3. Text Analysis

[0790] The server sends the stored text data to a natural language processing engine (generative AI model), which analyzes the grammatical structure and extracts information such as nouns, verbs, subjects, and predicates. The extracted information is then used to reconstruct the story.

[0791] 4. Narrative Generation

[0792] The server uses a story generation algorithm based on the analysis results to create a storyline and generate a narrative in simple language. The generated narrative is then stored in a database.

[0793] 5. Image Generation

[0794] The server sends the story text to an image generation engine, which generates image data corresponding to the story text. The generated image data is stored in a database.

[0795] 6. Picture Book Generation

[0796] The server combines the narrative text and image data to generate an e-book, which is then saved in PDF or ePub format, and a high-resolution print file is generated if necessary.

[0797] 7. Picture book distribution

[0798] The server delivers the generated e-book to the user's device, where the user can view the e-book within the app. In addition, printing services are available if necessary.

[0799] Specific examples

[0800] For example, suppose a child says, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went to a mysterious forest." In this case, the process will proceed as follows:

[0801] 1. Recording and uploading

[0802] The user records this content using the app and uploads it to the server.

[0803] 2. Audio Data Processing

[0804] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[0805] 3. Text Analysis

[0806] The server analyzes this using a natural language processing engine and extracts nouns (me, dragon, rainbow bridge, mysterious forest), verbs (found, crossed, went), etc.

[0807] 4. Narrative Generation

[0808] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[0809] 5. Image Generation

[0810] The server generates image data for "a flying dragon," "a rainbow bridge," and "a mysterious forest" based on the story text.

[0811] 6. Picture Book Generation

[0812] The server combines the narrative text and image data to generate an e-book, which is then saved in PDF or ePub format.

[0813] 7. Picture book distribution

[0814] The server delivers the generated e-book to the user's app, allowing the user to view the picture book within the app. In addition, printing services are available if necessary.

[0815] Examples of prompt statements

[0816] The generative AI model is given a prompt such as, "Create a story about a child finding a flying dragon that crosses the Rainbow Bridge and goes into the forest."

[0817] Throughout this process, what the child says is translated into a story in real time and presented as a visually engaging e-book. This system is a great way to foster children's creativity and create memorable memories for parents and children.

[0818] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0819] Step 1: Record and upload

[0820] User:

[0821] Input: What the child says.

[0822] What it does: The user launches the application on their smartphone or tablet and taps the record button to record what their child says.

[0823] Output: Recorded audio data.

[0824] Once you have finished recording, tap the upload button to send the recording data to the server.

[0825] Step 2: Receiving and converting audio data

[0826] server:

[0827] Input: Recorded audio data.

[0828] Specific operation: The server receives the voice data sent by the user.

[0829] The received voice data is sent to a voice recognition engine (e.g., a general voice recognition API).

[0830] Output: Converted to text by a speech recognition engine.

[0831] A voice recognition engine converts the voice data into text data.

[0832] Step 3: Save the text data

[0833] server:

[0834] Input: Text data.

[0835] Specific operation: The server temporarily stores the converted text data in a database.

[0836] Output: Text data stored in a database.

[0837] The converted text data is stored in a database so that it can be used for further processing.

[0838] Step 4: Text Analysis

[0839] server:

[0840] Input: Saved text data.

[0841] Specific operation: The server sends the stored text data to a natural language processing engine (e.g., a generative AI model).

[0842] A natural language processing engine analyzes the text data and extracts information such as nouns, verbs, subjects, and predicates.

[0843] Output: Parsed information (nouns, verbs, etc.).

[0844] Reconstruct the story based on the extracted information.

[0845] Step 5: Narrative generation

[0846] server:

[0847] Input: Parsed information.

[0848] Specific operation: The server creates a storyline using a narrative generation algorithm based on the analysis results of the natural language processing engine.

[0849] Generate narrative sentences in simple language.

[0850] Output: The generated narrative text.

[0851] The generated narrative text is stored in a database.

[0852] Step 6: Image generation

[0853] server:

[0854] Input: The generated narrative text.

[0855] Specific operation: The server sends the narrative text to an image generation engine (e.g., general image generation technology).

[0856] An image generation engine generates image data based on the narrative text.

[0857] Output: The generated image data.

[0858] The generated image data is stored in a database.

[0859] Step 7: Generate eBook

[0860] server:

[0861] Input: Generated narrative text and image data.

[0862] Specific operation: The server combines the narrative text and image data to generate an e-book.

[0863] Save your e-books in PDF or ePub format.

[0864] Output: The generated e-book.

[0865] High-resolution print files are also generated if needed.

[0866] Step 8: Distributing your picture book

[0867] server:

[0868] Input: The generated e-book.

[0869] Specific operation: The server delivers the generated e-book to the user's device.

[0870] Users will be able to read e-books within the app.

[0871] Output: An e-book that can be read by the user.

[0872] Printing services are also available if required.

[0873] By following these steps in sequence, what the child says is converted into a story in real time and presented as an e-book that can be enjoyed visually.

[0874] (Application example 1)

[0875] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0876] Today's parents and children are busy, with limited time to spend together and opportunities to foster creativity. Therefore, there is a need for a way to give shape to children's creativity and allow parents and children to spend fun time together. There is also a need for a system that can easily convert what a child says into a story and create a digital picture book with beautiful illustrations. Furthermore, there is a need for a way to easily view and distribute the created picture book.

[0877] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0878] In this invention, the server includes means for recording a child's voice data, voice recognition means for converting the recorded voice data into text data, natural language processing means for analyzing the text data and generating a narrative, image generation means for generating picture data based on the generated narrative, picture book generation means for generating an electronic picture book by combining the narrative and picture data, distribution means for providing the generated electronic picture book to a user, and a smartphone application for making the generated electronic picture book viewable. This allows what a child says to be easily converted into a story and picture book that can be enjoyed by parents and children.

[0879] "Children's voice data" refers to data that has been recorded and stored in digital format, containing what a child says.

[0880] A "recording means" is any device or software that records what a child says as a digital audio file.

[0881] "Speech recognition means" refers to technology or tools for analyzing recorded voice data and converting it into text information.

[0882] "Text data" is character information extracted using a voice recognition means.

[0883] "Natural language processing means" refers to technologies and engines for analyzing text data and generating narrative text and sentence structures.

[0884] A "narrative text" is a piece of writing that describes a story or narrative generated by natural language processing means.

[0885] "Image generation means" refers to techniques and tools for generating visual content based on narrative text.

[0886] "Pictorial data" is visual content generated using image generation means.

[0887] "Picture book generation means" refers to techniques and tools for creating an electronic picture book by combining the generated story text and picture data.

[0888] "Distribution means" refers to the technology and platform for providing the generated digital picture book to users.

[0889] A "smartphone application" is an app that runs on a smartphone and allows users to view and operate the generated digital picture book.

[0890] A "generative AI model" is a machine learning model that generates images and other data from text or prompts.

[0891] A "prompt" is an instruction or text used as input to a generative AI model.

[0892] System Overview

[0893] This invention relates to a system that creates picture books that give shape to children's creativity and can be enjoyed by parents and children. The entire system consists of the following components:

[0894] 1. Audio recording means

[0895] The device (smartphone) has a voice recording function to record what the child says. This voice recording function has the function to save the voice data in digital format and upload it to a server. As a concrete example, a voice recording API is used.

[0896] 2. Voice Recognition Method

[0897] The server receives the recorded voice data and converts it into text data using a speech recognition tool, such as an online API like Google Cloud Speech-to-Text API.

[0898] 3. Natural Language Processing Methods

[0899] The server uses natural language processing to analyze the converted text data. For example, a natural language processing engine such as spaCy is used as a text analysis tool. Story elements (nouns, verbs, etc.) are extracted from the analyzed text data, and a simple narrative is generated.

[0900] 4. Image Generation Method

[0901] The server generates pictorial data based on the narrative text. It uses a generative AI model, such as OpenAI's DALL-E, to generate images. When a prompt is entered, an image is generated based on it.

[0902] 5. Picture book generation method

[0903] The server combines the story text and the picture data to generate an electronic picture book. This picture book generation method uses a PDF generation library (e.g., ReportLab) to create an electronic picture book in PDF format.

[0904] 6. Distribution Method

[0905] The server has a distribution means for providing the generated digital picture book to users. Specifically, the server distributes the generated digital picture book through a smartphone application, allowing users to easily view it.

[0906] Hardware and software used

[0907] Hardware: Smartphones, servers

[0908] Software: Voice Recording API, Google Cloud Speech-to-Text API, spaCy, OpenAI DALL-E, ReportLab

[0909] Specific flow of program processing

[0910] For example, consider the example of a child saying, "One day, I became a wizard and defeated the dragon that lived in the castle."

[0911] 1. The device records this content using the voice recording API and uploads it to the server.

[0912] 2. The server converts the audio data into text data using the Google Cloud Speech-to-Text API.

[0913] 3. The server analyzes the converted text data using spaCy and generates a narrative text.

[0914] 4. The server sends the generated story text to OpenAI DALL-E, which generates picture data using the prompt text "Become a wizard and defeat a dragon."

[0915] 5. The server combines the story text and picture data using ReportLab to generate an electronic picture book in PDF format.

[0916] 6. The server distributes the generated e-book to the user via a smartphone application.

[0917] Prompt Sentence Examples

[0918] "One day, I became a wizard and defeated the dragon that lived in the castle."

[0919] Using this system, what a child says can be easily converted into stories and visual content that can be enjoyed by both parents and children.

[0920] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0921] Step 1:

[0922] Recording and uploading

[0923] The user launches the application on their device (smartphone) and uses the voice recording API to record what the child is imagining. The recorded data is saved in digital format. When the user finishes recording, they press the upload button, and the recorded audio data is sent to the server.

[0924] Input: Child's voice

[0925] Output: Recorded audio data (digital format)

[0926] Step 2:

[0927] Audio data conversion

[0928] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API, and the converted text data is stored in a database.

[0929] Input: Recorded audio data

[0930] Output: Text data

[0931] Step 3:

[0932] Text analytics

[0933] The server analyzes the text data using a natural language processing engine (e.g., spaCy) and extracts information such as nouns, verbs, subjects, and predicates. Based on the extracted information, it reconstructs the story and generates a simple narrative sentence. This narrative sentence is then stored in a database.

[0934] Input: Text data

[0935] Output: Analyzed information, narrative text

[0936] Step 4:

[0937] Story Generation

[0938] The server uses a narrative generation algorithm to create a storyline based on the information generated by the natural language processing engine, and generates a narrative in simple language. The generated narrative is then stored in a database.

[0939] Input: Parsed information

[0940] Output: Narrative text

[0941] Step 5:

[0942] Image generation

[0943] The server sends the generated story text to OpenAI's generative AI model (e.g., DALL-E), which generates corresponding picture data. It generates images by providing prompts for story elements (e.g., "wizard," "dragon," "castle," etc.) as input. This picture data is stored in a database.

[0944] Input: Narrative text, prompt text

[0945] Output: Painting data

[0946] Step 6:

[0947] Picture book generation

[0948] The server combines the story text and the picture data and generates an e-book file using a PDF generation library (e.g., ReportLab). The generated e-book is saved in PDF format.

[0949] Input: Narrative text, picture data

[0950] Output: Digital picture book file (PDF format)

[0951] Step 7:

[0952] Picture book distribution

[0953] The server distributes the generated digital picture book file to the user's terminal via a distribution means, and the user can view the digital picture book through a smartphone application.

[0954] Input: Digital picture book file

[0955] Output: Digital picture book delivery to user devices

[0956] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0957] This invention relates to a system that creates picture books that give shape to children's creativity and can be enjoyed by parents and children. This system not only automatically generates stories based on children's fantasies and creates picture books based on them, but also uses an emotion engine to adjust the story text and picture data according to the user's emotional state.

[0958] Overall system overview

[0959] The system includes the following major components:

[0960] 1. Audio recording method (used by the user)

[0961] Users use the application to record what their child says, and the audio recordings are stored digitally and uploaded to a server.

[0962] 2. Speech recognition method (runs on the server)

[0963] The server receives the voice data and converts it into text using a voice recognition engine, which uses an online API.

[0964] 3. Natural language processing means (running on the server)

[0965] The server analyzes the converted text data and generates a narrative using a natural language processing engine.

[0966] 4. Image generation method (runs on the server)

[0967] The server generates pictorial data based on the generated narrative text using an image generation engine.

[0968] 5. Picture book generation method (runs on the server)

[0969] The server combines the story text and image data to generate an electronic picture book, which is saved in PDF or ePub format.

[0970] 6. Delivery method (runs on the server)

[0971] The server distributes the generated digital picture books to users, who can view them within the app and use the printing service if necessary.

[0972] 7. Emotion recognition method (runs on the server)

[0973] The server is equipped with an emotion engine that recognizes emotional information in the recorded voice data, thereby analyzing the user's emotional state and storing it in a database.

[0974] 8. Emotional Adaptation Method (Runs on Server)

[0975] Based on the analysis results of the emotion engine, the server adjusts the content of the narrative text to be generated and the color tone and style of the painting data.

[0976] Program processing flow

[0977] The specific flow of program processing in this system will be explained below.

[0978] 1. Recording and uploading

[0979] The user launches the app and records what the child is imagining. After recording, the user presses the upload button to send the audio data to the server.

[0980] 2. Audio Data Processing

[0981] The server stores the received voice data and sends it to a voice recognition engine (e.g., an online API).

[0982] 3. Voice Recognition

[0983] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[0984] 4. Text Analysis

[0985] The server receives the returned text data and temporarily stores it in a database.

[0986] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure.

[0987] 5. Narrative Generation

[0988] The server uses a story generation algorithm to create a storyline based on the analysis results, and generates a narrative in simple language. The generated narrative is then stored in a database.

[0989] 6. Emotion recognition

[0990] The server sends the recorded voice data to the emotion engine to recognize the user's emotional state, and stores the information on the emotional state in a database.

[0991] 7. Story and Art Coordination

[0992] The server adjusts the content of the narrative sentence to be generated based on the emotional state recognized by the emotion engine.

[0993] The server further sends instructions to the image generation engine to adjust the color tone and style of the generated painting data based on the emotional state.

[0994] 8. Image Generation

[0995] The server generates pictorial data based on the narrative text using an image generation engine, and the pictorial data is adjusted based on the context of the story and the user's emotional state.

[0996] 9. Picture Book Generation

[0997] The server combines the story text and the picture data to generate an electronic picture book file (e.g., PDF or ePub format).

[0998] 10. Distribution

[0999] The server distributes the generated electronic picture book file to the user's terminal.

[1000] Users can view the digital picture books distributed within the app, and if necessary, can use the printing service to save them in paper form.

[1001] Specific examples

[1002] For example, suppose a child says, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went to a mysterious forest." Also, suppose the recording contains the emotions of a child talking happily.

[1003] 1. Recording and uploading

[1004] The user records this content using the app and uploads it to the server.

[1005] 2. Audio Data Processing

[1006] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[1007] 3. Text Analysis

[1008] The server analyzes this using a natural language processing engine and extracts nouns (me, dragon, rainbow bridge, mysterious forest), verbs (found, crossed, went), etc.

[1009] 4. Narrative Generation

[1010] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[1011] 5. Emotion recognition

[1012] The server sends the recorded voice data to the emotion engine, which recognizes that the user is speaking with a happy tone. Based on this emotional information, positive and fun elements are added to the story.

[1013] 6. Story and picture coordination

[1014] Based on the information from the emotion engine, the server adds more excitement and adventure to the story and instructs the image generation engine to draw bright and colorful pictures.

[1015] 7. Image Generation

[1016] The server generates art data for a "flying dragon," a "rainbow bridge," and a "mysterious forest" based on the story text. This art data is adjusted to create a bright and cheerful atmosphere.

[1017] 8. Picture Book Generation

[1018] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[1019] 9. Distribution

[1020] The server delivers the generated digital picture book to the user's app, where the user can view the book within the app. Furthermore, if necessary, the user can use a printing service to save the book on paper.

[1021] Through this process, what the child says is transformed into a story, and a picture book that reflects the child's emotions is generated. This system is a groundbreaking way to foster children's creativity and further enrich memories between parents and children.

[1022] The processing flow will be explained below.

[1023] Step 1:

[1024] The user launches the app and uses the recording feature to record the child's imaginary speech. When the recording is finished, the user presses the stop recording button.

[1025] Step 2:

[1026] The user presses the upload button to upload the recorded audio data to the server. The device sends the recorded data to the server.

[1027] Step 3:

[1028] The server stores the received voice data and sends it to the voice recognition engine.

[1029] Step 4:

[1030] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[1031] Step 5:

[1032] The server receives the returned text data and temporarily stores it in a database.

[1033] Step 6:

[1034] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure.

[1035] Step 7:

[1036] The natural language processing engine extracts nouns, verbs, subjects, predicates, etc. from text data and performs analysis to understand the context of the text.

[1037] Step 8:

[1038] The server receives the analysis results and creates a storyline using a narrative generation algorithm.

[1039] Step 9:

[1040] The server generates a story in simple language and stores the generated story in a database.

[1041] Step 10:

[1042] The server sends the recorded voice data to the emotion engine to recognize the user's emotional state. The emotion engine analyzes the emotional information (e.g., joy, sadness, surprise) from the voice and sends the emotional state back to the server.

[1043] Step 11:

[1044] The server adjusts the content of the story it generates based on the emotional state recognized by the emotion engine. For example, if the user is having fun, it adds positive elements to the story.

[1045] Step 12:

[1046] The server further transmits the narrative text to the image generation engine to adjust the color tone and style of the generated painting data based on the emotional state.

[1047] Step 13:

[1048] The image generation engine generates pictorial data based on the narrative text and returns the generated pictorial data to the server, where it is adjusted based on the emotional information.

[1049] Step 14:

[1050] The server combines the story text and the picture data to generate an electronic picture book file (e.g., PDF or ePub format).

[1051] Step 15:

[1052] The server distributes the generated electronic picture book file to the user's terminal.

[1053] Step 16:

[1054] Users can view the digital picture books delivered within the app, and if necessary, can save them in paper form using the printing service.

[1055] Example 2

[1056] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1057] Conventional picture book generation systems have struggled to automatically generate content that reflects a child's creativity and emotions. Furthermore, there was no technology that could accurately convert a child's speech into text and generate appropriate picture data based on that context, or that could adjust the story and picture data to reflect a child's emotional state. This made it difficult to provide personalized digital picture books that parents and children could enjoy.

[1058] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1059] In this invention, the server includes recording means for recording a child's voice data, voice recognition means for converting the recorded voice data into text data, natural language processing means for analyzing the text data and generating a narrative, image generation means for generating picture data based on the generated narrative, picture book generation means for generating an electronic picture book by combining the narrative and picture data, distribution means for providing the generated electronic picture book to a user, emotion engine means for recognizing an emotional state from the recorded voice data, and emotion adaptation means for adjusting the narrative and picture data based on the recognized emotional state. This allows for the automatic generation of a personalized electronic picture book that reflects the child's creativity and emotions, allowing parents and children to enjoy it together.

[1060] "Recording means" means a device or method for recording a child's voice data in digital form.

[1061] "Speech recognition means" refers to the technology or algorithms used to convert recorded voice data into text data.

[1062] "Natural language processing means" is a technology for analyzing text data generated by speech recognition means and constructing meaningful narrative text.

[1063] "Image generation means" refers to algorithms and techniques for generating pictorial data based on the generated narrative text.

[1064] "Picture book generation means" refers to a technique or method for synthesizing narrative text and picture data to generate a formatted electronic picture book.

[1065] "Distribution means" refers to the technology or system for delivering the generated electronic picture book to users.

[1066] The "emotion engine means" is a technology for recognizing an emotional state from recorded voice data and adjusting narrative text and picture data based on that.

[1067] The "emotion adaptation means" is a technique for dynamically changing the content and style of narrative text and picture data based on the emotional state recognized by the emotion engine means.

[1068] The present invention relates to a system for generating picture books that give shape to children's creativity and allow parents and children to enjoy them together. This system not only automatically generates stories based on children's fantasies and creates picture books based on those stories, but also uses an emotion engine to adjust the story text and picture data according to the user's emotional state. A specific embodiment of this system is described below.

[1069] Hardware and software used

[1070] 1. Audio recording method (used by the user)

[1071] Users use an application on their smartphone or tablet to record what their child says, and the audio data is stored digitally and uploaded to a server.

[1072] 2. Speech recognition method (runs on the server)

[1073] The server receives the voice data and converts it into text data using an online API (e.g., Google Cloud Speech-to-Text) as a voice recognition engine.

[1074] 3. Natural language processing means (running on the server)

[1075] The server analyzes the converted text data and generates a narrative using a natural language processing engine (e.g., OpenAI's GPT-3).

[1076] 4. Image generation method (runs on the server)

[1077] The server generates pictorial data based on the generated narrative text using an image generation engine (e.g., DALL-E).

[1078] 5. Picture book generation method (runs on the server)

[1079] The server combines the story text and image data to generate an electronic picture book, which is saved in PDF or ePub format.

[1080] 6. Delivery method (runs on the server)

[1081] The server distributes the generated digital picture books to users, who can view them within the app and use the printing service if necessary.

[1082] 7. Emotion recognition method (runs on the server)

[1083] The server is equipped with an emotion engine that recognizes emotional information in the recorded voice data, thereby analyzing the user's emotional state and storing it in a database.

[1084] 8. Emotional Adaptation Method (Runs on Server)

[1085] Based on the analysis results of the emotion engine, the server adjusts the content of the narrative text to be generated and the color tone and style of the painting data.

[1086] Example of a system

[1087] For example, suppose a child says, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went to a mysterious forest." This recording contains the emotions of a child talking happily.

[1088] 1. Recording and uploading

[1089] The user records this content using the app and uploads it to the server.

[1090] 2. Audio Data Processing

[1091] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[1092] 3. Text Analysis

[1093] The server analyzes this using a natural language processing engine and extracts nouns ("I," "dragon," "Rainbow Bridge," "Mysterious Forest"), verbs ("I found it," "crossed," "I went"), etc.

[1094] 4. Narrative Generation

[1095] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[1096] 5. Emotion recognition

[1097] The server sends the recorded voice data to the emotion engine, which recognizes that the user is speaking with a happy tone. Based on this emotional information, positive and fun elements are added to the story.

[1098] 6. Story and picture coordination

[1099] Based on the information from the emotion engine, the server adds more excitement and adventure to the story and instructs the image generation engine to draw bright and colorful pictures.

[1100] 7. Image Generation

[1101] The server generates art data for a "flying dragon," a "rainbow bridge," and a "mysterious forest" based on the story text. This art data is adjusted to create a bright and cheerful atmosphere.

[1102] 8. Picture Book Generation

[1103] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[1104] 9. Distribution

[1105] The server delivers the generated digital picture book to the user's app, where the user can view the book within the app. Furthermore, if necessary, the user can use a printing service to save the book on paper.

[1106] The following are examples of prompt sentences that can be input to a generative AI model:

[1107] Example prompt sentence:

[1108] Generate a new adventure story based on a story my child told me. My child said with a happy expression, "One day, I found a flying dragon. The dragon crossed the Rainbow Bridge and went to a mysterious forest." Based on this, generate a story and illustrations that contain many positive elements.

[1109] In this way, a system is realized that can generate a story based on what a child says and automatically generate an electronic picture book that reflects the user's emotional state, providing a personalized picture book that can be enjoyed by both parents and children.

[1110] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1111] Step 1:

[1112] Recording and uploading

[1113] The user launches the app, taps the record button to record the child's imaginary speech, and then taps the upload button to send the audio data to the server.

[1114] Input: The audio spoken by the user.

[1115] Output: Audio data uploaded to the server.

[1116] How it works: When the user presses the "Record" button on the smartphone app screen, the built-in microphone activates and records what the child is saying. When the recording is finished, the user taps the "Upload" button, and the app sends the audio data to the server.

[1117] Step 2:

[1118] Audio data processing

[1119] The server temporarily stores the received voice data and sends it to the voice recognition engine.

[1120] Input: The audio data sent to the server.

[1121] Output: The audio data sent to the speech recognition engine.

[1122] Specific operation: When the server receives the voice data, it stores it in temporary storage and then sends it to a voice recognition engine (e.g., Google Cloud Speech-to-Text) via API.

[1123] Step 3:

[1124] Voice Recognition

[1125] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[1126] Input: The audio data sent to the speech recognition engine.

[1127] Output: The text data sent back to the server.

[1128] Specific operation: The speech recognition engine analyzes the voice signal and converts the voice into text. The text data is sent back to the server as an API response.

[1129] Step 4:

[1130] Text analytics

[1131] The server receives the returned text data and temporarily stores it in a database, after which the text data is sent to a natural language processing engine for grammatical analysis.

[1132] Input: The returned text data.

[1133] Output: Data with grammatical structure parsed.

[1134] How it works: The server stores the text data in a database and sends it to a natural language processing engine (e.g., OpenAI's GPT-3). The analysis results are returned and stored on the server.

[1135] Step 5:

[1136] Story Generation

[1137] The server uses a narrative generation algorithm to create a story based on the analysis results, and generates a narrative text, which is then stored in a database.

[1138] Input: Analysis results.

[1139] Output: The generated narrative text.

[1140] Specific operation: The server generates the main plot of the story from the analysis results and converts it into simple sentences. The generated story is then saved in the database again.

[1141] Step 6:

[1142] emotion recognition

[1143] The server sends the recorded voice data to the emotion engine to recognize the user's emotional state, and stores the information on the emotional state in a database.

[1144] Input: Recorded audio data.

[1145] Output: Emotional state information.

[1146] How it works: The server sends the voice data to the emotion engine, which interprets emotions from the tone and expressions of the voice. Once the emotional state is recognized, it is stored in a database.

[1147] Step 7:

[1148] Story and picture coordination

[1149] The server adjusts the content of the narrative text to be generated based on the emotional state recognized by the emotion engine, and also sends instructions to the image generation engine to adjust the color tone and style of the generated painting data.

[1150] Input: Emotional state information.

[1151] Output: Adjusted narrative text and pictorial data instructions.

[1152] Specific actions: If the emotional state is "happy," add positive elements to the story. Send specific instructions to the image generation engine to adjust the color tone and style of the painting data to be bright and cheerful.

[1153] Step 8:

[1154] Picture generation

[1155] The server generates pictorial data based on the narrative text using an image generation engine, and the pictorial data is adjusted based on the context of the story and the user's emotional state.

[1156] Input: Narrative text and tailored instructions.

[1157] Output: The generated painting data.

[1158] How it works: The server sends the story text to the image generation engine, which creates an illustration based on the specified style and emotion. The generated illustration is then sent back to the server.

[1159] Step 9:

[1160] Picture book generation

[1161] The server combines the story text and the picture data to generate an electronic picture book file, which is saved in PDF or ePub format.

[1162] Input: Narrative text and picture data.

[1163] Output: Digital picture book file (PDF or ePub format).

[1164] Specific operation: The server inputs the story text and picture data into the template engine and determines the layout of the digital picture book. The generated picture book is output in PDF or ePub format and saved in storage.

[1165] Step 10:

[1166] delivery

[1167] The server delivers the generated digital picture book file to the user's device. The user can view the delivered digital picture book within the app and can also use the printing service if necessary.

[1168] Input: E-book file.

[1169] Output: An electronic picture book that can be viewed by the user.

[1170] Specific operation: The server sends the user a link to the generated e-book file. The user clicks the link through the app to view the e-book. They can also access the printing service if they wish.

[1171] (Application example 2)

[1172] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1173] Systems that generate picture books that give shape to children's creativity and can be enjoyed by parents and children alike are required to provide a more immersive, interactive experience that reflects the content and emotions of children.However, conventional systems often generate picture books with static content, and are unable to provide an interactive experience where parents and children can actually interact with the characters and story world in a virtual space.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for recording a child's voice data; means for converting the recorded voice data into text data; means for analyzing the text data and generating a narrative; means for generating picture data based on the generated narrative; means for adjusting the generated narrative and picture data and displaying them in a virtual space; means for interactively experiencing the generated electronic picture book in the virtual space; means for generating an electronic picture book by combining the narrative and picture data; and means for providing the generated electronic picture book to a user. This enables a dynamic and interactive story experience that reflects a child's creativity.

[1175] "Children's voice data" is digital voice data that records what a child is saying.

[1176] "Speech recognition means" refers to a technical means for converting recorded voice data into text data.

[1177] "Text data" is data that has been converted from audio data into text format.

[1178] "Natural language processing means" refers to technical means for analyzing text data and generating narrative text.

[1179] A "narrative text" is a story-form text generated from analyzed text data.

[1180] "Image generation means" refers to technical means for generating picture data based on the generated narrative text.

[1181] "Picture data" is digital image data generated based on a narrative text.

[1182] "Adjustment means" refers to technical means for appropriately adjusting the generated narrative text and pictorial data for display in a virtual space.

[1183] The "virtual space" is a virtual digital space in which the generated electronic picture book and other digital content are displayed.

[1184] The "experiential means" refers to a technical means that allows a user to interactively experience the generated digital picture book in a virtual space.

[1185] The "picture book generation means" is a technical means for generating an electronic picture book by combining a story and picture data.

[1186] "Distribution means" refers to the technical means for providing the generated electronic picture book to the user.

[1187] "Interactive" refers to the ability of users to directly interact with and respond to generated content.

[1188] This invention relates to a virtual picture book creation system that allows parents and children to express their creativity and have fun together. This system automatically creates a realistic, interactive picture book that reflects the child's words and emotions, allowing parents and children to experience the world of the picture book in a virtual space.

[1189] Overall system overview

[1190] The system includes the following major components:

[1191] 1. Audio recording means (user terminal)

[1192] Users use the application to record what their child says, and the audio recordings are stored digitally and uploaded to a server.

[1193] 2. Speech recognition means (server)

[1194] The server receives the voice data and converts it into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1195] 3. Natural language processing means (server)

[1196] The server analyzes the converted text data and generates a narrative using a natural language processing engine (e.g., GPT-4).

[1197] 4. Image generation means (server)

[1198] The server generates pictorial data based on the generated narrative text, using an image generation engine (e.g., DALL-E).

[1199] 5. Coordination means (server)

[1200] The server adjusts the content of the narrative text and the color tone and style of the painting data based on the emotional state recognized by an emotion engine (e.g., IBM Watson Tone Analyzer).

[1201] 6. Experience Method (User Device)

[1202] Users can interactively experience the generated picture book in a virtual space using smart glasses or a head-mounted display (e.g., Magic Leap, Oculus Rift).

[1203] 7. Picture book generation means (server)

[1204] The server combines the story text and the picture data to generate an electronic picture book, which is saved in PDF or ePub format.

[1205] 8. Delivery Method (Server)

[1206] The server distributes the generated digital picture book to the user, who can then view the picture book within the app.

[1207] Specific examples

[1208] The following is a specific example of the system.

[1209] For example, consider a situation where a system generates a story and pictures based on a child saying, "One day I want to play in a castle with a princess." The recording contains the emotion of the child talking happily.

[1210] 1. Audio recording and uploading

[1211] The user (parent or child) starts the app and records what the child says. After recording is complete, they press the upload button to send the audio data to the server.

[1212] 2. Voice Recognition

[1213] The server sends the received voice data to a speech recognition engine and converts it into text data, for example, "One day I want to play at the castle with the princess."

[1214] 3. Natural Language Processing

[1215] The server sends the converted text data to a natural language processing engine to generate a narrative sentence, such as "One day, the child had fun playing in the castle with the princess."

[1216] 4. Emotion recognition and regulation

[1217] The server sends the recorded voice data to the emotion engine, which recognizes positive emotions. Based on this emotional information, the color tone and style of the story text and painting data are adjusted to be brighter.

[1218] 5. Image Generation

[1219] Based on the story text, the server uses an image generation engine to generate pictures of "princesses" and "castles," resulting in colorful and fun images that fit the context of the story.

[1220] 6. Virtual Space Display and Experience

[1221] Users wear smart glasses or a head-mounted display and experience the picture book generated in a virtual space. In the virtual space, users can interact with the characters in the book by touching them. For example, the characters may walk around the castle or talk to them.

[1222] 7. Picture book generation and distribution

[1223] The server combines the story text and picture data to create an electronic picture book and delivers it to the user's device. Users can view the electronic picture book created within the app, and if necessary, can use a printing service to save it on paper.

[1224] Prompt Sentence Examples

[1225] Prompt: "Based on a child's joyful statement, 'One day I want to play in a castle with a princess,' please generate a story with a positive and fun atmosphere."

[1226] In this way, a dynamic and interactive storytelling experience is created that reflects the child's creativity in real time.

[1227] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1228] Step 1:

[1229] Audio recording and uploading

[1230] The user launches the application and records what the child says.

[1231] Input: User spoken voice data

[1232] What it does: Uses the device's microphone to collect audio and store it digitally.

[1233] Output: Recorded audio data (e.g. audio_file.wav)

[1234] Step 2:

[1235] Sending audio data

[1236] After the user has finished recording, he or she presses the upload button to send the audio data to the server.

[1237] Input: Recorded audio data (audio_file.wav)

[1238] Specific operation: The application sends an HTTP request to the server and uploads the audio data.

[1239] Output: Audio data received on the server side

[1240] Step 3:

[1241] Voice Recognition

[1242] The server sends the received voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data.

[1243] Input: Audio data uploaded to the server

[1244] Specific operation: The speech recognition engine analyzes the voice data and converts it into a string of characters.

[1245] Output: Text data obtained by speech recognition (e.g., "One day I want to play at the castle with the princess.")

[1246] Step 4:

[1247] Natural Language Processing (NLP)

[1248] The server sends the converted text data to a natural language processing engine (e.g., GPT-4) to generate a narrative.

[1249] Input: Text data (e.g., "One day I want to play in a castle with a princess.")

[1250] Specific operation: The NLP engine performs grammatical analysis and generates a narrative storyline.

[1251] Output: Generated story (e.g., "One day, the child had fun playing in the castle with the princess.")

[1252] Step 5:

[1253] Emotion Recognition and Regulation

[1254] The server sends the recorded voice data to an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotional state.

[1255] Input: Recorded audio data

[1256] Specific operation: The emotion engine recognizes emotions in the voice and returns the analysis results.

[1257] Output: Sentiment analysis result (e.g., positive)

[1258] Step 6:

[1259] Story and picture coordination

[1260] Based on the analysis results of the emotion engine, the server adjusts the content of the narrative text to be generated and the color tone and style of the painting data.

[1261] Input: Narrative text, sentiment analysis results

[1262] Specific actions: Add positive elements to the story and send instructions to the image generation engine (e.g., DALL-E) for bright colors and a happy atmosphere.

[1263] Output: Coordinated narrative and image generation instructions

[1264] Step 7:

[1265] Image generation

[1266] The server generates picture data based on the story using an image generation engine.

[1267] Input: tailored narrative text and image generation instructions

[1268] How it works: The image generation engine draws story scenes and characters and outputs them as digital images.

[1269] Output: Generated painting data (e.g., colorful painting)

[1270] Step 8:

[1271] Virtual space display and experience

[1272] Users can experience the picture book generated in a virtual space using smart glasses or a head-mounted display.

[1273] Input: Generated narrative text and picture data

[1274] Specific operation: The application displays narrative text and pictorial data in a virtual space, allowing users to experience it interactively.

[1275] Output: An interactive picture book experience in a virtual space

[1276] Step 9:

[1277] Picture book generation

[1278] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[1279] Input: Generated narrative text and picture data

[1280] Specific actions: Integrate the narrative text and picture data into a format and save it as an electronic file.

[1281] Output: Generated e-book (e.g. ebook.pdf)

[1282] Step 10:

[1283] delivery

[1284] The server distributes the generated electronic picture book to the user terminal.

[1285] Input: Generated e-book

[1286] Specific operation: The server sends the electronic picture book file to the user's application.

[1287] Output: Digital picture book delivered to the user's device

[1288] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1289] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1290] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1291] [Third embodiment]

[1292] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1293] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1294] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1295] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1296] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1297] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1298] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1299] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1300] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1301] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1302] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1303] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1304] This invention relates to a system that creates picture books that can give shape to a child's creativity and be enjoyed by parents and children. This system can automatically generate a story based on a child's imagination and create a picture book based on that story.

[1305] Overall system overview

[1306] The system includes the following major components:

[1307] 1. Audio recording method (used by the user)

[1308] Users use the application to record what their child says, and the audio recordings are stored digitally and uploaded to a server.

[1309] 2. Speech recognition method (runs on the server)

[1310] The server receives the voice data and converts it into text data using a voice recognition engine, which uses an online API (for example, a general voice recognition API).

[1311] 3. Natural language processing means (running on the server)

[1312] The server analyzes the converted text data and generates a narrative using a natural language processing engine.

[1313] 4. Image generation method (runs on the server)

[1314] The server generates pictorial data based on the generated narrative text using an image generation engine (e.g., using common image generation technology).

[1315] 5. Picture book generation method (runs on the server)

[1316] The server combines the story text and image data to generate an electronic picture book, which is saved in PDF or ePub format.

[1317] 6. Delivery method (runs on the server)

[1318] The server distributes the generated digital picture books to users, who can view them within the app and use the printing service if necessary.

[1319] Program processing flow

[1320] The specific flow of program processing in this system will be explained below.

[1321] Recording and uploading

[1322] User

[1323] The user launches the app, records what the child is imagining, and then presses the upload button to send the audio data to the server.

[1324] Audio data processing

[1325] server

[1326] The server sends the received voice data to a speech recognition engine and converts it into text data, which is then temporarily stored in a database.

[1327] Text analytics

[1328] server

[1329] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure and extracts information such as nouns, verbs, subjects, and predicates. The extracted information is then used to reconstruct the story.

[1330] Story Generation

[1331] server

[1332] The server uses a story generation algorithm to create a storyline based on the analysis results, and generates a narrative in simple language. The generated narrative is then stored in a database.

[1333] Picture generation

[1334] server

[1335] The server sends the story to an image generation engine, which generates pictorial data corresponding to the story, and stores the pictorial data in a database.

[1336] Picture book generation

[1337] server

[1338] The server combines the story text and the picture data to generate and store an electronic picture book file. If necessary, it also generates a high-resolution file for printing.

[1339] Picture book distribution

[1340] server

[1341] The server delivers the generated digital picture book file to the user's app, where the user can view the picture book and use the printing service.

[1342] Specific examples

[1343] For example, a child might say, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[1344] 1. Recording and uploading

[1345] The user records this content using the app and uploads it to the server.

[1346] 2. Audio Data Processing

[1347] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[1348] 3. Text Analysis

[1349] The server analyzes this using a natural language processing engine and extracts nouns (me, dragon, rainbow bridge, mysterious forest), verbs (found, crossed, went), etc.

[1350] 4. Narrative Generation

[1351] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[1352] 5. Image Generation

[1353] The server generates picture data for "a flying dragon," "a rainbow bridge," and "a mysterious forest" based on the story text.

[1354] 6. Picture Book Generation

[1355] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[1356] 7. Picture book distribution

[1357] The server delivers the generated e-book to the user's app, allowing the user to view the book within the app. Furthermore, printing services are also available if necessary.

[1358] This process transforms what the child says into a story in real time and provides it as a visually appealing picture book. This system is a groundbreaking way to foster children's creativity and create memories for parents and children.

[1359] The processing flow will be explained below.

[1360] Step 1:

[1361] The user launches the app and uses the recording feature to record the child's imaginary speech. When the recording is finished, the user presses the stop recording button.

[1362] Step 2:

[1363] The user presses the upload button to upload the recorded audio data to the server. The device sends the recorded data to the server.

[1364] Step 3:

[1365] The server stores the received voice data and sends it to a voice recognition engine (e.g., a common online API).

[1366] Step 4:

[1367] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[1368] Step 5:

[1369] The server receives the returned text data and temporarily stores it in a database.

[1370] Step 6:

[1371] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure.

[1372] Step 7:

[1373] The natural language processing engine extracts nouns, verbs, subjects, predicates, etc. from text data and performs analysis to understand the context of the text.

[1374] Step 8:

[1375] The server receives the analysis results and creates a storyline using a narrative generation algorithm.

[1376] Step 9:

[1377] The server generates a story in simple language and stores the generated story in a database.

[1378] Step 10:

[1379] The server sends the stored narrative to an image generation engine (eg, general image generation technology).

[1380] Step 11:

[1381] The image generation engine generates picture data based on the story text and returns the generated picture data to the server.

[1382] Step 12:

[1383] The server receives the returned painting data and stores it in a database.

[1384] Step 13:

[1385] The server combines the stored story text and picture data to generate an electronic picture book file (e.g., PDF or ePub format).

[1386] Step 14:

[1387] The server distributes the generated electronic picture book file to the user's terminal.

[1388] Step 15:

[1389] Users can view the digital picture books delivered within the app, and if necessary, can save them in paper form using the printing service.

[1390] Example 1

[1391] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1392] Existing picture book creation systems make it difficult to give form to children's creativity, and lack an easy way for parents and children to enjoy creating together. Furthermore, there was no system that efficiently converts recorded audio into text and then automatically generates illustrations that correspond to the story text based on that. This limited parent-child communication and restricted opportunities to develop children's creativity.

[1393] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1394] In this invention, the server includes means for recording a child's voice data, means for converting the recorded voice data into text data, means for analyzing the text data and generating a story, means for generating image data based on the generated story, means for generating an electronic book by combining the story and image data, and means for providing the generated electronic book to users. This makes it possible for parents and children to easily give shape to their children's creativity and create an electronic picture book that they can enjoy together.

[1395] A "means for recording a child's voice data" is a device or application that digitally records what a child says and stores that voice data.

[1396] "A speech recognition means for converting recorded speech data into text data" is a technology or engine that analyzes recorded speech data and converts it into corresponding text data.

[1397] "Natural language processing means for analyzing text data and generating narrative text" refers to a technology or algorithm that understands the language structure based on text data and reconstructs it as narrative text.

[1398] The "image generation means for generating image data based on the generated narrative text" refers to technology or software for automatically creating visual pictures or images that correspond to the content of the narrative text.

[1399] "Electronic book generation means for combining narrative text and image data to generate an electronic book" refers to a technology or system that combines narrative text and corresponding images into a single electronic book format (such as PDF or ePub).

[1400] The "distribution means for providing the generated electronic book to the user" refers to a technology or service for transmitting the completed electronic book to the user's terminal so that it can be viewed or printed.

[1401] This invention relates to a system that creates e-books that can give shape to children's creativity and be enjoyed by parents and children. This system can automatically generate stories based on children's fantasies and create picture books based on those stories.

[1402] Overall system overview

[1403] The system includes the following major components:

[1404] 1. Audio recording method (used by the device)

[1405] The device uses an application to record what the child says and saves the audio data, which is then uploaded digitally to a server.

[1406] 2. Speech recognition method (runs on the server)

[1407] The server receives the voice data and converts it into text data using a voice recognition engine (e.g., a general voice recognition API).

[1408] 3. Natural language processing means (running on the server)

[1409] The server analyzes the converted text data and generates a narrative using a natural language processing engine (e.g., a generative AI model).

[1410] 4. Image generation method (runs on the server)

[1411] The server generates image data based on the generated narrative text using an image generation engine (e.g., a general image generation technology).

[1412] 5. E-book generation means (running on the server)

[1413] The server combines the narrative text and image data to generate an e-book, which is saved in PDF or ePub format.

[1414] 6. Delivery method (runs on the server)

[1415] The server distributes the generated e-books to users, who can view them within the app and use printing services if necessary.

[1416] Program processing flow

[1417] 1. Recording and uploading

[1418] The user launches the app, records what the child says, and when they're done, they press the upload button to send the audio data to the server.

[1419] 2. Audio Data Processing

[1420] The server sends the received voice data to a voice recognition engine and converts it into text data, which is then temporarily stored in a database.

[1421] 3. Text Analysis

[1422] The server sends the stored text data to a natural language processing engine (generative AI model), which analyzes the grammatical structure and extracts information such as nouns, verbs, subjects, and predicates. The extracted information is then used to reconstruct the story.

[1423] 4. Narrative Generation

[1424] The server uses a story generation algorithm based on the analysis results to create a storyline and generate a narrative in simple language. The generated narrative is then stored in a database.

[1425] 5. Image Generation

[1426] The server sends the story text to an image generation engine, which generates image data corresponding to the story text. The generated image data is stored in a database.

[1427] 6. Picture Book Generation

[1428] The server combines the narrative text and image data to generate an e-book, which is then saved in PDF or ePub format, and a high-resolution print file is generated if necessary.

[1429] 7. Picture book distribution

[1430] The server delivers the generated e-book to the user's device, where the user can view the e-book within the app. In addition, printing services are available if necessary.

[1431] Specific examples

[1432] For example, suppose a child says, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went to a mysterious forest." In this case, the process will proceed as follows:

[1433] 1. Recording and uploading

[1434] The user records this content using the app and uploads it to the server.

[1435] 2. Audio Data Processing

[1436] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[1437] 3. Text Analysis

[1438] The server analyzes this using a natural language processing engine and extracts nouns (me, dragon, rainbow bridge, mysterious forest), verbs (found, crossed, went), etc.

[1439] 4. Narrative Generation

[1440] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[1441] 5. Image Generation

[1442] The server generates image data for "a flying dragon," "a rainbow bridge," and "a mysterious forest" based on the story text.

[1443] 6. Picture Book Generation

[1444] The server combines the narrative text and image data to generate an e-book, which is then saved in PDF or ePub format.

[1445] 7. Picture book distribution

[1446] The server delivers the generated e-book to the user's app, allowing the user to view the picture book within the app. In addition, printing services are available if necessary.

[1447] Examples of prompt statements

[1448] The generative AI model is given a prompt such as, "Create a story about a child finding a flying dragon that crosses the Rainbow Bridge and goes into the forest."

[1449] Throughout this process, what the child says is translated into a story in real time and presented as a visually engaging e-book. This system is a great way to foster children's creativity and create memorable memories for parents and children.

[1450] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1451] Step 1: Record and upload

[1452] User:

[1453] Input: What the child says.

[1454] What it does: The user launches the application on their smartphone or tablet and taps the record button to record what their child says.

[1455] Output: Recorded audio data.

[1456] Once you have finished recording, tap the upload button to send the recording data to the server.

[1457] Step 2: Receiving and converting audio data

[1458] server:

[1459] Input: Recorded audio data.

[1460] Specific operation: The server receives the voice data sent by the user.

[1461] The received voice data is sent to a voice recognition engine (e.g., a general voice recognition API).

[1462] Output: Converted to text by a speech recognition engine.

[1463] A voice recognition engine converts the voice data into text data.

[1464] Step 3: Save the text data

[1465] server:

[1466] Input: Text data.

[1467] Specific operation: The server temporarily stores the converted text data in a database.

[1468] Output: Text data stored in a database.

[1469] The converted text data is stored in a database so that it can be used for further processing.

[1470] Step 4: Text Analysis

[1471] server:

[1472] Input: Saved text data.

[1473] Specific operation: The server sends the stored text data to a natural language processing engine (e.g., a generative AI model).

[1474] A natural language processing engine analyzes the text data and extracts information such as nouns, verbs, subjects, and predicates.

[1475] Output: Parsed information (nouns, verbs, etc.).

[1476] Reconstruct the story based on the extracted information.

[1477] Step 5: Narrative generation

[1478] server:

[1479] Input: Parsed information.

[1480] Specific operation: The server creates a storyline using a narrative generation algorithm based on the analysis results of the natural language processing engine.

[1481] Generate narrative sentences in simple language.

[1482] Output: The generated narrative text.

[1483] The generated narrative text is stored in a database.

[1484] Step 6: Image generation

[1485] server:

[1486] Input: The generated narrative text.

[1487] Specific operation: The server sends the narrative text to an image generation engine (e.g., general image generation technology).

[1488] An image generation engine generates image data based on the narrative text.

[1489] Output: The generated image data.

[1490] The generated image data is stored in a database.

[1491] Step 7: Generate eBook

[1492] server:

[1493] Input: Generated narrative text and image data.

[1494] Specific operation: The server combines the narrative text and image data to generate an e-book.

[1495] Save your e-books in PDF or ePub format.

[1496] Output: The generated e-book.

[1497] High-resolution print files are also generated if needed.

[1498] Step 8: Distributing your picture book

[1499] server:

[1500] Input: The generated e-book.

[1501] Specific operation: The server delivers the generated e-book to the user's device.

[1502] Users will be able to read e-books within the app.

[1503] Output: An e-book that can be read by the user.

[1504] Printing services are also available if required.

[1505] By following these steps in sequence, what the child says is converted into a story in real time and presented as an e-book that can be enjoyed visually.

[1506] (Application example 1)

[1507] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1508] Today's parents and children are busy, with limited time to spend together and opportunities to foster creativity. Therefore, there is a need for a way to give shape to children's creativity and allow parents and children to spend fun time together. There is also a need for a system that can easily convert what a child says into a story and create a digital picture book with beautiful illustrations. Furthermore, there is a need for a way to easily view and distribute the created picture book.

[1509] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1510] In this invention, the server includes means for recording a child's voice data, voice recognition means for converting the recorded voice data into text data, natural language processing means for analyzing the text data and generating a narrative, image generation means for generating picture data based on the generated narrative, picture book generation means for generating an electronic picture book by combining the narrative and picture data, distribution means for providing the generated electronic picture book to a user, and a smartphone application for making the generated electronic picture book viewable. This allows what a child says to be easily converted into a story and picture book that can be enjoyed by parents and children.

[1511] "Children's voice data" refers to data that has been recorded and stored in digital format, containing what a child says.

[1512] A "recording means" is any device or software that records what a child says as a digital audio file.

[1513] "Speech recognition means" refers to technology or tools for analyzing recorded voice data and converting it into text information.

[1514] "Text data" is character information extracted using a voice recognition means.

[1515] "Natural language processing means" refers to technologies and engines for analyzing text data and generating narrative text and sentence structures.

[1516] A "narrative text" is a piece of writing that describes a story or narrative generated by natural language processing means.

[1517] "Image generation means" refers to techniques and tools for generating visual content based on narrative text.

[1518] "Pictorial data" is visual content generated using image generation means.

[1519] "Picture book generation means" refers to techniques and tools for creating an electronic picture book by combining the generated story text and picture data.

[1520] "Distribution means" refers to the technology and platform for providing the generated digital picture book to users.

[1521] A "smartphone application" is an app that runs on a smartphone and allows users to view and operate the generated digital picture book.

[1522] A "generative AI model" is a machine learning model that generates images and other data from text or prompts.

[1523] A "prompt" is an instruction or text used as input to a generative AI model.

[1524] System Overview

[1525] This invention relates to a system that creates picture books that give shape to children's creativity and can be enjoyed by parents and children. The entire system consists of the following components:

[1526] 1. Audio recording means

[1527] The device (smartphone) has a voice recording function to record what the child says. This voice recording function has the function to save the voice data in digital format and upload it to a server. As a concrete example, a voice recording API is used.

[1528] 2. Voice Recognition Method

[1529] The server receives the recorded voice data and converts it into text data using a speech recognition tool, such as an online API like Google Cloud Speech-to-Text API.

[1530] 3. Natural Language Processing Methods

[1531] The server uses natural language processing to analyze the converted text data. For example, a natural language processing engine such as spaCy is used as a text analysis tool. Story elements (nouns, verbs, etc.) are extracted from the analyzed text data, and a simple narrative is generated.

[1532] 4. Image Generation Method

[1533] The server generates pictorial data based on the narrative text. It uses a generative AI model, such as OpenAI's DALL-E, to generate images. When a prompt is entered, an image is generated based on it.

[1534] 5. Picture book generation method

[1535] The server combines the story text and the picture data to generate an electronic picture book. This picture book generation method uses a PDF generation library (e.g., ReportLab) to create an electronic picture book in PDF format.

[1536] 6. Distribution Method

[1537] The server has a distribution means for providing the generated digital picture book to users. Specifically, the server distributes the generated digital picture book through a smartphone application, allowing users to easily view it.

[1538] Hardware and software used

[1539] Hardware: Smartphones, servers

[1540] Software: Voice Recording API, Google Cloud Speech-to-Text API, spaCy, OpenAI DALL-E, ReportLab

[1541] Specific flow of program processing

[1542] For example, consider the example of a child saying, "One day, I became a wizard and defeated the dragon that lived in the castle."

[1543] 1. The device records this content using the voice recording API and uploads it to the server.

[1544] 2. The server converts the audio data into text data using the Google Cloud Speech-to-Text API.

[1545] 3. The server analyzes the converted text data using spaCy and generates a narrative text.

[1546] 4. The server sends the generated story text to OpenAI DALL-E, which generates picture data using the prompt text "Become a wizard and defeat a dragon."

[1547] 5. The server combines the story text and picture data using ReportLab to generate an electronic picture book in PDF format.

[1548] 6. The server distributes the generated e-book to the user via a smartphone application.

[1549] Prompt Sentence Examples

[1550] "One day, I became a wizard and defeated the dragon that lived in the castle."

[1551] Using this system, what a child says can be easily converted into stories and visual content that can be enjoyed by both parents and children.

[1552] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1553] Step 1:

[1554] Recording and uploading

[1555] The user launches the application on their device (smartphone) and uses the voice recording API to record what the child is imagining. The recorded data is saved in digital format. When the user finishes recording, they press the upload button, and the recorded audio data is sent to the server.

[1556] Input: Child's voice

[1557] Output: Recorded audio data (digital format)

[1558] Step 2:

[1559] Audio data conversion

[1560] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API, and the converted text data is stored in a database.

[1561] Input: Recorded audio data

[1562] Output: Text data

[1563] Step 3:

[1564] Text analytics

[1565] The server analyzes the text data using a natural language processing engine (e.g., spaCy) and extracts information such as nouns, verbs, subjects, and predicates. Based on the extracted information, it reconstructs the story and generates a simple narrative sentence. This narrative sentence is then stored in a database.

[1566] Input: Text data

[1567] Output: Analyzed information, narrative text

[1568] Step 4:

[1569] Story Generation

[1570] The server uses a narrative generation algorithm to create a storyline based on the information generated by the natural language processing engine, and generates a narrative in simple language. The generated narrative is then stored in a database.

[1571] Input: Parsed information

[1572] Output: Narrative text

[1573] Step 5:

[1574] Image generation

[1575] The server sends the generated story text to OpenAI's generative AI model (e.g., DALL-E), which generates corresponding picture data. It generates images by providing prompts for story elements (e.g., "wizard," "dragon," "castle," etc.) as input. This picture data is stored in a database.

[1576] Input: Narrative text, prompt text

[1577] Output: Painting data

[1578] Step 6:

[1579] Picture book generation

[1580] The server combines the story text and the picture data and generates an e-book file using a PDF generation library (e.g., ReportLab). The generated e-book is saved in PDF format.

[1581] Input: Narrative text, picture data

[1582] Output: Digital picture book file (PDF format)

[1583] Step 7:

[1584] Picture book distribution

[1585] The server distributes the generated digital picture book file to the user's terminal via a distribution means, and the user can view the digital picture book through a smartphone application.

[1586] Input: Digital picture book file

[1587] Output: Digital picture book delivery to user devices

[1588] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1589] This invention relates to a system that creates picture books that give shape to children's creativity and can be enjoyed by parents and children. This system not only automatically generates stories based on children's fantasies and creates picture books based on them, but also uses an emotion engine to adjust the story text and picture data according to the user's emotional state.

[1590] Overall system overview

[1591] The system includes the following major components:

[1592] 1. Audio recording method (used by the user)

[1593] Users use the application to record what their child says, and the audio recordings are stored digitally and uploaded to a server.

[1594] 2. Speech recognition method (runs on the server)

[1595] The server receives the voice data and converts it into text using a voice recognition engine, which uses an online API.

[1596] 3. Natural language processing means (running on the server)

[1597] The server analyzes the converted text data and generates a narrative using a natural language processing engine.

[1598] 4. Image generation method (runs on the server)

[1599] The server generates pictorial data based on the generated narrative text using an image generation engine.

[1600] 5. Picture book generation method (runs on the server)

[1601] The server combines the story text and image data to generate an electronic picture book, which is saved in PDF or ePub format.

[1602] 6. Delivery method (runs on the server)

[1603] The server distributes the generated digital picture books to users, who can view them within the app and use the printing service if necessary.

[1604] 7. Emotion recognition method (runs on the server)

[1605] The server is equipped with an emotion engine that recognizes emotional information in the recorded voice data, thereby analyzing the user's emotional state and storing it in a database.

[1606] 8. Emotional Adaptation Method (Runs on Server)

[1607] Based on the analysis results of the emotion engine, the server adjusts the content of the narrative text to be generated and the color tone and style of the painting data.

[1608] Program processing flow

[1609] The specific flow of program processing in this system will be explained below.

[1610] 1. Recording and uploading

[1611] The user launches the app and records what the child is imagining. After recording, the user presses the upload button to send the audio data to the server.

[1612] 2. Audio Data Processing

[1613] The server stores the received voice data and sends it to a voice recognition engine (e.g., an online API).

[1614] 3. Voice Recognition

[1615] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[1616] 4. Text Analysis

[1617] The server receives the returned text data and temporarily stores it in a database.

[1618] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure.

[1619] 5. Narrative Generation

[1620] The server uses a story generation algorithm to create a storyline based on the analysis results, and generates a narrative in simple language. The generated narrative is then stored in a database.

[1621] 6. Emotion recognition

[1622] The server sends the recorded voice data to the emotion engine to recognize the user's emotional state, and stores the information on the emotional state in a database.

[1623] 7. Story and Art Coordination

[1624] The server adjusts the content of the narrative sentence to be generated based on the emotional state recognized by the emotion engine.

[1625] The server further sends instructions to the image generation engine to adjust the color tone and style of the generated painting data based on the emotional state.

[1626] 8. Image Generation

[1627] The server generates pictorial data based on the narrative text using an image generation engine, and the pictorial data is adjusted based on the context of the story and the user's emotional state.

[1628] 9. Picture Book Generation

[1629] The server combines the story text and the picture data to generate an electronic picture book file (e.g., PDF or ePub format).

[1630] 10. Distribution

[1631] The server distributes the generated electronic picture book file to the user's terminal.

[1632] Users can view the digital picture books distributed within the app, and if necessary, can use the printing service to save them in paper form.

[1633] Specific examples

[1634] For example, suppose a child says, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went to a mysterious forest." Also, suppose the recording contains the emotions of a child talking happily.

[1635] 1. Recording and uploading

[1636] The user records this content using the app and uploads it to the server.

[1637] 2. Audio Data Processing

[1638] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[1639] 3. Text Analysis

[1640] The server analyzes this using a natural language processing engine and extracts nouns (me, dragon, rainbow bridge, mysterious forest), verbs (found, crossed, went), etc.

[1641] 4. Narrative Generation

[1642] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[1643] 5. Emotion recognition

[1644] The server sends the recorded voice data to the emotion engine, which recognizes that the user is speaking with a happy tone. Based on this emotional information, positive and fun elements are added to the story.

[1645] 6. Story and picture coordination

[1646] Based on the information from the emotion engine, the server adds more excitement and adventure to the story and instructs the image generation engine to draw bright and colorful pictures.

[1647] 7. Image Generation

[1648] The server generates art data for a "flying dragon," a "rainbow bridge," and a "mysterious forest" based on the story text. This art data is adjusted to create a bright and cheerful atmosphere.

[1649] 8. Picture Book Generation

[1650] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[1651] 9. Distribution

[1652] The server delivers the generated digital picture book to the user's app, where the user can view the book within the app. Furthermore, if necessary, the user can use a printing service to save the book on paper.

[1653] Through this process, what the child says is transformed into a story, and a picture book that reflects the child's emotions is generated. This system is a groundbreaking way to foster children's creativity and further enrich memories between parents and children.

[1654] The processing flow will be explained below.

[1655] Step 1:

[1656] The user launches the app and uses the recording feature to record the child's imaginary speech. When the recording is finished, the user presses the stop recording button.

[1657] Step 2:

[1658] The user presses the upload button to upload the recorded audio data to the server. The device sends the recorded data to the server.

[1659] Step 3:

[1660] The server stores the received voice data and sends it to the voice recognition engine.

[1661] Step 4:

[1662] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[1663] Step 5:

[1664] The server receives the returned text data and temporarily stores it in a database.

[1665] Step 6:

[1666] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure.

[1667] Step 7:

[1668] The natural language processing engine extracts nouns, verbs, subjects, predicates, etc. from text data and performs analysis to understand the context of the text.

[1669] Step 8:

[1670] The server receives the analysis results and creates a storyline using a narrative generation algorithm.

[1671] Step 9:

[1672] The server generates a story in simple language and stores the generated story in a database.

[1673] Step 10:

[1674] The server sends the recorded voice data to the emotion engine to recognize the user's emotional state. The emotion engine analyzes the emotional information (e.g., joy, sadness, surprise) from the voice and sends the emotional state back to the server.

[1675] Step 11:

[1676] The server adjusts the content of the story it generates based on the emotional state recognized by the emotion engine. For example, if the user is having fun, it adds positive elements to the story.

[1677] Step 12:

[1678] The server further transmits the narrative text to the image generation engine to adjust the color tone and style of the generated painting data based on the emotional state.

[1679] Step 13:

[1680] The image generation engine generates pictorial data based on the narrative text and returns the generated pictorial data to the server, where it is adjusted based on the emotional information.

[1681] Step 14:

[1682] The server combines the story text and the picture data to generate an electronic picture book file (e.g., PDF or ePub format).

[1683] Step 15:

[1684] The server distributes the generated electronic picture book file to the user's terminal.

[1685] Step 16:

[1686] Users can view the digital picture books delivered within the app, and if necessary, can save them in paper form using the printing service.

[1687] Example 2

[1688] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1689] Conventional picture book generation systems have struggled to automatically generate content that reflects a child's creativity and emotions. Furthermore, there was no technology that could accurately convert a child's speech into text and generate appropriate picture data based on that context, or that could adjust the story and picture data to reflect a child's emotional state. This made it difficult to provide personalized digital picture books that parents and children could enjoy.

[1690] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1691] In this invention, the server includes recording means for recording a child's voice data, voice recognition means for converting the recorded voice data into text data, natural language processing means for analyzing the text data and generating a narrative, image generation means for generating picture data based on the generated narrative, picture book generation means for generating an electronic picture book by combining the narrative and picture data, distribution means for providing the generated electronic picture book to a user, emotion engine means for recognizing an emotional state from the recorded voice data, and emotion adaptation means for adjusting the narrative and picture data based on the recognized emotional state. This allows for the automatic generation of a personalized electronic picture book that reflects the child's creativity and emotions, allowing parents and children to enjoy it together.

[1692] "Recording means" means a device or method for recording a child's voice data in digital form.

[1693] "Speech recognition means" refers to the technology or algorithms used to convert recorded voice data into text data.

[1694] "Natural language processing means" is a technology for analyzing text data generated by speech recognition means and constructing meaningful narrative text.

[1695] "Image generation means" refers to algorithms and techniques for generating pictorial data based on the generated narrative text.

[1696] "Picture book generation means" refers to a technique or method for synthesizing narrative text and picture data to generate a formatted electronic picture book.

[1697] "Distribution means" refers to the technology or system for delivering the generated electronic picture book to users.

[1698] The "emotion engine means" is a technology for recognizing an emotional state from recorded voice data and adjusting narrative text and picture data based on that.

[1699] The "emotion adaptation means" is a technique for dynamically changing the content and style of narrative text and picture data based on the emotional state recognized by the emotion engine means.

[1700] The present invention relates to a system for generating picture books that give shape to children's creativity and allow parents and children to enjoy them together. This system not only automatically generates stories based on children's fantasies and creates picture books based on those stories, but also uses an emotion engine to adjust the story text and picture data according to the user's emotional state. A specific embodiment of this system is described below.

[1701] Hardware and software used

[1702] 1. Audio recording method (used by the user)

[1703] Users use an application on their smartphone or tablet to record what their child says, and the audio data is stored digitally and uploaded to a server.

[1704] 2. Speech recognition method (runs on the server)

[1705] The server receives the voice data and converts it into text data using an online API (e.g., Google Cloud Speech-to-Text) as a voice recognition engine.

[1706] 3. Natural language processing means (running on the server)

[1707] The server analyzes the converted text data and generates a narrative using a natural language processing engine (e.g., OpenAI's GPT-3).

[1708] 4. Image generation method (runs on the server)

[1709] The server generates pictorial data based on the generated narrative text using an image generation engine (e.g., DALL-E).

[1710] 5. Picture book generation method (runs on the server)

[1711] The server combines the story text and image data to generate an electronic picture book, which is saved in PDF or ePub format.

[1712] 6. Delivery method (runs on the server)

[1713] The server distributes the generated digital picture books to users, who can view them within the app and use the printing service if necessary.

[1714] 7. Emotion recognition method (runs on the server)

[1715] The server is equipped with an emotion engine that recognizes emotional information in the recorded voice data, thereby analyzing the user's emotional state and storing it in a database.

[1716] 8. Emotional Adaptation Method (Runs on Server)

[1717] Based on the analysis results of the emotion engine, the server adjusts the content of the narrative text to be generated and the color tone and style of the painting data.

[1718] Example of a system

[1719] For example, suppose a child says, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went to a mysterious forest." This recording contains the emotions of a child talking happily.

[1720] 1. Recording and uploading

[1721] The user records this content using the app and uploads it to the server.

[1722] 2. Audio Data Processing

[1723] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[1724] 3. Text Analysis

[1725] The server analyzes this using a natural language processing engine and extracts nouns ("I," "dragon," "Rainbow Bridge," "Mysterious Forest"), verbs ("I found it," "crossed," "I went"), etc.

[1726] 4. Narrative Generation

[1727] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[1728] 5. Emotion recognition

[1729] The server sends the recorded voice data to the emotion engine, which recognizes that the user is speaking with a happy tone. Based on this emotional information, positive and fun elements are added to the story.

[1730] 6. Story and picture coordination

[1731] Based on the information from the emotion engine, the server adds more excitement and adventure to the story and instructs the image generation engine to draw bright and colorful pictures.

[1732] 7. Image Generation

[1733] The server generates art data for a "flying dragon," a "rainbow bridge," and a "mysterious forest" based on the story text. This art data is adjusted to create a bright and cheerful atmosphere.

[1734] 8. Picture Book Generation

[1735] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[1736] 9. Distribution

[1737] The server delivers the generated digital picture book to the user's app, where the user can view the book within the app. Furthermore, if necessary, the user can use a printing service to save the book on paper.

[1738] The following are examples of prompt sentences that can be input to a generative AI model:

[1739] Example prompt sentence:

[1740] Generate a new adventure story based on a story my child told me. My child said with a happy expression, "One day, I found a flying dragon. The dragon crossed the Rainbow Bridge and went to a mysterious forest." Based on this, generate a story and illustrations that contain many positive elements.

[1741] In this way, a system is realized that can generate a story based on what a child says and automatically generate an electronic picture book that reflects the user's emotional state, providing a personalized picture book that can be enjoyed by both parents and children.

[1742] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1743] Step 1:

[1744] Recording and uploading

[1745] The user launches the app, taps the record button to record the child's imaginary speech, and then taps the upload button to send the audio data to the server.

[1746] Input: The audio spoken by the user.

[1747] Output: Audio data uploaded to the server.

[1748] How it works: When the user presses the "Record" button on the smartphone app screen, the built-in microphone activates and records what the child is saying. When the recording is finished, the user taps the "Upload" button, and the app sends the audio data to the server.

[1749] Step 2:

[1750] Audio data processing

[1751] The server temporarily stores the received voice data and sends it to the voice recognition engine.

[1752] Input: The audio data sent to the server.

[1753] Output: The audio data sent to the speech recognition engine.

[1754] Specific operation: When the server receives the voice data, it stores it in temporary storage and then sends it to a voice recognition engine (e.g., Google Cloud Speech-to-Text) via API.

[1755] Step 3:

[1756] Voice Recognition

[1757] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[1758] Input: The audio data sent to the speech recognition engine.

[1759] Output: The text data sent back to the server.

[1760] Specific operation: The speech recognition engine analyzes the voice signal and converts the voice into text. The text data is sent back to the server as an API response.

[1761] Step 4:

[1762] Text analytics

[1763] The server receives the returned text data and temporarily stores it in a database, after which the text data is sent to a natural language processing engine for grammatical analysis.

[1764] Input: The returned text data.

[1765] Output: Data with grammatical structure parsed.

[1766] How it works: The server stores the text data in a database and sends it to a natural language processing engine (e.g., OpenAI's GPT-3). The analysis results are returned and stored on the server.

[1767] Step 5:

[1768] Story Generation

[1769] The server uses a narrative generation algorithm to create a story based on the analysis results, and generates a narrative text, which is then stored in a database.

[1770] Input: Analysis results.

[1771] Output: The generated narrative text.

[1772] Specific operation: The server generates the main plot of the story from the analysis results and converts it into simple sentences. The generated story is then saved in the database again.

[1773] Step 6:

[1774] emotion recognition

[1775] The server sends the recorded voice data to the emotion engine to recognize the user's emotional state, and stores the information on the emotional state in a database.

[1776] Input: Recorded audio data.

[1777] Output: Emotional state information.

[1778] How it works: The server sends the voice data to the emotion engine, which interprets emotions from the tone and expressions of the voice. Once the emotional state is recognized, it is stored in a database.

[1779] Step 7:

[1780] Story and picture coordination

[1781] The server adjusts the content of the narrative text to be generated based on the emotional state recognized by the emotion engine, and also sends instructions to the image generation engine to adjust the color tone and style of the generated painting data.

[1782] Input: Emotional state information.

[1783] Output: Adjusted narrative text and pictorial data instructions.

[1784] Specific actions: If the emotional state is "happy," add positive elements to the story. Send specific instructions to the image generation engine to adjust the color tone and style of the painting data to be bright and cheerful.

[1785] Step 8:

[1786] Picture generation

[1787] The server generates pictorial data based on the narrative text using an image generation engine, and the pictorial data is adjusted based on the context of the story and the user's emotional state.

[1788] Input: Narrative text and tailored instructions.

[1789] Output: The generated painting data.

[1790] How it works: The server sends the story text to the image generation engine, which creates an illustration based on the specified style and emotion. The generated illustration is then sent back to the server.

[1791] Step 9:

[1792] Picture book generation

[1793] The server combines the story text and the picture data to generate an electronic picture book file, which is saved in PDF or ePub format.

[1794] Input: Narrative text and picture data.

[1795] Output: Digital picture book file (PDF or ePub format).

[1796] Specific operation: The server inputs the story text and picture data into the template engine and determines the layout of the digital picture book. The generated picture book is output in PDF or ePub format and saved in storage.

[1797] Step 10:

[1798] delivery

[1799] The server delivers the generated digital picture book file to the user's device. The user can view the delivered digital picture book within the app and can also use the printing service if necessary.

[1800] Input: E-book file.

[1801] Output: An electronic picture book that can be viewed by the user.

[1802] Specific operation: The server sends the user a link to the generated e-book file. The user clicks the link through the app to view the e-book. They can also access the printing service if they wish.

[1803] (Application example 2)

[1804] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1805] Systems that generate picture books that give shape to children's creativity and can be enjoyed by parents and children alike are required to provide a more immersive, interactive experience that reflects the content and emotions of children.However, conventional systems often generate picture books with static content, and are unable to provide an interactive experience where parents and children can actually interact with the characters and story world in a virtual space.

[1806] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for recording a child's voice data; means for converting the recorded voice data into text data; means for analyzing the text data and generating a narrative; means for generating picture data based on the generated narrative; means for adjusting the generated narrative and picture data and displaying them in a virtual space; means for interactively experiencing the generated electronic picture book in the virtual space; means for generating an electronic picture book by combining the narrative and picture data; and means for providing the generated electronic picture book to a user. This enables a dynamic and interactive story experience that reflects a child's creativity.

[1807] "Children's voice data" is digital voice data that records what a child is saying.

[1808] "Speech recognition means" refers to a technical means for converting recorded voice data into text data.

[1809] "Text data" is data that has been converted from audio data into text format.

[1810] "Natural language processing means" refers to technical means for analyzing text data and generating narrative text.

[1811] A "narrative text" is a story-form text generated from analyzed text data.

[1812] "Image generation means" refers to technical means for generating picture data based on the generated narrative text.

[1813] "Picture data" is digital image data generated based on a narrative text.

[1814] "Adjustment means" refers to technical means for appropriately adjusting the generated narrative text and pictorial data for display in a virtual space.

[1815] The "virtual space" is a virtual digital space in which the generated electronic picture book and other digital content are displayed.

[1816] The "experiential means" refers to a technical means that allows a user to interactively experience the generated digital picture book in a virtual space.

[1817] The "picture book generation means" is a technical means for generating an electronic picture book by combining a story and picture data.

[1818] "Distribution means" refers to the technical means for providing the generated electronic picture book to the user.

[1819] "Interactive" refers to the ability of users to directly interact with and respond to generated content.

[1820] This invention relates to a virtual picture book creation system that allows parents and children to express their creativity and have fun together. This system automatically creates a realistic, interactive picture book that reflects the child's words and emotions, allowing parents and children to experience the world of the picture book in a virtual space.

[1821] Overall system overview

[1822] The system includes the following major components:

[1823] 1. Audio recording means (user terminal)

[1824] Users use the application to record what their child says, and the audio recordings are stored digitally and uploaded to a server.

[1825] 2. Speech recognition means (server)

[1826] The server receives the voice data and converts it into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1827] 3. Natural language processing means (server)

[1828] The server analyzes the converted text data and generates a narrative using a natural language processing engine (e.g., GPT-4).

[1829] 4. Image generation means (server)

[1830] The server generates pictorial data based on the generated narrative text, using an image generation engine (e.g., DALL-E).

[1831] 5. Coordination means (server)

[1832] The server adjusts the content of the narrative text and the color tone and style of the painting data based on the emotional state recognized by an emotion engine (e.g., IBM Watson Tone Analyzer).

[1833] 6. Experience Method (User Device)

[1834] Users can interactively experience the generated picture book in a virtual space using smart glasses or a head-mounted display (e.g., Magic Leap, Oculus Rift).

[1835] 7. Picture book generation means (server)

[1836] The server combines the story text and the picture data to generate an electronic picture book, which is saved in PDF or ePub format.

[1837] 8. Delivery Method (Server)

[1838] The server distributes the generated digital picture book to the user, who can then view the picture book within the app.

[1839] Specific examples

[1840] The following is a specific example of the system.

[1841] For example, consider a situation where a system generates a story and pictures based on a child saying, "One day I want to play in a castle with a princess." The recording contains the emotion of the child talking happily.

[1842] 1. Audio recording and uploading

[1843] The user (parent or child) starts the app and records what the child says. After recording is complete, they press the upload button to send the audio data to the server.

[1844] 2. Voice Recognition

[1845] The server sends the received voice data to a speech recognition engine and converts it into text data, for example, "One day I want to play at the castle with the princess."

[1846] 3. Natural Language Processing

[1847] The server sends the converted text data to a natural language processing engine to generate a narrative sentence, such as "One day, the child had fun playing in the castle with the princess."

[1848] 4. Emotion recognition and regulation

[1849] The server sends the recorded voice data to the emotion engine, which recognizes positive emotions. Based on this emotional information, the color tone and style of the story text and painting data are adjusted to be brighter.

[1850] 5. Image Generation

[1851] Based on the story text, the server uses an image generation engine to generate pictures of "princesses" and "castles," resulting in colorful and fun images that fit the context of the story.

[1852] 6. Virtual Space Display and Experience

[1853] Users wear smart glasses or a head-mounted display and experience the picture book generated in a virtual space. In the virtual space, users can interact with the characters in the book by touching them. For example, the characters may walk around the castle or talk to them.

[1854] 7. Picture book generation and distribution

[1855] The server combines the story text and picture data to create an electronic picture book and delivers it to the user's device. Users can view the electronic picture book created within the app, and if necessary, can use a printing service to save it on paper.

[1856] Prompt Sentence Examples

[1857] Prompt: "Based on a child's joyful statement, 'One day I want to play in a castle with a princess,' please generate a story with a positive and fun atmosphere."

[1858] In this way, a dynamic and interactive storytelling experience is created that reflects the child's creativity in real time.

[1859] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1860] Step 1:

[1861] Audio recording and uploading

[1862] The user launches the application and records what the child says.

[1863] Input: User spoken voice data

[1864] What it does: Uses the device's microphone to collect audio and store it digitally.

[1865] Output: Recorded audio data (e.g. audio_file.wav)

[1866] Step 2:

[1867] Sending audio data

[1868] After the user has finished recording, he or she presses the upload button to send the audio data to the server.

[1869] Input: Recorded audio data (audio_file.wav)

[1870] Specific operation: The application sends an HTTP request to the server and uploads the audio data.

[1871] Output: Audio data received on the server side

[1872] Step 3:

[1873] Voice Recognition

[1874] The server sends the received voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data.

[1875] Input: Audio data uploaded to the server

[1876] Specific operation: The speech recognition engine analyzes the voice data and converts it into a string of characters.

[1877] Output: Text data obtained by speech recognition (e.g., "One day I want to play at the castle with the princess.")

[1878] Step 4:

[1879] Natural Language Processing (NLP)

[1880] The server sends the converted text data to a natural language processing engine (e.g., GPT-4) to generate a narrative.

[1881] Input: Text data (e.g., "One day I want to play in a castle with a princess.")

[1882] Specific operation: The NLP engine performs grammatical analysis and generates a narrative storyline.

[1883] Output: Generated story (e.g., "One day, the child had fun playing in the castle with the princess.")

[1884] Step 5:

[1885] Emotion Recognition and Regulation

[1886] The server sends the recorded voice data to an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotional state.

[1887] Input: Recorded audio data

[1888] Specific operation: The emotion engine recognizes emotions in the voice and returns the analysis results.

[1889] Output: Sentiment analysis result (e.g., positive)

[1890] Step 6:

[1891] Story and picture coordination

[1892] Based on the analysis results of the emotion engine, the server adjusts the content of the narrative text to be generated and the color tone and style of the painting data.

[1893] Input: Narrative text, sentiment analysis results

[1894] Specific actions: Add positive elements to the story and send instructions to the image generation engine (e.g., DALL-E) for bright colors and a happy atmosphere.

[1895] Output: Coordinated narrative and image generation instructions

[1896] Step 7:

[1897] Image generation

[1898] The server generates picture data based on the story using an image generation engine.

[1899] Input: tailored narrative text and image generation instructions

[1900] How it works: The image generation engine draws story scenes and characters and outputs them as digital images.

[1901] Output: Generated painting data (e.g., colorful painting)

[1902] Step 8:

[1903] Virtual space display and experience

[1904] Users can experience the picture book generated in a virtual space using smart glasses or a head-mounted display.

[1905] Input: Generated narrative text and picture data

[1906] Specific operation: The application displays narrative text and pictorial data in a virtual space, allowing users to experience it interactively.

[1907] Output: An interactive picture book experience in a virtual space

[1908] Step 9:

[1909] Picture book generation

[1910] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[1911] Input: Generated narrative text and picture data

[1912] Specific actions: Integrate the narrative text and picture data into a format and save it as an electronic file.

[1913] Output: Generated e-book (e.g. ebook.pdf)

[1914] Step 10:

[1915] delivery

[1916] The server distributes the generated electronic picture book to the user terminal.

[1917] Input: Generated e-book

[1918] Specific operation: The server sends the electronic picture book file to the user's application.

[1919] Output: Digital picture book delivered to the user's device

[1920] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1921] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1922] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1923] [Fourth embodiment]

[1924] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1925] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1926] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1927] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1928] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1929] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1930] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1931] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1932] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1933] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1934] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1935] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1936] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1937] This invention relates to a system that creates picture books that can give shape to a child's creativity and be enjoyed by parents and children. This system can automatically generate a story based on a child's imagination and create a picture book based on that story.

[1938] Overall system overview

[1939] The system includes the following major components:

[1940] 1. Audio recording method (used by the user)

[1941] Users use the application to record what their child says, and the audio recordings are stored digitally and uploaded to a server.

[1942] 2. Speech recognition method (runs on the server)

[1943] The server receives the voice data and converts it into text data using a voice recognition engine, which uses an online API (for example, a general voice recognition API).

[1944] 3. Natural language processing means (running on the server)

[1945] The server analyzes the converted text data and generates a narrative using a natural language processing engine.

[1946] 4. Image generation method (runs on the server)

[1947] The server generates pictorial data based on the generated narrative text using an image generation engine (e.g., using common image generation technology).

[1948] 5. Picture book generation method (runs on the server)

[1949] The server combines the story text and image data to generate an electronic picture book, which is saved in PDF or ePub format.

[1950] 6. Delivery method (runs on the server)

[1951] The server distributes the generated digital picture books to users, who can view them within the app and use the printing service if necessary.

[1952] Program processing flow

[1953] The specific flow of program processing in this system will be explained below.

[1954] Recording and uploading

[1955] User

[1956] The user launches the app, records what the child is imagining, and then presses the upload button to send the audio data to the server.

[1957] Audio data processing

[1958] server

[1959] The server sends the received voice data to a speech recognition engine and converts it into text data, which is then temporarily stored in a database.

[1960] Text analytics

[1961] server

[1962] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure and extracts information such as nouns, verbs, subjects, and predicates. The extracted information is then used to reconstruct the story.

[1963] Story Generation

[1964] server

[1965] The server uses a story generation algorithm to create a storyline based on the analysis results, and generates a narrative in simple language. The generated narrative is then stored in a database.

[1966] Picture generation

[1967] server

[1968] The server sends the story to an image generation engine, which generates pictorial data corresponding to the story, and stores the pictorial data in a database.

[1969] Picture book generation

[1970] server

[1971] The server combines the story text and the picture data to generate and store an electronic picture book file. If necessary, it also generates a high-resolution file for printing.

[1972] Picture book distribution

[1973] server

[1974] The server delivers the generated digital picture book file to the user's app, where the user can view the picture book and use the printing service.

[1975] Specific examples

[1976] For example, a child might say, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[1977] 1. Recording and uploading

[1978] The user records this content using the app and uploads it to the server.

[1979] 2. Audio Data Processing

[1980] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[1981] 3. Text Analysis

[1982] The server analyzes this using a natural language processing engine and extracts nouns (me, dragon, rainbow bridge, mysterious forest), verbs (found, crossed, went), etc.

[1983] 4. Narrative Generation

[1984] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[1985] 5. Image Generation

[1986] The server generates picture data for "a flying dragon," "a rainbow bridge," and "a mysterious forest" based on the story text.

[1987] 6. Picture Book Generation

[1988] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[1989] 7. Picture book distribution

[1990] The server delivers the generated e-book to the user's app, allowing the user to view the book within the app. Furthermore, printing services are also available if necessary.

[1991] This process transforms what the child says into a story in real time and provides it as a visually appealing picture book. This system is a groundbreaking way to foster children's creativity and create memories for parents and children.

[1992] The processing flow will be explained below.

[1993] Step 1:

[1994] The user launches the app and uses the recording feature to record the child's imaginary speech. When the recording is finished, the user presses the stop recording button.

[1995] Step 2:

[1996] The user presses the upload button to upload the recorded audio data to the server. The device sends the recorded data to the server.

[1997] Step 3:

[1998] The server stores the received voice data and sends it to a voice recognition engine (e.g., a common online API).

[1999] Step 4:

[2000] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[2001] Step 5:

[2002] The server receives the returned text data and temporarily stores it in a database.

[2003] Step 6:

[2004] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure.

[2005] Step 7:

[2006] The natural language processing engine extracts nouns, verbs, subjects, predicates, etc. from text data and performs analysis to understand the context of the text.

[2007] Step 8:

[2008] The server receives the analysis results and creates a storyline using a narrative generation algorithm.

[2009] Step 9:

[2010] The server generates a story in simple language and stores the generated story in a database.

[2011] Step 10:

[2012] The server sends the stored narrative to an image generation engine (eg, general image generation technology).

[2013] Step 11:

[2014] The image generation engine generates picture data based on the story text and returns the generated picture data to the server.

[2015] Step 12:

[2016] The server receives the returned painting data and stores it in a database.

[2017] Step 13:

[2018] The server combines the stored story text and picture data to generate an electronic picture book file (e.g., PDF or ePub format).

[2019] Step 14:

[2020] The server distributes the generated electronic picture book file to the user's terminal.

[2021] Step 15:

[2022] Users can view the digital picture books delivered within the app, and if necessary, can save them in paper form using the printing service.

[2023] Example 1

[2024] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2025] Existing picture book creation systems make it difficult to give form to children's creativity, and lack an easy way for parents and children to enjoy creating together. Furthermore, there was no system that efficiently converts recorded audio into text and then automatically generates illustrations that correspond to the story text based on that. This limited parent-child communication and restricted opportunities to develop children's creativity.

[2026] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2027] In this invention, the server includes means for recording a child's voice data, means for converting the recorded voice data into text data, means for analyzing the text data and generating a story, means for generating image data based on the generated story, means for generating an electronic book by combining the story and image data, and means for providing the generated electronic book to users. This makes it possible for parents and children to easily give shape to their children's creativity and create an electronic picture book that they can enjoy together.

[2028] A "means for recording a child's voice data" is a device or application that digitally records what a child says and stores that voice data.

[2029] "A speech recognition means for converting recorded speech data into text data" is a technology or engine that analyzes recorded speech data and converts it into corresponding text data.

[2030] "Natural language processing means for analyzing text data and generating narrative text" refers to a technology or algorithm that understands the language structure based on text data and reconstructs it as narrative text.

[2031] The "image generation means for generating image data based on the generated narrative text" refers to technology or software for automatically creating visual pictures or images that correspond to the content of the narrative text.

[2032] "Electronic book generation means for combining narrative text and image data to generate an electronic book" refers to a technology or system that combines narrative text and corresponding images into a single electronic book format (such as PDF or ePub).

[2033] The "distribution means for providing the generated electronic book to the user" refers to a technology or service for transmitting the completed electronic book to the user's terminal so that it can be viewed or printed.

[2034] This invention relates to a system that creates e-books that can give shape to children's creativity and be enjoyed by parents and children. This system can automatically generate stories based on children's fantasies and create picture books based on those stories.

[2035] Overall system overview

[2036] The system includes the following major components:

[2037] 1. Audio recording method (used by the device)

[2038] The device uses an application to record what the child says and saves the audio data, which is then uploaded digitally to a server.

[2039] 2. Speech recognition method (runs on the server)

[2040] The server receives the voice data and converts it into text data using a voice recognition engine (e.g., a general voice recognition API).

[2041] 3. Natural language processing means (running on the server)

[2042] The server analyzes the converted text data and generates a narrative using a natural language processing engine (e.g., a generative AI model).

[2043] 4. Image generation method (runs on the server)

[2044] The server generates image data based on the generated narrative text using an image generation engine (e.g., a general image generation technology).

[2045] 5. E-book generation means (running on the server)

[2046] The server combines the narrative text and image data to generate an e-book, which is saved in PDF or ePub format.

[2047] 6. Delivery method (runs on the server)

[2048] The server distributes the generated e-books to users, who can view them within the app and use printing services if necessary.

[2049] Program processing flow

[2050] 1. Recording and uploading

[2051] The user launches the app, records what the child says, and when they're done, they press the upload button to send the audio data to the server.

[2052] 2. Audio Data Processing

[2053] The server sends the received voice data to a voice recognition engine and converts it into text data, which is then temporarily stored in a database.

[2054] 3. Text Analysis

[2055] The server sends the stored text data to a natural language processing engine (generative AI model), which analyzes the grammatical structure and extracts information such as nouns, verbs, subjects, and predicates. The extracted information is then used to reconstruct the story.

[2056] 4. Narrative Generation

[2057] The server uses a story generation algorithm based on the analysis results to create a storyline and generate a narrative in simple language. The generated narrative is then stored in a database.

[2058] 5. Image Generation

[2059] The server sends the story text to an image generation engine, which generates image data corresponding to the story text. The generated image data is stored in a database.

[2060] 6. Picture Book Generation

[2061] The server combines the narrative text and image data to generate an e-book, which is then saved in PDF or ePub format, and a high-resolution print file is generated if necessary.

[2062] 7. Picture book distribution

[2063] The server delivers the generated e-book to the user's device, where the user can view the e-book within the app. In addition, printing services are available if necessary.

[2064] Specific examples

[2065] For example, suppose a child says, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went to a mysterious forest." In this case, the process will proceed as follows:

[2066] 1. Recording and uploading

[2067] The user records this content using the app and uploads it to the server.

[2068] 2. Audio Data Processing

[2069] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[2070] 3. Text Analysis

[2071] The server analyzes this using a natural language processing engine and extracts nouns (me, dragon, rainbow bridge, mysterious forest), verbs (found, crossed, went), etc.

[2072] 4. Narrative Generation

[2073] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[2074] 5. Image Generation

[2075] The server generates image data for "a flying dragon," "a rainbow bridge," and "a mysterious forest" based on the story text.

[2076] 6. Picture Book Generation

[2077] The server combines the narrative text and image data to generate an e-book, which is then saved in PDF or ePub format.

[2078] 7. Picture book distribution

[2079] The server delivers the generated e-book to the user's app, allowing the user to view the picture book within the app. In addition, printing services are available if necessary.

[2080] Examples of prompt statements

[2081] The generative AI model is given a prompt such as, "Create a story about a child finding a flying dragon that crosses the Rainbow Bridge and goes into the forest."

[2082] Throughout this process, what the child says is translated into a story in real time and presented as a visually engaging e-book. This system is a great way to foster children's creativity and create memorable memories for parents and children.

[2083] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2084] Step 1: Record and upload

[2085] User:

[2086] Input: What the child says.

[2087] What it does: The user launches the application on their smartphone or tablet and taps the record button to record what their child says.

[2088] Output: Recorded audio data.

[2089] Once you have finished recording, tap the upload button to send the recording data to the server.

[2090] Step 2: Receiving and converting audio data

[2091] server:

[2092] Input: Recorded audio data.

[2093] Specific operation: The server receives the voice data sent by the user.

[2094] The received voice data is sent to a voice recognition engine (e.g., a general voice recognition API).

[2095] Output: Converted to text by a speech recognition engine.

[2096] A voice recognition engine converts the voice data into text data.

[2097] Step 3: Save the text data

[2098] server:

[2099] Input: Text data.

[2100] Specific operation: The server temporarily stores the converted text data in a database.

[2101] Output: Text data stored in a database.

[2102] The converted text data is stored in a database so that it can be used for further processing.

[2103] Step 4: Text Analysis

[2104] server:

[2105] Input: Saved text data.

[2106] Specific operation: The server sends the stored text data to a natural language processing engine (e.g., a generative AI model).

[2107] A natural language processing engine analyzes the text data and extracts information such as nouns, verbs, subjects, and predicates.

[2108] Output: Parsed information (nouns, verbs, etc.).

[2109] Reconstruct the story based on the extracted information.

[2110] Step 5: Narrative generation

[2111] server:

[2112] Input: Parsed information.

[2113] Specific operation: The server creates a storyline using a narrative generation algorithm based on the analysis results of the natural language processing engine.

[2114] Generate narrative sentences in simple language.

[2115] Output: The generated narrative text.

[2116] The generated narrative text is stored in a database.

[2117] Step 6: Image generation

[2118] server:

[2119] Input: The generated narrative text.

[2120] Specific operation: The server sends the narrative text to an image generation engine (e.g., general image generation technology).

[2121] An image generation engine generates image data based on the narrative text.

[2122] Output: The generated image data.

[2123] The generated image data is stored in a database.

[2124] Step 7: Generate eBook

[2125] server:

[2126] Input: Generated narrative text and image data.

[2127] Specific operation: The server combines the narrative text and image data to generate an e-book.

[2128] Save your e-books in PDF or ePub format.

[2129] Output: The generated e-book.

[2130] High-resolution print files are also generated if needed.

[2131] Step 8: Distributing your picture book

[2132] server:

[2133] Input: The generated e-book.

[2134] Specific operation: The server delivers the generated e-book to the user's device.

[2135] Users will be able to read e-books within the app.

[2136] Output: An e-book that can be read by the user.

[2137] Printing services are also available if required.

[2138] By following these steps in sequence, what the child says is converted into a story in real time and presented as an e-book that can be enjoyed visually.

[2139] (Application example 1)

[2140] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2141] Today's parents and children are busy, with limited time to spend together and opportunities to foster creativity. Therefore, there is a need for a way to give shape to children's creativity and allow parents and children to spend fun time together. There is also a need for a system that can easily convert what a child says into a story and create a digital picture book with beautiful illustrations. Furthermore, there is a need for a way to easily view and distribute the created picture book.

[2142] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2143] In this invention, the server includes means for recording a child's voice data, voice recognition means for converting the recorded voice data into text data, natural language processing means for analyzing the text data and generating a narrative, image generation means for generating picture data based on the generated narrative, picture book generation means for generating an electronic picture book by combining the narrative and picture data, distribution means for providing the generated electronic picture book to a user, and a smartphone application for making the generated electronic picture book viewable. This allows what a child says to be easily converted into a story and picture book that can be enjoyed by parents and children.

[2144] "Children's voice data" refers to data that has been recorded and stored in digital format, containing what a child says.

[2145] A "recording means" is any device or software that records what a child says as a digital audio file.

[2146] "Speech recognition means" refers to technology or tools for analyzing recorded voice data and converting it into text information.

[2147] "Text data" is character information extracted using a voice recognition means.

[2148] "Natural language processing means" refers to technologies and engines for analyzing text data and generating narrative text and sentence structures.

[2149] A "narrative text" is a piece of writing that describes a story or narrative generated by natural language processing means.

[2150] "Image generation means" refers to techniques and tools for generating visual content based on narrative text.

[2151] "Pictorial data" is visual content generated using image generation means.

[2152] "Picture book generation means" refers to techniques and tools for creating an electronic picture book by combining the generated story text and picture data.

[2153] "Distribution means" refers to the technology and platform for providing the generated digital picture book to users.

[2154] A "smartphone application" is an app that runs on a smartphone and allows users to view and operate the generated digital picture book.

[2155] A "generative AI model" is a machine learning model that generates images and other data from text or prompts.

[2156] A "prompt" is an instruction or text used as input to a generative AI model.

[2157] System Overview

[2158] This invention relates to a system that creates picture books that give shape to children's creativity and can be enjoyed by parents and children. The entire system consists of the following components:

[2159] 1. Audio recording means

[2160] The device (smartphone) has a voice recording function to record what the child says. This voice recording function has the function to save the voice data in digital format and upload it to a server. As a concrete example, a voice recording API is used.

[2161] 2. Voice Recognition Method

[2162] The server receives the recorded voice data and converts it into text data using a speech recognition tool, such as an online API like Google Cloud Speech-to-Text API.

[2163] 3. Natural Language Processing Methods

[2164] The server uses natural language processing to analyze the converted text data. For example, a natural language processing engine such as spaCy is used as a text analysis tool. Story elements (nouns, verbs, etc.) are extracted from the analyzed text data, and a simple narrative is generated.

[2165] 4. Image Generation Method

[2166] The server generates pictorial data based on the narrative text. It uses a generative AI model, such as OpenAI's DALL-E, to generate images. When a prompt is entered, an image is generated based on it.

[2167] 5. Picture book generation method

[2168] The server combines the story text and the picture data to generate an electronic picture book. This picture book generation method uses a PDF generation library (e.g., ReportLab) to create an electronic picture book in PDF format.

[2169] 6. Distribution Method

[2170] The server has a distribution means for providing the generated digital picture book to users. Specifically, the server distributes the generated digital picture book through a smartphone application, allowing users to easily view it.

[2171] Hardware and software used

[2172] Hardware: Smartphones, servers

[2173] Software: Voice Recording API, Google Cloud Speech-to-Text API, spaCy, OpenAI DALL-E, ReportLab

[2174] Specific flow of program processing

[2175] For example, consider the example of a child saying, "One day, I became a wizard and defeated the dragon that lived in the castle."

[2176] 1. The device records this content using the voice recording API and uploads it to the server.

[2177] 2. The server converts the audio data into text data using the Google Cloud Speech-to-Text API.

[2178] 3. The server analyzes the converted text data using spaCy and generates a narrative text.

[2179] 4. The server sends the generated story text to OpenAI DALL-E, which generates picture data using the prompt text "Become a wizard and defeat a dragon."

[2180] 5. The server combines the story text and picture data using ReportLab to generate an electronic picture book in PDF format.

[2181] 6. The server distributes the generated e-book to the user via a smartphone application.

[2182] Prompt Sentence Examples

[2183] "One day, I became a wizard and defeated the dragon that lived in the castle."

[2184] Using this system, what a child says can be easily converted into stories and visual content that can be enjoyed by both parents and children.

[2185] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2186] Step 1:

[2187] Recording and uploading

[2188] The user launches the application on their device (smartphone) and uses the voice recording API to record what the child is imagining. The recorded data is saved in digital format. When the user finishes recording, they press the upload button, and the recorded audio data is sent to the server.

[2189] Input: Child's voice

[2190] Output: Recorded audio data (digital format)

[2191] Step 2:

[2192] Audio data conversion

[2193] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API, and the converted text data is stored in a database.

[2194] Input: Recorded audio data

[2195] Output: Text data

[2196] Step 3:

[2197] Text analytics

[2198] The server analyzes the text data using a natural language processing engine (e.g., spaCy) and extracts information such as nouns, verbs, subjects, and predicates. Based on the extracted information, it reconstructs the story and generates a simple narrative sentence. This narrative sentence is then stored in a database.

[2199] Input: Text data

[2200] Output: Analyzed information, narrative text

[2201] Step 4:

[2202] Story Generation

[2203] The server uses a narrative generation algorithm to create a storyline based on the information generated by the natural language processing engine, and generates a narrative in simple language. The generated narrative is then stored in a database.

[2204] Input: Parsed information

[2205] Output: Narrative text

[2206] Step 5:

[2207] Image generation

[2208] The server sends the generated story text to OpenAI's generative AI model (e.g., DALL-E), which generates corresponding picture data. It generates images by providing prompts for story elements (e.g., "wizard," "dragon," "castle," etc.) as input. This picture data is stored in a database.

[2209] Input: Narrative text, prompt text

[2210] Output: Painting data

[2211] Step 6:

[2212] Picture book generation

[2213] The server combines the story text and the picture data and generates an e-book file using a PDF generation library (e.g., ReportLab). The generated e-book is saved in PDF format.

[2214] Input: Narrative text, picture data

[2215] Output: Digital picture book file (PDF format)

[2216] Step 7:

[2217] Picture book distribution

[2218] The server distributes the generated digital picture book file to the user's terminal via a distribution means, and the user can view the digital picture book through a smartphone application.

[2219] Input: Digital picture book file

[2220] Output: Digital picture book delivery to user devices

[2221] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2222] This invention relates to a system that creates picture books that give shape to children's creativity and can be enjoyed by parents and children. This system not only automatically generates stories based on children's fantasies and creates picture books based on them, but also uses an emotion engine to adjust the story text and picture data according to the user's emotional state.

[2223] Overall system overview

[2224] The system includes the following major components:

[2225] 1. Audio recording method (used by the user)

[2226] Users use the application to record what their child says, and the audio recordings are stored digitally and uploaded to a server.

[2227] 2. Speech recognition method (runs on the server)

[2228] The server receives the voice data and converts it into text using a voice recognition engine, which uses an online API.

[2229] 3. Natural language processing means (running on the server)

[2230] The server analyzes the converted text data and generates a narrative using a natural language processing engine.

[2231] 4. Image generation method (runs on the server)

[2232] The server generates pictorial data based on the generated narrative text using an image generation engine.

[2233] 5. Picture book generation method (runs on the server)

[2234] The server combines the story text and image data to generate an electronic picture book, which is saved in PDF or ePub format.

[2235] 6. Delivery method (runs on the server)

[2236] The server distributes the generated digital picture books to users, who can view them within the app and use the printing service if necessary.

[2237] 7. Emotion recognition method (runs on the server)

[2238] The server is equipped with an emotion engine that recognizes emotional information in the recorded voice data, thereby analyzing the user's emotional state and storing it in a database.

[2239] 8. Emotional Adaptation Method (Runs on Server)

[2240] Based on the analysis results of the emotion engine, the server adjusts the content of the narrative text to be generated and the color tone and style of the painting data.

[2241] Program processing flow

[2242] The specific flow of program processing in this system will be explained below.

[2243] 1. Recording and uploading

[2244] The user launches the app and records what the child is imagining. After recording, the user presses the upload button to send the audio data to the server.

[2245] 2. Audio Data Processing

[2246] The server stores the received voice data and sends it to a voice recognition engine (e.g., an online API).

[2247] 3. Voice Recognition

[2248] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[2249] 4. Text Analysis

[2250] The server receives the returned text data and temporarily stores it in a database.

[2251] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure.

[2252] 5. Narrative Generation

[2253] The server uses a story generation algorithm to create a storyline based on the analysis results, and generates a narrative in simple language. The generated narrative is then stored in a database.

[2254] 6. Emotion recognition

[2255] The server sends the recorded voice data to the emotion engine to recognize the user's emotional state, and stores the information on the emotional state in a database.

[2256] 7. Story and Art Coordination

[2257] The server adjusts the content of the narrative sentence to be generated based on the emotional state recognized by the emotion engine.

[2258] The server further sends instructions to the image generation engine to adjust the color tone and style of the generated painting data based on the emotional state.

[2259] 8. Image Generation

[2260] The server generates pictorial data based on the narrative text using an image generation engine, and the pictorial data is adjusted based on the context of the story and the user's emotional state.

[2261] 9. Picture Book Generation

[2262] The server combines the story text and the picture data to generate an electronic picture book file (e.g., PDF or ePub format).

[2263] 10. Distribution

[2264] The server distributes the generated electronic picture book file to the user's terminal.

[2265] Users can view the digital picture books distributed within the app, and if necessary, can use the printing service to save them in paper form.

[2266] Specific examples

[2267] For example, suppose a child says, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went to a mysterious forest." Also, suppose the recording contains the emotions of a child talking happily.

[2268] 1. Recording and uploading

[2269] The user records this content using the app and uploads it to the server.

[2270] 2. Audio Data Processing

[2271] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[2272] 3. Text Analysis

[2273] The server analyzes this using a natural language processing engine and extracts nouns (me, dragon, rainbow bridge, mysterious forest), verbs (found, crossed, went), etc.

[2274] 4. Narrative Generation

[2275] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[2276] 5. Emotion recognition

[2277] The server sends the recorded voice data to the emotion engine, which recognizes that the user is speaking with a happy tone. Based on this emotional information, positive and fun elements are added to the story.

[2278] 6. Story and picture coordination

[2279] Based on the information from the emotion engine, the server adds more excitement and adventure to the story and instructs the image generation engine to draw bright and colorful pictures.

[2280] 7. Image Generation

[2281] The server generates art data for a "flying dragon," a "rainbow bridge," and a "mysterious forest" based on the story text. This art data is adjusted to create a bright and cheerful atmosphere.

[2282] 8. Picture Book Generation

[2283] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[2284] 9. Distribution

[2285] The server delivers the generated digital picture book to the user's app, where the user can view the book within the app. Furthermore, if necessary, the user can use a printing service to save the book on paper.

[2286] Through this process, what the child says is transformed into a story, and a picture book that reflects the child's emotions is generated. This system is a groundbreaking way to foster children's creativity and further enrich memories between parents and children.

[2287] The processing flow will be explained below.

[2288] Step 1:

[2289] The user launches the app and uses the recording feature to record the child's imaginary speech. When the recording is finished, the user presses the stop recording button.

[2290] Step 2:

[2291] The user presses the upload button to upload the recorded audio data to the server. The device sends the recorded data to the server.

[2292] Step 3:

[2293] The server stores the received voice data and sends it to the voice recognition engine.

[2294] Step 4:

[2295] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[2296] Step 5:

[2297] The server receives the returned text data and temporarily stores it in a database.

[2298] Step 6:

[2299] The server sends the stored text data to a natural language processing engine, which analyzes the grammatical structure.

[2300] Step 7:

[2301] The natural language processing engine extracts nouns, verbs, subjects, predicates, etc. from text data and performs analysis to understand the context of the text.

[2302] Step 8:

[2303] The server receives the analysis results and creates a storyline using a narrative generation algorithm.

[2304] Step 9:

[2305] The server generates a story in simple language and stores the generated story in a database.

[2306] Step 10:

[2307] The server sends the recorded voice data to the emotion engine to recognize the user's emotional state. The emotion engine analyzes the emotional information (e.g., joy, sadness, surprise) from the voice and sends the emotional state back to the server.

[2308] Step 11:

[2309] The server adjusts the content of the story it generates based on the emotional state recognized by the emotion engine. For example, if the user is having fun, it adds positive elements to the story.

[2310] Step 12:

[2311] The server further transmits the narrative text to the image generation engine to adjust the color tone and style of the generated painting data based on the emotional state.

[2312] Step 13:

[2313] The image generation engine generates pictorial data based on the narrative text and returns the generated pictorial data to the server, where it is adjusted based on the emotional information.

[2314] Step 14:

[2315] The server combines the story text and the picture data to generate an electronic picture book file (e.g., PDF or ePub format).

[2316] Step 15:

[2317] The server distributes the generated electronic picture book file to the user's terminal.

[2318] Step 16:

[2319] Users can view the digital picture books delivered within the app, and if necessary, can save them in paper form using the printing service.

[2320] Example 2

[2321] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2322] Conventional picture book generation systems have struggled to automatically generate content that reflects a child's creativity and emotions. Furthermore, there was no technology that could accurately convert a child's speech into text and generate appropriate picture data based on that context, or that could adjust the story and picture data to reflect a child's emotional state. This made it difficult to provide personalized digital picture books that parents and children could enjoy.

[2323] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2324] In this invention, the server includes recording means for recording a child's voice data, voice recognition means for converting the recorded voice data into text data, natural language processing means for analyzing the text data and generating a narrative, image generation means for generating picture data based on the generated narrative, picture book generation means for generating an electronic picture book by combining the narrative and picture data, distribution means for providing the generated electronic picture book to a user, emotion engine means for recognizing an emotional state from the recorded voice data, and emotion adaptation means for adjusting the narrative and picture data based on the recognized emotional state. This allows for the automatic generation of a personalized electronic picture book that reflects the child's creativity and emotions, allowing parents and children to enjoy it together.

[2325] "Recording means" means a device or method for recording a child's voice data in digital form.

[2326] "Speech recognition means" refers to the technology or algorithms used to convert recorded voice data into text data.

[2327] "Natural language processing means" is a technology for analyzing text data generated by speech recognition means and constructing meaningful narrative text.

[2328] "Image generation means" refers to algorithms and techniques for generating pictorial data based on the generated narrative text.

[2329] "Picture book generation means" refers to a technique or method for synthesizing narrative text and picture data to generate a formatted electronic picture book.

[2330] "Distribution means" refers to the technology or system for delivering the generated electronic picture book to users.

[2331] The "emotion engine means" is a technology for recognizing an emotional state from recorded voice data and adjusting narrative text and picture data based on that.

[2332] The "emotion adaptation means" is a technique for dynamically changing the content and style of narrative text and picture data based on the emotional state recognized by the emotion engine means.

[2333] The present invention relates to a system for generating picture books that give shape to children's creativity and allow parents and children to enjoy them together. This system not only automatically generates stories based on children's fantasies and creates picture books based on those stories, but also uses an emotion engine to adjust the story text and picture data according to the user's emotional state. A specific embodiment of this system is described below.

[2334] Hardware and software used

[2335] 1. Audio recording method (used by the user)

[2336] Users use an application on their smartphone or tablet to record what their child says, and the audio data is stored digitally and uploaded to a server.

[2337] 2. Speech recognition method (runs on the server)

[2338] The server receives the voice data and converts it into text data using an online API (e.g., Google Cloud Speech-to-Text) as a voice recognition engine.

[2339] 3. Natural language processing means (running on the server)

[2340] The server analyzes the converted text data and generates a narrative using a natural language processing engine (e.g., OpenAI's GPT-3).

[2341] 4. Image generation method (runs on the server)

[2342] The server generates pictorial data based on the generated narrative text using an image generation engine (e.g., DALL-E).

[2343] 5. Picture book generation method (runs on the server)

[2344] The server combines the story text and image data to generate an electronic picture book, which is saved in PDF or ePub format.

[2345] 6. Delivery method (runs on the server)

[2346] The server distributes the generated digital picture books to users, who can view them within the app and use the printing service if necessary.

[2347] 7. Emotion recognition method (runs on the server)

[2348] The server is equipped with an emotion engine that recognizes emotional information in the recorded voice data, thereby analyzing the user's emotional state and storing it in a database.

[2349] 8. Emotional Adaptation Method (Runs on Server)

[2350] Based on the analysis results of the emotion engine, the server adjusts the content of the narrative text to be generated and the color tone and style of the painting data.

[2351] Example of a system

[2352] For example, suppose a child says, "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went to a mysterious forest." This recording contains the emotions of a child talking happily.

[2353] 1. Recording and uploading

[2354] The user records this content using the app and uploads it to the server.

[2355] 2. Audio Data Processing

[2356] The server receives the voice data and converts it into text data using a voice recognition engine: "One day, I found a dragon flying in the sky. The dragon crossed the Rainbow Bridge and went into a mysterious forest."

[2357] 3. Text Analysis

[2358] The server analyzes this using a natural language processing engine and extracts nouns ("I," "dragon," "Rainbow Bridge," "Mysterious Forest"), verbs ("I found it," "crossed," "I went"), etc.

[2359] 4. Narrative Generation

[2360] Based on this, the server generates a simple story: "One day, I found a dragon flying in the sky. The dragon crossed the rainbow bridge and went to a mysterious forest."

[2361] 5. Emotion recognition

[2362] The server sends the recorded voice data to the emotion engine, which recognizes that the user is speaking with a happy tone. Based on this emotional information, positive and fun elements are added to the story.

[2363] 6. Story and picture coordination

[2364] Based on the information from the emotion engine, the server adds more excitement and adventure to the story and instructs the image generation engine to draw bright and colorful pictures.

[2365] 7. Image Generation

[2366] The server generates art data for a "flying dragon," a "rainbow bridge," and a "mysterious forest" based on the story text. This art data is adjusted to create a bright and cheerful atmosphere.

[2367] 8. Picture Book Generation

[2368] The server combines the story text and picture data to generate an electronic picture book, which is then saved in PDF or ePub format.

[2369] 9. Distribution

[2370] The server delivers the generated digital picture book to the user's app, where the user can view the book within the app. Furthermore, if necessary, the user can use a printing service to save the book on paper.

[2371] The following are examples of prompt sentences that can be input to a generative AI model:

[2372] Example prompt sentence:

[2373] Generate a new adventure story based on a story my child told me. My child said with a happy expression, "One day, I found a flying dragon. The dragon crossed the Rainbow Bridge and went to a mysterious forest." Based on this, generate a story and illustrations that contain many positive elements.

[2374] In this way, a system is realized that can generate a story based on what a child says and automatically generate an electronic picture book that reflects the user's emotional state, providing a personalized picture book that can be enjoyed by both parents and children.

[2375] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2376] Step 1:

[2377] Recording and uploading

[2378] The user launches the app, taps the record button to record the child's imaginary speech, and then taps the upload button to send the audio data to the server.

[2379] Input: The audio spoken by the user.

[2380] Output: Audio data uploaded to the server.

[2381] How it works: When the user presses the "Record" button on the smartphone app screen, the built-in microphone activates and records what the child is saying. When the recording is finished, the user taps the "Upload" button, and the app sends the audio data to the server.

[2382] Step 2:

[2383] Audio data processing

[2384] The server temporarily stores the received voice data and sends it to the voice recognition engine.

[2385] Input: The audio data sent to the server.

[2386] Output: The audio data sent to the speech recognition engine.

[2387] Specific operation: When the server receives the voice data, it stores it in temporary storage and then sends it to a voice recognition engine (e.g., Google Cloud Speech-to-Text) via API.

[2388] Step 3:

[2389] Voice Recognition

[2390] The voice recognition engine converts the voice data into text data and returns the converted text data to the server.

[2391] Input: The audio data sent to the speech recognition engine.

[2392] Output: The text data sent back to the server.

[2393] Specific operation: The speech recognition engine analyzes the voice signal and converts the voice into text. The text data is sent back to the server as an API response.

[2394] Step 4:

[2395] Text analytics

[2396] The server receives the returned text data and temporarily stores it in a database, after which the text data is sent to a natural language processing engine for grammatical analysis.

[2397] Input: The returned text data.

[2398] Output: Data with grammatical structure parsed.

[2399] How it works: The server stores the text data in a database and sends it to a natural language processing engine (e.g., OpenAI's GPT-3). The analysis results are returned and stored on the server.

[2400] Step 5:

[2401] Story Generation

[2402] The server uses a narrative generation algorithm to create a story based on the analysis results, and generates a narrative text, which is then stored in a database.

[2403] Input: Analysis results.

[2404] Output: The generated narrative text.

[2405] Specific operation: The server generates the main plot of the story from the analysis results and converts it into simple sentences. The generated story is then saved in the database again.

[2406] Step 6:

[2407] emotion recognition

[2408] The server sends the recorded voice data to the emotion engine to recognize the user's emotional state, and stores the information on the emotional state in a database.

[2409] Input: Recorded audio data.

[2410] Output: Emotional state information.

[2411] How it works: The server sends the voice data to the emotion engine, which interprets emotions from the tone and expressions of the voice. Once the emotional state is recognized, it is stored in a database.

[2412] Step 7:

[2413] Story and picture coordination

[2414] The server adjusts the content of the narrative text to be generated based on the emotional state recognized by the emotion engine, and also sends instructions to the image generation engine to adjust the color tone and style of the generated painting data.

[2415] Input: Emotional state information.

[2416] Output: Adjusted narrative text and pictorial data instructions.

[2417] Specific actions: If the emotional state is "happy," add positive elements to the story. Send specific instructions to the image generation engine to adjust the color tone and style of the painting data to be bright and cheerful.

[2418] Step 8:

[2419] Picture generation

[2420] The server generates pictorial data based on the narrative text using an image generation engine, and the pictorial data is adjusted based on the context of the story and the user's emotional state.

[2421] Input: Narrative text and tailored instructions.

[2422] Output: The generated painting data.

[2423] How it works: The server sends the story text to the image generation engine, which creates an illustration based on the specified style and emotion. The generated illustration is then sent back to the server.

[2424] Step 9:

[2425] Picture book generation

[2426] The server combines the story text and the picture data to generate an electronic picture book file, which is saved in PDF or ePub format.

[2427] Input: Narrative text and picture data.

[2428] Output: Digital picture book file (PDF or ePub format).

[2429] Specific operation: The server inputs the story text and picture data into the template engine and determines the layout of the digital picture book. The generated picture book is output in PDF or ePub format and saved in storage.

[2430] Step 10:

[2431] delivery

[2432] The server delivers the generated digital picture book file to the user's device. The user can view the delivered digital picture book within the app and can also use the printing service if necessary.

[2433] Input: E-book file.

[2434] Output: An electronic picture book that can be viewed by the user.

[2435] Specific operation: The server sends the user a link to the generated e-book file. The user clicks the link through the app to view the e-book. They can also access the printing service if they wish.

[2436] (Application example 2)

[2437] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2438] Systems that generate picture books that give shape to children's creativity and can be enjoyed by parents and children alike are required to provide a more immersive, interactive experience that reflects the content and emotions of children.However, conventional systems often generate picture books with static content, and are unable to provide an interactive experience where parents and children can actually interact with the characters and story world in a virtual space.

[2439] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for recording a child's voice data; means for converting the recorded voice data into text data; means for analyzing the text data and generating a narrative; means for generating picture data based on the generated narrative; means for adjusting the generated narrative and picture data and displaying them in a virtual space; means for interactively experiencing the generated electronic picture book in the virtual space; means for generating an electronic picture book by combining the narrative and picture data; and means for providing the generated electronic picture book to a user. This enables a dynamic and interactive story experience that reflects a child's creativity.

[2440] "Children's voice data" is digital voice data that records what a child is saying.

[2441] "Speech recognition means" refers to a technical means for converting recorded voice data into text data.

[2442] "Text data" is data that has been converted from audio data into text format.

[2443] "Natural language processing means" refers to technical means for analyzing text data and generating narrative text.

[2444] A "narrative text" is a story-form text generated from analyzed text data.

[2445] "Image generation means" refers to technical means for generating picture data based on the generated narrative text.

[2446] "Picture data" is digital image data generated based on a narrative text.

[2447] "Adjustment means" refers to technical means for appropriately adjusting the generated narrative text and pictorial data for display in a virtual space.

[2448] The "virtual space" is a virtual digital space in which the generated electronic picture book and other digital content are displayed.

[2449] The "experiential means" refers to a technical means that allows a user to interactively experience the generated digital picture book in a virtual space.

[2450] The "picture book generation means" is a technical means for generating an electronic picture book by combining a story and picture data.

[2451] "Distribution means" refers to the technical means for providing the generated electronic picture book to the user.

[2452] "Interactive" refers to the ability of users to directly interact with and respond to generated content.

[2453] This invention relates to a virtual picture book creation system that allows parents and children to express their creativity and have fun together. This system automatically creates a realistic, interactive picture book that reflects the child's words and emotions, allowing parents and children to experience the world of the picture book in a virtual space.

[2454] Overall system overview

[2455] The system includes the following major components:

[2456] 1. Audio recording means (user terminal)

[2457] Users use the application to record what their child says, and the audio recordings are stored digitally and uploaded to a server.

[2458] 2. Speech recognition means (server)

[2459] The server receives the voice data and converts it into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[2460] 3. Natural language processing means (server)

[2461] The server analyzes the converted text data and generates a narrative using a natural language processing engine (e.g., GPT-4).

[2462] 4. Image generation means (server)

[2463] The server generates pictorial data based on the generated narrative text, using an image generation engine (e.g., DALL-E).

[2464] 5. Coordination means (server)

[2465] The server adjusts the content of the narrative text and the color tone and style of the painting data based on the emotional state recognized by an emotion engine (e.g., IBM Watson Tone Analyzer).

[2466] 6. Experience Method (User Device)

[2467] Users can interactively experience the generated picture book in a virtual space using smart glasses or a head-mounted display (e.g., Magic Leap, Oculus Rift).

[2468] 7. Picture book generation means (server)

[2469] The server combines the story text and the picture data to generate an electronic picture book, which is saved in PDF or ePub format.

[2470] 8. Delivery Method (Server)

[2471] The server distributes the generated digital picture book to the user, who can then view the picture book within the app.

[2472] Specific examples

[2473] The following is a specific example of the system.

[2474] For example, consider a situation where a system generates a story and pictures based on a child saying, "One day I want to play in a castle with a princess." The recording contains the emotion of the child talking happily.

[2475] 1. Audio recording and uploading

[2476] The user (parent or child) starts the app and records what the child says. After recording is complete, they press the upload button to send the audio data to the server.

[2477] 2. Voice Recognition

[2478] The server sends the received voice data to a speech recognition engine and converts it into text data, for example, "One day I want to play at the castle with the princess."

[2479] 3. Natural Language Processing

[2480] The server sends the converted text data to a natural language processing engine to generate a narrative sentence, such as "One day, the child had fun playing in the castle with the princess."

[2481] 4. Emotion recognition and regulation

[2482] The server sends the recorded voice data to the emotion engine, which recognizes positive emotions. Based on this emotional information, the color tone and style of the story text and painting data are adjusted to be brighter.

[2483] 5. Image Generation

[2484] Based on the story text, the server uses an image generation engine to generate pictures of "princesses" and "castles," resulting in colorful and fun images that fit the context of the story.

[2485] 6. Virtual Space Display and Experience

[2486] Users wear smart glasses or a head-mounted display and experience the picture book generated in a virtual space. In the virtual space, users can interact with the characters in the book by touching them. For example, the characters may walk around the castle or talk to them.

[2487] 7. Picture book generation and distribution

[2488] The server combines the story text and picture data to create an electronic picture book and delivers it to the user's device. Users can view the electronic picture book created within the app, and if necessary, can use a printing service to save it on paper.

[2489] Prompt Sentence Examples

[2490] Prompt: "Based on a child's joyful statement, 'One day I want to play in a castle with a princess,' please generate a story with a positive and fun atmosphere."

[2491] In this way, a dynamic and interactive storytelling experience is created that reflects the child's creativity in real time.

[2492] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2493] Step 1:

[2494] Audio recording and uploading

[2495] The user launches the application and records what the child says.

[2496] Input: User spoken voice data

[2497] What it does: Uses the device's microphone to collect audio and store it digitally.

[2498] Output: Recorded audio data (e.g. audio_file.wav)

[2499] Step 2:

[2500] Sending audio data

[2501] After the user has finished recording, he or she presses the upload button to send the audio data to the server.

[2502] Input: Recorded audio data (audio_file.wav)

[2503] Specific operation: The application sends an HTTP request to the server and uploads the audio data.

[2504] Output: Audio data received on the server side

[2505] Step 3:

[2506] Voice Recognition

[2507] The server sends the received voice data to a voice recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data.

[2508] Input: Audio data uploaded to the server

[2509] Specific operation: The speech recognition engine analyzes the voice data and converts it into a string of characters.

[2510] Output: Text data obtained by speech recognition (e.g., "One day I want to play at the castle with the princess.")

[2511] Step 4:

[2512] Natural Language Processing (NLP)

[2513] The server sends the converted text data to a natural language processing engine (e.g., GPT-4) to generate a narrative.

[2514] Input: Text data (e.g., "One day I want to play in a castle with a princess.")

[2515] Specific operation: The NLP engine performs grammatical analysis and generates a narrative storyline.

[2516] Output: Generated story (e.g., "One day, the child had fun playing in the castle with the princess.")

[2517] Step 5:

[2518] Emotion Recognition and Regulation

[2519] The server sends the recorded voice data to an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotional state.

[2520] Input: Recorded audio data

[2521] Specific operation: The emotion engine recognizes emotions in the voice and returns the analysis results.

[2522] Output: Sentiment analysis result (e.g., positive)

[2523] Step 6:

[2524] Story and picture coordination

[2525] Based on the analysis results of the emotion engine, the server adjusts the content of the narrative text to be generated and the color tone and style of the painting data.

[2526] Input: Narrative text, sentiment analysis results

[2527] Specific actions: Add positive elements to the story and send instructions to the image generation engine (e.g., DALL-E) for bright colors and a happy atmosphere.

[2528] Output: Coordinated narrative and image generation instructions

[2529] Step 7:

[2530] Image generation

[2531] The server generates picture data...

Claims

1. a means for recording voice data of the child; a voice recognition means for converting recorded voice data into text data; natural language processing means for analyzing text data and generating narrative text; an image generating means for generating picture data based on the generated narrative text; a picture book generating means for generating an electronic picture book by combining a story and picture data; a distribution means for providing the generated electronic picture book to a user; A system including:

2. 2. The system of claim 1, wherein the speech recognition means uses an online API to convert speech data into text data.

3. The system according to claim 1 , further comprising a function of providing an electronic picture book created to embody the pictorial data in a printable format.

Images