system

Abstract

We provide the system. [Solution] A means of inputting a picture drawn by a child, A means for analyzing the aforementioned drawing and identifying the main objects, A means for automatically generating a story based on the analyzed object, A means for customizing the generated story based on the child's age, interests, and strengths, A means for generating a picture book containing the customized story, A system including means for providing the generated picture books online.

Description

Technical Field

[0004] ,

[0006] , , , ,

[0005] , , , , ,

[0003] , , , , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003] <000001​​​​​​​​​​​​​​​​​​​​​​​A method for inputting drawings made by children

[0007] A means of analyzing a drawing and identifying the main objects.

[0008] A means of automatically generating a story based on analyzed objects.

[0009] A way to customize generated stories based on a child's age, interests, and strengths.

[0010] A means of generating picture books that include customized stories.

[0011] A method for providing the generated picture books online.

[0012] This makes it possible to quickly and easily create personalized picture books that leverage children's creativity. Furthermore, adding educational content to the picture books can enhance their learning effectiveness. In addition to online delivery, printed versions can also be ordered, allowing children to enjoy them as physical objects.

[0013] "Children's drawings" refers to paintings and illustrations created freely by children.

[0014] "Input method" refers to a device or interface for digitally importing a child's drawing into the system.

[0015] "Means of analysis" refers to devices or algorithms that analyze an input image through digital image processing to identify the main objects (e.g., people, animals, background, etc.).

[0016] A "major object" is an element in a painting that is particularly prominent or deemed important for generating a story.

[0017] The "means for automatically generating a story" refers to a generative AI model or algorithm for automatically creating a story based on the analyzed objects.

[0018] The "means for customizing" refers to a device or algorithm for adjusting the generated story according to the age, interests, and areas of strength of the child.

[0019] The "means for generating a picture book" refers to a device or process for creating a picture book including the customized story in digital or printed form.

[0020] The "means for providing online" refers to a web server or digital distribution platform for making the generated picture book viewable by users through the Internet.

[0021] "Educational content" refers to scientific facts, learning elements, and information with educational value added to the picture book.

[0022] The "means for ordering a printed version" refers to an order reception and processing system for manufacturing and delivering a printed version of the picture book provided online.

Brief Description of Drawings

[0023] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6]This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0024] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0025] First, let's explain the terminology used in the following explanation.

[0026] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0027] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0028] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0029] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0031] [First Embodiment]

[0032] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0033] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0036] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0039] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0043] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0044] This invention relates to a system for automatically generating original picture books based on drawings made by children. Specifically, the system generates and provides picture books through the following process.

[0045] 1. Upload the image and enter the information.

[0046] User:

[0047] Users upload drawings made by their children using the application or website. Users also enter necessary information such as the child's age, interests, and areas of expertise.

[0048] Terminal:

[0049] The device sends the image file uploaded by the user and any additional information entered to the server.

[0050] 2. Receiving and analyzing image data

[0051] server:

[0052] The server receives the uploaded image data and accompanying information and stores them in a database. Next, the server invokes a generative AI model for image analysis to identify the main objects depicted in the image (people, animals, background, etc.).

[0053] Specific example:

[0054] For example, if a child draws a picture of a dinosaur, the server recognizes the picture as a dinosaur object and saves it in the database with the label "dinosaur".

[0055] 3. Automatic story generation

[0056] server:

[0057] The server automatically generates a story using a generative AI model based on the analysis results. The generated story uses objects depicted in illustrations as plot elements and characters.

[0058] Specific example:

[0059] If a picture of a dinosaur is entered, the server will generate a story themed around "dinosaur adventures," depicting the challenges the dinosaur faces and how it overcomes them.

[0060] 4. Customization and Addition of Educational Content

[0061] server:

[0062] The server customizes the generated stories based on the child's age, interests, and areas of expertise. Furthermore, it adds scientific facts and learning elements to enhance their educational value.

[0063] Specific example:

[0064] If a 6-year-old child is interested in dinosaurs, the server will incorporate scientific facts about dinosaur species and their ecology into the story. For example, a scenario could be created where a dinosaur discovers a new cave and finds fossils within, incorporating learning about different dinosaur species along the way.

[0065] 5. Generation and provision of picture book data

[0066] server:

[0067] The server converts the customized story into an ebook format (such as EPUB or PDF) and provides a link for users to view it online. Printed versions of the picture book can also be ordered if desired.

[0068] Specific example:

[0069] Users can browse picture books based on children's drawings online and also order printed copies. When a user orders a printed copy, the server sends the data to the printer, and the printed book is delivered a few days later.

[0070] The system of this invention fosters children's creativity, deepens the parent-child bond, and provides educational value. This system is highly effective in quickly and easily generating original picture books and expanding learning opportunities.

[0071] The following describes the processing flow.

[0072] Step 1:

[0073] User:

[0074] The user opens an application or website for generating picture books and clicks the "Upload Picture" button.

[0075] Step 2:

[0076] Terminal:

[0077] The terminal displays a file selection window, providing a screen where the user can select the image file they want to upload.

[0078] Step 3:

[0079] User:

[0080] The user selects an image file of a picture, enters the necessary information (such as the child's age, areas of interest, and areas of expertise), and presses the "Submit" button.

[0081] Step 4:

[0082] Terminal:

[0083] The terminal sends data, including the selected image file and the entered information, to the server.

[0084] Step 5:

[0085] server:

[0086] The server receives the data. It saves the image file and additional information to the database and verifies that the upload was successful.

[0087] Step 6:

[0088] server:

[0089] The server invokes a generative AI model to analyze the stored image files. The AI ​​model analyzes the image files and identifies key objects.

[0090] Step 7:

[0091] server:

[0092] The server labels each object based on the analysis results and stores them in the database. For example, if the picture depicts a dinosaur, it will be labeled "dinosaur".

[0093] Step 8:

[0094] server:

[0095] The server uses the analysis results and accompanying information to invoke a generative AI model to generate a story. The generative AI model automatically generates a story based on the input prompts.

[0096] Step 9:

[0097] server:

[0098] The server retrieves the generated story and customizes it based on user-provided information (age, interests, areas of expertise).

[0099] Step 10:

[0100] server:

[0101] The server adds educational content to customized stories. For example, if a picture of a dinosaur is included, it will add scientific facts and edutainment elements about dinosaurs.

[0102] Step 11:

[0103] server:

[0104] The server integrates the completed story and educational content and converts it into an ebook format (EPUB or PDF).

[0105] Step 12:

[0106] server:

[0107] The server uploads the generated ebook data to the web server and generates a link that allows users to view it online.

[0108] Step 13:

[0109] Terminal:

[0110] The terminal retrieves a link provided by the server and provides an interface that allows the user to view picture books online.

[0111] Step 14:

[0112] User:

[0113] Users can view and check the content of picture books generated online.

[0114] Step 15:

[0115] User:

[0116] If the user wishes, they can click the "Order Printed Version" button on the online interface.

[0117] Step 16:

[0118] Terminal:

[0119] The terminal displays an order form for entering shipping and payment information.

[0120] Step 17:

[0121] User:

[0122] The user enters the required information and presses the "Confirm Order" button.

[0123] Step 18:

[0124] server:

[0125] The server receives order data, processes payments, and sends print data to the printer.

[0126] Step 19:

[0127] Printing company:

[0128] The printing company prints the picture book based on the received data and delivers it to the specified address.

[0129] Step 20:

[0130] server:

[0131] The server monitors the progress of the order and notifies the user when shipping is confirmed.

[0132] (Example 1)

[0133] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0134] Currently, there is no system on the market that automatically generates original picture books using children's drawings. This means that parents and educators lack effective tools to further cultivate children's creativity and enhance their educational value. Furthermore, there are limited means of easily generating and providing high-quality picture books. Given this situation, there is a need for a system that can easily generate original picture books based on children's drawings while also adding educational value.

[0135] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0136] In this invention, the server includes means for storing children's drawings in a database, means for analyzing the stored drawings using a generation AI model to identify key objects, and means for automatically generating a story based on the identified objects. This enables the automatic generation of original picture books based on children's drawings and allows for customization that includes educational elements.

[0137] "Children's drawings" refer to illustrations and paintings created by children using their own creativity.

[0138] "Means of input" refer to the functions or interfaces that allow users to provide data to the system.

[0139] "Means of uploading" refers to the functions or interfaces that allow users to transfer data stored on their local storage to a server via the internet.

[0140] "Means of transmission" refers to the functions and protocols used to send data from a terminal to a server.

[0141] A "server" is a computer system that stores and processes data over a network.

[0142] "Means of saving to a database" refers to functions and systems for storing and managing received data in an organized format.

[0143] A "generative AI model" is an artificial intelligence algorithm used to learn from large amounts of data and generate new data.

[0144] "Key objects" refer to important elements or components present in an image that are identified through image analysis.

[0145] "Methods for automatically generating stories" refers to functions that use generative AI models to create narratives based on specified objects and information.

[0146] "Customization options" refer to features that allow you to adjust and modify the content of stories generated based on a child's age, interests, and strengths.

[0147] An "eBook format" refers to a standard format for saving and displaying books in digital format (e.g., EPUB, PDF).

[0148] "Means of providing content online" refers to functions that make generated content accessible to users via the internet.

[0149] "Educational content" refers to information and materials intended to improve learning and knowledge.

[0150] A "printed version" refers to digital content printed on physical paper.

[0151] This invention relates to a system that automatically generates original picture books based on drawings made by children. The system includes a process in which a user uploads a child's drawing, a server analyzes the drawing, generates and customizes a story, and finally delivers it as an ebook.

[0152] Hardware and software to be used

[0153] Hardware:

[0154] Server: A high-performance computer used for data storage, analysis, and execution of generated AI models.

[0155] Device: A PC or smartphone used by users to upload images and information.

[0156] software:

[0157] Web application or mobile application: Provides a user interface.

[0158] Database: A data management system for storing received images and information (e.g., MySQL, PostgreSQL).

[0159] Generative AI models: AI algorithms used for image analysis and story generation (e.g., TensorFlow, PyTorch, GPT-3).

[0160] Image processing libraries: Used for image preprocessing (e.g., OpenCV).

[0161] eBook generation tools: Tools for converting generated stories into eBook formats (e.g., Calibre).

[0162] Program Processing Overview

[0163] User:

[0164] Users upload drawings made by their children to the application or website and enter information such as the child's age, interests, and areas of expertise. The uploaded data is sent from the device to the server. For example:

[0165] Click the upload button and select your image file.

[0166] Enter your age, interests, and areas of expertise in the text fields.

[0167] Click the "Submit Data" button.

[0168] Terminal:

[0169] The terminal receives the image file and input information uploaded by the user, packets them, and sends them to the server using the HTTPS protocol. For example:

[0170] The image file and input information are saved to temporary storage.

[0171] The packet should include data and necessary metadata (user ID, timestamp, etc.).

[0172] Generate a POST request and send the data.

[0173] server:

[0174] The server performs the following actions:

[0175] 1. Save the received image data and additional information to the database.

[0176] 2. Analyze the image using a generative AI model to identify the main objects.

[0177] 3. Based on the identified object, a story is automatically generated that aligns with the prompt text.

[0178] Specific example:

[0179] If a child draws a picture of a dinosaur, the server uses a generative AI model to identify the dinosaur and generates the prompt "Generate a dinosaur adventure story." A story is then generated based on this prompt.

[0180] Next, the generated story is customized based on the child's age, interests, and areas of expertise. Furthermore, educational content (e.g., information about dinosaur species and their ecology) is incorporated into the story.

[0181] After the story is completed, it will be converted into an ebook format (e.g., EPUB, PDF), and users will be provided with a link to view the picture book online. Printed copies can also be ordered upon request, in which case the data will be sent to the printing company.

[0182] This invention enables the rapid and easy creation of picture books that foster children's creativity, deepen parent-child bonds, and provide educational value.

[0183] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0184] Step 1:

[0185] User:

[0186] Users access the application or website and click the "Upload Picture" button on the screen. Next, they select and upload an image file of a picture drawn by their child from their local storage. They also enter necessary information such as the child's age, interests, and areas of expertise.

[0187] Specific actions:

[0188] Use a web browser or smartphone app.

[0189] Select the image file and perform the upload operation.

[0190] Enter the required information and press the submit button.

[0191] Inputs and outputs:

[0192] Input: Image files of drawings made by the child, information about the child's age, interests, and areas of expertise.

[0193] Output: The uploaded image file and input information are sent to the device.

[0194] Step 2:

[0195] Terminal:

[0196] The device temporarily stores the image file uploaded by the user and any additional information entered, and then sends it to the server using the HTTPS protocol.

[0197] Specific actions:

[0198] Uploaded images and input information are saved to temporary storage.

[0199] The packet should include data and necessary metadata (user ID, timestamp, etc.).

[0200] Generate a POST request to the server and send the data.

[0201] Inputs and outputs:

[0202] Input: Image files submitted by the user, age, interests, and areas of expertise.

[0203] Output: Packetized data is sent to the server.

[0204] Step 3:

[0205] server:

[0206] The server extracts image data and input information from the received packets and stores them in a database. Next, it calls a generative AI model to analyze the stored image data and identify key objects.

[0207] Specific actions:

[0208] The received data is converted to the appropriate format and saved to the database.

[0209] Image features are extracted using a generative AI model (e.g., TensorFlow or PyTorch).

[0210] Identify and label the main objects based on the features.

[0211] Inputs and outputs:

[0212] Input: Image file and input information sent from the terminal.

[0213] Output: Analysis results stored in the database, labels of identified key objects.

[0214] Step 4:

[0215] server:

[0216] Based on the analysis results, the server calls a generative AI model for story generation (e.g., GPT-3) to generate prompt text. The story is then automatically generated based on the generated prompt text.

[0217] Specific actions:

[0218] The labels extracted from the analysis results are incorporated into the prompt message (e.g., "Generate a dinosaur adventure story").

[0219] The prompt text is input into the AI ​​model to generate a story.

[0220] Save the generated story to the database.

[0221] Inputs and outputs:

[0222] Input: Labels of the identified main objects.

[0223] Output: The generated story.

[0224] Step 5:

[0225] server:

[0226] The server customizes the generated story based on the child's age, interests, and areas of expertise. At the same time, it incorporates educational content (e.g., scientific facts and learning elements) into the story.

[0227] Specific actions:

[0228] The story content is adjusted based on the user's age, interests, and areas of expertise.

[0229] Retrieve the relevant educational content from the database and add it to the story.

[0230] Inputs and outputs:

[0231] Input: Generated story, user-entered age, interests, and areas of expertise.

[0232] Output: Customized story.

[0233] Step 6:

[0234] server:

[0235] The server converts customized stories into ebook formats (e.g., EPUB, PDF). Users are provided with a link to view the stories online. Printed versions can also be ordered upon request.

[0236] Specific actions:

[0237] Call an API (e.g., Calibre) to convert the story into an ebook format.

[0238] The generated ebook format file is saved to cloud storage (e.g., AWS S3), and a link is generated.

[0239] When a print order is placed, the order data is sent to the printing company's system.

[0240] Inputs and outputs:

[0241] Input: A customized story.

[0242] Output: ebook format file, online viewing link, print order data.

[0243] (Application Example 1)

[0244] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0245] In the modern era, there is a lack of systems that can easily provide educational and enjoyable content while fostering children's creativity. Furthermore, while there is a wealth of digitized learning content, systems that generate original stories based on children's own drawings still do not exist. Therefore, there is a need for a system that can easily provide educational content tailored to each child's individual interests and age.

[0246] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0247] In this invention, the server includes means for inputting a drawing made by a child, means for analyzing the drawing and identifying the main objects, means for automatically generating a story based on the analyzed objects, means for customizing the generated story based on the child's age, interests, and areas of expertise, means for generating the picture book in ebook format, means for providing the generated ebook online and enabling orders for printed versions, and means for using a generation AI model throughout the entire picture book generation process. This makes it possible for children themselves to become creators and automatically generate and provide customized educational content tailored to their interests and age. Furthermore, since it can be provided in ebook format and used as a printed material, it can meet a variety of needs.

[0248] "A means of inputting children's drawings" refers to a system that allows children to upload illustrations and drawings they have drawn on paper or on digital devices to the system via devices such as smartphones and computers.

[0249] "The means of analyzing the drawn image and identifying the main objects" refers to a system that uses an image analysis algorithm to recognize the main elements such as people, animals, and backgrounds from the uploaded image and register them in a database.

[0250] "Means for automatically generating a story based on the analyzed objects" refers to a mechanism that automatically creates a story by setting relevant plots and characters using a generation AI model based on recognized objects.

[0251] "Means for customizing the generated story based on the child's age, interests, and areas of expertise" refers to a mechanism that optimizes and individually customizes an automatically generated story according to the input data on the child's age, interests, and areas of expertise.

[0252] "A method for generating picture books in ebook format" refers to a system that converts customized stories into digital formats (such as EPUB or PDF) and generates them as ebooks.

[0253] "The means of providing the generated e-books online and enabling orders for printed versions" refers to a system that provides the generated e-books to users via the internet and also accepts orders for printed versions according to the user's wishes.

[0254] "Means of using a generative AI model throughout the entire picture book generation process" refers to a system that uses a generative AI model throughout the entire process, from image analysis to storytelling, to achieve a high level of automation and personalization.

[0255] The system of this invention automatically generates original picture books based on drawings made by children and provides them online. This system includes a series of processes in which the user inputs a drawing made by a child, the system analyzes the drawing to generate a story, and the system provides the story in ebook format.

[0256] The entire system consists of a smartphone application and a server. The primary hardware used is a smartphone and a server. The software includes a generative AI model (TensorFlow), Flask (a server-side web application framework), and React Native (a mobile application framework).

[0257] 1. User input and image upload

[0258] The user opens a smartphone application and takes a picture of or uploads a drawing made by their child. Along with the image file of the drawing, they also enter information such as the child's age, interests, and areas of expertise. This data is sent from the smartphone to the server.

[0259] 2. Image analysis and database storage on the server.

[0260] The server analyzes the received image file using a generation AI model (using TensorFlow). As a result of the image analysis, the main objects (e.g., people, animals, background) are identified. The analysis results are stored in a database (e.g., AWS RDS).

[0261] 3. Story Generation

[0262] The server automatically generates a story using a generative AI model based on the analyzed objects. In this generation process, appropriate plots and characters are set based on the uploaded image. For example, if an image of a dinosaur is uploaded, a story themed around "dinosaur adventures" will be generated.

[0263] 4. Customization and addition of educational elements

[0264] The generated stories are customized based on the child's age and interests. Furthermore, educational elements (e.g., scientific facts and learning elements) are added. For example, a story about a dinosaur discovering a new cave might incorporate information about different types of dinosaurs and their behavior.

[0265] 5. Generation and provision of picture book ebooks

[0266] The customized story is converted into an ebook format (e.g., EPUB or PDF). Users can view the generated ebook online. Printed versions can also be ordered if desired. The server sends the data to the printer, and the printed picture book is delivered to the user within a few days.

[0267] Examples of specific cases and prompt statements

[0268] As a concrete example, using a child's drawing of a dinosaur as a reference, use the following prompt:

[0269] I've uploaded a drawing of a dinosaur made by a child. Based on this drawing, please generate an adventure story in which the dinosaur discovers a new cave and finds fossils inside. Also, please add information to the story that teaches about different types of dinosaurs and their ecology.

[0270] This system allows children to become creators themselves, automatically generating and providing customized educational content tailored to their interests and age. It can also be provided in e-book format, and print versions are available upon request, accommodating diverse needs.

[0271] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0272] Step 1:

[0273] The user opens the smartphone application and takes a picture of the child's drawing or uploads it within the application. Additionally, the user enters the child's age, interests, areas of expertise, etc. These data are sent from the smartphone to the server. The smartphone sends the image file and text data in an HTTP request.

[0274] Input: Image file of the child's drawing, child's age, interests, areas of expertise

[0275] Output: Image file and text data sent to the server

[0276] Step 2:

[0277] The server saves the received image file and analyzes the image using a generative AI model. Specifically, it uses an image analysis algorithm (TensorFlow) to identify the main objects (people, animals, background, etc.) and saves the analysis results in a database.

[0278] Input: Image file, text data

[0279] Output: List of analyzed objects, saving to the database

[0280] Specific operations:

[0281] Receiving and saving the image file

[0282] Image analysis using a generative AI model

[0283] Identifying the main objects

[0284] Saving the analysis results to the database (e.g., AWS RDS)

[0285] Step 3:

[0286] The server automatically generates a story using the generated AI model based on the information of the analyzed object. Data on the age, interests, and fields of expertise entered by the user are also utilized at this stage.

[0287] Input: List of analyzed objects, child's age, interests, fields of expertise

[0288] Output: Automatically generated story

[0289] Specific operations:

[0290] Combine the analysis results and additional information to generate a prompt sentence

[0291] Input the prompt sentence into the generated AI model to generate a story

[0292] Step 4:

[0293] The server customizes the automatically generated story based on the child's age, interests, and fields of expertise, and adds educational elements. For example, incorporate scientific facts and learning elements into the story.

[0294] Input: Automatically generated story, child's age, interests, fields of expertise

[0295] Output: Customized story

[0296] Specific operations: [[ID=四十二]]

[0297] Add educational elements to the automatically generated story

[0298] Optimize the story based on the child's age and interests

[0299] Step 5:

[0300] The server converts the customized story into an e - book format (e.g., EPUB or PDF). Then, it generates a link to provide the generated e - book online and notifies the user.

[0301] Input: Customized story

[0302] Output: E - book file, online link

[0303] Specific operations:

[0304] Convert the story into an e - book format [[ID=X18]]

[0305] Generate a link for online viewing

[0306] [[ID=X24]]Notify the user of the link (e.g., send an email)

[0307] Step 6:

[0308] The user views the e - book generated online or, if desired, also places an order for a printed version. The server sends the order data for the printed version to the printer and arranges for the printing and delivery of the picture book.

[0309] Input: User's order for the printed version

[0310] Output: Printed picture book

[0311] Specific operations:

[0312] Receive the order information for the printed version

[0313] Send the order data to the printer X

[0314] The printer prints the picture book and delivers it to the user

[0315] Through this series of processes, an original picture book that makes the most of children's creativity is easily generated and provided in both electronic and paper versions. Note: There seems to be a formatting issue with the original text where some tags might be misaligned or have incorrect line breaks. I've tried to translate as accurately as possible while maintaining the structure. Also, the "X" marked lines might need further clarification in the original for a more precise translation.

[0316] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0317] This invention relates to a system that automatically generates original picture books based on drawings made by children and further customizes the content by recognizing the user's emotions. Specifically, the invention describes a form that generates and provides picture books through the following process and combines it with emotion recognition.

[0318] 1. Upload the image and enter the information.

[0319] User:

[0320] Users upload drawings made by their children using the application or website. Users also enter necessary information such as the child's age, interests, and areas of expertise.

[0321] Terminal:

[0322] The device sends the image file uploaded by the user and any additional information entered to the server.

[0323] 2. Receiving and analyzing image data

[0324] server:

[0325] The server receives the uploaded image data and accompanying information and stores them in a database. Next, the server invokes a generative AI model for image analysis to identify the main objects depicted in the image (people, animals, background, etc.).

[0326] Specific example:

[0327] For example, if a child draws a picture of a dinosaur, the server recognizes the picture as a dinosaur object and saves it in the database with the label "dinosaur".

[0328] 3. Automatic story generation

[0329] server:

[0330] The server automatically generates a story using a generative AI model based on the analysis results. The generated story uses objects depicted in illustrations as plot elements and characters.

[0331] Specific example:

[0332] If a picture of a dinosaur is entered, the server will generate a story themed around "dinosaur adventures," depicting the challenges the dinosaur faces and how it overcomes them.

[0333] 4. Customization and Addition of Educational Content

[0334] server:

[0335] The server customizes the generated stories based on the child's age, interests, and areas of expertise. Furthermore, it adds scientific facts and learning elements to enhance their educational value.

[0336] Specific example:

[0337] If a 6-year-old child is interested in dinosaurs, the server will incorporate scientific facts about dinosaur species and their ecology into the story. For example, a scenario could be created where a dinosaur discovers a new cave and finds fossils within, incorporating learning about different dinosaur species along the way.

[0338] 5. Utilizing the Emotion Engine

[0339] server:

[0340] The server uses an emotion engine to analyze the user's emotions in real time. The emotion engine analyzes the user's facial expressions and voice through the camera and microphone to acquire emotional data.

[0341] Specific example:

[0342] While the user is browsing the picture book, the server analyzes the user's facial expressions through an emotion engine and recognizes emotions such as surprise and joy.

[0343] 6. Emotion-based real-time story adjustment

[0344] server:

[0345] The server adjusts the story content in real time based on feedback from the emotion engine. The story's progression and character actions change dynamically each time the user's emotions shift.

[0346] Specific example:

[0347] If the user shows a surprised expression in response to a story development, the server will add elements in the next scene that allow them to continue enjoying the surprise. Conversely, if the emotional data indicates sadness, the server will adjust the story and introduce a more positive development.

[0348] 7. Generation and provision of picture book data

[0349] server:

[0350] The server integrates customized stories and educational content, as well as emotionally-based adjustments, and converts them into ebook formats (EPUB and PDF).

[0351] server:

[0352] The server uploads the generated ebook data to the web server and generates a link that allows users to view it online. Printed versions of the picture books can also be ordered if desired.

[0353] Specific example:

[0354] Users can browse picture books based on children's drawings online and also order printed copies. When a user orders a printed copy, the server sends the data to the printer, and the printed book is delivered a few days later.

[0355] The system of this invention fosters children's creativity, deepens the parent-child bond, and provides educational value. Furthermore, by combining it with an emotion engine, it provides a more personalized experience that responds to the user's emotions, generating even greater emotion and excitement. This system is highly effective not only in quickly and easily generating original picture books and expanding learning opportunities, but also in providing a dynamic experience that responds to emotions.

[0356] The following describes the processing flow.

[0357] Step 1:

[0358] User:

[0359] The user opens an application or website for generating picture books and clicks the "Upload Picture" button.

[0360] Step 2:

[0361] Terminal:

[0362] The terminal displays a file selection window, providing a screen where the user can select the image file they want to upload.

[0363] Step 3:

[0364] User:

[0365] The user selects an image file of a picture, then enters the necessary information (such as the child's age, areas of interest, and areas of expertise), and presses the "Submit" button.

[0366] Step 4:

[0367] Terminal:

[0368] The terminal sends data, including the selected image file and the entered information, to the server.

[0369] Step 5:

[0370] server:

[0371] The server receives the data. It saves the image file and additional information to the database and verifies that the upload was successful.

[0372] Step 6:

[0373] server:

[0374] The server invokes a generative AI model to analyze the stored image files. The AI ​​model analyzes the image files and identifies key objects.

[0375] Step 7:

[0376] server:

[0377] The server labels each object based on the analysis results and stores them in the database. For example, if the image depicts a dinosaur, it will be labeled "dinosaur".

[0378] Step 8:

[0379] server:

[0380] The server uses the analysis results and accompanying information to invoke a generative AI model to generate a story. The generative AI model automatically generates a story based on the input prompts.

[0381] Step 9:

[0382] server:

[0383] The server retrieves the generated story and customizes it based on user-provided information (age, interests, areas of expertise).

[0384] Step 10:

[0385] server:

[0386] The server adds educational content to customized stories. For example, if a picture of a dinosaur is included, it will add scientific facts and educational elements about dinosaurs.

[0387] Step 11:

[0388] server:

[0389] The server uses an emotion engine to acquire data from the user's device's camera and microphone in order to analyze the user's emotions in real time.

[0390] Step 12:

[0391] Terminal:

[0392] The device uses an emotion engine to analyze the user's facial expressions and voice tone, and generates emotion data.

[0393] Step 13:

[0394] server:

[0395] Based on feedback from the emotion engine, the server analyzes the user's emotions and recognizes feelings such as positive, negative, and surprise.

[0396] Step 14:

[0397] server:

[0398] The server adjusts the story content in real time based on the recognized emotion data. For example, if the user makes a surprised expression, the next scene will be designed to add a further element of surprise.

[0399] Step 15:

[0400] server:

[0401] Based on emotional data, the server dynamically changes the story's progression and character actions to provide content that aligns with the user's interests and emotions.

[0402] Step 16:

[0403] server:

[0404] The server converts the final integrated document data into an ebook format (EPUB or PDF) and uploads the generated ebook data to the web server.

[0405] Step 17:

[0406] server:

[0407] The server generates a link that allows the user to view the picture book online and provides the user with the link.

[0408] Step 18:

[0409] Terminal:

[0410] The terminal retrieves a link provided by the server and provides an interface that allows the user to view picture books online.

[0411] Step 19:

[0412] User:

[0413] Users can view picture books generated online. They can also click the "Order a Printed Version" button if they wish.

[0414] Step 20:

[0415] Terminal:

[0416] The terminal displays an order form for entering shipping and payment information.

[0417] Step 21:

[0418] User:

[0419] The user enters the required information and presses the "Confirm Order" button.

[0420] Step 22:

[0421] server:

[0422] The server receives order data, processes payments, and sends print data to the printer.

[0423] Step 23:

[0424] Printing company:

[0425] The printing company prints the picture book based on the received data and delivers it to the specified address.

[0426] Step 24:

[0427] server:

[0428] The server monitors the progress of the order and notifies the user when shipping is confirmed.

[0429] (Example 2)

[0430] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0431] Children today have limited opportunities to exercise their creativity in the digital age. Furthermore, educational content is delivered in a uniform manner, lacking opportunities for learning tailored to each child's individual interests and strengths. In addition, the technology for personalizing content based on user emotions is still immature, making it difficult to provide a greater sense of immersion and emotional impact. This invention aims to solve these problems by automatically generating original picture books and providing children with educationally valuable and emotionally engaging content.

[0432] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for inputting a picture drawn by a child, means for analyzing the picture and identifying the main objects, means for automatically generating a story based on the analyzed objects, means for customizing the generated story based on the child's age, interests, and areas of expertise, means for adding educational content based on the customized story, means for analyzing the user's emotions and adjusting the story in real time, means for converting the generated story into an ebook format, means for providing the generated picture book online, and means for ordering a printed version of the generated picture book. This makes it possible to generate a personalized, educational, and emotionally engaging picture book based on a child's picture and adjust it in real time.

[0433] "A means of inputting children's drawings" refers to an interface for importing children's drawings into a system in an electronic format.

[0434] "Methods for analyzing drawings and identifying key objects" refers to technologies that analyze uploaded drawings and extract and identify specific elements (such as people, animals, backgrounds, etc.) from them.

[0435] "Means for automatically generating stories based on analyzed objects" refers to an algorithm or system for creating a narrative based on identified object information.

[0436] "A means of customizing generated stories based on a child's age, interests, and strengths" refers to a function that adjusts the generated stories to correspond with the information of the child that has been entered.

[0437] "Means of adding customized story-based educational content" refers to technologies for inserting educational information and learning elements into a tailored story.

[0438] "A means of analyzing user emotions and adjusting the story in real time" refers to a system that evaluates emotional data such as the user's facial expressions and voice in real time and dynamically changes the development of the story accordingly.

[0439] "Methods for converting generated stories into ebook formats" refers to methods for converting customized stories and educational content into ebook formats (such as EPUB or PDF).

[0440] "Methods for providing generated picture books online" refers to a system that makes completed ebooks accessible to users via the internet.

[0441] "Method for ordering a printed version of the generated picture book" refers to the procedure for users to order a printed version of the generated e-book as a physical book, if they so desire, via the internet.

[0442] This invention relates to a system that automatically generates original picture books based on drawings made by children and further customizes the content by recognizing the user's emotions. Specifically, the program is implemented as follows.

[0443] First, users upload drawings made by their children using an application or website. In addition, users enter information such as the child's age, interests, and areas of expertise. This entered drawing and information are sent to the server by the device. Image files are sent in JPEG or PNG format, and the input information is sent in JSON format.

[0444] The server stores the received image data and associated information in a database. It then calls a generative AI model (e.g., TensorFlow or PyTorch) for image analysis to identify the main objects depicted in the image (people, animals, background, etc.). For example, if an image of a dinosaur is input, the server recognizes "dinosaur" as the object and stores it as a label in the database.

[0445] Based on the analysis results, the server automatically generates a story using a generative AI model. The generative AI model is given instructions using prompts. A specific example of a prompt is: "Based on the input dinosaur picture, generate a story themed around a 'dinosaur adventure' that a 6-year-old child can understand. Include content where the dinosaur discovers a new cave, finds fossils inside, and learns about different types of dinosaurs along the way." As a result, a story template is generated and saved in the database.

[0446] Next, the server customizes the generated story based on the child's age, interests, and areas of expertise. For example, if a 6-year-old child is interested in dinosaurs, scientific facts about dinosaur species and their ecology will be incorporated into the story. Furthermore, learning content will be added to enhance its educational value.

[0447] The server uses an emotion engine to analyze the user's emotions in real time. This involves analyzing facial expressions and voice through the user's device's camera and microphone to acquire emotional data. For example, if a user shows expressions of surprise or joy while viewing a picture book, the server analyzes that emotion and adds elements in the next scene that allow them to continue enjoying the surprise. Conversely, if the user expresses sadness, a positive development is introduced.

[0448] Finally, the server integrates the customized story and educational content, as well as emotionally-based adjustments, and converts it into an ebook format (EPUB or PDF). This ebook data is uploaded to a web server, and users can view it online. Users can also order printed copies of the picture book if they wish. Specifically, the server sends the data to a printer, and the printed book is delivered to the user a few days later.

[0449] Through specific data processing and calculations performed at each processing step of the program, this invention leverages children's creativity, deepens the parent-child bond, and provides highly educational content. Furthermore, by combining it with an emotion engine, it provides a more personalized experience that responds to the user's emotions, generating even greater emotion and excitement.

[0450] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0451] Step 1:

[0452] Users upload drawings made by their children through the application or website and enter information such as the child's age, interests, and areas of expertise. Specifically, the user clicks the "Upload Drawing" button in the application, selects the image file, fills in the child's information in the input form, and clicks the "Submit" button.

[0453] input:

[0454] Children's drawings (image files), information such as the child's age, interests, and areas of expertise.

[0455] output:

[0456] The entered image and information (sent to the server).

[0457] Step 2:

[0458] The terminal sends the image file uploaded by the user and the input information to the server. The image file is sent in JPEG or PNG format, and the input information is sent in JSON format.

[0459] Specific operation:

[0460] The terminal responds to user input and sends image files and JSON-formatted information to the server as an HTTP request.

[0461] input:

[0462] Image files and input information submitted by the user.

[0463] output:

[0464] Image files and input information sent to the server.

[0465] Step 3:

[0466] The server stores the received image data and associated information in a database. This allows for centralized management of all necessary information.

[0467] Specific operation:

[0468] The server temporarily stores the image file, and then stores the image file and information in JSON format in the database.

[0469] input:

[0470] Image files and input information sent from the device.

[0471] output:

[0472] Image files and information stored in the database.

[0473] Step 4:

[0474] The server invokes a generative AI model for image analysis to identify the main objects depicted in the image. Image recognition models (such as TensorFlow or PyTorch) are used for the object identification process.

[0475] Specific operation:

[0476] The server generates and inputs the saved image files into an AI model. The model analyzes the images and identifies key objects.

[0477] input:

[0478] Image files stored in the database.

[0479] output:

[0480] Identified object information (e.g., dinosaur, tree, house, etc.).

[0481] Step 5:

[0482] The server generates prompt statements based on the object analysis results and inputs them into a generative AI model (e.g., OpenAI's GPT) to generate a story. A story template is created.

[0483] Specific operation:

[0484] The server generates a prompt message. Based on the information "a picture of a dinosaur" and "age 6," it generates a prompt message such as "Please generate a story about a dinosaur on an adventure." This prompt message is input into the AI ​​model to generate the story.

[0485] input:

[0486] Identified object information, user input information.

[0487] output:

[0488] A template for the generated story.

[0489] Step 6:

[0490] The server customizes the generated story based on the child's age, interests, and areas of expertise. For example, it might incorporate scientific facts about different types of dinosaurs and their behavior into the story.

[0491] Specific operation:

[0492] The server retrieves appropriate learning elements from the educational content database based on a story template and integrates them into the story.

[0493] input:

[0494] The generated story template and user input information.

[0495] output:

[0496] A customized story.

[0497] Step 7:

[0498] The server uses an emotion engine to analyze the user's emotions in real time. It uses the user's device's camera and microphone to analyze facial expressions and voice, and acquires emotional data.

[0499] Specific operation:

[0500] The server obtains access permissions to the user's camera and microphone, and analyzes facial expressions and voice in real time. It retrieves data through an emotion analysis API and evaluates the user's emotions.

[0501] input:

[0502] Real-time facial expression and voice data of the user.

[0503] output:

[0504] User sentiment data.

[0505] Step 8:

[0506] The server adjusts the story in real time based on feedback from the emotion engine. Every time the user's emotions change, the story's progression and the characters' actions are dynamically altered.

[0507] Specific operation:

[0508] The server modifies the next scene of the story based on emotional data. If positive emotions are detected, additional adventure elements are added. If negative emotions are detected, a positive development is introduced.

[0509] input:

[0510] User sentiment data, customized stories.

[0511] output:

[0512] A story that is adjusted in real time.

[0513] Step 9:

[0514] The server ultimately integrates customized stories, educational content, and emotionally-based adjustments, and converts them into ebook formats (EPUB or PDF). This is then uploaded to a web server, generating a link that users can access online.

[0515] Specific operation:

[0516] The server converts the story into an ebook format, uploads the file to the web server, and generates a URL.

[0517] input:

[0518] A well-structured story and content.

[0519] output:

[0520] eBook format file, viewing link.

[0521] Step 10:

[0522] The server then processes orders for printed copies of the generated picture books for users who wish to receive them.

[0523] Specific operation:

[0524] When a user orders a printed version, the server sends the data to the printer and proceeds with the process of delivering the printed picture book to the user.

[0525] input:

[0526] eBook format files, user print order information.

[0527] output:

[0528] Data sent to the printer, printed picture book delivered to the user.

[0529] (Application Example 2)

[0530] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0531] Conventional picture book generation systems are limited to one-way generation based on children's drawings and cannot adjust the story in real time in response to the user's emotions. Furthermore, the content of the picture books is templated, making it difficult to maximize children's creativity and learning effectiveness. This invention aims to solve these problems and provide a more personalized experience.

[0532] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for inputting a picture drawn by a child, means for analyzing the picture and identifying the main objects, means for automatically generating a story based on the analyzed objects, means for customizing the generated story based on the child's age, interests, and areas of expertise, means for analyzing the user's emotions, means for adjusting the story in real time based on the emotion data, means for generating a picture book containing the customized story, and means for providing the generated picture book online. This makes it possible to provide an original picture book that integrates learning elements based on a child's drawing, and further makes it possible to adjust the story in real time according to the user's emotions.

[0533] "Children's drawings" refers to artwork or sketches drawn by children, whether in digital or physical form.

[0534] "Analysis" refers to the process of breaking down images and data to understand their contents and identify specific objects and attributes.

[0535] "Key objects" refer to elements identified through image analysis that play a crucial role in the story's plot and characters.

[0536] "Methods for automatically generating stories" refer to technologies such as artificial intelligence that construct narratives based on input data and generate text and graphics.

[0537] "Customization" refers to the process of changing or adapting content and functions to suit the preferences and requirements of a specific user.

[0538] "User emotions" refer to the psychological state and feedback perceived through the user's facial expressions, tone of voice, and other similar actions.

[0539] "Emotional data" refers to digital information that indicates a user's emotional state, acquired using devices such as cameras and microphones.

[0540] "Adjusting in real time" refers to the process of making immediate changes and optimizations in response to the situation and data at hand.

[0541] "Means of generating picture books" refers to systems and software that convert stories, including text and illustrations, into digital or physical book formats.

[0542] "Methods of providing content online" refers to methods of delivering ebooks and digital content to users using the internet.

[0543] This invention describes a system that generates original picture books based on drawings made by children and customizes the content in real time based on the user's emotions. In a specific embodiment, the present invention is carried out using the following procedure and apparatus.

[0544] 1. Main hardware and software used

[0545] Smart glasses camera: Used to scan and upload children's drawings.

[0546] Server: The central component responsible for analyzing images, generating stories, analyzing emotions, and customizing picture books.

[0547] OpenCV: A library for capturing and preprocessing images.

[0548] Transformers (Generative AI Model): A library for analyzing images and generating stories.

[0549] EmotionRecognizer: Software for analyzing user emotions.

[0550] PDFGenerator: Used to generate the final picture book.

[0551] 2. Specific System Processing Overview

[0552] First, the user scans a child's drawing using smart glasses. This image is uploaded to a server. The server analyzes the image using OpenCV to identify key objects. Next, a generative AI model is used to automatically generate a story based on these objects.

[0553] The generated stories are customized based on the child's age, interests, and strengths, and educational content is also added. Next, the server uses EmotionRecognizer to analyze the user's emotions in real time and adjusts the story's progression based on the emotional data.

[0554] Finally, the customized story is generated in ebook format using PDFGenerator. This provides users with a link to view it online. Printed versions of the picture book can also be ordered upon request.

[0555] 3. Specific Examples and Examples of Prompt Statements

[0556] For example, if a child draws a picture of a dinosaur, they can use smart glasses to scan the drawing and upload it to a server. A generative AI model then generates a story themed around a dinosaur adventure. If the user shows a surprised expression, an emotion analysis engine detects this reaction and adds further surprising elements to the story. Finally, this customized picture book is delivered as an ebook.

[0557] Example of a prompt:

[0558] "Please have the generative AI model analyze the dinosaur images and generate a story."

[0559] "Use the emotion analysis engine to adjust the story's progression to match the user's emotions."

[0560] The above describes a specific embodiment for carrying out the present invention. This embodiment makes it possible to easily and quickly generate original picture books using children's drawings and to provide personalized stories that respond to emotions.

[0561] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0562] Step 1:

[0563] The user scans a child's drawing using smart glasses. The smart glasses' camera captures the drawing and generates image data. Here, the input is the image data of the drawing acquired through the smart glasses' camera, and the output is that image data.

[0564] Step 2:

[0565] The device uploads the acquired image data to the server. It uses a communication module within the smart glasses to transmit the images to the server via the internet. The input is the image data, and the output is the image data that has been sent to the server.

[0566] Step 3:

[0567] The server uses OpenCV to analyze uploaded images and identify key objects. Specifically, it extracts features of objects (e.g., dinosaurs) from the image data and identifies them. The input is the uploaded image data, and the output is a list of identified key objects.

[0568] Step 4:

[0569] The server uses a generative AI model to automatically generate a story based on identified key objects. The generative AI model uses the input key objects (e.g., dinosaurs) as plot elements and characters to create the story. The input is a list of key objects, and the output is the automatically generated story.

[0570] Step 5:

[0571] The server customizes the generated story based on the child's age, interests, and strengths. It adjusts the story content based on the child's information entered by the user and adds educational content as needed. The input consists of the automatically generated story and the child's information entered by the user, while the output is the customized story.

[0572] Step 6:

[0573] The server uses EmotionRecognizer to analyze the user's emotions in real time. Specifically, it analyzes data acquired from the smart glasses' camera and microphone to identify emotions from the user's facial expressions and tone of voice. The input is video and audio streams, and the output is the analyzed emotion data.

[0574] Step 7:

[0575] The server adjusts the story in real time based on sentiment data. For example, if a user reacts with surprise, the server adjusts the next development of the story accordingly. The input is sentiment data and a customized story, and the output is the adjusted story.

[0576] Step 8:

[0577] The server converts the customized and adjusted story into an ebook format using PDFGenerator. Specifically, it integrates the story text and associated images and saves them as EPUB or PDF files. The input is the adjusted story, and the output is the generated ebook.

[0578] Step 9:

[0579] The server uploads the generated ebook to a web server and generates a link that users can access online. It also allows users to order a printed version of the picture book if desired. The input is the generated ebook, and the output is the online link and the option to order a print copy.

[0580] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0581] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0582] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0583] [Second Embodiment]

[0584] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0585] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0586] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0587] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0588] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0589] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0590] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0591] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0592] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0593] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0594] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0595] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0596] This invention relates to a system for automatically generating original picture books based on drawings made by children. Specifically, the system generates and provides picture books through the following process.

[0597] 1. Upload the image and enter the information.

[0598] User:

[0599] Users upload drawings made by their children using the application or website. Users also enter necessary information such as the child's age, interests, and areas of expertise.

[0600] Terminal:

[0601] The device sends the image file uploaded by the user and any additional information entered to the server.

[0602] 2. Receiving and analyzing image data

[0603] server:

[0604] The server receives the uploaded image data and accompanying information and stores them in a database. Next, the server invokes a generative AI model for image analysis to identify the main objects depicted in the image (people, animals, background, etc.).

[0605] Specific example:

[0606] For example, if a child draws a picture of a dinosaur, the server recognizes the picture as a dinosaur object and saves it in the database with the label "dinosaur".

[0607] 3. Automatic story generation

[0608] server:

[0609] The server automatically generates a story using a generative AI model based on the analysis results. The generated story uses objects depicted in illustrations as plot elements and characters.

[0610] Specific example:

[0611] If a picture of a dinosaur is entered, the server will generate a story themed around "dinosaur adventures," depicting the challenges the dinosaur faces and how it overcomes them.

[0612] 4. Customization and Addition of Educational Content

[0613] server:

[0614] The server customizes the generated stories based on the child's age, interests, and areas of expertise. Furthermore, it adds scientific facts and learning elements to enhance their educational value.

[0615] Specific example:

[0616] If a 6-year-old child is interested in dinosaurs, the server will incorporate scientific facts about dinosaur species and their ecology into the story. For example, a scenario could be created where a dinosaur discovers a new cave and finds fossils within, incorporating learning about different dinosaur species along the way.

[0617] 5. Generation and provision of picture book data

[0618] server:

[0619] The server converts the customized story into an ebook format (such as EPUB or PDF) and provides a link for users to view it online. Printed versions of the picture book can also be ordered if desired.

[0620] Specific example:

[0621] Users can browse picture books based on children's drawings online and also order printed copies. When a user orders a printed copy, the server sends the data to the printer, and the printed book is delivered a few days later.

[0622] The system of this invention fosters children's creativity, deepens the parent-child bond, and provides educational value. This system is highly effective in quickly and easily generating original picture books and expanding learning opportunities.

[0623] The following describes the processing flow.

[0624] Step 1:

[0625] User:

[0626] The user opens an application or website for generating picture books and clicks the "Upload Picture" button.

[0627] Step 2:

[0628] Terminal:

[0629] The terminal displays a file selection window, providing a screen where the user can select the image file they want to upload.

[0630] Step 3:

[0631] User:

[0632] The user selects an image file of a picture, enters the necessary information (such as the child's age, areas of interest, and areas of expertise), and presses the "Submit" button.

[0633] Step 4:

[0634] Terminal:

[0635] The terminal sends data, including the selected image file and the entered information, to the server.

[0636] Step 5:

[0637] server:

[0638] The server receives the data. It saves the image file and additional information to the database and verifies that the upload was successful.

[0639] Step 6:

[0640] server:

[0641] The server invokes a generative AI model to analyze the stored image files. The AI ​​model analyzes the image files and identifies key objects.

[0642] Step 7:

[0643] server:

[0644] The server labels each object based on the analysis results and stores them in the database. For example, if the picture depicts a dinosaur, it will be labeled "dinosaur".

[0645] Step 8:

[0646] server:

[0647] The server uses the analysis results and accompanying information to invoke a generative AI model to generate a story. The generative AI model automatically generates a story based on the input prompts.

[0648] Step 9:

[0649] server:

[0650] The server retrieves the generated story and customizes it based on user-provided information (age, interests, areas of expertise).

[0651] Step 10:

[0652] server:

[0653] The server adds educational content to customized stories. For example, if a picture of a dinosaur is included, it will add scientific facts and edutainment elements about dinosaurs.

[0654] Step 11:

[0655] server:

[0656] The server integrates the completed story and educational content and converts it into an ebook format (EPUB or PDF).

[0657] Step 12:

[0658] server:

[0659] The server uploads the generated ebook data to the web server and generates a link that allows users to view it online.

[0660] Step 13:

[0661] Terminal:

[0662] The terminal retrieves a link provided by the server and provides an interface that allows the user to view picture books online.

[0663] Step 14:

[0664] User:

[0665] Users can view and check the content of picture books generated online.

[0666] Step 15:

[0667] User:

[0668] If the user wishes, they can click the "Order Printed Version" button on the online interface.

[0669] Step 16:

[0670] Terminal:

[0671] The terminal displays an order form for entering shipping and payment information.

[0672] Step 17:

[0673] User:

[0674] The user enters the required information and presses the "Confirm Order" button.

[0675] Step 18:

[0676] server:

[0677] The server receives order data, processes payments, and sends print data to the printer.

[0678] Step 19:

[0679] Printing company:

[0680] The printing company prints the picture book based on the received data and delivers it to the specified address.

[0681] Step 20:

[0682] server:

[0683] The server monitors the progress of the order and notifies the user when shipping is confirmed.

[0684] (Example 1)

[0685] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0686] Currently, there is no system on the market that automatically generates original picture books using children's drawings. This means that parents and educators lack effective tools to further cultivate children's creativity and enhance their educational value. Furthermore, there are limited means of easily generating and providing high-quality picture books. Given this situation, there is a need for a system that can easily generate original picture books based on children's drawings while also adding educational value.

[0687] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0688] In this invention, the server includes means for storing children's drawings in a database, means for analyzing the stored drawings using a generation AI model to identify key objects, and means for automatically generating a story based on the identified objects. This enables the automatic generation of original picture books based on children's drawings and allows for customization that includes educational elements.

[0689] "Children's drawings" refer to illustrations and paintings created by children using their own creativity.

[0690] "Means of input" refer to the functions or interfaces that allow users to provide data to the system.

[0691] "Means of uploading" refers to the functions or interfaces that allow users to transfer data stored on their local storage to a server via the internet.

[0692] "Means of transmission" refers to the functions and protocols used to send data from a terminal to a server.

[0693] A "server" is a computer system that stores and processes data over a network.

[0694] "Means of saving to a database" refers to functions and systems for storing and managing received data in an organized format.

[0695] A "generative AI model" is an artificial intelligence algorithm used to learn from large amounts of data and generate new data.

[0696] "Key objects" refer to important elements or components present in an image that are identified through image analysis.

[0697] "Methods for automatically generating stories" refers to functions that use generative AI models to create narratives based on specified objects and information.

[0698] "Customization options" refer to features that allow you to adjust and modify the content of stories generated based on a child's age, interests, and strengths.

[0699] An "eBook format" refers to a standard format for saving and displaying books in digital format (e.g., EPUB, PDF).

[0700] "Means of providing content online" refers to functions that make generated content accessible to users via the internet.

[0701] "Educational content" refers to information and materials intended to improve learning and knowledge.

[0702] A "printed version" refers to digital content printed on physical paper.

[0703] This invention relates to a system that automatically generates original picture books based on drawings made by children. The system includes a process in which a user uploads a child's drawing, a server analyzes the drawing, generates and customizes a story, and finally delivers it as an ebook.

[0704] Hardware and software to be used

[0705] Hardware:

[0706] Server: A high-performance computer used for data storage, analysis, and execution of generated AI models.

[0707] Device: A PC or smartphone used by users to upload images and information.

[0708] software:

[0709] Web application or mobile application: Provides a user interface.

[0710] Database: A data management system for storing received images and information (e.g., MySQL, PostgreSQL).

[0711] Generative AI models: AI algorithms used for image analysis and story generation (e.g., TensorFlow, PyTorch, GPT-3).

[0712] Image processing libraries: Used for image preprocessing (e.g., OpenCV).

[0713] eBook generation tools: Tools for converting generated stories into eBook formats (e.g., Calibre).

[0714] Program Processing Overview

[0715] User:

[0716] Users upload drawings made by their children to the application or website and enter information such as the child's age, interests, and areas of expertise. The uploaded data is sent from the device to the server. For example:

[0717] Click the upload button and select your image file.

[0718] Enter your age, interests, and areas of expertise in the text fields.

[0719] Click the "Submit Data" button.

[0720] Terminal:

[0721] The terminal receives the image file and input information uploaded by the user, packets them, and sends them to the server using the HTTPS protocol. For example:

[0722] The image file and input information are saved to temporary storage.

[0723] The packet should include data and necessary metadata (user ID, timestamp, etc.).

[0724] Generate a POST request and send the data.

[0725] server:

[0726] The server performs the following actions:

[0727] 1. Save the received image data and additional information to the database.

[0728] 2. Analyze the image using a generative AI model to identify the main objects.

[0729] 3. Based on the identified object, a story is automatically generated that aligns with the prompt text.

[0730] Specific example:

[0731] If a child draws a picture of a dinosaur, the server uses a generative AI model to identify the dinosaur and generates the prompt "Generate a dinosaur adventure story." A story is then generated based on this prompt.

[0732] Next, the generated story is customized based on the child's age, interests, and areas of expertise. Furthermore, educational content (e.g., information about dinosaur species and their ecology) is incorporated into the story.

[0733] After the story is completed, it will be converted into an ebook format (e.g., EPUB, PDF), and users will be provided with a link to view the picture book online. Printed copies can also be ordered upon request, in which case the data will be sent to the printing company.

[0734] This invention enables the rapid and easy creation of picture books that foster children's creativity, deepen parent-child bonds, and provide educational value.

[0735] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0736] Step 1:

[0737] User:

[0738] Users access the application or website and click the "Upload Picture" button on the screen. Next, they select and upload an image file of a picture drawn by their child from their local storage. They also enter necessary information such as the child's age, interests, and areas of expertise.

[0739] Specific actions:

[0740] Use a web browser or smartphone app.

[0741] Select the image file and perform the upload operation.

[0742] Enter the required information and press the submit button.

[0743] Inputs and outputs:

[0744] Input: Image files of drawings made by the child, information about the child's age, interests, and areas of expertise.

[0745] Output: The uploaded image file and input information are sent to the device.

[0746] Step 2:

[0747] Terminal:

[0748] The device temporarily stores the image file uploaded by the user and any additional information entered, and then sends it to the server using the HTTPS protocol.

[0749] Specific actions:

[0750] Uploaded images and input information are saved to temporary storage.

[0751] The packet should include data and necessary metadata (user ID, timestamp, etc.).

[0752] Generate a POST request to the server and send the data.

[0753] Inputs and outputs:

[0754] Input: Image files submitted by the user, age, interests, and areas of expertise.

[0755] Output: Packetized data is sent to the server.

[0756] Step 3:

[0757] server:

[0758] The server extracts image data and input information from the received packets and stores them in a database. Next, it calls a generative AI model to analyze the stored image data and identify key objects.

[0759] Specific actions:

[0760] The received data is converted to the appropriate format and saved to the database.

[0761] Image features are extracted using a generative AI model (e.g., TensorFlow or PyTorch).

[0762] Identify and label the main objects based on the features.

[0763] Inputs and outputs:

[0764] Input: Image file and input information sent from the terminal.

[0765] Output: Analysis results stored in the database, labels of identified key objects.

[0766] Step 4:

[0767] server:

[0768] Based on the analysis results, the server calls a generative AI model for story generation (e.g., GPT-3) to generate prompt text. The story is then automatically generated based on the generated prompt text.

[0769] Specific actions:

[0770] The labels extracted from the analysis results are incorporated into the prompt message (e.g., "Generate a dinosaur adventure story").

[0771] The prompt text is input into the AI ​​model to generate a story.

[0772] Save the generated story to the database.

[0773] Inputs and outputs:

[0774] Input: Labels of the identified main objects.

[0775] Output: The generated story.

[0776] Step 5:

[0777] server:

[0778] The server customizes the generated story based on the child's age, interests, and areas of expertise. At the same time, it incorporates educational content (e.g., scientific facts and learning elements) into the story.

[0779] Specific actions:

[0780] The story content is adjusted based on the user's age, interests, and areas of expertise.

[0781] Retrieve the relevant educational content from the database and add it to the story.

[0782] Inputs and outputs:

[0783] Input: Generated story, user-entered age, interests, and areas of expertise.

[0784] Output: Customized story.

[0785] Step 6:

[0786] server:

[0787] The server converts customized stories into ebook formats (e.g., EPUB, PDF). Users are provided with a link to view the stories online. Printed versions can also be ordered upon request.

[0788] Specific actions:

[0789] Call an API (e.g., Calibre) to convert the story into an ebook format.

[0790] The generated ebook format file is saved to cloud storage (e.g., AWS S3), and a link is generated.

[0791] When a print order is placed, the order data is sent to the printing company's system.

[0792] Inputs and outputs:

[0793] Input: A customized story.

[0794] Output: ebook format file, online viewing link, print order data.

[0795] (Application Example 1)

[0796] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0797] In the modern era, there is a lack of systems that can easily provide educational and enjoyable content while fostering children's creativity. Furthermore, while there is a wealth of digitized learning content, systems that generate original stories based on children's own drawings still do not exist. Therefore, there is a need for a system that can easily provide educational content tailored to each child's individual interests and age.

[0798] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0799] In this invention, the server includes means for inputting a drawing made by a child, means for analyzing the drawing and identifying the main objects, means for automatically generating a story based on the analyzed objects, means for customizing the generated story based on the child's age, interests, and areas of expertise, means for generating the picture book in ebook format, means for providing the generated ebook online and enabling orders for printed versions, and means for using a generation AI model throughout the entire picture book generation process. This makes it possible for children themselves to become creators and automatically generate and provide customized educational content tailored to their interests and age. Furthermore, since it can be provided in ebook format and used as a printed material, it can meet a variety of needs.

[0800] "A means of inputting children's drawings" refers to a system that allows children to upload illustrations and drawings they have drawn on paper or on digital devices to the system via devices such as smartphones and computers.

[0801] "The means of analyzing the drawn image and identifying the main objects" refers to a system that uses an image analysis algorithm to recognize the main elements such as people, animals, and backgrounds from the uploaded image and register them in a database.

[0802] "Means for automatically generating a story based on the analyzed objects" refers to a mechanism that automatically creates a story by setting relevant plots and characters using a generation AI model based on recognized objects.

[0803] "Means for customizing the generated story based on the child's age, interests, and areas of expertise" refers to a mechanism that optimizes and individually customizes an automatically generated story according to the input data on the child's age, interests, and areas of expertise.

[0804] "A method for generating picture books in ebook format" refers to a system that converts customized stories into digital formats (such as EPUB or PDF) and generates them as ebooks.

[0805] "The means of providing the generated e-books online and enabling orders for printed versions" refers to a system that provides the generated e-books to users via the internet and also accepts orders for printed versions according to the user's wishes.

[0806] "Means of using a generative AI model throughout the entire picture book generation process" refers to a system that uses a generative AI model throughout the entire process, from image analysis to storytelling, to achieve a high level of automation and personalization.

[0807] The system of this invention automatically generates original picture books based on drawings made by children and provides them online. This system includes a series of processes in which the user inputs a drawing made by a child, the system analyzes the drawing to generate a story, and the system provides the story in ebook format.

[0808] The entire system consists of a smartphone application and a server. The primary hardware used is a smartphone and a server. The software includes a generative AI model (TensorFlow), Flask (a server-side web application framework), and React Native (a mobile application framework).

[0809] 1. User input and image upload

[0810] The user opens a smartphone application and takes a picture of or uploads a drawing made by their child. Along with the image file of the drawing, they also enter information such as the child's age, interests, and areas of expertise. This data is sent from the smartphone to the server.

[0811] 2. Image analysis and database storage on the server.

[0812] The server analyzes the received image file using a generation AI model (using TensorFlow). As a result of the image analysis, the main objects (e.g., people, animals, background) are identified. The analysis results are stored in a database (e.g., AWS RDS).

[0813] 3. Story Generation

[0814] The server automatically generates a story using a generative AI model based on the analyzed objects. In this generation process, appropriate plots and characters are set based on the uploaded image. For example, if an image of a dinosaur is uploaded, a story themed around "dinosaur adventures" will be generated.

[0815] 4. Customization and addition of educational elements

[0816] The generated stories are customized based on the child's age and interests. Furthermore, educational elements (e.g., scientific facts and learning elements) are added. For example, a story about a dinosaur discovering a new cave might incorporate information about different types of dinosaurs and their behavior.

[0817] 5. Generation and provision of picture book ebooks

[0818] The customized story is converted into an ebook format (e.g., EPUB or PDF). Users can view the generated ebook online. Printed versions can also be ordered if desired. The server sends the data to the printer, and the printed picture book is delivered to the user within a few days.

[0819] Examples of specific cases and prompt statements

[0820] As a concrete example, using a child's drawing of a dinosaur as a reference, use the following prompt:

[0821] I've uploaded a drawing of a dinosaur made by a child. Based on this drawing, please generate an adventure story in which the dinosaur discovers a new cave and finds fossils inside. Also, please add information to the story that teaches about different types of dinosaurs and their ecology.

[0822] This system allows children to become creators themselves, automatically generating and providing customized educational content tailored to their interests and age. It can also be provided in e-book format, and print versions are available upon request, accommodating diverse needs.

[0823] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0824] Step 1:

[0825] The user opens a smartphone application and takes a picture of their child's drawing or uploads it within the application. They also enter information such as the child's age, interests, and areas of expertise. This data is sent from the smartphone to the server. The smartphone sends the image file and text data via an HTTP request.

[0826] Input: Image file of a child's drawing, child's age, interests, and areas of expertise.

[0827] Output: Image files and text data sent to the server

[0828] Step 2:

[0829] The server stores the received image files and analyzes them using a generating AI model. Specifically, it uses an image analysis algorithm (TensorFlow) to identify key objects (people, animals, background, etc.) and saves the analysis results to a database.

[0830] Input: Image file, text data

[0831] Output: List of analyzed objects, saved to database.

[0832] Specific actions:

[0833] Receiving and saving image files

[0834] Image analysis using generative AI models

[0835] Identifying the main objects

[0836] Save the analysis results to a database (e.g., AWS RDS).

[0837] Step 3:

[0838] The server automatically generates stories using a generative AI model based on the information from the analyzed objects. Data such as age, interests, and areas of expertise entered by the user are also used at this stage.

[0839] Input: List of parsed objects, child's age, interests, areas of expertise

[0840] Output: Auto-generated story

[0841] Specific actions:

[0842] Combine the analysis results and additional information to generate a prompt message.

[0843] Input prompt text into the generative AI model and generate a story.

[0844] Step 4:

[0845] The server customizes automatically generated stories based on the child's age, interests, and strengths, and adds educational elements. For example, it incorporates scientific facts and learning elements into the stories.

[0846] Input: Auto-generated story, child's age, interests, areas of expertise

[0847] Output: Customized story

[0848] Specific actions:

[0849] Add educational elements to automatically generated stories.

[0850] Optimize the story based on the child's age and interests.

[0851] Step 5:

[0852] The server converts the customized story into an ebook format (e.g., EPUB or PDF). It then generates a link to provide the generated ebook online and notifies the user.

[0853] Input: Customized story

[0854] Output: ebook file, online link

[0855] Specific actions:

[0856] Convert story to ebook format

[0857] Generate a link for online viewing.

[0858] Notify the user of the link (e.g., send an email).

[0859] Step 6:

[0860] Users can view online-generated ebooks or, if desired, order printed versions. The server sends the print order data to the printer and arranges for the printing and delivery of the picture books.

[0861] Input: User's print order

[0862] Output: Printed picture book

[0863] Specific actions:

[0864] Received print order information.

[0865] Send order data to the printing company

[0866] A printing company prints the picture book and delivers it to the user.

[0867] This process allows for the easy creation of original picture books that leverage children's creativity, and these books are available in both electronic and print formats.

[0868] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0869] This invention relates to a system that automatically generates original picture books based on drawings made by children and further customizes the content by recognizing the user's emotions. Specifically, the invention describes a form that generates and provides picture books through the following process and combines it with emotion recognition.

[0870] 1. Upload the image and enter the information.

[0871] User:

[0872] Users upload drawings made by their children using the application or website. Users also enter necessary information such as the child's age, interests, and areas of expertise.

[0873] Terminal:

[0874] The device sends the image file uploaded by the user and any additional information entered to the server.

[0875] 2. Receiving and analyzing image data

[0876] server:

[0877] The server receives the uploaded image data and accompanying information and stores them in a database. Next, the server invokes a generative AI model for image analysis to identify the main objects depicted in the image (people, animals, background, etc.).

[0878] Specific example:

[0879] For example, if a child draws a picture of a dinosaur, the server recognizes the picture as a dinosaur object and saves it in the database with the label "dinosaur".

[0880] 3. Automatic story generation

[0881] server:

[0882] The server automatically generates a story using a generative AI model based on the analysis results. The generated story uses objects depicted in illustrations as plot elements and characters.

[0883] Specific example:

[0884] If a picture of a dinosaur is entered, the server will generate a story themed around "dinosaur adventures," depicting the challenges the dinosaur faces and how it overcomes them.

[0885] 4. Customization and Addition of Educational Content

[0886] server:

[0887] The server customizes the generated stories based on the child's age, interests, and areas of expertise. Furthermore, it adds scientific facts and learning elements to enhance their educational value.

[0888] Specific example:

[0889] If a 6-year-old child is interested in dinosaurs, the server will incorporate scientific facts about dinosaur species and their ecology into the story. For example, a scenario could be created where a dinosaur discovers a new cave and finds fossils within, incorporating learning about different dinosaur species along the way.

[0890] 5. Utilizing the Emotion Engine

[0891] server:

[0892] The server uses an emotion engine to analyze the user's emotions in real time. The emotion engine analyzes the user's facial expressions and voice through the camera and microphone to acquire emotional data.

[0893] Specific example:

[0894] While the user is browsing the picture book, the server analyzes the user's facial expressions through an emotion engine and recognizes emotions such as surprise and joy.

[0895] 6. Emotion-based real-time story adjustment

[0896] server:

[0897] The server adjusts the story content in real time based on feedback from the emotion engine. The story's progression and character actions change dynamically each time the user's emotions shift.

[0898] Specific example:

[0899] If the user shows a surprised expression in response to a story development, the server will add elements in the next scene that allow them to continue enjoying the surprise. Conversely, if the emotional data indicates sadness, the server will adjust the story and introduce a more positive development.

[0900] 7. Generation and provision of picture book data

[0901] server:

[0902] The server integrates customized stories and educational content, as well as emotionally-based adjustments, and converts them into ebook formats (EPUB and PDF).

[0903] server:

[0904] The server uploads the generated ebook data to the web server and generates a link that allows users to view it online. Printed versions of the picture books can also be ordered if desired.

[0905] Specific example:

[0906] Users can browse picture books based on children's drawings online and also order printed copies. When a user orders a printed copy, the server sends the data to the printer, and the printed book is delivered a few days later.

[0907] The system of this invention fosters children's creativity, deepens the parent-child bond, and provides educational value. Furthermore, by combining it with an emotion engine, it provides a more personalized experience that responds to the user's emotions, generating even greater emotion and excitement. This system is highly effective not only in quickly and easily generating original picture books and expanding learning opportunities, but also in providing a dynamic experience that responds to emotions.

[0908] The following describes the processing flow.

[0909] Step 1:

[0910] User:

[0911] The user opens an application or website for generating picture books and clicks the "Upload Picture" button.

[0912] Step 2:

[0913] Terminal:

[0914] The terminal displays a file selection window, providing a screen where the user can select the image file they want to upload.

[0915] Step 3:

[0916] User:

[0917] The user selects an image file of a picture, then enters the necessary information (such as the child's age, areas of interest, and areas of expertise), and presses the "Submit" button.

[0918] Step 4:

[0919] Terminal:

[0920] The terminal sends data, including the selected image file and the entered information, to the server.

[0921] Step 5:

[0922] server:

[0923] The server receives the data. It saves the image file and additional information to the database and verifies that the upload was successful.

[0924] Step 6:

[0925] server:

[0926] The server invokes a generative AI model to analyze the stored image files. The AI ​​model analyzes the image files and identifies key objects.

[0927] Step 7:

[0928] server:

[0929] The server labels each object based on the analysis results and stores them in the database. For example, if the image depicts a dinosaur, it will be labeled "dinosaur".

[0930] Step 8:

[0931] server:

[0932] The server uses the analysis results and accompanying information to invoke a generative AI model to generate a story. The generative AI model automatically generates a story based on the input prompts.

[0933] Step 9:

[0934] server:

[0935] The server retrieves the generated story and customizes it based on user-provided information (age, interests, areas of expertise).

[0936] Step 10:

[0937] server:

[0938] The server adds educational content to customized stories. For example, if a picture of a dinosaur is included, it will add scientific facts and educational elements about dinosaurs.

[0939] Step 11:

[0940] server:

[0941] The server uses an emotion engine to acquire data from the user's device's camera and microphone in order to analyze the user's emotions in real time.

[0942] Step 12:

[0943] Terminal:

[0944] The device uses an emotion engine to analyze the user's facial expressions and voice tone, and generates emotion data.

[0945] Step 13:

[0946] server:

[0947] Based on feedback from the emotion engine, the server analyzes the user's emotions and recognizes feelings such as positive, negative, and surprise.

[0948] Step 14:

[0949] server:

[0950] The server adjusts the story content in real time based on the recognized emotion data. For example, if the user makes a surprised expression, the next scene will be designed to add a further element of surprise.

[0951] Step 15:

[0952] server:

[0953] Based on emotional data, the server dynamically changes the story's progression and character actions to provide content that aligns with the user's interests and emotions.

[0954] Step 16:

[0955] server:

[0956] The server converts the final integrated document data into an ebook format (EPUB or PDF) and uploads the generated ebook data to the web server.

[0957] Step 17:

[0958] server:

[0959] The server generates a link that allows the user to view the picture book online and provides the user with the link.

[0960] Step 18:

[0961] Terminal:

[0962] The terminal retrieves a link provided by the server and provides an interface that allows the user to view picture books online.

[0963] Step 19:

[0964] User:

[0965] Users can view picture books generated online. They can also click the "Order a Printed Version" button if they wish.

[0966] Step 20:

[0967] Terminal:

[0968] The terminal displays an order form for entering shipping and payment information.

[0969] Step 21:

[0970] User:

[0971] The user enters the required information and presses the "Confirm Order" button.

[0972] Step 22:

[0973] server:

[0974] The server receives order data, processes payments, and sends print data to the printer.

[0975] Step 23:

[0976] Printing company:

[0977] The printing company prints the picture book based on the received data and delivers it to the specified address.

[0978] Step 24:

[0979] server:

[0980] The server monitors the progress of the order and notifies the user when shipping is confirmed.

[0981] (Example 2)

[0982] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0983] Children today have limited opportunities to exercise their creativity in the digital age. Furthermore, educational content is delivered in a uniform manner, lacking opportunities for learning tailored to each child's individual interests and strengths. In addition, the technology for personalizing content based on user emotions is still immature, making it difficult to provide a greater sense of immersion and emotional impact. This invention aims to solve these problems by automatically generating original picture books and providing children with educationally valuable and emotionally engaging content.

[0984] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for inputting a picture drawn by a child, means for analyzing the picture and identifying the main objects, means for automatically generating a story based on the analyzed objects, means for customizing the generated story based on the child's age, interests, and areas of expertise, means for adding educational content based on the customized story, means for analyzing the user's emotions and adjusting the story in real time, means for converting the generated story into an ebook format, means for providing the generated picture book online, and means for ordering a printed version of the generated picture book. This makes it possible to generate a personalized, educational, and emotionally engaging picture book based on a child's picture and adjust it in real time.

[0985] "A means of inputting children's drawings" refers to an interface for importing children's drawings into a system in an electronic format.

[0986] "Methods for analyzing drawings and identifying key objects" refers to technologies that analyze uploaded drawings and extract and identify specific elements (such as people, animals, backgrounds, etc.) from them.

[0987] "Means for automatically generating stories based on analyzed objects" refers to an algorithm or system for creating a narrative based on identified object information.

[0988] "A means of customizing generated stories based on a child's age, interests, and strengths" refers to a function that adjusts the generated stories to correspond with the information of the child that has been entered.

[0989] "Means of adding customized story-based educational content" refers to technologies for inserting educational information and learning elements into a tailored story.

[0990] "A means of analyzing user emotions and adjusting the story in real time" refers to a system that evaluates emotional data such as the user's facial expressions and voice in real time and dynamically changes the development of the story accordingly.

[0991] "Methods for converting generated stories into ebook formats" refers to methods for converting customized stories and educational content into ebook formats (such as EPUB or PDF).

[0992] "Methods for providing generated picture books online" refers to a system that makes completed ebooks accessible to users via the internet.

[0993] "Method for ordering a printed version of the generated picture book" refers to the procedure for users to order a printed version of the generated e-book as a physical book, if they so desire, via the internet.

[0994] This invention relates to a system that automatically generates original picture books based on drawings made by children and further customizes the content by recognizing the user's emotions. Specifically, the program is implemented as follows.

[0995] First, users upload drawings made by their children using an application or website. In addition, users enter information such as the child's age, interests, and areas of expertise. This entered drawing and information are sent to the server by the device. Image files are sent in JPEG or PNG format, and the input information is sent in JSON format.

[0996] The server stores the received image data and associated information in a database. It then calls a generative AI model (e.g., TensorFlow or PyTorch) for image analysis to identify the main objects depicted in the image (people, animals, background, etc.). For example, if an image of a dinosaur is input, the server recognizes "dinosaur" as the object and stores it as a label in the database.

[0997] Based on the analysis results, the server automatically generates a story using a generative AI model. The generative AI model is given instructions using prompts. A specific example of a prompt is: "Based on the input dinosaur picture, generate a story themed around a 'dinosaur adventure' that a 6-year-old child can understand. Include content where the dinosaur discovers a new cave, finds fossils inside, and learns about different types of dinosaurs along the way." As a result, a story template is generated and saved in the database.

[0998] Next, the server customizes the generated story based on the child's age, interests, and areas of expertise. For example, if a 6-year-old child is interested in dinosaurs, scientific facts about dinosaur species and their ecology will be incorporated into the story. Furthermore, learning content will be added to enhance its educational value.

[0999] The server uses an emotion engine to analyze the user's emotions in real time. This involves analyzing facial expressions and voice through the user's device's camera and microphone to acquire emotional data. For example, if a user shows expressions of surprise or joy while viewing a picture book, the server analyzes that emotion and adds elements in the next scene that allow them to continue enjoying the surprise. Conversely, if the user expresses sadness, a positive development is introduced.

[1000] Finally, the server integrates the customized story and educational content, as well as emotionally-based adjustments, and converts it into an ebook format (EPUB or PDF). This ebook data is uploaded to a web server, and users can view it online. Users can also order printed copies of the picture book if they wish. Specifically, the server sends the data to a printer, and the printed book is delivered to the user a few days later.

[1001] Through specific data processing and calculations performed at each processing step of the program, this invention leverages children's creativity, deepens the parent-child bond, and provides highly educational content. Furthermore, by combining it with an emotion engine, it provides a more personalized experience that responds to the user's emotions, generating even greater emotion and excitement.

[1002] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1003] Step 1:

[1004] Users upload drawings made by their children through the application or website and enter information such as the child's age, interests, and areas of expertise. Specifically, the user clicks the "Upload Drawing" button in the application, selects the image file, fills in the child's information in the input form, and clicks the "Submit" button.

[1005] input:

[1006] Children's drawings (image files), information such as the child's age, interests, and areas of expertise.

[1007] output:

[1008] The entered image and information (sent to the server).

[1009] Step 2:

[1010] The terminal sends the image file uploaded by the user and the input information to the server. The image file is sent in JPEG or PNG format, and the input information is sent in JSON format.

[1011] Specific operation:

[1012] The terminal responds to user input and sends image files and JSON-formatted information to the server as an HTTP request.

[1013] input:

[1014] Image files and input information submitted by the user.

[1015] output:

[1016] Image files and input information sent to the server.

[1017] Step 3:

[1018] The server stores the received image data and associated information in a database. This allows for centralized management of all necessary information.

[1019] Specific operation:

[1020] The server temporarily stores the image file, and then stores the image file and information in JSON format in the database.

[1021] input:

[1022] Image files and input information sent from the device.

[1023] output:

[1024] Image files and information stored in the database.

[1025] Step 4:

[1026] The server invokes a generative AI model for image analysis to identify the main objects depicted in the image. Image recognition models (such as TensorFlow or PyTorch) are used for the object identification process.

[1027] Specific operation:

[1028] The server generates and inputs the saved image files into an AI model. The model analyzes the images and identifies key objects.

[1029] input:

[1030] Image files stored in the database.

[1031] output:

[1032] Identified object information (e.g., dinosaur, tree, house, etc.).

[1033] Step 5:

[1034] The server generates prompt statements based on the object analysis results and inputs them into a generative AI model (e.g., OpenAI's GPT) to generate a story. A story template is created.

[1035] Specific operation:

[1036] The server generates a prompt message. Based on the information "a picture of a dinosaur" and "age 6," it generates a prompt message such as "Please generate a story about a dinosaur on an adventure." This prompt message is input into the AI ​​model to generate the story.

[1037] input:

[1038] Identified object information, user input information.

[1039] output:

[1040] A template for the generated story.

[1041] Step 6:

[1042] The server customizes the generated story based on the child's age, interests, and areas of expertise. For example, it might incorporate scientific facts about different types of dinosaurs and their behavior into the story.

[1043] Specific operation:

[1044] The server retrieves appropriate learning elements from the educational content database based on a story template and integrates them into the story.

[1045] input:

[1046] The generated story template and user input information.

[1047] output:

[1048] A customized story.

[1049] Step 7:

[1050] The server uses an emotion engine to analyze the user's emotions in real time. It uses the user's device's camera and microphone to analyze facial expressions and voice, and acquires emotional data.

[1051] Specific operation:

[1052] The server obtains access permissions to the user's camera and microphone, and analyzes facial expressions and voice in real time. It retrieves data through an emotion analysis API and evaluates the user's emotions.

[1053] input:

[1054] Real-time facial expression and voice data of the user.

[1055] output:

[1056] User sentiment data.

[1057] Step 8:

[1058] The server adjusts the story in real time based on feedback from the emotion engine. Every time the user's emotions change, the story's progression and the characters' actions are dynamically altered.

[1059] Specific operation:

[1060] The server modifies the next scene of the story based on emotional data. If positive emotions are detected, additional adventure elements are added. If negative emotions are detected, a positive development is introduced.

[1061] input:

[1062] User sentiment data, customized stories.

[1063] output:

[1064] A story that is adjusted in real time.

[1065] Step 9:

[1066] The server ultimately integrates customized stories, educational content, and emotionally-based adjustments, and converts them into ebook formats (EPUB or PDF). This is then uploaded to a web server, generating a link that users can access online.

[1067] Specific operation:

[1068] The server converts the story into an ebook format, uploads the file to the web server, and generates a URL.

[1069] input:

[1070] A well-structured story and content.

[1071] output:

[1072] eBook format file, viewing link.

[1073] Step 10:

[1074] The server then processes orders for printed copies of the generated picture books for users who wish to receive them.

[1075] Specific operation:

[1076] When a user orders a printed version, the server sends the data to the printer and proceeds with the process of delivering the printed picture book to the user.

[1077] input:

[1078] eBook format files, user print order information.

[1079] output:

[1080] Data sent to the printer, printed picture book delivered to the user.

[1081] (Application Example 2)

[1082] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1083] Conventional picture book generation systems are limited to one-way generation based on children's drawings and cannot adjust the story in real time in response to the user's emotions. Furthermore, the content of the picture books is templated, making it difficult to maximize children's creativity and learning effectiveness. This invention aims to solve these problems and provide a more personalized experience.

[1084] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for inputting a picture drawn by a child, means for analyzing the picture and identifying the main objects, means for automatically generating a story based on the analyzed objects, means for customizing the generated story based on the child's age, interests, and areas of expertise, means for analyzing the user's emotions, means for adjusting the story in real time based on the emotion data, means for generating a picture book containing the customized story, and means for providing the generated picture book online. This makes it possible to provide an original picture book that integrates learning elements based on a child's drawing, and further makes it possible to adjust the story in real time according to the user's emotions.

[1085] "Children's drawings" refers to artwork or sketches drawn by children, whether in digital or physical form.

[1086] "Analysis" refers to the process of breaking down images and data to understand their contents and identify specific objects and attributes.

[1087] "Key objects" refer to elements identified through image analysis that play a crucial role in the story's plot and characters.

[1088] "Methods for automatically generating stories" refer to technologies such as artificial intelligence that construct narratives based on input data and generate text and graphics.

[1089] "Customization" refers to the process of changing or adapting content and functions to suit the preferences and requirements of a specific user.

[1090] "User emotions" refer to the psychological state and feedback perceived through the user's facial expressions, tone of voice, and other similar actions.

[1091] "Emotional data" refers to digital information that indicates a user's emotional state, acquired using devices such as cameras and microphones.

[1092] "Adjusting in real time" refers to the process of making immediate changes and optimizations in response to the situation and data at hand.

[1093] "Means of generating picture books" refers to systems and software that convert stories, including text and illustrations, into digital or physical book formats.

[1094] "Methods of providing content online" refers to methods of delivering ebooks and digital content to users using the internet.

[1095] This invention describes a system that generates original picture books based on drawings made by children and customizes the content in real time based on the user's emotions. In a specific embodiment, the present invention is carried out using the following procedure and apparatus.

[1096] 1. Main hardware and software used

[1097] Smart glasses camera: Used to scan and upload children's drawings.

[1098] Server: The central component responsible for analyzing images, generating stories, analyzing emotions, and customizing picture books.

[1099] OpenCV: A library for capturing and preprocessing images.

[1100] Transformers (Generative AI Model): A library for analyzing images and generating stories.

[1101] EmotionRecognizer: Software for analyzing user emotions.

[1102] PDFGenerator: Used to generate the final picture book.

[1103] 2. Specific System Processing Overview

[1104] First, the user scans a child's drawing using smart glasses. This image is uploaded to a server. The server analyzes the image using OpenCV to identify key objects. Next, a generative AI model is used to automatically generate a story based on these objects.

[1105] The generated stories are customized based on the child's age, interests, and strengths, and educational content is also added. Next, the server uses EmotionRecognizer to analyze the user's emotions in real time and adjusts the story's progression based on the emotional data.

[1106] Finally, the customized story is generated in ebook format using PDFGenerator. This provides users with a link to view it online. Printed versions of the picture book can also be ordered upon request.

[1107] 3. Specific Examples and Examples of Prompt Statements

[1108] For example, if a child draws a picture of a dinosaur, they can use smart glasses to scan the drawing and upload it to a server. A generative AI model then generates a story themed around a dinosaur adventure. If the user shows a surprised expression, an emotion analysis engine detects this reaction and adds further surprising elements to the story. Finally, this customized picture book is delivered as an ebook.

[1109] Example of a prompt:

[1110] "Please have the generative AI model analyze the dinosaur images and generate a story."

[1111] "Use the emotion analysis engine to adjust the story's progression to match the user's emotions."

[1112] The above describes a specific embodiment for carrying out the present invention. This embodiment makes it possible to easily and quickly generate original picture books using children's drawings and to provide personalized stories that respond to emotions.

[1113] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1114] Step 1:

[1115] The user scans a child's drawing using smart glasses. The smart glasses' camera captures the drawing and generates image data. Here, the input is the image data of the drawing acquired through the smart glasses' camera, and the output is that image data.

[1116] Step 2:

[1117] The device uploads the acquired image data to the server. It uses a communication module within the smart glasses to transmit the images to the server via the internet. The input is the image data, and the output is the image data that has been sent to the server.

[1118] Step 3:

[1119] The server uses OpenCV to analyze uploaded images and identify key objects. Specifically, it extracts features of objects (e.g., dinosaurs) from the image data and identifies them. The input is the uploaded image data, and the output is a list of identified key objects.

[1120] Step 4:

[1121] The server uses a generative AI model to automatically generate a story based on identified key objects. The generative AI model uses the input key objects (e.g., dinosaurs) as plot elements and characters to create the story. The input is a list of key objects, and the output is the automatically generated story.

[1122] Step 5:

[1123] The server customizes the generated story based on the child's age, interests, and strengths. It adjusts the story content based on the child's information entered by the user and adds educational content as needed. The input consists of the automatically generated story and the child's information entered by the user, while the output is the customized story.

[1124] Step 6:

[1125] The server uses EmotionRecognizer to analyze the user's emotions in real time. Specifically, it analyzes data acquired from the smart glasses' camera and microphone to identify emotions from the user's facial expressions and tone of voice. The input is video and audio streams, and the output is the analyzed emotion data.

[1126] Step 7:

[1127] The server adjusts the story in real time based on sentiment data. For example, if a user reacts with surprise, the server adjusts the next development of the story accordingly. The input is sentiment data and a customized story, and the output is the adjusted story.

[1128] Step 8:

[1129] The server converts the customized and adjusted story into an ebook format using PDFGenerator. Specifically, it integrates the story text and associated images and saves them as EPUB or PDF files. The input is the adjusted story, and the output is the generated ebook.

[1130] Step 9:

[1131] The server uploads the generated ebook to a web server and generates a link that users can access online. It also allows users to order a printed version of the picture book if desired. The input is the generated ebook, and the output is the online link and the option to order a print copy.

[1132] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1133] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1134] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1135] [Third Embodiment]

[1136] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1137] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1138] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1139] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1140] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1141] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1142] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1143] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1144] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1145] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1146] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1147] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1148] This invention relates to a system for automatically generating original picture books based on drawings made by children. Specifically, the system generates and provides picture books through the following process.

[1149] 1. Upload the image and enter the information.

[1150] User:

[1151] Users upload drawings made by their children using the application or website. Users also enter necessary information such as the child's age, interests, and areas of expertise.

[1152] Terminal:

[1153] The device sends the image file uploaded by the user and any additional information entered to the server.

[1154] 2. Receiving and analyzing image data

[1155] server:

[1156] The server receives the uploaded image data and accompanying information and stores them in a database. Next, the server invokes a generative AI model for image analysis to identify the main objects depicted in the image (people, animals, background, etc.).

[1157] Specific example:

[1158] For example, if a child draws a picture of a dinosaur, the server recognizes the picture as a dinosaur object and saves it in the database with the label "dinosaur".

[1159] 3. Automatic story generation

[1160] server:

[1161] The server automatically generates a story using a generative AI model based on the analysis results. The generated story uses objects depicted in illustrations as plot elements and characters.

[1162] Specific example:

[1163] If a picture of a dinosaur is entered, the server will generate a story themed around "dinosaur adventures," depicting the challenges the dinosaur faces and how it overcomes them.

[1164] 4. Customization and Addition of Educational Content

[1165] server:

[1166] The server customizes the generated stories based on the child's age, interests, and areas of expertise. Furthermore, it adds scientific facts and learning elements to enhance their educational value.

[1167] Specific example:

[1168] If a 6-year-old child is interested in dinosaurs, the server will incorporate scientific facts about dinosaur species and their ecology into the story. For example, a scenario could be created where a dinosaur discovers a new cave and finds fossils within, incorporating learning about different dinosaur species along the way.

[1169] 5. Generation and provision of picture book data

[1170] server:

[1171] The server converts the customized story into an ebook format (such as EPUB or PDF) and provides a link for users to view it online. Printed versions of the picture book can also be ordered if desired.

[1172] Specific example:

[1173] Users can browse picture books based on children's drawings online and also order printed copies. When a user orders a printed copy, the server sends the data to the printer, and the printed book is delivered a few days later.

[1174] The system of this invention fosters children's creativity, deepens the parent-child bond, and provides educational value. This system is highly effective in quickly and easily generating original picture books and expanding learning opportunities.

[1175] The following describes the processing flow.

[1176] Step 1:

[1177] User:

[1178] The user opens an application or website for generating picture books and clicks the "Upload Picture" button.

[1179] Step 2:

[1180] Terminal:

[1181] The terminal displays a file selection window, providing a screen where the user can select the image file they want to upload.

[1182] Step 3:

[1183] User:

[1184] The user selects an image file of a picture, enters the necessary information (such as the child's age, areas of interest, and areas of expertise), and presses the "Submit" button.

[1185] Step 4:

[1186] Terminal:

[1187] The terminal sends data, including the selected image file and the entered information, to the server.

[1188] Step 5:

[1189] server:

[1190] The server receives the data. It saves the image file and additional information to the database and verifies that the upload was successful.

[1191] Step 6:

[1192] server:

[1193] The server invokes a generative AI model to analyze the stored image files. The AI ​​model analyzes the image files and identifies key objects.

[1194] Step 7:

[1195] server:

[1196] The server labels each object based on the analysis results and stores them in the database. For example, if the picture depicts a dinosaur, it will be labeled "dinosaur".

[1197] Step 8:

[1198] server:

[1199] The server uses the analysis results and accompanying information to invoke a generative AI model to generate a story. The generative AI model automatically generates a story based on the input prompts.

[1200] Step 9:

[1201] server:

[1202] The server retrieves the generated story and customizes it based on user-provided information (age, interests, areas of expertise).

[1203] Step 10:

[1204] server:

[1205] The server adds educational content to customized stories. For example, if a picture of a dinosaur is included, it will add scientific facts and edutainment elements about dinosaurs.

[1206] Step 11:

[1207] server:

[1208] The server integrates the completed story and educational content and converts it into an ebook format (EPUB or PDF).

[1209] Step 12:

[1210] server:

[1211] The server uploads the generated ebook data to the web server and generates a link that allows users to view it online.

[1212] Step 13:

[1213] Terminal:

[1214] The terminal retrieves a link provided by the server and provides an interface that allows the user to view picture books online.

[1215] Step 14:

[1216] User:

[1217] Users can view and check the content of picture books generated online.

[1218] Step 15:

[1219] User:

[1220] If the user wishes, they can click the "Order Printed Version" button on the online interface.

[1221] Step 16:

[1222] Terminal:

[1223] The terminal displays an order form for entering shipping and payment information.

[1224] Step 17:

[1225] User:

[1226] The user enters the required information and presses the "Confirm Order" button.

[1227] Step 18:

[1228] server:

[1229] The server receives order data, processes payments, and sends print data to the printer.

[1230] Step 19:

[1231] Printing company:

[1232] The printing company prints the picture book based on the received data and delivers it to the specified address.

[1233] Step 20:

[1234] server:

[1235] The server monitors the progress of the order and notifies the user when shipping is confirmed.

[1236] (Example 1)

[1237] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1238] Currently, there is no system on the market that automatically generates original picture books using children's drawings. This means that parents and educators lack effective tools to further cultivate children's creativity and enhance their educational value. Furthermore, there are limited means of easily generating and providing high-quality picture books. Given this situation, there is a need for a system that can easily generate original picture books based on children's drawings while also adding educational value.

[1239] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1240] In this invention, the server includes means for storing children's drawings in a database, means for analyzing the stored drawings using a generation AI model to identify key objects, and means for automatically generating a story based on the identified objects. This enables the automatic generation of original picture books based on children's drawings and allows for customization that includes educational elements.

[1241] "Children's drawings" refer to illustrations and paintings created by children using their own creativity.

[1242] "Means of input" refer to the functions or interfaces that allow users to provide data to the system.

[1243] "Means of uploading" refers to the functions or interfaces that allow users to transfer data stored on their local storage to a server via the internet.

[1244] "Means of transmission" refers to the functions and protocols used to send data from a terminal to a server.

[1245] A "server" is a computer system that stores and processes data over a network.

[1246] "Means of saving to a database" refers to functions and systems for storing and managing received data in an organized format.

[1247] A "generative AI model" is an artificial intelligence algorithm used to learn from large amounts of data and generate new data.

[1248] "Key objects" refer to important elements or components present in an image that are identified through image analysis.

[1249] "Methods for automatically generating stories" refers to functions that use generative AI models to create narratives based on specified objects and information.

[1250] "Customization options" refer to features that allow you to adjust and modify the content of stories generated based on a child's age, interests, and strengths.

[1251] An "eBook format" refers to a standard format for saving and displaying books in digital format (e.g., EPUB, PDF).

[1252] "Means of providing content online" refers to functions that make generated content accessible to users via the internet.

[1253] "Educational content" refers to information and materials intended to improve learning and knowledge.

[1254] A "printed version" refers to digital content printed on physical paper.

[1255] This invention relates to a system that automatically generates original picture books based on drawings made by children. The system includes a process in which a user uploads a child's drawing, a server analyzes the drawing, generates and customizes a story, and finally delivers it as an ebook.

[1256] Hardware and software to be used

[1257] Hardware:

[1258] Server: A high-performance computer used for data storage, analysis, and execution of generated AI models.

[1259] Device: A PC or smartphone used by users to upload images and information.

[1260] software:

[1261] Web application or mobile application: Provides a user interface.

[1262] Database: A data management system for storing received images and information (e.g., MySQL, PostgreSQL).

[1263] Generative AI models: AI algorithms used for image analysis and story generation (e.g., TensorFlow, PyTorch, GPT-3).

[1264] Image processing libraries: Used for image preprocessing (e.g., OpenCV).

[1265] eBook generation tools: Tools for converting generated stories into eBook formats (e.g., Calibre).

[1266] Program Processing Overview

[1267] User:

[1268] Users upload drawings made by their children to the application or website and enter information such as the child's age, interests, and areas of expertise. The uploaded data is sent from the device to the server. For example:

[1269] Click the upload button and select your image file.

[1270] Enter your age, interests, and areas of expertise in the text fields.

[1271] Click the "Submit Data" button.

[1272] Terminal:

[1273] The terminal receives the image file and input information uploaded by the user, packets them, and sends them to the server using the HTTPS protocol. For example:

[1274] The image file and input information are saved to temporary storage.

[1275] The packet should include data and necessary metadata (user ID, timestamp, etc.).

[1276] Generate a POST request and send the data.

[1277] server:

[1278] The server performs the following actions:

[1279] 1. Save the received image data and additional information to the database.

[1280] 2. Analyze the image using a generative AI model to identify the main objects.

[1281] 3. Based on the identified object, a story is automatically generated that aligns with the prompt text.

[1282] Specific example:

[1283] If a child draws a picture of a dinosaur, the server uses a generative AI model to identify the dinosaur and generates the prompt "Generate a dinosaur adventure story." A story is then generated based on this prompt.

[1284] Next, the generated story is customized based on the child's age, interests, and areas of expertise. Furthermore, educational content (e.g., information about dinosaur species and their ecology) is incorporated into the story.

[1285] After the story is completed, it will be converted into an ebook format (e.g., EPUB, PDF), and users will be provided with a link to view the picture book online. Printed copies can also be ordered upon request, in which case the data will be sent to the printing company.

[1286] This invention enables the rapid and easy creation of picture books that foster children's creativity, deepen parent-child bonds, and provide educational value.

[1287] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1288] Step 1:

[1289] User:

[1290] Users access the application or website and click the "Upload Picture" button on the screen. Next, they select and upload an image file of a picture drawn by their child from their local storage. They also enter necessary information such as the child's age, interests, and areas of expertise.

[1291] Specific actions:

[1292] Use a web browser or smartphone app.

[1293] Select the image file and perform the upload operation.

[1294] Enter the required information and press the submit button.

[1295] Inputs and outputs:

[1296] Input: Image files of drawings made by the child, information about the child's age, interests, and areas of expertise.

[1297] Output: The uploaded image file and input information are sent to the device.

[1298] Step 2:

[1299] Terminal:

[1300] The device temporarily stores the image file uploaded by the user and any additional information entered, and then sends it to the server using the HTTPS protocol.

[1301] Specific actions:

[1302] Uploaded images and input information are saved to temporary storage.

[1303] The packet should include data and necessary metadata (user ID, timestamp, etc.).

[1304] Generate a POST request to the server and send the data.

[1305] Inputs and outputs:

[1306] Input: Image files submitted by the user, age, interests, and areas of expertise.

[1307] Output: Packetized data is sent to the server.

[1308] Step 3:

[1309] server:

[1310] The server extracts image data and input information from the received packets and stores them in a database. Next, it calls a generative AI model to analyze the stored image data and identify key objects.

[1311] Specific actions:

[1312] The received data is converted to the appropriate format and saved to the database.

[1313] Image features are extracted using a generative AI model (e.g., TensorFlow or PyTorch).

[1314] Identify and label the main objects based on the features.

[1315] Inputs and outputs:

[1316] Input: Image file and input information sent from the terminal.

[1317] Output: Analysis results stored in the database, labels of identified key objects.

[1318] Step 4:

[1319] server:

[1320] Based on the analysis results, the server calls a generative AI model for story generation (e.g., GPT-3) to generate prompt text. The story is then automatically generated based on the generated prompt text.

[1321] Specific actions:

[1322] The labels extracted from the analysis results are incorporated into the prompt message (e.g., "Generate a dinosaur adventure story").

[1323] The prompt text is input into the AI ​​model to generate a story.

[1324] Save the generated story to the database.

[1325] Inputs and outputs:

[1326] Input: Labels of the identified main objects.

[1327] Output: The generated story.

[1328] Step 5:

[1329] server:

[1330] The server customizes the generated story based on the child's age, interests, and areas of expertise. At the same time, it incorporates educational content (e.g., scientific facts and learning elements) into the story.

[1331] Specific actions:

[1332] The story content is adjusted based on the user's age, interests, and areas of expertise.

[1333] Retrieve the relevant educational content from the database and add it to the story.

[1334] Inputs and outputs:

[1335] Input: Generated story, user-entered age, interests, and areas of expertise.

[1336] Output: Customized story.

[1337] Step 6:

[1338] server:

[1339] The server converts customized stories into ebook formats (e.g., EPUB, PDF). Users are provided with a link to view the stories online. Printed versions can also be ordered upon request.

[1340] Specific actions:

[1341] Call an API (e.g., Calibre) to convert the story into an ebook format.

[1342] The generated ebook format file is saved to cloud storage (e.g., AWS S3), and a link is generated.

[1343] When a print order is placed, the order data is sent to the printing company's system.

[1344] Inputs and outputs:

[1345] Input: A customized story.

[1346] Output: ebook format file, online viewing link, print order data.

[1347] (Application Example 1)

[1348] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1349] In the modern era, there is a lack of systems that can easily provide educational and enjoyable content while fostering children's creativity. Furthermore, while there is a wealth of digitized learning content, systems that generate original stories based on children's own drawings still do not exist. Therefore, there is a need for a system that can easily provide educational content tailored to each child's individual interests and age.

[1350] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1351] In this invention, the server includes means for inputting a drawing made by a child, means for analyzing the drawing and identifying the main objects, means for automatically generating a story based on the analyzed objects, means for customizing the generated story based on the child's age, interests, and areas of expertise, means for generating the picture book in ebook format, means for providing the generated ebook online and enabling orders for printed versions, and means for using a generation AI model throughout the entire picture book generation process. This makes it possible for children themselves to become creators and automatically generate and provide customized educational content tailored to their interests and age. Furthermore, since it can be provided in ebook format and used as a printed material, it can meet a variety of needs.

[1352] "A means of inputting children's drawings" refers to a system that allows children to upload illustrations and drawings they have drawn on paper or on digital devices to the system via devices such as smartphones and computers.

[1353] "The means of analyzing the drawn image and identifying the main objects" refers to a system that uses an image analysis algorithm to recognize the main elements such as people, animals, and backgrounds from the uploaded image and register them in a database.

[1354] "Means for automatically generating a story based on the analyzed objects" refers to a mechanism that automatically creates a story by setting relevant plots and characters using a generation AI model based on recognized objects.

[1355] "Means for customizing the generated story based on the child's age, interests, and areas of expertise" refers to a mechanism that optimizes and individually customizes an automatically generated story according to the input data on the child's age, interests, and areas of expertise.

[1356] "A method for generating picture books in ebook format" refers to a system that converts customized stories into digital formats (such as EPUB or PDF) and generates them as ebooks.

[1357] "The means of providing the generated e-books online and enabling orders for printed versions" refers to a system that provides the generated e-books to users via the internet and also accepts orders for printed versions according to the user's wishes.

[1358] "Means of using a generative AI model throughout the entire picture book generation process" refers to a system that uses a generative AI model throughout the entire process, from image analysis to storytelling, to achieve a high level of automation and personalization.

[1359] The system of this invention automatically generates original picture books based on drawings made by children and provides them online. This system includes a series of processes in which the user inputs a drawing made by a child, the system analyzes the drawing to generate a story, and the system provides the story in ebook format.

[1360] The entire system consists of a smartphone application and a server. The primary hardware used is a smartphone and a server. The software includes a generative AI model (TensorFlow), Flask (a server-side web application framework), and React Native (a mobile application framework).

[1361] 1. User input and image upload

[1362] The user opens a smartphone application and takes a picture of or uploads a drawing made by their child. Along with the image file of the drawing, they also enter information such as the child's age, interests, and areas of expertise. This data is sent from the smartphone to the server.

[1363] 2. Image analysis and database storage on the server.

[1364] The server analyzes the received image file using a generation AI model (using TensorFlow). As a result of the image analysis, the main objects (e.g., people, animals, background) are identified. The analysis results are stored in a database (e.g., AWS RDS).

[1365] 3. Story Generation

[1366] The server automatically generates a story using a generative AI model based on the analyzed objects. In this generation process, appropriate plots and characters are set based on the uploaded image. For example, if an image of a dinosaur is uploaded, a story themed around "dinosaur adventures" will be generated.

[1367] 4. Customization and addition of educational elements

[1368] The generated stories are customized based on the child's age and interests. Furthermore, educational elements (e.g., scientific facts and learning elements) are added. For example, a story about a dinosaur discovering a new cave might incorporate information about different types of dinosaurs and their behavior.

[1369] 5. Generation and provision of picture book ebooks

[1370] The customized story is converted into an ebook format (e.g., EPUB or PDF). Users can view the generated ebook online. Printed versions can also be ordered if desired. The server sends the data to the printer, and the printed picture book is delivered to the user within a few days.

[1371] Examples of specific cases and prompt statements

[1372] As a concrete example, using a child's drawing of a dinosaur as a reference, use the following prompt:

[1373] I've uploaded a drawing of a dinosaur made by a child. Based on this drawing, please generate an adventure story in which the dinosaur discovers a new cave and finds fossils inside. Also, please add information to the story that teaches about different types of dinosaurs and their ecology.

[1374] This system allows children to become creators themselves, automatically generating and providing customized educational content tailored to their interests and age. It can also be provided in e-book format, and print versions are available upon request, accommodating diverse needs.

[1375] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1376] Step 1:

[1377] The user opens a smartphone application and takes a picture of their child's drawing or uploads it within the application. They also enter information such as the child's age, interests, and areas of expertise. This data is sent from the smartphone to the server. The smartphone sends the image file and text data via an HTTP request.

[1378] Input: Image file of a child's drawing, child's age, interests, and areas of expertise.

[1379] Output: Image files and text data sent to the server

[1380] Step 2:

[1381] The server stores the received image files and analyzes them using a generating AI model. Specifically, it uses an image analysis algorithm (TensorFlow) to identify key objects (people, animals, background, etc.) and saves the analysis results to a database.

[1382] Input: Image file, text data

[1383] Output: List of analyzed objects, saved to database.

[1384] Specific actions:

[1385] Receiving and saving image files

[1386] Image analysis using generative AI models

[1387] Identifying the main objects

[1388] Save the analysis results to a database (e.g., AWS RDS).

[1389] Step 3:

[1390] The server automatically generates stories using a generative AI model based on the information from the analyzed objects. Data such as age, interests, and areas of expertise entered by the user are also used at this stage.

[1391] Input: List of parsed objects, child's age, interests, areas of expertise

[1392] Output: Auto-generated story

[1393] Specific actions:

[1394] Combine the analysis results and additional information to generate a prompt message.

[1395] Input prompt text into the generative AI model and generate a story.

[1396] Step 4:

[1397] The server customizes automatically generated stories based on the child's age, interests, and strengths, and adds educational elements. For example, it incorporates scientific facts and learning elements into the stories.

[1398] Input: Auto-generated story, child's age, interests, areas of expertise

[1399] Output: Customized story

[1400] Specific actions:

[1401] Add educational elements to automatically generated stories.

[1402] Optimize the story based on the child's age and interests.

[1403] Step 5:

[1404] The server converts the customized story into an ebook format (e.g., EPUB or PDF). It then generates a link to provide the generated ebook online and notifies the user.

[1405] Input: Customized story

[1406] Output: ebook file, online link

[1407] Specific actions:

[1408] Convert story to ebook format

[1409] Generate a link for online viewing.

[1410] Notify the user of the link (e.g., send an email).

[1411] Step 6:

[1412] Users can view online-generated ebooks or, if desired, order printed versions. The server sends the print order data to the printer and arranges for the printing and delivery of the picture books.

[1413] Input: User's print order

[1414] Output: Printed picture book

[1415] Specific actions:

[1416] Received print order information.

[1417] Send order data to the printing company

[1418] A printing company prints the picture book and delivers it to the user.

[1419] This process allows for the easy creation of original picture books that leverage children's creativity, and these books are available in both electronic and print formats.

[1420] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1421] This invention relates to a system that automatically generates original picture books based on drawings made by children and further customizes the content by recognizing the user's emotions. Specifically, the invention describes a form that generates and provides picture books through the following process and combines it with emotion recognition.

[1422] 1. Upload the image and enter the information.

[1423] User:

[1424] Users upload drawings made by their children using the application or website. Users also enter necessary information such as the child's age, interests, and areas of expertise.

[1425] Terminal:

[1426] The device sends the image file uploaded by the user and any additional information entered to the server.

[1427] 2. Receiving and analyzing image data

[1428] server:

[1429] The server receives the uploaded image data and accompanying information and stores them in a database. Next, the server invokes a generative AI model for image analysis to identify the main objects depicted in the image (people, animals, background, etc.).

[1430] Specific example:

[1431] For example, if a child draws a picture of a dinosaur, the server recognizes the picture as a dinosaur object and saves it in the database with the label "dinosaur".

[1432] 3. Automatic story generation

[1433] server:

[1434] The server automatically generates a story using a generative AI model based on the analysis results. The generated story uses objects depicted in illustrations as plot elements and characters.

[1435] Specific example:

[1436] If a picture of a dinosaur is entered, the server will generate a story themed around "dinosaur adventures," depicting the challenges the dinosaur faces and how it overcomes them.

[1437] 4. Customization and Addition of Educational Content

[1438] server:

[1439] The server customizes the generated stories based on the child's age, interests, and areas of expertise. Furthermore, it adds scientific facts and learning elements to enhance their educational value.

[1440] Specific example:

[1441] If a 6-year-old child is interested in dinosaurs, the server will incorporate scientific facts about dinosaur species and their ecology into the story. For example, a scenario could be created where a dinosaur discovers a new cave and finds fossils within, incorporating learning about different dinosaur species along the way.

[1442] 5. Utilizing the Emotion Engine

[1443] server:

[1444] The server uses an emotion engine to analyze the user's emotions in real time. The emotion engine analyzes the user's facial expressions and voice through the camera and microphone to acquire emotional data.

[1445] Specific example:

[1446] While the user is browsing the picture book, the server analyzes the user's facial expressions through an emotion engine and recognizes emotions such as surprise and joy.

[1447] 6. Emotion-based real-time story adjustment

[1448] server:

[1449] The server adjusts the story content in real time based on feedback from the emotion engine. The story's progression and character actions change dynamically each time the user's emotions shift.

[1450] Specific example:

[1451] If the user shows a surprised expression in response to a story development, the server will add elements in the next scene that allow them to continue enjoying the surprise. Conversely, if the emotional data indicates sadness, the server will adjust the story and introduce a more positive development.

[1452] 7. Generation and provision of picture book data

[1453] server:

[1454] The server integrates customized stories and educational content, as well as emotionally-based adjustments, and converts them into ebook formats (EPUB and PDF).

[1455] server:

[1456] The server uploads the generated ebook data to the web server and generates a link that allows users to view it online. Printed versions of the picture books can also be ordered if desired.

[1457] Specific example:

[1458] Users can browse picture books based on children's drawings online and also order printed copies. When a user orders a printed copy, the server sends the data to the printer, and the printed book is delivered a few days later.

[1459] The system of this invention fosters children's creativity, deepens the parent-child bond, and provides educational value. Furthermore, by combining it with an emotion engine, it provides a more personalized experience that responds to the user's emotions, generating even greater emotion and excitement. This system is highly effective not only in quickly and easily generating original picture books and expanding learning opportunities, but also in providing a dynamic experience that responds to emotions.

[1460] The following describes the processing flow.

[1461] Step 1:

[1462] User:

[1463] The user opens an application or website for generating picture books and clicks the "Upload Picture" button.

[1464] Step 2:

[1465] Terminal:

[1466] The terminal displays a file selection window, providing a screen where the user can select the image file they want to upload.

[1467] Step 3:

[1468] User:

[1469] The user selects an image file of a picture, then enters the necessary information (such as the child's age, areas of interest, and areas of expertise), and presses the "Submit" button.

[1470] Step 4:

[1471] Terminal:

[1472] The terminal sends data, including the selected image file and the entered information, to the server.

[1473] Step 5:

[1474] server:

[1475] The server receives the data. It saves the image file and additional information to the database and verifies that the upload was successful.

[1476] Step 6:

[1477] server:

[1478] The server invokes a generative AI model to analyze the stored image files. The AI ​​model analyzes the image files and identifies key objects.

[1479] Step 7:

[1480] server:

[1481] The server labels each object based on the analysis results and stores them in the database. For example, if the image depicts a dinosaur, it will be labeled "dinosaur".

[1482] Step 8:

[1483] server:

[1484] The server uses the analysis results and accompanying information to invoke a generative AI model to generate a story. The generative AI model automatically generates a story based on the input prompts.

[1485] Step 9:

[1486] server:

[1487] The server retrieves the generated story and customizes it based on user-provided information (age, interests, areas of expertise).

[1488] Step 10:

[1489] server:

[1490] The server adds educational content to customized stories. For example, if a picture of a dinosaur is included, it will add scientific facts and educational elements about dinosaurs.

[1491] Step 11:

[1492] server:

[1493] The server uses an emotion engine to acquire data from the user's device's camera and microphone in order to analyze the user's emotions in real time.

[1494] Step 12:

[1495] Terminal:

[1496] The device uses an emotion engine to analyze the user's facial expressions and voice tone, and generates emotion data.

[1497] Step 13:

[1498] server:

[1499] Based on feedback from the emotion engine, the server analyzes the user's emotions and recognizes feelings such as positive, negative, and surprise.

[1500] Step 14:

[1501] server:

[1502] The server adjusts the story content in real time based on the recognized emotion data. For example, if the user makes a surprised expression, the next scene will be designed to add a further element of surprise.

[1503] Step 15:

[1504] server:

[1505] Based on emotional data, the server dynamically changes the story's progression and character actions to provide content that aligns with the user's interests and emotions.

[1506] Step 16:

[1507] server:

[1508] The server converts the final integrated document data into an ebook format (EPUB or PDF) and uploads the generated ebook data to the web server.

[1509] Step 17:

[1510] server:

[1511] The server generates a link that allows the user to view the picture book online and provides the user with the link.

[1512] Step 18:

[1513] Terminal:

[1514] The terminal retrieves a link provided by the server and provides an interface that allows the user to view picture books online.

[1515] Step 19:

[1516] User:

[1517] Users can view picture books generated online. They can also click the "Order a Printed Version" button if they wish.

[1518] Step 20:

[1519] Terminal:

[1520] The terminal displays an order form for entering shipping and payment information.

[1521] Step 21:

[1522] User:

[1523] The user enters the required information and presses the "Confirm Order" button.

[1524] Step 22:

[1525] server:

[1526] The server receives order data, processes payments, and sends print data to the printer.

[1527] Step 23:

[1528] Printing company:

[1529] The printing company prints the picture book based on the received data and delivers it to the specified address.

[1530] Step 24:

[1531] server:

[1532] The server monitors the progress of the order and notifies the user when shipping is confirmed.

[1533] (Example 2)

[1534] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1535] Children today have limited opportunities to exercise their creativity in the digital age. Furthermore, educational content is delivered in a uniform manner, lacking opportunities for learning tailored to each child's individual interests and strengths. In addition, the technology for personalizing content based on user emotions is still immature, making it difficult to provide a greater sense of immersion and emotional impact. This invention aims to solve these problems by automatically generating original picture books and providing children with educationally valuable and emotionally engaging content.

[1536] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for inputting a picture drawn by a child, means for analyzing the picture and identifying the main objects, means for automatically generating a story based on the analyzed objects, means for customizing the generated story based on the child's age, interests, and areas of expertise, means for adding educational content based on the customized story, means for analyzing the user's emotions and adjusting the story in real time, means for converting the generated story into an ebook format, means for providing the generated picture book online, and means for ordering a printed version of the generated picture book. This makes it possible to generate a personalized, educational, and emotionally engaging picture book based on a child's picture and adjust it in real time.

[1537] "A means of inputting children's drawings" refers to an interface for importing children's drawings into a system in an electronic format.

[1538] "Methods for analyzing drawings and identifying key objects" refers to technologies that analyze uploaded drawings and extract and identify specific elements (such as people, animals, backgrounds, etc.) from them.

[1539] "Means for automatically generating stories based on analyzed objects" refers to an algorithm or system for creating a narrative based on identified object information.

[1540] "A means of customizing generated stories based on a child's age, interests, and strengths" refers to a function that adjusts the generated stories to correspond with the information of the child that has been entered.

[1541] "Means of adding customized story-based educational content" refers to technologies for inserting educational information and learning elements into a tailored story.

[1542] "A means of analyzing user emotions and adjusting the story in real time" refers to a system that evaluates emotional data such as the user's facial expressions and voice in real time and dynamically changes the development of the story accordingly.

[1543] "Methods for converting generated stories into ebook formats" refers to methods for converting customized stories and educational content into ebook formats (such as EPUB or PDF).

[1544] "Methods for providing generated picture books online" refers to a system that makes completed ebooks accessible to users via the internet.

[1545] "Method for ordering a printed version of the generated picture book" refers to the procedure for users to order a printed version of the generated e-book as a physical book, if they so desire, via the internet.

[1546] This invention relates to a system that automatically generates original picture books based on drawings made by children and further customizes the content by recognizing the user's emotions. Specifically, the program is implemented as follows.

[1547] First, users upload drawings made by their children using an application or website. In addition, users enter information such as the child's age, interests, and areas of expertise. This entered drawing and information are sent to the server by the device. Image files are sent in JPEG or PNG format, and the input information is sent in JSON format.

[1548] The server stores the received image data and associated information in a database. It then calls a generative AI model (e.g., TensorFlow or PyTorch) for image analysis to identify the main objects depicted in the image (people, animals, background, etc.). For example, if an image of a dinosaur is input, the server recognizes "dinosaur" as the object and stores it as a label in the database.

[1549] Based on the analysis results, the server automatically generates a story using a generative AI model. The generative AI model is given instructions using prompts. A specific example of a prompt is: "Based on the input dinosaur picture, generate a story themed around a 'dinosaur adventure' that a 6-year-old child can understand. Include content where the dinosaur discovers a new cave, finds fossils inside, and learns about different types of dinosaurs along the way." As a result, a story template is generated and saved in the database.

[1550] Next, the server customizes the generated story based on the child's age, interests, and areas of expertise. For example, if a 6-year-old child is interested in dinosaurs, scientific facts about dinosaur species and their ecology will be incorporated into the story. Furthermore, learning content will be added to enhance its educational value.

[1551] The server uses an emotion engine to analyze the user's emotions in real time. This involves analyzing facial expressions and voice through the user's device's camera and microphone to acquire emotional data. For example, if a user shows expressions of surprise or joy while viewing a picture book, the server analyzes that emotion and adds elements in the next scene that allow them to continue enjoying the surprise. Conversely, if the user expresses sadness, a positive development is introduced.

[1552] Finally, the server integrates the customized story and educational content, as well as emotionally-based adjustments, and converts it into an ebook format (EPUB or PDF). This ebook data is uploaded to a web server, and users can view it online. Users can also order printed copies of the picture book if they wish. Specifically, the server sends the data to a printer, and the printed book is delivered to the user a few days later.

[1553] Through specific data processing and calculations performed at each processing step of the program, this invention leverages children's creativity, deepens the parent-child bond, and provides highly educational content. Furthermore, by combining it with an emotion engine, it provides a more personalized experience that responds to the user's emotions, generating even greater emotion and excitement.

[1554] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1555] Step 1:

[1556] Users upload drawings made by their children through the application or website and enter information such as the child's age, interests, and areas of expertise. Specifically, the user clicks the "Upload Drawing" button in the application, selects the image file, fills in the child's information in the input form, and clicks the "Submit" button.

[1557] input:

[1558] Children's drawings (image files), information such as the child's age, interests, and areas of expertise.

[1559] output:

[1560] The entered image and information (sent to the server).

[1561] Step 2:

[1562] The terminal sends the image file uploaded by the user and the input information to the server. The image file is sent in JPEG or PNG format, and the input information is sent in JSON format.

[1563] Specific operation:

[1564] The terminal responds to user input and sends image files and JSON-formatted information to the server as an HTTP request.

[1565] input:

[1566] Image files and input information submitted by the user.

[1567] output:

[1568] Image files and input information sent to the server.

[1569] Step 3:

[1570] The server stores the received image data and associated information in a database. This allows for centralized management of all necessary information.

[1571] Specific operation:

[1572] The server temporarily stores the image file, and then stores the image file and information in JSON format in the database.

[1573] input:

[1574] Image files and input information sent from the device.

[1575] output:

[1576] Image files and information stored in the database.

[1577] Step 4:

[1578] The server invokes a generative AI model for image analysis to identify the main objects depicted in the image. Image recognition models (such as TensorFlow or PyTorch) are used for the object identification process.

[1579] Specific operation:

[1580] The server generates and inputs the saved image files into an AI model. The model analyzes the images and identifies key objects.

[1581] input:

[1582] Image files stored in the database.

[1583] output:

[1584] Identified object information (e.g., dinosaur, tree, house, etc.).

[1585] Step 5:

[1586] The server generates prompt statements based on the object analysis results and inputs them into a generative AI model (e.g., OpenAI's GPT) to generate a story. A story template is created.

[1587] Specific operation:

[1588] The server generates a prompt message. Based on the information "a picture of a dinosaur" and "age 6," it generates a prompt message such as "Please generate a story about a dinosaur on an adventure." This prompt message is input into the AI ​​model to generate the story.

[1589] input:

[1590] Identified object information, user input information.

[1591] output:

[1592] A template for the generated story.

[1593] Step 6:

[1594] The server customizes the generated story based on the child's age, interests, and areas of expertise. For example, it might incorporate scientific facts about different types of dinosaurs and their behavior into the story.

[1595] Specific operation:

[1596] The server retrieves appropriate learning elements from the educational content database based on a story template and integrates them into the story.

[1597] input:

[1598] The generated story template and user input information.

[1599] output:

[1600] A customized story.

[1601] Step 7:

[1602] The server uses an emotion engine to analyze the user's emotions in real time. It uses the user's device's camera and microphone to analyze facial expressions and voice, and acquires emotional data.

[1603] Specific operation:

[1604] The server obtains access permissions to the user's camera and microphone, and analyzes facial expressions and voice in real time. It retrieves data through an emotion analysis API and evaluates the user's emotions.

[1605] input:

[1606] Real-time facial expression and voice data of the user.

[1607] output:

[1608] User sentiment data.

[1609] Step 8:

[1610] The server adjusts the story in real time based on feedback from the emotion engine. Every time the user's emotions change, the story's progression and the characters' actions are dynamically altered.

[1611] Specific operation:

[1612] The server modifies the next scene of the story based on emotional data. If positive emotions are detected, additional adventure elements are added. If negative emotions are detected, a positive development is introduced.

[1613] input:

[1614] User sentiment data, customized stories.

[1615] output:

[1616] A story that is adjusted in real time.

[1617] Step 9:

[1618] The server ultimately integrates customized stories, educational content, and emotionally-based adjustments, and converts them into ebook formats (EPUB or PDF). This is then uploaded to a web server, generating a link that users can access online.

[1619] Specific operation:

[1620] The server converts the story into an ebook format, uploads the file to the web server, and generates a URL.

[1621] input:

[1622] A well-structured story and content.

[1623] output:

[1624] eBook format file, viewing link.

[1625] Step 10:

[1626] The server then processes orders for printed copies of the generated picture books for users who wish to receive them.

[1627] Specific operation:

[1628] When a user orders a printed version, the server sends the data to the printer and proceeds with the process of delivering the printed picture book to the user.

[1629] input:

[1630] eBook format files, user print order information.

[1631] output:

[1632] Data sent to the printer, printed picture book delivered to the user.

[1633] (Application Example 2)

[1634] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1635] Conventional picture book generation systems are limited to one-way generation based on children's drawings and cannot adjust the story in real time in response to the user's emotions. Furthermore, the content of the picture books is templated, making it difficult to maximize children's creativity and learning effectiveness. This invention aims to solve these problems and provide a more personalized experience.

[1636] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for inputting a picture drawn by a child, means for analyzing the picture and identifying the main objects, means for automatically generating a story based on the analyzed objects, means for customizing the generated story based on the child's age, interests, and areas of expertise, means for analyzing the user's emotions, means for adjusting the story in real time based on the emotion data, means for generating a picture book containing the customized story, and means for providing the generated picture book online. This makes it possible to provide an original picture book that integrates learning elements based on a child's drawing, and further makes it possible to adjust the story in real time according to the user's emotions.

[1637] "Children's drawings" refers to artwork or sketches drawn by children, whether in digital or physical form.

[1638] "Analysis" refers to the process of breaking down images and data to understand their contents and identify specific objects and attributes.

[1639] "Key objects" refer to elements identified through image analysis that play a crucial role in the story's plot and characters.

[1640] "Methods for automatically generating stories" refer to technologies such as artificial intelligence that construct narratives based on input data and generate text and graphics.

[1641] "Customization" refers to the process of changing or adapting content and functions to suit the preferences and requirements of a specific user.

[1642] "User emotions" refer to the psychological state and feedback perceived through the user's facial expressions, tone of voice, and other similar actions.

[1643] "Emotional data" refers to digital information that indicates a user's emotional state, acquired using devices such as cameras and microphones.

[1644] "Adjusting in real time" refers to the process of making immediate changes and optimizations in response to the situation and data at hand.

[1645] "Means of generating picture books" refers to systems and software that convert stories, including text and illustrations, into digital or physical book formats.

[1646] "Methods of providing content online" refers to methods of delivering ebooks and digital content to users using the internet.

[1647] This invention describes a system that generates original picture books based on drawings made by children and customizes the content in real time based on the user's emotions. In a specific embodiment, the present invention is carried out using the following procedure and apparatus.

[1648] 1. Main hardware and software used

[1649] Smart glasses camera: Used to scan and upload children's drawings.

[1650] Server: The central component responsible for analyzing images, generating stories, analyzing emotions, and customizing picture books.

[1651] OpenCV: A library for capturing and preprocessing images.

[1652] Transformers (Generative AI Model): A library for analyzing images and generating stories.

[1653] EmotionRecognizer: Software for analyzing user emotions.

[1654] PDFGenerator: Used to generate the final picture book.

[1655] 2. Specific System Processing Overview

[1656] First, the user scans a child's drawing using smart glasses. This image is uploaded to a server. The server analyzes the image using OpenCV to identify key objects. Next, a generative AI model is used to automatically generate a story based on these objects.

[1657] The generated stories are customized based on the child's age, interests, and strengths, and educational content is also added. Next, the server uses EmotionRecognizer to analyze the user's emotions in real time and adjusts the story's progression based on the emotional data.

[1658] Finally, the customized story is generated in ebook format using PDFGenerator. This provides users with a link to view it online. Printed versions of the picture book can also be ordered upon request.

[1659] 3. Specific Examples and Examples of Prompt Statements

[1660] For example, if a child draws a picture of a dinosaur, they can use smart glasses to scan the drawing and upload it to a server. A generative AI model then generates a story themed around a dinosaur adventure. If the user shows a surprised expression, an emotion analysis engine detects this reaction and adds further surprising elements to the story. Finally, this customized picture book is delivered as an ebook.

[1661] Example of a prompt:

[1662] "Please have the generative AI model analyze the dinosaur images and generate a story."

[1663] "Use the emotion analysis engine to adjust the story's progression to match the user's emotions."

[1664] The above describes a specific embodiment for carrying out the present invention. This embodiment makes it possible to easily and quickly generate original picture books using children's drawings and to provide personalized stories that respond to emotions.

[1665] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1666] Step 1:

[1667] The user scans a child's drawing using smart glasses. The smart glasses' camera captures the drawing and generates image data. Here, the input is the image data of the drawing acquired through the smart glasses' camera, and the output is that image data.

[1668] Step 2:

[1669] The device uploads the acquired image data to the server. It uses a communication module within the smart glasses to transmit the images to the server via the internet. The input is the image data, and the output is the image data that has been sent to the server.

[1670] Step 3:

[1671] The server uses OpenCV to analyze uploaded images and identify key objects. Specifically, it extracts features of objects (e.g., dinosaurs) from the image data and identifies them. The input is the uploaded image data, and the output is a list of identified key objects.

[1672] Step 4:

[1673] The server uses a generative AI model to automatically generate a story based on identified key objects. The generative AI model uses the input key objects (e.g., dinosaurs) as plot elements and characters to create the story. The input is a list of key objects, and the output is the automatically generated story.

[1674] Step 5:

[1675] The server customizes the generated story based on the child's age, interests, and strengths. It adjusts the story content based on the child's information entered by the user and adds educational content as needed. The input consists of the automatically generated story and the child's information entered by the user, while the output is the customized story.

[1676] Step 6:

[1677] The server uses EmotionRecognizer to analyze the user's emotions in real time. Specifically, it analyzes data acquired from the smart glasses' camera and microphone to identify emotions from the user's facial expressions and tone of voice. The input is video and audio streams, and the output is the analyzed emotion data.

[1678] Step 7:

[1679] The server adjusts the story in real time based on sentiment data. For example, if a user reacts with surprise, the server adjusts the next development of the story accordingly. The input is sentiment data and a customized story, and the output is the adjusted story.

[1680] Step 8:

[1681] The server converts the customized and adjusted story into an ebook format using PDFGenerator. Specifically, it integrates the story text and associated images and saves them as EPUB or PDF files. The input is the adjusted story, and the output is the generated ebook.

[1682] Step 9:

[1683] The server uploads the generated ebook to a web server and generates a link that users can access online. It also allows users to order a printed version of the picture book if desired. The input is the generated ebook, and the output is the online link and the option to order a print copy.

[1684] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1685] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1686] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1687] [Fourth Embodiment]

[1688] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1689] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1690] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1691] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1692] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1693] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1694] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1695] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1696] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1697] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1698] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1699] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1700] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1701] This invention relates to a system for automatically generating original picture books based on drawings made by children. Specifically, the system generates and provides picture books through the following process.

[1702] 1. Upload the image and enter the information.

[1703] User:

[1704] Users upload drawings made by their children using the application or website. Users also enter necessary information such as the child's age, interests, and areas of expertise.

[1705] Terminal:

[1706] The device sends the image file uploaded by the user and any additional information entered to the server.

[1707] 2. Receiving and analyzing image data

[1708] server:

[1709] The server receives the uploaded image data and accompanying information and stores them in a database. Next, the server invokes a generative AI model for image analysis to identify the main objects depicted in the image (people, animals, background, etc.).

[1710] Specific example:

[1711] For example, if a child draws a picture of a dinosaur, the server recognizes the picture as a dinosaur object and saves it in the database with the label "dinosaur".

[1712] 3. Automatic story generation

[1713] server:

[1714] The server automatically generates a story using a generative AI model based on the analysis results. The generated story uses objects depicted in illustrations as plot elements and characters.

[1715] Specific example:

[1716] If a picture of a dinosaur is entered, the server will generate a story themed around "dinosaur adventures," depicting the challenges the dinosaur faces and how it overcomes them.

[1717] 4. Customization and Addition of Educational Content

[1718] server:

[1719] The server customizes the generated stories based on the child's age, interests, and areas of expertise. Furthermore, it adds scientific facts and learning elements to enhance their educational value.

[1720] Specific example:

[1721] If a 6-year-old child is interested in dinosaurs, the server will incorporate scientific facts about dinosaur species and their ecology into the story. For example, a scenario could be created where a dinosaur discovers a new cave and finds fossils within, incorporating learning about different dinosaur species along the way.

[1722] 5. Generation and provision of picture book data

[1723] server:

[1724] The server converts the customized story into an ebook format (such as EPUB or PDF) and provides a link for users to view it online. Printed versions of the picture book can also be ordered if desired.

[1725] Specific example:

[1726] Users can browse picture books based on children's drawings online and also order printed copies. When a user orders a printed copy, the server sends the data to the printer, and the printed book is delivered a few days later.

[1727] The system of this invention fosters children's creativity, deepens the parent-child bond, and provides educational value. This system is highly effective in quickly and easily generating original picture books and expanding learning opportunities.

[1728] The following describes the processing flow.

[1729] Step 1:

[1730] User:

[1731] The user opens an application or website for generating picture books and clicks the "Upload Picture" button.

[1732] Step 2:

[1733] Terminal:

[1734] The terminal displays a file selection window, providing a screen where the user can select the image file they want to upload.

[1735] Step 3:

[1736] User:

[1737] The user selects an image file of a picture, enters the necessary information (such as the child's age, areas of interest, and areas of expertise), and presses the "Submit" button.

[1738] Step 4:

[1739] Terminal:

[1740] The terminal sends data, including the selected image file and the entered information, to the server.

[1741] Step 5:

[1742] server:

[1743] The server receives the data. It saves the image file and additional information to the database and verifies that the upload was successful.

[1744] Step 6:

[1745] server:

[1746] The server invokes a generative AI model to analyze the stored image files. The AI ​​model analyzes the image files and identifies key objects.

[1747] Step 7:

[1748] server:

[1749] The server labels each object based on the analysis results and stores them in the database. For example, if the picture depicts a dinosaur, it will be labeled "dinosaur".

[1750] Step 8:

[1751] server:

[1752] The server uses the analysis results and accompanying information to invoke a generative AI model to generate a story. The generative AI model automatically generates a story based on the input prompts.

[1753] Step 9:

[1754] server:

[1755] The server retrieves the generated story and customizes it based on user-provided information (age, interests, areas of expertise).

[1756] Step 10:

[1757] server:

[1758] The server adds educational content to customized stories. For example, if a picture of a dinosaur is included, it will add scientific facts and edutainment elements about dinosaurs.

[1759] Step 11:

[1760] server:

[1761] The server integrates the completed story and educational content and converts it into an ebook format (EPUB or PDF).

[1762] Step 12:

[1763] server:

[1764] The server uploads the generated ebook data to the web server and generates a link that allows users to view it online.

[1765] Step 13:

[1766] Terminal:

[1767] The terminal retrieves a link provided by the server and provides an interface that allows the user to view picture books online.

[1768] Step 14:

[1769] User:

[1770] Users can view and check the content of picture books generated online.

[1771] Step 15:

[1772] User:

[1773] If the user wishes, they can click the "Order Printed Version" button on the online interface.

[1774] Step 16:

[1775] Terminal:

[1776] The terminal displays an order form for entering shipping and payment information.

[1777] Step 17:

[1778] User:

[1779] The user enters the required information and presses the "Confirm Order" button.

[1780] Step 18:

[1781] server:

[1782] The server receives order data, processes payments, and sends print data to the printer.

[1783] Step 19:

[1784] Printing company:

[1785] The printing company prints the picture book based on the received data and delivers it to the specified address.

[1786] Step 20:

[1787] server:

[1788] The server monitors the progress of the order and notifies the user when shipping is confirmed.

[1789] (Example 1)

[1790] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1791] Currently, there is no system on the market that automatically generates original picture books using children's drawings. This means that parents and educators lack effective tools to further cultivate children's creativity and enhance their educational value. Furthermore, there are limited means of easily generating and providing high-quality picture books. Given this situation, there is a need for a system that can easily generate original picture books based on children's drawings while also adding educational value.

[1792] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1793] In this invention, the server includes means for storing children's drawings in a database, means for analyzing the stored drawings using a generation AI model to identify key objects, and means for automatically generating a story based on the identified objects. This enables the automatic generation of original picture books based on children's drawings and allows for customization that includes educational elements.

[1794] "Children's drawings" refer to illustrations and paintings created by children using their own creativity.

[1795] "Means of input" refer to the functions or interfaces that allow users to provide data to the system.

[1796] "Means of uploading" refers to the functions or interfaces that allow users to transfer data stored on their local storage to a server via the internet.

[1797] "Means of transmission" refers to the functions and protocols used to send data from a terminal to a server.

[1798] A "server" is a computer system that stores and processes data over a network.

[1799] "Means of saving to a database" refers to functions and systems for storing and managing received data in an organized format.

[1800] A "generative AI model" is an artificial intelligence algorithm used to learn from large amounts of data and generate new data.

[1801] "Key objects" refer to important elements or components present in an image that are identified through image analysis.

[1802] "Methods for automatically generating stories" refers to functions that use generative AI models to create narratives based on specified objects and information.

[1803] "Customization options" refer to features that allow you to adjust and modify the content of stories generated based on a child's age, interests, and strengths.

[1804] An "eBook format" refers to a standard format for saving and displaying books in digital format (e.g., EPUB, PDF).

[1805] "Means of providing content online" refers to functions that make generated content accessible to users via the internet.

[1806] "Educational content" refers to information and materials intended to improve learning and knowledge.

[1807] A "printed version" refers to digital content printed on physical paper.

[1808] This invention relates to a system that automatically generates original picture books based on drawings made by children. The system includes a process in which a user uploads a child's drawing, a server analyzes the drawing, generates and customizes a story, and finally delivers it as an ebook.

[1809] Hardware and software to be used

[1810] Hardware:

[1811] Server: A high-performance computer used for data storage, analysis, and execution of generated AI models.

[1812] Device: A PC or smartphone used by users to upload images and information.

[1813] software:

[1814] Web application or mobile application: Provides a user interface.

[1815] Database: A data management system for storing received images and information (e.g., MySQL, PostgreSQL).

[1816] Generative AI models: AI algorithms used for image analysis and story generation (e.g., TensorFlow, PyTorch, GPT-3).

[1817] Image processing libraries: Used for image preprocessing (e.g., OpenCV).

[1818] eBook generation tools: Tools for converting generated stories into eBook formats (e.g., Calibre).

[1819] Program Processing Overview

[1820] User:

[1821] Users upload drawings made by their children to the application or website and enter information such as the child's age, interests, and areas of expertise. The uploaded data is sent from the device to the server. For example:

[1822] Click the upload button and select your image file.

[1823] Enter your age, interests, and areas of expertise in the text fields.

[1824] Click the "Submit Data" button.

[1825] Terminal:

[1826] The terminal receives the image file and input information uploaded by the user, packets them, and sends them to the server using the HTTPS protocol. For example:

[1827] The image file and input information are saved to temporary storage.

[1828] The packet should include data and necessary metadata (user ID, timestamp, etc.).

[1829] Generate a POST request and send the data.

[1830] server:

[1831] The server performs the following actions:

[1832] 1. Save the received image data and additional information to the database.

[1833] 2. Analyze the image using a generative AI model to identify the main objects.

[1834] 3. Based on the identified object, a story is automatically generated that aligns with the prompt text.

[1835] Specific example:

[1836] If a child draws a picture of a dinosaur, the server uses a generative AI model to identify the dinosaur and generates the prompt "Generate a dinosaur adventure story." A story is then generated based on this prompt.

[1837] Next, the generated story is customized based on the child's age, interests, and areas of expertise. Furthermore, educational content (e.g., information about dinosaur species and their ecology) is incorporated into the story.

[1838] After the story is completed, it will be converted into an ebook format (e.g., EPUB, PDF), and users will be provided with a link to view the picture book online. Printed copies can also be ordered upon request, in which case the data will be sent to the printing company.

[1839] This invention enables the rapid and easy creation of picture books that foster children's creativity, deepen parent-child bonds, and provide educational value.

[1840] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1841] Step 1:

[1842] User:

[1843] Users access the application or website and click the "Upload Picture" button on the screen. Next, they select and upload an image file of a picture drawn by their child from their local storage. They also enter necessary information such as the child's age, interests, and areas of expertise.

[1844] Specific actions:

[1845] Use a web browser or smartphone app.

[1846] Select the image file and perform the upload operation.

[1847] Enter the required information and press the submit button.

[1848] Inputs and outputs:

[1849] Input: Image files of drawings made by the child, information about the child's age, interests, and areas of expertise.

[1850] Output: The uploaded image file and input information are sent to the device.

[1851] Step 2:

[1852] Terminal:

[1853] The device temporarily stores the image file uploaded by the user and any additional information entered, and then sends it to the server using the HTTPS protocol.

[1854] Specific actions:

[1855] Uploaded images and input information are saved to temporary storage.

[1856] The packet should include data and necessary metadata (user ID, timestamp, etc.).

[1857] Generate a POST request to the server and send the data.

[1858] Inputs and outputs:

[1859] Input: Image files submitted by the user, age, interests, and areas of expertise.

[1860] Output: Packetized data is sent to the server.

[1861] Step 3:

[1862] server:

[1863] The server extracts image data and input information from the received packets and stores them in a database. Next, it calls a generative AI model to analyze the stored image data and identify key objects.

[1864] Specific actions:

[1865] The received data is converted to the appropriate format and saved to the database.

[1866] Image features are extracted using a generative AI model (e.g., TensorFlow or PyTorch).

[1867] Identify and label the main objects based on the features.

[1868] Inputs and outputs:

[1869] Input: Image file and input information sent from the terminal.

[1870] Output: Analysis results stored in the database, labels of identified key objects.

[1871] Step 4:

[1872] server:

[1873] Based on the analysis results, the server calls a generative AI model for story generation (e.g., GPT-3) to generate prompt text. The story is then automatically generated based on the generated prompt text.

[1874] Specific actions:

[1875] The labels extracted from the analysis results are incorporated into the prompt message (e.g., "Generate a dinosaur adventure story").

[1876] The prompt text is input into the AI ​​model to generate a story.

[1877] Save the generated story to the database.

[1878] Inputs and outputs:

[1879] Input: Labels of the identified main objects.

[1880] Output: The generated story.

[1881] Step 5:

[1882] server:

[1883] The server customizes the generated story based on the child's age, interests, and areas of expertise. At the same time, it incorporates educational content (e.g., scientific facts and learning elements) into the story.

[1884] Specific actions:

[1885] The story content is adjusted based on the user's age, interests, and areas of expertise.

[1886] Retrieve the relevant educational content from the database and add it to the story.

[1887] Inputs and outputs:

[1888] Input: Generated story, user-entered age, interests, and areas of expertise.

[1889] Output: Customized story.

[1890] Step 6:

[1891] server:

[1892] The server converts customized stories into ebook formats (e.g., EPUB, PDF). Users are provided with a link to view the stories online. Printed versions can also be ordered upon request.

[1893] Specific actions:

[1894] Call an API (e.g., Calibre) to convert the story into an ebook format.

[1895] The generated ebook format file is saved to cloud storage (e.g., AWS S3), and a link is generated.

[1896] When a print order is placed, the order data is sent to the printing company's system.

[1897] Inputs and outputs:

[1898] Input: A customized story.

[1899] Output: ebook format file, online viewing link, print order data.

[1900] (Application Example 1)

[1901] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1902] In the modern era, there is a lack of systems that can easily provide educational and enjoyable content while fostering children's creativity. Furthermore, while there is a wealth of digitized learning content, systems that generate original stories based on children's own drawings still do not exist. Therefore, there is a need for a system that can easily provide educational content tailored to each child's individual interests and age.

[1903] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1904] In this invention, the server includes means for inputting a drawing made by a child, means for analyzing the drawing and identifying the main objects, means for automatically generating a story based on the analyzed objects, means for customizing the generated story based on the child's age, interests, and areas of expertise, means for generating the picture book in ebook format, means for providing the generated ebook online and enabling orders for printed versions, and means for using a generation AI model throughout the entire picture book generation process. This makes it possible for children themselves to become creators and automatically generate and provide customized educational content tailored to their interests and age. Furthermore, since it can be provided in ebook format and used as a printed material, it can meet a variety of needs.

[1905] "A means of inputting children's drawings" refers to a system that allows children to upload illustrations and drawings they have drawn on paper or on digital devices to the system via devices such as smartphones and computers.

[1906] "The means of analyzing the drawn image and identifying the main objects" refers to a system that uses an image analysis algorithm to recognize the main elements such as people, animals, and backgrounds from the uploaded image and register them in a database.

[1907] "Means for automatically generating a story based on the analyzed objects" refers to a mechanism that automatically creates a story by setting relevant plots and characters using a generation AI model based on recognized objects.

[1908] "Means for customizing the generated story based on the child's age, interests, and areas of expertise" refers to a mechanism that optimizes and individually customizes an automatically generated story according to the input data on the child's age, interests, and areas of expertise.

[1909] "A method for generating picture books in ebook format" refers to a system that converts customized stories into digital formats (such as EPUB or PDF) and generates them as ebooks.

[1910] "The means of providing the generated e-books online and enabling orders for printed versions" refers to a system that provides the generated e-books to users via the internet and also accepts orders for printed versions according to the user's wishes.

[1911] "Means of using a generative AI model throughout the entire picture book generation process" refers to a system that uses a generative AI model throughout the entire process, from image analysis to storytelling, to achieve a high level of automation and personalization.

[1912] The system of this invention automatically generates original picture books based on drawings made by children and provides them online. This system includes a series of processes in which the user inputs a drawing made by a child, the system analyzes the drawing to generate a story, and the system provides the story in ebook format.

[1913] The entire system consists of a smartphone application and a server. The primary hardware used is a smartphone and a server. The software includes a generative AI model (TensorFlow), Flask (a server-side web application framework), and React Native (a mobile application framework).

[1914] 1. User input and image upload

[1915] The user opens a smartphone application and takes a picture of or uploads a drawing made by their child. Along with the image file of the drawing, they also enter information such as the child's age, interests, and areas of expertise. This data is sent from the smartphone to the server.

[1916] 2. Image analysis and database storage on the server.

[1917] The server analyzes the received image file using a generation AI model (using TensorFlow). As a result of the image analysis, the main objects (e.g., people, animals, background) are identified. The analysis results are stored in a database (e.g., AWS RDS).

[1918] 3. Story Generation

[1919] The server automatically generates a story using a generative AI model based on the analyzed objects. In this generation process, appropriate plots and characters are set based on the uploaded image. For example, if an image of a dinosaur is uploaded, a story themed around "dinosaur adventures" will be generated.

[1920] 4. Customization and addition of educational elements

[1921] The generated stories are customized based on the child's age and interests. Furthermore, educational elements (e.g., scientific facts and learning elements) are added. For example, a story about a dinosaur discovering a new cave might incorporate information about different types of dinosaurs and their behavior.

[1922] 5. Generation and provision of picture book ebooks

[1923] The customized story is converted into an ebook format (e.g., EPUB or PDF). Users can view the generated ebook online. Printed versions can also be ordered if desired. The server sends the data to the printer, and the printed picture book is delivered to the user within a few days.

[1924] Examples of specific cases and prompt statements

[1925] As a concrete example, using a child's drawing of a dinosaur as a reference, use the following prompt:

[1926] I've uploaded a drawing of a dinosaur made by a child. Based on this drawing, please generate an adventure story in which the dinosaur discovers a new cave and finds fossils inside. Also, please add information to the story that teaches about different types of dinosaurs and their ecology.

[1927] This system allows children to become creators themselves, automatically generating and providing customized educational content tailored to their interests and age. It can also be provided in e-book format, and print versions are available upon request, accommodating diverse needs.

[1928] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1929] Step 1:

[1930] The user opens a smartphone application and takes a picture of their child's drawing or uploads it within the application. They also enter information such as the child's age, interests, and areas of expertise. This data is sent from the smartphone to the server. The smartphone sends the image file and text data via an HTTP request.

[1931] Input: Image file of a child's drawing, child's age, interests, and areas of expertise.

[1932] Output: Image files and text data sent to the server

[1933] Step 2:

[1934] The server stores the received image files and analyzes them using a generating AI model. Specifically, it uses an image analysis algorithm (TensorFlow) to identify key objects (people, animals, background, etc.) and saves the analysis results to a database.

[1935] Input: Image file, text data

[1936] Output: List of analyzed objects, saved to database.

[1937] Specific actions:

[1938] Receiving and saving image files

[1939] Image analysis using generative AI models

[1940] Identifying the main objects

[1941] Save the analysis results to a database (e.g., AWS RDS).

[1942] Step 3:

[1943] The server automatically generates stories using a generative AI model based on the information from the analyzed objects. Data such as age, interests, and areas of expertise entered by the user are also used at this stage.

[1944] Input: List of parsed objects, child's age, interests, areas of expertise

[1945] Output: Auto-generated story

[1946] Specific actions:

[1947] Combine the analysis results and additional information to generate a prompt message.

[1948] Input prompt text into the generative AI model and generate a story.

[1949] Step 4:

[1950] The server customizes automatically generated stories based on the child's age, interests, and strengths, and adds educational elements. For example, it incorporates scientific facts and learning elements into the stories.

[1951] Input: Auto-generated story, child's age, interests, areas of expertise

[1952] Output: Customized story

[1953] Specific actions:

[1954] Add educational elements to automatically generated stories.

[1955] Optimize the story based on the child's age and interests.

[1956] Step 5:

[1957] The server converts the customized story into an ebook format (e.g., EPUB or PDF). It then generates a link to provide the generated ebook online and notifies the user.

[1958] Input: Customized story

[1959] Output: ebook file, online link

[1960] Specific actions:

[1961] Convert story to ebook format

[1962] Generate a link for online viewing.

[1963] Notify the user of the link (e.g., send an email).

[1964] Step 6:

[1965] Users can view online-generated ebooks or, if desired, order printed versions. The server sends the print order data to the printer and arranges for the printing and delivery of the picture books.

[1966] Input: User's print order

[1967] Output: Printed picture book

[1968] Specific actions:

[1969] Received print order information.

[1970] Send order data to the printing company

[1971] A printing company prints the picture book and delivers it to the user.

[1972] This process allows for the easy creation of original picture books that leverage children's creativity, and these books are available in both electronic and print formats.

[1973] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1974] This invention relates to a system that automatically generates original picture books based on drawings made by children and further customizes the content by recognizing the user's emotions. Specifically, the invention describes a form that generates and provides picture books through the following process and combines it with emotion recognition.

[1975] 1. Upload the image and enter the information.

[1976] User:

[1977] Users upload drawings made by their children using the application or website. Users also enter necessary information such as the child's age, interests, and areas of expertise.

[1978] Terminal:

[1979] The device sends the image file uploaded by the user and any additional information entered to the server.

[1980] 2. Receiving and analyzing image data

[1981] server:

[1982] The server receives the uploaded image data and accompanying information and stores them in a database. Next, the server invokes a generative AI model for image analysis to identify the main objects depicted in the image (people, animals, background, etc.).

[1983] Specific example:

[1984] For example, if a child draws a picture of a dinosaur, the server recognizes the picture as a dinosaur object and saves it in the database with the label "dinosaur".

[1985] 3. Automatic story generation

[1986] server:

[1987] The server automatically generates a story using a generative AI model based on the analysis results. The generated story uses objects depicted in illustrations as plot elements and characters.

[1988] Specific example:

[1989] If a picture of a dinosaur is entered, the server will generate a story themed around "dinosaur adventures," depicting the challenges the dinosaur faces and how it overcomes them.

[1990] 4. Customization and Addition of Educational Content

[1991] server:

[1992] The server customizes the generated stories based on the child's age, interests, and areas of expertise. Furthermore, it adds scientific facts and learning elements to enhance their educational value.

[1993] Specific example:

[1994] If a 6-year-old child is interested in dinosaurs, the server will incorporate scientific facts about dinosaur species and their ecology into the story. For example, a scenario could be created where a dinosaur discovers a new cave and finds fossils within, incorporating learning about different dinosaur species along the way.

[1995] 5. Utilizing the Emotion Engine

[1996] server:

[1997] The server uses an emotion engine to analyze the user's emotions in real time. The emotion engine analyzes the user's facial expressions and voice through the camera and microphone to acquire emotional data.

[1998] Specific example:

[1999] While the user is browsing the picture book, the server analyzes the user's facial expressions through an emotion engine and recognizes emotions such as surprise and joy.

[2000] 6. Emotion-based real-time story adjustment

[2001] server:

[2002] The server adjusts the story content in real time based on feedback from the emotion engine. The story's progression and character actions change dynamically each time the user's emotions shift.

[2003] Specific example:

[2004] If the user shows a surprised expression in response to a story development, the server will add elements in the next scene that allow them to continue enjoying the surprise. Conversely, if the emotional data indicates sadness, the server will adjust the story and introduce a more positive development.

[2005] 7. Generation and provision of picture book data

[2006] server:

[2007] The server integrates customized stories and educational content, as well as emotionally-based adjustments, and converts them into ebook formats (EPUB and PDF).

[2008] server:

[2009] The server uploads the generated ebook data to the web server and generates a link that allows users to view it online. Printed versions of the picture books can also be ordered if desired.

[2010] Specific example:

[2011] Users can browse picture books based on children's drawings online and also order printed copies. When a user orders a printed copy, the server sends the data to the printer, and the printed book is delivered a few days later.

[2012] The system of this invention fosters children's creativity, deepens the parent-child bond, and provides educational value. Furthermore, by combining it with an emotion engine, it provides a more personalized experience that responds to the user's emotions, generating even greater emotion and excitement. This system is highly effective not only in quickly and easily generating original picture books and expanding learning opportunities, but also in providing a dynamic experience that responds to emotions.

[2013] The following describes the processing flow.

[2014] Step 1:

[2015] User:

[2016] The user opens an application or website for generating picture books and clicks the "Upload Picture" button.

[2017] Step 2:

[2018] Terminal:

[2019] The terminal displays a file selection window, providing a screen where the user can select the image file they want to upload.

[2020] Step 3:

[2021] User:

[2022] The user selects an image file of a picture, then enters the necessary information (such as the child's age, areas of interest, and areas of expertise), and presses the "Submit" button.

[2023] Step 4:

[2024] Terminal:

[2025] The terminal sends data, including the selected image file and the entered information, to the server.

[2026] Step 5:

[2027] server:

[2028] The server receives the data. It saves the image file and additional information to the database and verifies that the upload was successful.

[2029] Step 6:

[2030] server:

[2031] The server invokes a generative AI model to analyze the stored image files. The AI ​​model analyzes the image files and identifies key objects.

[2032] Step 7:

[2033] server:

[2034] The server labels each object based on the analysis results and stores them in the database. For example, if the image depicts a dinosaur, it will be labeled "dinosaur".

[2035] Step 8:

[2036] server:

[2037] The server uses the analysis results and accompanying information to invoke a generative AI model to generate a story. The generative AI model automatically generates a story based on the input prompts.

[2038] Step 9:

[2039] server:

[2040] The server retrieves the generated story and customizes it based on user-provided information (age, interests, areas of expertise).

[2041] Step 10:

[2042] server:

[2043] The server adds educational content to customized stories. For example, if a picture of a dinosaur is included, it will add scientific facts and educational elements about dinosaurs.

[2044] Step 11:

[2045] server:

[2046] The server uses an emotion engine to acquire data from the user's device's camera and microphone in order to analyze the user's emotions in real time.

[2047] Step 12:

[2048] Terminal:

[2049] The device uses an emotion engine to analyze the user's facial expressions and voice tone, and generates emotion data.

[2050] Step 13:

[2051] server:

[2052] Based on feedback from the emotion engine, the server analyzes the user's emotions and recognizes feelings such as positive, negative, and surprise.

[2053] Step 14:

[2054] server:

[2055] The server adjusts the story content in real time based on the recognized emotion data. For example, if the user makes a surprised expression, the next scene will be designed to add a further element of surprise.

[2056] Step 15:

[2057] server:

[2058] Based on emotional data, the server dynamically changes the story's progression and character actions to provide content that aligns with the user's interests and emotions.

[2059] Step 16:

[2060] server:

[2061] The server converts the final integrated document data into an ebook format (EPUB or PDF) and uploads the generated ebook data to the web server.

[2062] Step 17:

[2063] server:

[2064] The server generates a link that allows the user to view the picture book online and provides the user with the link.

[2065] Step 18:

[2066] Terminal:

[2067] The terminal retrieves a link provided by the server and provides an interface that allows the user to view picture books online.

[2068] Step 19:

[2069] User:

[2070] Users can view picture books generated online. They can also click the "Order a Printed Version" button if they wish.

[2071] Step 20:

[2072] Terminal:

[2073] The terminal displays an order form for entering shipping and payment information.

[2074] Step 21:

[2075] User:

[2076] The user enters the required information and presses the "Confirm Order" button.

[2077] Step 22:

[2078] server:

[2079] The server receives order data, processes payments, and sends print data to the printer.

[2080] Step 23:

[2081] Printing company:

[2082] The printing company prints the picture book based on the received data and delivers it to the specified address.

[2083] Step 24:

[2084] server:

[2085] The server monitors the progress of the order and notifies the user when shipping is confirmed.

[2086] (Example 2)

[2087] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2088] Children today have limited opportunities to exercise their creativity in the digital age. Furthermore, educational content is delivered in a uniform manner, lacking opportunities for learning tailored to each child's individual interests and strengths. In addition, the technology for personalizing content based on user emotions is still immature, making it difficult to provide a greater sense of immersion and emotional impact. This invention aims to solve these problems by automatically generating original picture books and providing children with educationally valuable and emotionally engaging content.

[2089] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for inputting a picture drawn by a child, means for analyzing the picture and identifying the main objects, means for automatically generating a story based on the analyzed objects, means for customizing the generated story based on the child's age, interests, and areas of expertise, means for adding educational content based on the customized story, means for analyzing the user's emotions and adjusting the story in real time, means for converting the generated story into an ebook format, means for providing the generated picture book online, and means for ordering a printed version of the generated picture book. This makes it possible to generate a personalized, educational, and emotionally engaging picture book based on a child's picture and adjust it in real time.

[2090] "A means of inputting children's drawings" refers to an interface for importing children's drawings into a system in an electronic format.

[2091] "Methods for analyzing drawings and identifying key objects" refers to technologies that analyze uploaded drawings and extract and identify specific elements (such as people, animals, backgrounds, etc.) from them.

[2092] "Means for automatically generating stories based on analyzed objects" refers to an algorithm or system for creating a narrative based on identified object information.

[2093] "A means of customizing generated stories based on a child's age, interests, and strengths" refers to a function that adjusts the generated stories to correspond with the information of the child that has been entered.

[2094] "Means of adding customized story-based educational content" refers to technologies for inserting educational information and learning elements into a tailored story.

[2095] "A means of analyzing user emotions and adjusting the story in real time" refers to a system that evaluates emotional data such as the user's facial expressions and voice in real time and dynamically changes the development of the story accordingly.

[2096] "Methods for converting generated stories into ebook formats" refers to methods for converting customized stories and educational content into ebook formats (such as EPUB or PDF).

[2097] "Methods for providing generated picture books online" refers to a system that makes completed ebooks accessible to users via the internet.

[2098] "Method for ordering a printed version of the generated picture book" refers to the procedure for users to order a printed version of the generated e-book as a physical book, if they so desire, via the internet.

[2099] This invention relates to a system that automatically generates original picture books based on drawings made by children and further customizes the content by recognizing the user's emotions. Specifically, the program is implemented as follows.

[2100] First, users upload drawings made by their children using an application or website. In addition, users enter information such as the child's age, interests, and areas of expertise. This entered drawing and information are sent to the server by the device. Image files are sent in JPEG or PNG format, and the input information is sent in JSON format.

[2101] The server stores the received image data and associated information in a database. It then calls a generative AI model (e.g., TensorFlow or PyTorch) for image analysis to identify the main objects depicted in the image (people, animals, background, etc.). For example, if an image of a dinosaur is input, the server recognizes "dinosaur" as the object and stores it as a label in the database.

[2102] Based on the analysis results, the server automatically generates a story using a generative AI model. The generative AI model is given instructions using prompts. A specific example of a prompt is: "Based on the input dinosaur picture, generate a story themed around a 'dinosaur adventure' that a 6-year-old child can understand. Include content where the dinosaur discovers a new cave, finds fossils inside, and learns about different types of dinosaurs along the way." As a result, a story template is generated and saved in the database.

[2103] Next, the server customizes the generated story based on the child's age, interests, and areas of expertise. For example, if a 6-year-old child is interested in dinosaurs, scientific facts about dinosaur species and their ecology will be incorporated into the story. Furthermore, learning content will be added to enhance its educational value.

[2104] The server uses an emotion engine to analyze the user's emotions in real time. This involves analyzing facial expressions and voice through the user's device's camera and microphone to acquire emotional data. For example, if a user shows expressions of surprise or joy while viewing a picture book, the server analyzes that emotion and adds elements in the next scene that allow them to continue enjoying the surprise. Conversely, if the user expresses sadness, a positive development is introduced.

[2105] Finally, the server integrates the customized story and educational content, as well as emotionally-based adjustments, and converts it into an ebook format (EPUB or PDF). This ebook data is uploaded to a web server, and users can view it online. Users can also order printed copies of the picture book if they wish. Specifically, the server sends the data to a printer, and the printed book is delivered to the user a few days later.

[2106] Through specific data processing and calculations performed at each processing step of the program, this invention leverages children's creativity, deepens the parent-child bond, and provides highly educational content. Furthermore, by combining it with an emotion engine, it provides a more personalized experience that responds to the user's emotions, generating even greater emotion and excitement.

[2107] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2108] Step 1:

[2109] Users upload drawings made by their children through the application or website and enter information such as the child's age, interests, and areas of expertise. Specifically, the user clicks the "Upload Drawing" button in the application, selects the image file, fills in the child's information in the input form, and clicks the "Submit" button.

[2110] input:

[2111] Children's drawings (image files), information such as the child's age, interests, and areas of expertise.

[2112] output:

[2113] The entered image and information (sent to the server).

[2114] Step 2:

[2115] The terminal sends the image file uploaded by the user and the input information to the server. The image file is sent in JPEG or PNG format, and the input information is sent in JSON format.

[2116] Specific operation:

[2117] The terminal responds to user input and sends image files and JSON-formatted information to the server as an HTTP request.

[2118] input:

[2119] Image files and input information submitted by the user.

[2120] output:

[2121] Image files and input information sent to the server.

[2122] Step 3:

[2123] The server stores the received image data and associated information in a database. This allows for centralized management of all necessary information.

[2124] Specific operation:

[2125] The server temporarily stores the image file, and then stores the image file and information in JSON format in the database.

[2126] input:

[2127] Image files and input information sent from the device.

[2128] output:

[2129] Image files and information stored in the database.

[2130] Step 4:

[2131] The server invokes a generative AI model for image analysis to identify the main objects depicted in the image. Image recognition models (such as TensorFlow or PyTorch) are used for the object identification process.

[2132] Specific operation:

[2133] The server generates and inputs the saved image files into an AI model. The model analyzes the images and identifies key objects.

[2134] input:

[2135] Image files stored in the database.

[2136] output:

[2137] Identified object information (e.g., dinosaur, tree, house, etc.).

[2138] Step 5:

[2139] The server generates prompt statements based on the object analysis results and inputs them into a generative AI model (e.g., OpenAI's GPT) to generate a story. A story template is created.

[2140] Specific operation:

[2141] The server generates a prompt message. Based on the information "a picture of a dinosaur" and "age 6," it generates a prompt message such as "Please generate a story about a dinosaur on an adventure." This prompt message is input into the AI ​​model to generate the story.

[2142] input:

[2143] Identified object information, user input information.

[2144] output:

[2145] A template for the generated story.

[2146] Step 6:

[2147] The server customizes the generated story based on the child's age, interests, and areas of expertise. For example, it might incorporate scientific facts about different types of dinosaurs and their behavior into the story.

[2148] Specific operation:

[2149] The server retrieves appropriate learning elements from the educational content database based on a story template and integrates them into the story.

[2150] input:

[2151] The generated story template and user input information.

[2152] output:

[2153] A customized story.

[2154] Step 7:

[2155] The server uses an emotion engine to analyze the user's emotions in real time. It uses the user's device's camera and microphone to analyze facial expressions and voice, and acquires emotional data.

[2156] Specific operation:

[2157] The server obtains access permissions to the user's camera and microphone, and analyzes facial expressions and voice in real time. It retrieves data through an emotion analysis API and evaluates the user's emotions.

[2158] input:

[2159] Real-time facial expression and voice data of the user.

[2160] output:

[2161] User sentiment data.

[2162] Step 8:

[2163] The server adjusts the story in real time based on feedback from the emotion engine. Every time the user's emotions change, the story's progression and the characters' actions are dynamically altered.

[2164] Specific operation:

[2165] The server modifies the next scene of the story based on emotional data. If positive emotions are detected, additional adventure elements are added. If negative emotions are detected, a positive development is introduced.

[2166] input:

[2167] User sentiment data, customized stories.

[2168] output:

[2169] A story that is adjusted in real time.

[2170] Step 9:

[2171] The server ultimately integrates customized stories, educational content, and emotionally-based adjustments, and converts them into ebook formats (EPUB or PDF). This is then uploaded to a web server, generating a link that users can access online.

[2172] Specific operation:

[2173] The server converts the story into an ebook format, uploads the file to the web server, and generates a URL.

[2174] input:

[2175] A well-structured story and content.

[2176] output:

[2177] eBook format file, viewing link.

[2178] Step 10:

[2179] The server then processes orders for printed copies of the generated picture books for users who wish to receive them.

[2180] Specific operation:

[2181] When a user orders a printed version, the server sends the data to the printer and proceeds with the process of delivering the printed picture book to the user.

[2182] input:

[2183] eBook format files, user print order information.

[2184] output:

[2185] Data sent to the printer, printed picture book delivered to the user.

[2186] (Application Example 2)

[2187] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2188] Conventional picture book generation systems are limited to one-way generation based on children's drawings and cannot adjust the story in real time in response to the user's emotions. Furthermore, the content of the picture books is templated, making it difficult to maximize children's creativity and learning effectiveness. This invention aims to solve these problems and provide a more personalized experience.

[2189] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for inputting a picture drawn by a child, means for analyzing the picture and identifying the main objects, means for automatically generating a story based on the analyzed objects, means for customizing the generated story based on the child's age, interests, and areas of expertise, means for analyzing the user's emotions, means for adjusting the story in real time based on the emotion data, means for generating a picture book containing the customized story, and means for providing the generated picture book online. This makes it possible to provide an original picture book that integrates learning elements based on a child's drawing, and further makes it possible to adjust the story in real time according to the user's emotions.

[2190] "Children's drawings" refers to artwork or sketches drawn by children, whether in digital or physical form.

[2191] "Analysis" refers to the process of breaking down images and data to understand their contents and identify specific objects and attributes.

[2192] "Key objects" refer to elements identified through image analysis that play a crucial role in the story's plot and characters.

[2193] "Methods for automatically generating stories" refer to technologies such as artificial intelligence that construct narratives based on input data and generate text and graphics.

[2194] "Customization" refers to the process of changing or adapting content and functions to suit the preferences and requirements of a specific user.

[2195] "User emotions" refer to the psychological state and feedback perceived through the user's facial expressions, tone of voice, and other similar actions.

[2196] "Emotional data" refers to digital information that indicates a user's emotional state, acquired using devices such as cameras and microphones.

[2197] "Adjusting in real time" refers to the process of making immediate changes and optimizations in response to the situation and data at hand.

[2198] "Means of generating picture books" refers to systems and software that convert stories, including text and illustrations, into digital or physical book formats.

[2199] "Methods of providing content online" refers to methods of delivering ebooks and digital content to users using the internet.

[2200] This invention describes a system that generates original picture books based on drawings made by children and customizes the content in real time based on the user's emotions. In a specific embodiment, the present invention is carried out using the following procedure and apparatus.

[2201] 1. Main hardware and software used

[2202] Smart glasses camera: Used to scan and upload children's drawings.

[2203] Server: The central component responsible for analyzing images, generating stories, analyzing emotions, and customizing picture books.

[2204] OpenCV: A library for capturing and preprocessing images.

[2205] Transformers (Generative AI Model): A library for analyzing images and generating stories.

[2206] EmotionRecognizer: Software for analyzing user emotions.

[2207] PDFGenerator: Used to generate the final picture book.

[2208] 2. Specific System Processing Overview

[2209] First, the user scans a child's drawing using smart glasses. This image is uploaded to a server. The server analyzes the image using OpenCV to identify key objects. Next, a generative AI model is used to automatically generate a story based on these objects.

[2210] The generated stories are customized based on the child's age, interests, and strengths, and educational content is also added. Next, the server uses EmotionRecognizer to analyze the user's emotions in real time and adjusts the story's progression based on the emotional data.

[2211] Finally, the customized story is generated in ebook format using PDFGenerator. This provides users with a link to view it online. Printed versions of the picture book can also be ordered upon request.

[2212] 3. Specific Examples and Examples of Prompt Statements

[2213] For example, if a child draws a picture of a dinosaur, they can use smart glasses to scan the drawing and upload it to a server. A generative AI model then generates a story themed around a dinosaur adventure. If the user shows a surprised expression, an emotion analysis engine detects this reaction and adds further surprising elements to the story. Finally, this customized picture book is delivered as an ebook.

[2214] Example of a prompt:

[2215] "Please have the generative AI model analyze the dinosaur images and generate a story."

[2216] "Use the emotion analysis engine to adjust the story's progression to match the user's emotions."

[2217] The above describes a specific embodiment for carrying out the present invention. This embodiment makes it possible to easily and quickly generate original picture books using children's drawings and to provide personalized stories that respond to emotions.

[2218] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2219] Step 1:

[2220] The user scans a child's drawing using smart glasses. The smart glasses' camera captures the drawing and generates image data. Here, the input is the image data of the drawing acquired through the smart glasses' camera, and the output is that image data.

[2221] Step 2:

[2222] The device uploads the acquired image data to the server. It uses a communication module within the smart glasses to transmit the images to the server via the internet. The input is the image data, and the output is the image data that has been sent to the server.

[2223] Step 3:

[2224] The server uses OpenCV to analyze uploaded images and identify key objects. Specifically, it extracts features of objects (e.g., dinosaurs) from the image data and identifies them. The input is the uploaded image data, and the output is a list of identified key objects.

[2225] Step 4:

[2226] The server uses a generative AI model to automatically generate a story based on identified key objects. The generative AI model uses the input key objects (e.g., dinosaurs) as plot elements and characters to create the story. The input is a list of key objects, and the output is the automatically generated story.

[2227] Step 5:

[2228] The server customizes the generated story based on the child's age, interests, and strengths. It adjusts the story content based on the child's information entered by the user and adds educational content as needed. The input consists of the automatically generated story and the child's information entered by the user, while the output is the customized story.

[2229] Step 6:

[2230] The server uses EmotionRecognizer to analyze the user's emotions in real time. Specifically, it analyzes data acquired from the smart glasses' camera and microphone to identify emotions from the user's facial expressions and tone of voice. The input is video and audio streams, and the output is the analyzed emotion data.

[2231] Step 7:

[2232] The server adjusts the story in real time based on sentiment data. For example, if a user reacts with surprise, the server adjusts the next development of the story accordingly. The input is sentiment data and a customized story, and the output is the adjusted story.

[2233] Step 8:

[2234] The server converts the customized and adjusted story into an ebook format using PDFGenerator. Specifically, it integrates the story text and associated images and saves them as EPUB or PDF files. The input is the adjusted story, and the output is the generated ebook.

[2235] Step 9:

[2236] The server uploads the generated ebook to a web server and generates a link that users can access online. It also allows users to order a printed version of the picture book if desired. The input is the generated ebook, and the output is the online link and the option to order a print copy.

[2237] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2238] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2239] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[2240] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2241] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[2242] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[2243] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[2244] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[2245] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[2246] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[2247] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[2248] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[2249] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[2250] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2251] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[2252] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[2253] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[2254] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[2255] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[2256] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[2257] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[2258] The following is further disclosed regarding the embodiments described above.

[2259] (Claim 1)

[2260] A means of inputting a child's drawing,

[2261] A means for analyzing the aforementioned drawing and identifying the main objects,

[2262] A means for automatically generating a story based on the analyzed object,

[2263] A means for customizing the generated story based on the child's age, interests, and strengths,

[2264] A means for generating a picture book containing the customized story,

[2265] A system including means for providing the generated picture books online.

[2266] (Claim 2)

[2267] The system according to claim 1, further comprising means for adding educational content to the customized picture book.

[2268] (Claim 3)

[2269] The system according to claim 1, further comprising means for ordering printed copies of the generated picture book.

[2270] "Example 1"

[2271] (Claim 1)

[2272] A means of inputting a child's drawing,

[2273] A means for uploading the aforementioned input picture along with information such as the child's age, interests, and areas of expertise,

[2274] Means for sending the uploaded image and information to the server,

[2275] The means for storing the image received by the server in a database,

[2276] A means for analyzing the aforementioned saved image using a generation AI model to identify the main object,

[2277] A means for automatically generating a story based on the identified object,

[2278] A means for customizing the automatically generated story based on the child's age, interests, and strengths,

[2279] A means for converting the customized story into an ebook format,

[2280] A system including means for providing the generated picture books online.

[2281] (Claim 2)

[2282] The system according to claim 1, further comprising means for adding educational content to the customized picture book.

[2283] (Claim 3)

[2284] The system according to claim 1, further comprising means for ordering printed copies of the generated picture book.

[2285] "Application Example 1"

[2286] (Claim 1)

[2287] A means of inputting a child's drawing,

[2288] A means for analyzing the aforementioned drawing and identifying the main objects,

[2289] A means for automatically generating a story based on the analyzed object,

[2290] A means for customizing the generated story based on the child's age, interests, and strengths,

[2291] A method for generating the picture book in ebook format,

[2292] A means of providing the generated e-book online and also allowing orders for print versions,

[2293] A system that includes means for using a generation AI model throughout the entire picture book generation process.

[2294] (Claim 2)

[2295] The system according to claim 1, which adds educational content to the generated story, including content customized based on the child's age and interests.

[2296] (Claim 3)

[2297] The system according to claim 1, further comprising means for ordering a printed picture book based on the generated ebook.

[2298] "Example 2 of combining an emotion engine"

[2299] (Claim 1)

[2300] A means of inputting a child's drawing,

[2301] A means for analyzing the aforementioned drawing and identifying the main objects,

[2302] A means for automatically generating a story based on the analyzed object,

[2303] A means for customizing the generated story based on the child's age, interests, and strengths,

[2304] A means for adding educational content based on the aforementioned customized story,

[2305] A means to analyze user emotions and adjust the story in real time,

[2306] A means for converting the generated story into an ebook format,

[2307] A means of providing the generated picture book online,

[2308] A means for ordering the printed version of the generated picture book

[2309] Includes system.

[2310] (Claim 2)

[2311] The system according to claim 1, characterized in that the means for analyzing emotions analyzes the user's facial expressions and voice.

[2312] (Claim 3)

[2313] The system according to claim 1, characterized in that it is capable of real-time story adjustment based on the aforementioned emotions.

[2314] "Application example 2 when combining with an emotional engine"

[2315] (Claim 1)

[2316] A means of inputting a child's drawing,

[2317] A means for analyzing the aforementioned drawing and identifying the main objects,

[2318] A means for automatically generating a story based on the analyzed object,

[2319] A means for customizing the generated story based on the child's age, interests, and strengths,

[2320] A means of analyzing user emotions,

[2321] A means for adjusting the story in real time based on the aforementioned emotional data,

[2322] A means for generating a picture book containing the customized story,

[2323] A system including means for providing the generated picture books online.

[2324] (Claim 2)

[2325] The system according to claim 1, further comprising means for adding educational content to the customized picture book.

[2326] (Claim 3)

[2327] The system according to claim 1, further comprising means for ordering printed copies of the generated picture book. [Explanation of Symbols]

[2328] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of inputting a child's drawing, A means for analyzing the aforementioned drawing and identifying the main objects, A means for automatically generating a story based on the analyzed object, A means for customizing the generated story based on the child's age, interests, and strengths, A means for generating a picture book containing the customized story, A system including means for providing the generated picture books online.

2. The system according to claim 1, further comprising means for adding educational content to the customized picture book.

3. The system according to claim 1, further comprising means for ordering printed copies of the generated picture book.

Images