System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting registration information by a user and transmitting the registration information to a server; means for storing the registration information in a database by the server; means for inputting photographic data and educational content by the user and transmitting the photographic data and the educational content to the server; means for generating a story and an illustration by the server based on the photographic data and the educational content; and means for displaying the story and the illustration received by the terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Parents of young children often find it difficult to find educational picture books that fit the content they want their children to learn. Furthermore, commercially available picture books tend to be generic and not personalized to each child's interests or themes they want to learn. The present invention aims to easily create picture books tailored to specific lessons and family situations, providing an environment where children can learn while having fun. [Means for solving the problem]

[0005] The present invention achieves the object to be solved by a system including the following means.

[0006] The system includes a means for a user to input registration information and send it to a server, a means for the server to save the registration information in a database, a means for a user to input photo data and educational content and send it to the server, a means for the server to generate a story and create illustrations based on the photo data and educational content, a means for the server to transmit the generated story and illustrations to a terminal, and a means for the terminal to display the received story and illustrations, thereby reducing the burden on parents and providing appropriate learning opportunities for children.

[0007] "User" refers to an individual who uses this system, specifically a parent or guardian or educator.

[0008] A "terminal" is an electronic device used by a user to operate, specifically a smartphone, tablet, PC, or other device.

[0009] "Server" refers to a central processing unit that receives, stores, and processes user input information and distributes generated content to terminals.

[0010] "Registration information" refers to personal information and authentication information provided by users to use the system, specifically name, email address, password, etc.

[0011] "Photo Data" refers to image files of family members and children that users upload to the system.

[0012] "Educational content" refers to the lessons or themes that the user wants their child to learn, specifically, learning goals such as "greetings are important."

[0013] A "story" is a narrative generated based on educational content, and refers to a narrative structure created using specified photographs and information.

[0014] "Illustrations" are visual images generated from uploaded photos and used as characters within stories.

[0015] A "database" is a collection of information stored in a server, including user information, photo data, educational content, generated content, and the like.

[0016] "Reactions" refers to the actions and reactions that children show when using the app, specifically facial expressions and words recorded by the camera and microphone. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The system of the present invention generates a personalized educational picture book through the mutual cooperation of a user, a terminal, and a server. A specific embodiment of the system will be described below.

[0039] User Registration and Login

[0040] 1. User Registration

[0041] The user downloads the app to their device and launches it.

[0042] The device displays a registration form to the user, which includes fields for entering name, email address, password, etc.

[0043] The user enters the required information and presses the registration button.

[0044] The terminal transmits the entered registration information to the server.

[0045] The server stores the received registration information in a database and notifies the terminal that registration has been completed.

[0046] The terminal displays a message to the user indicating that registration is complete.

[0047] 2. Log in

[0048] The user launches the app again and goes to the login screen.

[0049] The terminal prompts the user to enter an email address and password.

[0050] The user enters the information and clicks the login button.

[0051] The terminal sends the input information to the server for authentication.

[0052] If the server is successful in authenticating the user, it sends the user's home screen to the device. If authentication fails, it displays an error message.

[0053] Upload photos and assign lessons

[0054] 1. Upload a photo

[0055] The user selects "Upload Photo" from the menu.

[0056] The device prompts the user to select a family photo to upload.

[0057] The user selects a photo and presses the upload button.

[0058] The device sends the photo data to the server.

[0059] The server receives the photo data and stores it in a database.

[0060] 2. Designation of lessons learned

[0061] The user is taken to the "Specify Lessons Learned" screen.

[0062] The device prompts the user to enter more information about the lesson or story they want to learn.

[0063] The user inputs a lesson such as "Greetings are important."

[0064] The terminal transmits the input lesson information to the server.

[0065] The server receives the lesson information and stores it in a database.

[0066] Creating and viewing stories

[0067] 1. Story generation

[0068] The server starts generating a story based on the photo data and the lesson information.

[0069] The server uses conversational AI to automatically generate a story that follows the specified lesson.

[0070] The server generates illustrations of the characters based on the uploaded photos.

[0071] 2. Integration and transmission of picture book data

[0072] The server combines the generated story and illustrations to create picture book data.

[0073] The server sends the completed picture book data to the terminal.

[0074] 3. Display of picture books

[0075] The terminal displays the received picture book data to the user.

[0076] Users can read picture books together with their children and spend time learning.

[0077] Reaction monitoring (optional)

[0078] 1. Use of camera and microphone

[0079] The device will use its camera and microphone to give you the option to record your child's responses.

[0080] If the user selects this option, the device will start recording.

[0081] 2. Submitting and analyzing reaction data

[0082] The terminal transmits the recorded reaction data to the server.

[0083] The server analyzes the response data and uses it to generate the next story.

[0084] Gathering feedback and incorporating it into next time

[0085] 1. Enter your feedback

[0086] After finishing reading the picture book, the device displays a feedback form to the user.

[0087] The user inputs customization requests for the story and illustrations.

[0088] 2. Sending and storing feedback data

[0089] The terminal transmits the input feedback data to the server.

[0090] The server stores the feedback in a database and reflects it in the next story generation.

[0091] The above is a specific embodiment of the present invention. This system allows parents to easily create educational picture books tailored to their children and provide them with learning that is appropriate for their children.

[0092] The processing flow will be explained below.

[0093] Step 1: User Registration and Login

[0094] 1. Install the app

[0095] The user downloads the app to their device and launches it.

[0096] 2. Displaying the user registration form

[0097] The terminal displays a registration form to the user.

[0098] 3. Enter your registration information

[0099] The user enters their name, email address, and password.

[0100] 4. Submission of Registration Information

[0101] The terminal transmits the input information to the server.

[0102] 5. Saving your registration information

[0103] The server verifies the received registration information and stores it in a database.

[0104] 6. Notification of registration completion

[0105] The server sends a registration completion message to the terminal.

[0106] The terminal displays a registration complete message to the user.

[0107] 7. Display the login screen

[0108] The device displays the login screen.

[0109] 8. Enter your authentication information

[0110] The user enters their email address and password.

[0111] 9. Submitting authentication information

[0112] The terminal transmits the entered authentication information to the server.

[0113] 10. Credential Verification

[0114] The server verifies the authentication information and sends the result to the device.

[0115] 11. Notification of Authentication Completion

[0116] If the server is successful in the authentication, it sends the user's home screen data to the terminal.

[0117] The device displays the user's home screen and completes the login.

[0118] Step 2: Upload photos and specify lessons

[0119] 1. Display the photo selection screen

[0120] The terminal displays a screen that allows the user to select a family photo.

[0121] 2. Select a photo

[0122] The user selects a family photo to upload.

[0123] 3. Sending photo data

[0124] The terminal transmits the selected photo data to the server.

[0125] 4. Saving photo data

[0126] The server stores the received photo data in a database.

[0127] 5. Display the lesson specification screen

[0128] The terminal displays a screen that prompts the user to input the lesson they want to learn.

[0129] 6. Input of lessons learned

[0130] The user inputs a lesson such as "Greetings are important."

[0131] 7. Submission of lessons learned data

[0132] The terminal transmits the input lesson data to the server.

[0133] 8. Lessons Learned Data Storage

[0134] The server stores the received lesson data in a database.

[0135] Step 3: Generate and view stories

[0136] 1. Start creating a story

[0137] The server retrieves the photo data and lesson data from the database.

[0138] 2. Story Generation

[0139] Based on the data acquired by the server, a story is generated using conversational AI.

[0140] 3. Illustration Generation

[0141] The server generates illustrations of the characters based on the uploaded photos.

[0142] 4. Picture book data integration

[0143] The server combines the generated story and illustrations to create picture book data.

[0144] 5. Sending picture book data

[0145] The server sends the completed picture book data to the terminal.

[0146] 6. Display of picture books

[0147] The terminal displays the received picture book data to the user.

[0148] 7. Picture book reading

[0149] The user reads a picture book with the child.

[0150] Step 4: Monitoring the reaction (optional)

[0151] 1. Turn on the camera and microphone

[0152] The device will activate the camera and microphone according to the user's selection.

[0153] 2. Recording the reaction

[0154] The device records the child's responses.

[0155] 3. Submitting reaction data

[0156] The terminal transmits the recorded reaction data to the server.

[0157] 4. Saving and analyzing reaction data

[0158] The server stores the received response data and uses it for the next story generation.

[0159] Step 5: Gather feedback

[0160] 1. Display the feedback form

[0161] The device displays a feedback form after the picture book has finished reading.

[0162] 2. Enter your customization requests

[0163] The user inputs customization requests for the story and illustrations.

[0164] 3. Sending feedback data

[0165] The terminal transmits the input feedback data to the server.

[0166] 4. Feedback Data Storage

[0167] The server stores the received feedback in a database.

[0168] 5. Reflection in next story generation

[0169] The server analyzes the feedback data and reflects it in the next story generation.

[0170] Example 1

[0171] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0172] In modern education, there is a demand for personalized educational materials tailored to each child, but existing systems have difficulty in providing such personalized education. Furthermore, because the content and format of educational materials are fixed, it is difficult to provide education that reflects the interests and reactions of children. Therefore, a system is needed that can effectively provide individual educational experiences tailored to each child.

[0173] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0174] In this invention, the server includes means for a user to input registration information and transmit it to the server, means for the server to store the registration information in a database, means for a user to input photo data and educational information and transmit it to the server, means for the server to automatically generate a story and create illustrations using a generative AI model based on the photo data and educational information, means for the server to combine the generated story and illustrations to create picture book data and transmit it to the terminal, and means for the terminal to display the picture book data received. This enables users to easily create personalized educational picture books and provide learning content tailored to their children.

[0175] A "user" is an individual or group that uses this system and is the entity that creates and uses educational picture books.

[0176] A "terminal" is a device used by a user, such as a computer, smartphone, or tablet, that provides an interface with the system.

[0177] A "server" is the central part of a system and is a computer system that stores, processes, and manages data.

[0178] "Registration information" refers to information such as name, email address, and password required for a user to access the system.

[0179] The "database" is a data storage system for storing and managing user registration information, photo data, educational information, generated picture book data, and the like.

[0180] "Photo data" refers to image files uploaded by users and used for characters and backgrounds in picture books.

[0181] "Educational information" is detailed information about the lessons and stories that the user wants to learn as the content of the picture book.

[0182] A "generative AI model" is an artificial intelligence that automatically generates text and images based on data, such as ChatGPT.

[0183] A "story" is a narrative created by a generative AI model based on educational information.

[0184] "Illustrations" are images of characters and backgrounds generated based on uploaded photo data.

[0185] "Picture book data" is data in the form of an e-book created by combining the generated story and illustrations.

[0186] "Customization requests" are specific requests for changes or improvements that users input regarding stories or illustrations.

[0187] "Reaction data" refers to audio and video data recorded with a camera or microphone of a child's reaction when a picture book is read to them.

[0188] The system of the present invention generates a personalized educational picture book through the mutual cooperation of a user, a terminal, and a server. A specific embodiment of the system will be described below.

[0189] User Registration and Login Implementation

[0190] First, the user downloads the application to their device and launches it. The device displays a registration form to the user, and the user enters registration information such as their name, email address, and password. The entered registration information is sent from the device to the server, and the server stores the information in a database. The server then notifies the device that registration is complete, and the device displays a message to the user that registration is complete. When the user launches the application again, the user enters their email address and password on the login screen and sends them to the server via the device. If the server is successful in its authentication, the user's home screen is sent to the device.

[0191] Uploading photos and assigning lessons

[0192] When the user selects "Upload Photos" from the menu, the device prompts the user to select a family photo to upload. When the user selects a photo and presses the upload button, the photo data is sent from the device to the server. The server receives the photo data and stores it in a database. Next, the user moves to the "Specify Lesson" screen and enters educational information, such as "Greetings are important." The entered educational information is sent to the server via the device, and the server stores the educational information in a database.

[0193] Story Generation and Display Embodiments

[0194] The server begins generating a story based on the photo data and educational information. It uses generative AI models such as ChatGPT to automatically generate a story that follows the specified moral. It also generates illustrations of the characters based on the uploaded photos. The server then combines the generated story and illustrations to create picture book data and sends it to the device. The device then displays the received picture book data to the user, who can then read the book together with their child.

[0195] Reaction Monitoring Embodiments (Optional)

[0196] The device provides an option to record the child's reactions using the camera and microphone. When the user selects this option, the device starts recording. The recorded reaction data is sent from the device to a server, which analyzes the reaction data and uses it to generate the next story. This analysis includes emotion analysis using a voice recognition API.

[0197] How to collect feedback and incorporate it into future projects

[0198] After reading the picture book, the device displays a feedback form to the user. The user enters any customization requests for the story or illustrations and sends them from the device to the server. The server stores the customization requests in a database and reflects them in the next story generation.

[0199] Specific examples

[0200] Users download the app, enter their registration information, and send it to the server. Next, they upload a family photo and specify a lesson, such as "greetings are important." The server uses a generative AI model such as ChatGPT to generate a story and creates illustrations of the characters based on the photos. The completed picture book data is sent to the device, and users can provide learning opportunities by reading the book together with their children.

[0201] An example prompt is "Generate a children's story with the moral that saying hello is important. The story will feature a little bear, and include illustrations based on uploaded family photos."

[0202] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0203] Step 1:

[0204] A user downloads the app and enters their registration information.

[0205] The user downloads the app to their device and launches it. The device displays a form for the user to enter their name, email address, and password. The user enters the required information and presses the "Register" button. The entered information is temporarily stored in the device's memory.

[0206] Input: Name, email address, password

[0207] Output: The registration information is temporarily stored in the device's memory.

[0208] Step 2:

[0209] The terminal sends the registration information to the server, which stores it in a database.

[0210] The terminal sends the registration information entered by the user to the server. The server receives the registration information and stores it in a database using an SQL query. The server sends a notification of registration completion to the terminal. The terminal displays a message to the user notifying them that registration was successful.

[0211] Input: Registration information (name, email address, password)

[0212] Output: The registration information is saved in the database and a message indicating successful registration is displayed.

[0213] Step 3:

[0214] The user logs in again and authenticates.

[0215] The user launches the app again and goes to the login screen. The device prompts the user to enter their email address and password. The user enters the information and presses the "Login" button. The device sends the entered information to the server, which then references the database and performs authentication. If authentication is successful, the server sends the user's home screen to the device.

[0216] Input: Email address, password

[0217] Output: If authentication is successful, the home screen is displayed.

[0218] Step 4:

[0219] The user uploads a photo.

[0220] The user selects "Upload photo" from the menu. The device displays a file selection dialog, and the user selects the photo to upload. The user selects a photo and presses the "Upload" button. The device sends the photo data to the server, and the server saves the received photo data in a database.

[0221] Input: Photo data

[0222] Output: The photo data is saved in the database.

[0223] Step 5:

[0224] The user inputs the lesson information.

[0225] The user moves to the "Specify Lesson" screen and inputs the lesson information. The terminal displays an input form for educational information, and the user inputs a lesson such as "Greetings are important." The user inputs the information and presses the "Send" button. The terminal sends the input educational information to the server, which stores it in a database.

[0226] Input: Lesson learned information

[0227] Output: Lessons learned information is stored in a database.

[0228] Step 6:

[0229] The server uses a generative AI model to generate a story and create illustrations.

[0230] The server starts generating a story based on the photo data and lesson information. It inputs a prompt (e.g., "Greetings are important") into a generative AI model such as ChatGPT to generate a story. The server uses an image processing library to generate illustrations of people based on the uploaded photos.

[0231] Input: Photo data, lesson information

[0232] Output: Generated stories and illustrations

[0233] Step 7:

[0234] The server integrates the generated story and illustrations into picture book data and sends it to the terminal.

[0235] The server combines the generated story and illustrations to create picture book data. The server uses a PDF generation library to combine the text and illustrations into PDF format. The completed picture book data is sent to the device.

[0236] Input: Generated story and illustrations

[0237] Output: Picture book data is sent to the terminal.

[0238] Step 8:

[0239] The device displays the picture book, and the user reads it together with the child.

[0240] The device displays the received picture book data. Using the device's built-in PDF viewer, the user can read the book together with their child.

[0241] Input: Picture book data

[0242] Output: The picture book is displayed on the device.

[0243] Step 9:

[0244] Optionally, the device records the child's responses and transmits them to a server.

[0245] The device provides an option to record the child's reactions using the camera and microphone. When the user selects this option, the device starts recording and sends the reaction data to the server. The server receives the reaction data and performs sentiment analysis using a speech recognition API. The results are used to generate the next story.

[0246] Input: Child's reaction (audio and video data)

[0247] Output: The analysis results are reflected in the next story generation.

[0248] Step 10:

[0249] The user inputs customization requests, which the server reflects in the next story generation.

[0250] After reading the picture book, the device displays a feedback form to the user. The user enters and submits any requests for customization of the story or illustrations. The device then sends the feedback data to the server, which stores it in a database and reflects it in the next story generation.

[0251] Input: Customization request

[0252] Output: Customization requests are reflected in the next story generation.

[0253] (Application example 1)

[0254] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0255] There is a demand for efficient training in specific operating procedures and safety instructions in educational settings and factories. However, traditional training methods have had issues in meeting individual needs, making it difficult to achieve efficient training. It has also been difficult to provide real-time feedback on the reactions of employees and operators and reflect them in training content.

[0256] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0257] In this invention, the server includes: a means for a user to input registration information and transmit it to the server; a means for the server to store the registration information in a database; a means for a user to input photo data and training content and transmit it to the server; a means for the server to generate a story and create illustrations based on the photo data and training content; a means for the server to transmit the generated story and illustrations to a terminal; a means for the terminal to display the received story and illustrations; a means for inputting training content including operating procedures and safety instructions for equipment installed in a factory and transmitting it to the server; a means for the server to automatically generate a story regarding operating procedures and safety instructions based on analyzed photo data; and a means for installing the generated story on a robot and showing operating procedures in real time, thereby enabling efficient education and training according to individual needs.

[0258] A "user" is a person who uses the system to input and submit registration information, photo data, and educational content.

[0259] "Server" refers to a computer system that receives, stores, analyzes, and processes registration information, photo data, and educational content sent by users.

[0260] A "terminal" is a device that a user operates to input and display data, and has the function of receiving and displaying data from a server.

[0261] "Registration information" refers to personal information such as name, email address, and password that a user enters when accessing the system for the first time.

[0262] "Photo data" refers to image files that record training and operating procedures and are uploaded by users to the system.

[0263] "Educational content" refers to information that is the subject of education, such as procedures, instructions, lessons, etc. that users should learn.

[0264] "Stories" are a series of educational stories generated by the server based on photo data and educational content.

[0265] "Illustrations" are visual diagrams or pictures based on the story that are generated by the server.

[0266] "Factory equipment" refers to various operating devices and machines used in factories.

[0267] An "operating procedure" is a set of procedures or methods for safely and efficiently operating equipment in a factory.

[0268] "Safety instructions" are instructions and guidelines for ensuring safety when using equipment in a factory.

[0269] "Real-time feedback" refers to immediate user reactions and data obtained during education and training.

[0270] A "robot" is a programmable machine that executes and presents specified operating procedures and safety instructions to the user.

[0271] The present invention provides a personalized training system that allows users to efficiently learn specific operating procedures and safety instructions within a factory. This system functions in cooperation with users, terminals, a server, and robots.

[0272] First, a user accesses the system via a terminal and enters their registration information. The terminal then sends the registration information (name, email address, password, etc.) entered by the user to the server. The server receives this information and stores it in a database.

[0273] After registration, the user uploads photographs of the equipment in the factory to the terminal, which then sends the photographs to the server, where they are stored.

[0274] Next, the user inputs the desired operational procedures and safety instructions into the device and sends the training content to the server. The server then uses a generative AI model based on the photo data and training content to automatically generate a story based on the specific operational procedures and safety instructions. The server then creates illustrations that correspond to the story.

[0275] The generated story and illustrations are sent from the server to the device, where they are displayed, allowing the user to learn the necessary steps and instructions.

[0276] Furthermore, the system is equipped with a real-time feedback function. The device uses a camera and microphone to record the user's reactions and transmits the data to the server. The server analyzes the recorded reaction data and reflects it in the next story generation.

[0277] For example, if a user wants to learn the safety inspection procedures for conveyor belts in a factory, they can send a prompt like this to the server:

[0278] "Generate a story based on the operating procedure of the equipment shown in the photo and the specified lesson. Lesson: 'Review safety procedures'. OCR text: 'Start operation by pressing the green button and ensure the emergency stop is accessible.'"

[0279] The generated story is then installed in a robot, which then provides practical training to the user by showing them operational procedures in real time. For example, the robot can assist the user in actually operating a conveyor belt.

[0280] This system enables efficient education and training tailored to individual needs, ensuring safety within the factory and helping employees learn efficient work procedures.

[0281] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0282] Step 1:

[0283] The user operates the terminal and enters registration information into the system. Specifically, the user enters their name, email address, and password, and presses the send button. The entered data is sent from the terminal to the server. The server receives this registration information and stores it in a database. It then notifies the terminal that registration is complete.

[0284] Step 2:

[0285] The user logs in via the device. The user enters their email address and password and presses the authentication button. The entered information is sent from the device to the server. The server performs authentication, and if successful, sends the home screen to the device; if unsuccessful, an error message is displayed.

[0286] Step 3:

[0287] A user uploads a photo of equipment in a factory. The user selects "Upload photo" from the menu on the terminal, selects a photo, and presses the send button. The terminal sends the photo data to the server. The server stores the sent photo data in a database.

[0288] Step 4:

[0289] The user specifies the operating procedures and safety instructions they want to learn. The user inputs the specific lessons and training content into the terminal and presses the send button. The input data is sent from the terminal to the server. The server stores this information in a database.

[0290] Step 5:

[0291] The server generates a story based on the photo data and educational content. First, the server performs OCR analysis of the uploaded photo data and extracts text information. Next, the generative AI model receives a prompt as input, which automatically generates a story and illustrations. Example prompt: "Generate a story based on the operating procedure for the equipment shown in the photo and the specified lesson. Lesson: 'Review safety procedures'. OCR text: 'Start operation by pressing the green button and ensure the emergency stop is accessible.'"

[0292] Step 6:

[0293] The generated story and illustrations are sent to the device. The server aggregates the generated data and sends it to the device. The device receives it and displays it to the user. The user can learn the educational content through the displayed story and illustrations.

[0294] Step 7:

[0295] Get real-time feedback. The device uses a camera and microphone to record the user's reactions during training. The recorded data is sent from the device to a server, which receives this reaction data and stores it in a database.

[0296] Step 8:

[0297] The server analyzes the response data and reflects it in the next story generation. The server uses a data analysis tool to analyze the user's response data. Based on the analysis results, the prompt sentences of the generative AI model are adjusted and applied to the next story generation.

[0298] Step 9:

[0299] The generated story is installed on the robot, and operating procedures are shown in real time. The server sends the generated story to the robot, and the robot uses the received data to show operating procedures and safety instructions to the user in real time. This allows the user to receive practical training.

[0300] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0301] The system of the present invention generates a personalized educational picture book through cooperation between a user, a terminal, a server, and an emotion engine. A specific embodiment of the system will be described below.

[0302] User Registration and Login

[0303] 1. User Registration

[0304] The user downloads the app to their device and launches it.

[0305] The device displays a registration form to the user, which includes fields for entering name, email address, password, etc.

[0306] The user enters the required information and presses the registration button.

[0307] The terminal transmits the entered registration information to the server.

[0308] The server stores the received registration information in a database and notifies the terminal that registration has been completed.

[0309] The terminal displays a message to the user indicating that registration is complete.

[0310] 2. Log in

[0311] The user launches the app again and goes to the login screen.

[0312] The terminal prompts the user to enter an email address and password.

[0313] The user enters the information and clicks the login button.

[0314] The terminal sends the input information to the server for authentication.

[0315] If the server is successful in authenticating the user, it sends the user's home screen to the device. If authentication fails, it displays an error message.

[0316] Upload photos and assign lessons

[0317] 1. Upload a photo

[0318] The user selects "Upload Photo" from the menu.

[0319] The device prompts the user to select a family photo to upload.

[0320] The user selects a photo and presses the upload button.

[0321] The device sends the photo data to the server.

[0322] The server receives the photo data and stores it in a database.

[0323] 2. Designation of lessons learned

[0324] The user is taken to the "Specify Lessons Learned" screen.

[0325] The device prompts the user to enter more information about the lesson or story they want to learn.

[0326] The user inputs a lesson such as "Greetings are important."

[0327] The terminal transmits the input lesson information to the server.

[0328] The server receives the lesson information and stores it in a database.

[0329] Creating and viewing stories

[0330] 1. Story Generation

[0331] The server starts generating a story based on the photo data and the lesson information.

[0332] The server uses conversational AI to automatically generate a story that follows the specified lesson.

[0333] The server generates illustrations of the characters based on the uploaded photos.

[0334] 2. Integration and transmission of picture book data

[0335] The server combines the generated story and illustrations to create picture book data.

[0336] The server sends the completed picture book data to the terminal.

[0337] 3. Display of picture books

[0338] The terminal displays the received picture book data to the user.

[0339] Users can read picture books together with their children and spend time learning.

[0340] Reaction monitoring and emotion recognition (optional)

[0341] 1. Activating the Emotional Engine

[0342] The device activates the camera and microphone and starts the emotion engine according to the user's selection.

[0343] 2. Reaction recording and emotion recognition

[0344] The device uses a camera and microphone to record the child's reactions and uses an emotion engine to recognize emotions from those reactions.

[0345] The recognized emotional information is processed in real time.

[0346] 3. Sending Emotional Data

[0347] The device transmits the recognized emotion data to the server.

[0348] The server receives the emotion data and stores it in a database.

[0349] 4. Emotional Data Analysis

[0350] The server analyzes the stored emotional data and uses it to generate the next story.

[0351] Gathering feedback and incorporating it into next time

[0352] 1. Enter your feedback

[0353] After finishing reading the picture book, the device displays a feedback form to the user.

[0354] The user inputs customization requests for the story and illustrations.

[0355] 2. Sending and storing feedback data

[0356] The terminal transmits the input feedback data to the server.

[0357] The server stores the feedback in a database and reflects it in the next story generation.

[0358] The above is a specific embodiment of the present invention. This system allows parents to easily create educational picture books tailored to their children and provide learning that is appropriate for their children. Furthermore, by using an emotion engine, more personalized content can be provided based on the child's emotions.

[0359] The processing flow will be explained below.

[0360] Step 1: User Registration and Login

[0361] 1. Install the app

[0362] The user downloads the app to their device and launches it.

[0363] 2. Displaying the user registration form

[0364] The terminal displays a registration form to the user.

[0365] 3. Enter your registration information

[0366] The user enters their name, email address, and password.

[0367] 4. Submission of Registration Information

[0368] The terminal transmits the input information to the server.

[0369] 5. Saving your registration information

[0370] The server verifies the received registration information and stores it in a database.

[0371] 6. Notification of registration completion

[0372] The server sends a registration completion message to the terminal.

[0373] The terminal displays a registration complete message to the user.

[0374] 7. Display the login screen

[0375] The device displays the login screen.

[0376] 8. Enter your authentication information

[0377] The user enters their email address and password.

[0378] 9. Submitting authentication information

[0379] The terminal transmits the entered authentication information to the server.

[0380] 10. Credential Verification

[0381] The server verifies the authentication information and sends the result to the device.

[0382] 11. Notification of Authentication Completion

[0383] If the server is successful in the authentication, it sends the user's home screen data to the terminal.

[0384] The device displays the user's home screen and completes the login.

[0385] Step 2: Upload photos and specify lessons

[0386] 1. Display the photo selection screen

[0387] The terminal displays a screen that allows the user to select a family photo.

[0388] 2. Select a photo

[0389] The user selects a family photo to upload.

[0390] 3. Sending photo data

[0391] The terminal transmits the selected photo data to the server.

[0392] 4. Saving photo data

[0393] The server stores the received photo data in a database.

[0394] 5. Display the lesson specification screen

[0395] The terminal displays a screen that prompts the user to input the lesson they want to learn.

[0396] 6. Input of lessons learned

[0397] The user inputs a lesson such as "Greetings are important."

[0398] 7. Submission of lessons learned data

[0399] The terminal transmits the input lesson data to the server.

[0400] 8. Lessons Learned Data Storage

[0401] The server stores the received lesson data in a database.

[0402] Step 3: Generate and view stories

[0403] 1. Start creating a story

[0404] The server retrieves the photo data and lesson data from the database.

[0405] 2. Story Generation

[0406] Based on the data acquired by the server, a story is generated using conversational AI.

[0407] 3. Illustration Generation

[0408] The server generates illustrations of the characters based on the uploaded photos.

[0409] 4. Picture book data integration

[0410] The server combines the generated story and illustrations to create picture book data.

[0411] 5. Sending picture book data

[0412] The server sends the completed picture book data to the terminal.

[0413] 6. Display of picture books

[0414] The terminal displays the received picture book data to the user.

[0415] 7. Picture book reading

[0416] The user reads a picture book with the child.

[0417] Step 4: Response Monitoring and Emotion Recognition (Optional)

[0418] 1. Activating the Emotional Engine

[0419] The device activates the camera and microphone and starts the emotion engine according to the user's selection.

[0420] 2. Recording the reaction

[0421] The device uses a camera and microphone to record the child's responses.

[0422] 3. Emotional Recognition

[0423] The device analyzes the recorded responses using an emotion engine to recognize emotions.

[0424] 4. Sending Emotional Data

[0425] The device transmits the recognized emotion data to the server.

[0426] The server stores the received emotion data in a database.

[0427] 5. Emotional Data Analysis

[0428] The server analyzes the stored emotional data and uses it to generate the next story.

[0429] Step 5: Gather feedback and incorporate it into next steps

[0430] 1. Display the feedback form

[0431] The device displays a feedback form after the picture book has finished reading.

[0432] 2. Enter your customization requests

[0433] The user inputs customization requests for the story and illustrations.

[0434] 3. Sending feedback data

[0435] The terminal transmits the input feedback data to the server.

[0436] 4. Feedback Data Storage

[0437] The server stores the received feedback in a database.

[0438] 5. Reflection in next story generation

[0439] The server analyzes the feedback data and reflects it in the next story generation.

[0440] Example 2

[0441] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0442] Conventional educational picture books have had difficulty generating stories that fit each child's interests and learning needs. Furthermore, they were unable to incorporate children's reactions as feedback in real time, making it difficult to change the content of a picture book once it was created. Furthermore, they were unable to provide content that responded to children's emotions.

[0443] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0444] In this invention, the server includes means for a user to input registration information and transmit it to the server, means for the server to store the registration information in a database, means for a user to input image data and educational content and transmit it to the server, means for the server to generate a story and create illustrations based on the image data and educational content, and means for the server to transmit the generated story and illustrations to the terminal, thereby enabling the generation of a personalized educational picture book in response to a user's request.

[0445] "User" refers to an individual or organization who enters registration information to use this system and receives services.

[0446] "Server" refers to a central computer system that receives information sent by users, stores it in a database, and performs any necessary processing.

[0447] "Registration information" refers to personal information such as name, email address, and password that a user enters to begin using the system.

[0448] "Database" refers to a collection of various data managed and stored by a server, such as registration information, image data, and educational content.

[0449] "Image data" refers to data containing visual information, such as photographs and pictures, uploaded by users.

[0450] "Educational content" refers to detailed information entered by the user about the lessons or stories they want their children to learn.

[0451] "Stories" refer to sentences and scenarios generated based on educational content using generative AI models.

[0452] "Illustration" refers to a visual picture or diagram generated based on uploaded image data or a story.

[0453] "Terminal" refers to a device such as a computer or smartphone that a user uses to operate the system.

[0454] "Reaction" refers to the child's reactions, including their behavior, facial expressions, and voice while reading an educational picture book.

[0455] "Customization requests" refer to improvements or specific requests that users input to be reflected in the next story generation or illustration creation.

[0456] A "generative AI model" refers to artificial intelligence technology that automatically generates stories based on educational content specified by the user.

[0457] MODE FOR CARRYING OUT THE INVENTION

[0458] The system of the present invention generates personalized educational picture books through cooperation between users, terminals, a server, and an emotion engine. The specific configuration and operation of the system are described below.

[0459] System configuration

[0460] This system consists of a terminal used by the user, a central management server, and an emotion engine that is used as needed.

[0461] Device: Computer, smartphone, tablet, etc.

[0462] Server: A high-performance computer that contains a database, generative AI models, and a network interface for communicating with devices over the internet.

[0463] Emotion Engine: A hardware device with a camera and microphone attached, optionally used by the user.

[0464] User Registration and Login

[0465] The user first downloads the app to their device and launches it. The device then displays a registration form to the user. When the user enters registration information such as their name, email address, and password and presses the registration button, the device sends this information to the server. The server stores the received registration information in a database and notifies the device that registration is complete. The user then launches the app again, enters their email address and password on the login screen, and sends them to the server for authentication.

[0466] Upload photos and assign lessons

[0467] The user selects "Upload Photos" from the menu, selects a family photo, and sends it to the server. The server saves the received photo data in a database. The user then moves to the "Specify Lesson" screen and inputs a lesson, such as "Greetings are important." The device sends this lesson information to the server, which saves it in a database.

[0468] Creating and viewing stories

[0469] The server automatically generates a story using a generative AI model (such as GPT-4) based on the received photo data and lesson information. It also generates illustrations of the characters based on the uploaded photos. The server then integrates the generated story and illustrations to create picture book data and sends it to the device. The user receives the picture book data on their device and reads it together with their child.

[0470] Reaction monitoring and emotion recognition (optional)

[0471] Depending on the user's selection, the device's camera and microphone are activated and the emotion engine is used. While the child is reading the picture book, the device uses the camera and microphone to record the child's reactions and sends the data to the server. The server analyzes the recorded data and reflects it in the next story generation.

[0472] Gathering feedback and incorporating it into next time

[0473] After reading the picture book, the device displays a feedback form to the user. The user enters their requests for customization of the story and illustrations and sends them from the device to the server. The server stores the feedback data in a database and reflects it in the next story generation.

[0474] Specific examples and examples of prompts for generative AI models

[0475] Examples:

[0476] The user specifies the lesson, "Greetings are important," and uploads a family photo. The server follows the following process: receives the photo data and lesson information ⇒ generates a story using a generative AI model (e.g., GPT-4) ⇒ generates illustrations of the characters ⇒ integrates the picture book data ⇒ sends the data to the device. The user opens the generated picture book and teaches their child the importance of greetings such as "good morning" and "thank you."

[0477] Example prompt sentence:

[0478] text

[0479] Lesson: Greetings are important

[0480] Photo: Family photo

[0481] Story content: Describe how greetings are important throughout a family's day. Include scenes such as "Good morning" in the morning, "Itadakimasu" at mealtimes, "I'm home" when you get home, and "Good night" before going to bed.

[0482] This system allows parents to easily create educational picture books tailored to their children and provide learning that is appropriate for them. Furthermore, by using an emotion engine, more personalized content can be provided based on the child's emotions.

[0483] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0484] Step 1:

[0485] User Registration

[0486] The user downloads the application to their device and launches it. The device displays a registration form that prompts the user to enter their name, email address, password, etc. The user enters the required information and presses the registration button. The entered information is sent from the device to the server. The server stores the received registration information in a database and notifies the device that registration is complete. The device displays a message to the user that registration is complete.

[0487] Input: Name, email address, password

[0488] Output: Registration complete message

[0489] Step 2:

[0490] Log in

[0491] The user launches the application and goes to the login screen. The device prompts the user to enter their email address and password. The user enters the information and presses the login button. The entered login information is sent from the device to the server, and the server performs authentication. If authentication is successful, the server sends the user's home screen data to the device, which displays it. If authentication fails, the server sends an error message to the device, which displays it.

[0492] Input: Email address, password

[0493] Output: Home screen data or error message

[0494] Step 3:

[0495] Upload a photo

[0496] The user selects "Upload Photos" from the menu. The device displays a screen prompting the user to select a family photo. When the user selects a photo and presses the upload button, the photo data is sent from the device to the server. The server stores the received photo data in a database.

[0497] Input: Photo data

[0498] Output: Confirmation of saved photo data

[0499] Step 4:

[0500] Designation of lessons learned

[0501] The user moves to the "Specify Lesson" screen and enters detailed information about the lesson or story they want to learn. The device sends this information to the server, which then stores the received lesson information in a database.

[0502] Input: Detailed information about the lesson or story (e.g., greetings are important)

[0503] Output: Confirmation of saved lesson information

[0504] Step 5:

[0505] Story Generation

[0506] The server generates a story using a generative AI model (e.g., GPT-4) based on the received photo data and lesson information, and also generates illustrations of the characters based on the uploaded photos.

[0507] Input: Photo data, lesson information

[0508] Output: Generated story and illustrations

[0509] Step 6:

[0510] Integration and transmission of picture book data

[0511] The server integrates the generated story and illustrations to create a picture book, and the completed picture book data is sent from the server to the device.

[0512] Input: Story, illustration

[0513] Output: Picture book data

[0514] Step 7:

[0515] Picture book display

[0516] The device displays the received picture book data to the user, who then reads the picture book together with their child, spending time learning.

[0517] Input: Picture book data

[0518] Output: The displayed picture book

[0519] Step 8:

[0520] Reaction monitoring and emotion recognition (optional)

[0521] If the user selects it, the device activates the camera and microphone to record the child's reactions. The emotion engine analyzes this data and transmits the recognized emotion data from the device to the server. The server receives the emotion data and stores it in a database.

[0522] Input: Child's reaction data via camera and microphone

[0523] Output: Analyzed sentiment data

[0524] Step 9:

[0525] Gathering feedback and incorporating it into next time

[0526] After reading the picture book, the device displays a feedback form to the user. The user enters their requests for customization of the story and illustrations and sends them from the device to the server. The server stores the feedback data in a database and reflects it in the next story generation.

[0527] Input: Customization request

[0528] Output: A confirmation of saved feedback data

[0529] (Application example 2)

[0530] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0531] Conventional educational picture book systems have issues with the quality and personalization of story generation, even when users enter registration information and provide photos and lesson information. It is also difficult to grasp children's reactions in real time and reflect them in the next story. This makes it impossible to provide a truly personalized educational experience that meets the needs of each user and child.

[0532] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input registration information and transmit it to the server; a means for the server to store the registration information in a database; a means for a user to input photo data and educational content and transmit it to the server; a means for the server to generate a story and create illustrations based on the photo data and educational content; a means for the server to transmit the generated story and illustration to the terminal; a means for the terminal to display the received story and illustration; a means for a user to input feedback and customization requests for the generated story and illustration and transmit it to the server; a means for the server to store the received feedback and customization requests in a database and reflect them in the next story generation; a means for the terminal to record the child's responses using a camera or microphone and transmit it to the server; a means for the server to analyze the recorded child's responses using a generative AI model and reflect them in the next story generation; and a means for an application to run on a smartphone or other available terminal. This makes it possible to provide a personalized educational experience tailored to the needs of each user and child.

[0533] A "user" is a person who uses the personalized educational picture book system.

[0534] "Server" means the central processing unit that receives and processes information from users, stores data, and transmits generated stories and illustrations.

[0535] A "terminal" is a device used by a user, such as a smartphone or tablet, that is used to display stories and illustrations received from the server.

[0536] "Photo data" refers to image files that users upload to the system and are used as characters and backgrounds for educational picture books.

[0537] "Educational content" refers to the lesson or learning material specified by the user, and is the main input information for story generation.

[0538] "Story generation" is the process in which the server automatically creates a story using conversational AI based on the input photo data and educational content.

[0539] "Illustration creation" is the process in which the server draws the characters and backgrounds in the story based on the uploaded photo data.

[0540] "Feedback" refers to the evaluation and customization requests provided by users for the generated stories and illustrations.

[0541] "Customization Request" means a specific change or addition you would like to make provided as part of your Feedback.

[0542] "Response recording" is the process by which the device collects the child's responses using a camera and microphone.

[0543] A "generative AI model" refers to an artificial intelligence technology that performs story and sentiment analysis based on input data.

[0544] "Prompts" refer to instructions or questions used when performing story or sentiment analysis on a generative AI model.

[0545] MODE FOR CARRYING OUT THE INVENTION

[0546] This invention is a system that generates educational picture books using a user interface, where users use devices such as smartphones and tablets. At the heart of the system is a server that processes and stores input from users and uses a generative AI model to create personalized stories and illustrations.

[0547] User Registration and Login

[0548] 1. The user installs and launches a dedicated application on their device.

[0549] 2. The device presents the user with a registration form and asks them to enter registration information such as their name, email address, and password.

[0550] 3. The user enters the required information and sends it to the server.

[0551] 4. The server saves the registration information in its database and completes the user registration.

[0552] Upload photos and assign lessons

[0553] 1. The user selects "Upload Photo" from the menu, selects a family photo, and sends it to the server.

[0554] 2. The server receives the photo data and stores it in a database.

[0555] 3. The user inputs the lesson or learning they want to teach and sends it to the server.

[0556] 4. The server receives the lesson information and stores it in a database.

[0557] Creating and viewing stories

[0558] 1. Based on the photo data and lesson information received by the server, a generative AI model is used to automatically generate a story and illustrations.

[0559] 2. The server combines the generated story and illustrations to create picture book data.

[0560] 3. The server sends the completed picture book data to the terminal.

[0561] 4. The device displays the received picture book data to the user.

[0562] Emotion Recognition and Feedback

[0563] 1. The device uses a camera and microphone to record the child's responses.

[0564] 2. The device sends the recorded data to the server and executes the emotion engine.

[0565] 3. The server uses an emotion engine to analyze the child's reaction data and reflect it in the next story generation.

[0566] 4. The user enters feedback and customization requests from the device and sends them to the server.

[0567] 5. The server stores the feedback and customization requests in a database and reflects them in the next story generation.

[0568] Hardware and software used

[0569] Device: The smartphone or tablet used by the user.

[0570] Server: Central processing unit that stores and processes registration information, photo data, lesson information, emotion data, and feedback.

[0571] Generative AI models: Artificial intelligence techniques used for story and illustration generation, as well as sentiment analysis.

[0572] Open source libraries: HTTP communication using the requests library, conversational AI models using the transformers library, image and video analysis using opencv.

[0573] Examples of concrete examples and prompts

[0574] As a concrete example, if a parent wants to teach their child the importance of greetings, the steps would be as follows:

[0575] 1. A parent uploads a family photo on their smartphone and enters a lesson about the importance of greetings.

[0576] 2. The server uses the generative AI model to generate a story using the prompt: "Create a story about a family going hiking with their parents. Include the lesson that greetings are important. Use the family photo below as the characters."

[0577] 3. The child's reaction when reading the picture book is recorded with a camera and sent to the server. The emotional data recorded at that time is reflected in the next story generation.

[0578] In this way, the system of the present invention can provide personalized education that meets the needs of each individual user and encourage effective learning for children.

[0579] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0580] Step 1:

[0581] The user installs and launches a dedicated application on their device. The device then provides the user with a registration form, prompting them to enter registration information such as their name, email address, and password. The user enters the required information and sends it to the server. The server saves the registration information in a database, completing the user registration. The input is the user's personal information, and the output is the registration information saved in the database.

[0582] Step 2:

[0583] The user selects "Upload Photo" from the menu, selects a family photo, and sends it to the server. The server receives the photo data and stores it in a database. The user then enters the lesson or learning content they want to teach and sends it to the server. The server receives the lesson information and stores it in the database. The input is the photo data and lesson information, and the output is the photo data and lesson information stored in the database.

[0584] Step 3:

[0585] Based on the photo data and lesson information received by the server, a generative AI model is used to automatically generate a story and illustrations. The prompt text "Write a story about a family going hiking with their parents. Include the lesson that greetings are important. Also, use the family photos below as characters." is input into the generative AI model. The server combines the generated story and illustrations to create picture book data. The input is photo data, lesson information, and the prompt text, and the output is the generated story and illustrations.

[0586] Step 4:

[0587] The server sends the completed picture book data to the terminal. The terminal displays the received picture book data to the user. The user reads the picture book together with the child and spends time learning. The input is the picture book data, and the output is the picture book displayed on the terminal.

[0588] Step 5:

[0589] The device uses a camera and microphone to record the child's reactions. The recorded data includes facial expressions and tone of voice. The device sends the recorded data to the server, which runs the emotion engine. The server uses the emotion engine to analyze the child's reaction data. The input is the child's reaction data, and the output is the analyzed emotion data.

[0590] Step 6:

[0591] The user inputs feedback and customization requests from the terminal and sends them to the server. The server stores the feedback and customization requests in a database and reflects them in the next story generation. The input is the feedback and customization requests, and the output is the feedback and customization requests stored in the database.

[0592] Step 7:

[0593] The next time a story is generated, the server uses the customization requests and emotional data stored in the database to input prompts into the generative AI model. This generates a story optimized for each user or child. The input is past customization requests and emotional data, and the output is a new story optimized for the user.

[0594] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0595] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0596] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0597] [Second embodiment]

[0598] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0599] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0600] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0601] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0602] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0603] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0604] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0605] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0606] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0607] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0608] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0609] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0610] The system of the present invention generates a personalized educational picture book through the mutual cooperation of a user, a terminal, and a server. A specific embodiment of the system will be described below.

[0611] User Registration and Login

[0612] 1. User Registration

[0613] The user downloads the app to their device and launches it.

[0614] The device displays a registration form to the user, which includes fields for entering name, email address, password, etc.

[0615] The user enters the required information and presses the registration button.

[0616] The terminal transmits the entered registration information to the server.

[0617] The server stores the received registration information in a database and notifies the terminal that registration has been completed.

[0618] The terminal displays a message to the user indicating that registration is complete.

[0619] 2. Log in

[0620] The user launches the app again and goes to the login screen.

[0621] The terminal prompts the user to enter an email address and password.

[0622] The user enters the information and clicks the login button.

[0623] The terminal sends the input information to the server for authentication.

[0624] If the server is successful in authenticating the user, it sends the user's home screen to the device. If authentication fails, it displays an error message.

[0625] Upload photos and assign lessons

[0626] 1. Upload a photo

[0627] The user selects "Upload Photo" from the menu.

[0628] The device prompts the user to select a family photo to upload.

[0629] The user selects a photo and presses the upload button.

[0630] The device sends the photo data to the server.

[0631] The server receives the photo data and stores it in a database.

[0632] 2. Designation of lessons learned

[0633] The user is taken to the "Specify Lessons Learned" screen.

[0634] The device prompts the user to enter more information about the lesson or story they want to learn.

[0635] The user inputs a lesson such as "Greetings are important."

[0636] The terminal transmits the input lesson information to the server.

[0637] The server receives the lesson information and stores it in a database.

[0638] Creating and viewing stories

[0639] 1. Story generation

[0640] The server starts generating a story based on the photo data and the lesson information.

[0641] The server uses conversational AI to automatically generate a story that follows the specified lesson.

[0642] The server generates illustrations of the characters based on the uploaded photos.

[0643] 2. Integration and transmission of picture book data

[0644] The server combines the generated story and illustrations to create picture book data.

[0645] The server sends the completed picture book data to the terminal.

[0646] 3. Display of picture books

[0647] The terminal displays the received picture book data to the user.

[0648] Users can read picture books together with their children and spend time learning.

[0649] Reaction monitoring (optional)

[0650] 1. Use of camera and microphone

[0651] The device will use its camera and microphone to give you the option to record your child's responses.

[0652] If the user selects this option, the device will start recording.

[0653] 2. Submitting and analyzing reaction data

[0654] The terminal transmits the recorded reaction data to the server.

[0655] The server analyzes the response data and uses it to generate the next story.

[0656] Gathering feedback and incorporating it into next time

[0657] 1. Enter your feedback

[0658] After finishing reading the picture book, the device displays a feedback form to the user.

[0659] The user inputs customization requests for the story and illustrations.

[0660] 2. Sending and storing feedback data

[0661] The terminal transmits the input feedback data to the server.

[0662] The server stores the feedback in a database and reflects it in the next story generation.

[0663] The above is a specific embodiment of the present invention. This system allows parents to easily create educational picture books tailored to their children and provide them with learning that is appropriate for their children.

[0664] The processing flow will be explained below.

[0665] Step 1: User Registration and Login

[0666] 1. Install the app

[0667] The user downloads the app to their device and launches it.

[0668] 2. Displaying the user registration form

[0669] The terminal displays a registration form to the user.

[0670] 3. Enter your registration information

[0671] The user enters their name, email address, and password.

[0672] 4. Submission of Registration Information

[0673] The terminal transmits the input information to the server.

[0674] 5. Saving your registration information

[0675] The server verifies the received registration information and stores it in a database.

[0676] 6. Notification of registration completion

[0677] The server sends a registration completion message to the terminal.

[0678] The terminal displays a registration complete message to the user.

[0679] 7. Display the login screen

[0680] The device displays the login screen.

[0681] 8. Enter your authentication information

[0682] The user enters their email address and password.

[0683] 9. Submitting authentication information

[0684] The terminal transmits the entered authentication information to the server.

[0685] 10. Credential Verification

[0686] The server verifies the authentication information and sends the result to the device.

[0687] 11. Notification of Authentication Completion

[0688] If the server is successful in the authentication, it sends the user's home screen data to the terminal.

[0689] The device displays the user's home screen and completes the login.

[0690] Step 2: Upload photos and specify lessons

[0691] 1. Display the photo selection screen

[0692] The terminal displays a screen that allows the user to select a family photo.

[0693] 2. Select a photo

[0694] The user selects a family photo to upload.

[0695] 3. Sending photo data

[0696] The terminal transmits the selected photo data to the server.

[0697] 4. Saving photo data

[0698] The server stores the received photo data in a database.

[0699] 5. Display the lesson specification screen

[0700] The terminal displays a screen that prompts the user to input the lesson they want to learn.

[0701] 6. Input of lessons learned

[0702] The user inputs a lesson such as "Greetings are important."

[0703] 7. Submission of lessons learned data

[0704] The terminal transmits the input lesson data to the server.

[0705] 8. Lessons Learned Data Storage

[0706] The server stores the received lesson data in a database.

[0707] Step 3: Generate and view stories

[0708] 1. Start creating a story

[0709] The server retrieves the photo data and lesson data from the database.

[0710] 2. Story Generation

[0711] Based on the data acquired by the server, a story is generated using conversational AI.

[0712] 3. Illustration Generation

[0713] The server generates illustrations of the characters based on the uploaded photos.

[0714] 4. Picture book data integration

[0715] The server combines the generated story and illustrations to create picture book data.

[0716] 5. Sending picture book data

[0717] The server sends the completed picture book data to the terminal.

[0718] 6. Display of picture books

[0719] The terminal displays the received picture book data to the user.

[0720] 7. Picture book reading

[0721] The user reads a picture book with the child.

[0722] Step 4: Monitoring the reaction (optional)

[0723] 1. Turn on the camera and microphone

[0724] The device will activate the camera and microphone according to the user's selection.

[0725] 2. Recording the reaction

[0726] The device records the child's responses.

[0727] 3. Submitting reaction data

[0728] The terminal transmits the recorded reaction data to the server.

[0729] 4. Saving and analyzing reaction data

[0730] The server stores the received response data and uses it for the next story generation.

[0731] Step 5: Gather feedback

[0732] 1. Display the feedback form

[0733] The device displays a feedback form after the picture book has finished reading.

[0734] 2. Enter your customization requests

[0735] The user inputs customization requests for the story and illustrations.

[0736] 3. Sending feedback data

[0737] The terminal transmits the input feedback data to the server.

[0738] 4. Feedback Data Storage

[0739] The server stores the received feedback in a database.

[0740] 5. Reflection in next story generation

[0741] The server analyzes the feedback data and reflects it in the next story generation.

[0742] Example 1

[0743] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0744] In modern education, there is a demand for personalized educational materials tailored to each child, but existing systems have difficulty in providing such personalized education. Furthermore, because the content and format of educational materials are fixed, it is difficult to provide education that reflects the interests and reactions of children. Therefore, a system is needed that can effectively provide individual educational experiences tailored to each child.

[0745] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0746] In this invention, the server includes means for a user to input registration information and transmit it to the server, means for the server to store the registration information in a database, means for a user to input photo data and educational information and transmit it to the server, means for the server to automatically generate a story and create illustrations using a generative AI model based on the photo data and educational information, means for the server to combine the generated story and illustrations to create picture book data and transmit it to the terminal, and means for the terminal to display the picture book data received. This enables users to easily create personalized educational picture books and provide learning content tailored to their children.

[0747] A "user" is an individual or group that uses this system and is the entity that creates and uses educational picture books.

[0748] A "terminal" is a device used by a user, such as a computer, smartphone, or tablet, that provides an interface with the system.

[0749] A "server" is the central part of a system and is a computer system that stores, processes, and manages data.

[0750] "Registration information" refers to information such as name, email address, and password required for a user to access the system.

[0751] The "database" is a data storage system for storing and managing user registration information, photo data, educational information, generated picture book data, and the like.

[0752] "Photo data" refers to image files uploaded by users and used for characters and backgrounds in picture books.

[0753] "Educational information" is detailed information about the lessons and stories that the user wants to learn as the content of the picture book.

[0754] A "generative AI model" is an artificial intelligence that automatically generates text and images based on data, such as ChatGPT.

[0755] A "story" is a narrative created by a generative AI model based on educational information.

[0756] "Illustrations" are images of characters and backgrounds generated based on uploaded photo data.

[0757] "Picture book data" is data in the form of an e-book created by combining the generated story and illustrations.

[0758] "Customization requests" are specific requests for changes or improvements that users input regarding stories or illustrations.

[0759] "Reaction data" refers to audio and video data recorded with a camera or microphone of a child's reaction when a picture book is read to them.

[0760] The system of the present invention generates a personalized educational picture book through the mutual cooperation of a user, a terminal, and a server. A specific embodiment of the system will be described below.

[0761] User Registration and Login Implementation

[0762] First, the user downloads the application to their device and launches it. The device displays a registration form to the user, and the user enters registration information such as their name, email address, and password. The entered registration information is sent from the device to the server, and the server stores the information in a database. The server then notifies the device that registration is complete, and the device displays a message to the user that registration is complete. When the user launches the application again, the user enters their email address and password on the login screen and sends them to the server via the device. If the server is successful in its authentication, the user's home screen is sent to the device.

[0763] Uploading photos and assigning lessons

[0764] When the user selects "Upload Photos" from the menu, the device prompts the user to select a family photo to upload. When the user selects a photo and presses the upload button, the photo data is sent from the device to the server. The server receives the photo data and stores it in a database. Next, the user moves to the "Specify Lesson" screen and enters educational information, such as "Greetings are important." The entered educational information is sent to the server via the device, and the server stores the educational information in a database.

[0765] Story Generation and Display Embodiments

[0766] The server begins generating a story based on the photo data and educational information. It uses generative AI models such as ChatGPT to automatically generate a story that follows the specified moral. It also generates illustrations of the characters based on the uploaded photos. The server then combines the generated story and illustrations to create picture book data and sends it to the device. The device then displays the received picture book data to the user, who can then read the book together with their child.

[0767] Reaction Monitoring Embodiments (Optional)

[0768] The device provides an option to record the child's reactions using the camera and microphone. When the user selects this option, the device starts recording. The recorded reaction data is sent from the device to a server, which analyzes the reaction data and uses it to generate the next story. This analysis includes emotion analysis using a voice recognition API.

[0769] How to collect feedback and incorporate it into future projects

[0770] After reading the picture book, the device displays a feedback form to the user. The user enters any customization requests for the story or illustrations and sends them from the device to the server. The server stores the customization requests in a database and reflects them in the next story generation.

[0771] Specific examples

[0772] Users download the app, enter their registration information, and send it to the server. Next, they upload a family photo and specify a lesson, such as "greetings are important." The server uses a generative AI model such as ChatGPT to generate a story and creates illustrations of the characters based on the photos. The completed picture book data is sent to the device, and users can provide learning opportunities by reading the book together with their children.

[0773] An example prompt is "Generate a children's story with the moral that saying hello is important. The story will feature a little bear, and include illustrations based on uploaded family photos."

[0774] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0775] Step 1:

[0776] A user downloads the app and enters their registration information.

[0777] The user downloads the app to their device and launches it. The device displays a form for the user to enter their name, email address, and password. The user enters the required information and presses the "Register" button. The entered information is temporarily stored in the device's memory.

[0778] Input: Name, email address, password

[0779] Output: The registration information is temporarily stored in the device's memory.

[0780] Step 2:

[0781] The terminal sends the registration information to the server, which stores it in a database.

[0782] The terminal sends the registration information entered by the user to the server. The server receives the registration information and stores it in a database using an SQL query. The server sends a notification of registration completion to the terminal. The terminal displays a message to the user notifying them that registration was successful.

[0783] Input: Registration information (name, email address, password)

[0784] Output: The registration information is saved in the database and a message indicating successful registration is displayed.

[0785] Step 3:

[0786] The user logs in again and authenticates.

[0787] The user launches the app again and goes to the login screen. The device prompts the user to enter their email address and password. The user enters the information and presses the "Login" button. The device sends the entered information to the server, which then references the database and performs authentication. If authentication is successful, the server sends the user's home screen to the device.

[0788] Input: Email address, password

[0789] Output: If authentication is successful, the home screen is displayed.

[0790] Step 4:

[0791] The user uploads a photo.

[0792] The user selects "Upload photo" from the menu. The device displays a file selection dialog, and the user selects the photo to upload. The user selects a photo and presses the "Upload" button. The device sends the photo data to the server, and the server saves the received photo data in a database.

[0793] Input: Photo data

[0794] Output: The photo data is saved in the database.

[0795] Step 5:

[0796] The user inputs the lesson information.

[0797] The user moves to the "Specify Lesson" screen and inputs the lesson information. The terminal displays an input form for educational information, and the user inputs a lesson such as "Greetings are important." The user inputs the information and presses the "Send" button. The terminal sends the input educational information to the server, which stores it in a database.

[0798] Input: Lesson learned information

[0799] Output: Lessons learned information is stored in a database.

[0800] Step 6:

[0801] The server uses a generative AI model to generate a story and create illustrations.

[0802] The server starts generating a story based on the photo data and lesson information. It inputs a prompt (e.g., "Greetings are important") into a generative AI model such as ChatGPT to generate a story. The server uses an image processing library to generate illustrations of people based on the uploaded photos.

[0803] Input: Photo data, lesson information

[0804] Output: Generated stories and illustrations

[0805] Step 7:

[0806] The server integrates the generated story and illustrations into picture book data and sends it to the terminal.

[0807] The server combines the generated story and illustrations to create picture book data. The server uses a PDF generation library to combine the text and illustrations into PDF format. The completed picture book data is sent to the device.

[0808] Input: Generated story and illustrations

[0809] Output: Picture book data is sent to the terminal.

[0810] Step 8:

[0811] The device displays the picture book, and the user reads it together with the child.

[0812] The device displays the received picture book data. Using the device's built-in PDF viewer, the user can read the book together with their child.

[0813] Input: Picture book data

[0814] Output: The picture book is displayed on the device.

[0815] Step 9:

[0816] Optionally, the device records the child's responses and transmits them to a server.

[0817] The device provides an option to record the child's reactions using the camera and microphone. When the user selects this option, the device starts recording and sends the reaction data to the server. The server receives the reaction data and performs sentiment analysis using a speech recognition API. The results are used to generate the next story.

[0818] Input: Child's reaction (audio and video data)

[0819] Output: The analysis results are reflected in the next story generation.

[0820] Step 10:

[0821] The user inputs customization requests, which the server reflects in the next story generation.

[0822] After reading the picture book, the device displays a feedback form to the user. The user enters and submits any requests for customization of the story or illustrations. The device then sends the feedback data to the server, which stores it in a database and reflects it in the next story generation.

[0823] Input: Customization request

[0824] Output: Customization requests are reflected in the next story generation.

[0825] (Application example 1)

[0826] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0827] There is a demand for efficient training in specific operating procedures and safety instructions in educational settings and factories. However, traditional training methods have had issues in meeting individual needs, making it difficult to achieve efficient training. It has also been difficult to provide real-time feedback on the reactions of employees and operators and reflect them in training content.

[0828] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0829] In this invention, the server includes: a means for a user to input registration information and transmit it to the server; a means for the server to store the registration information in a database; a means for a user to input photo data and training content and transmit it to the server; a means for the server to generate a story and create illustrations based on the photo data and training content; a means for the server to transmit the generated story and illustrations to a terminal; a means for the terminal to display the received story and illustrations; a means for inputting training content including operating procedures and safety instructions for equipment installed in a factory and transmitting it to the server; a means for the server to automatically generate a story regarding operating procedures and safety instructions based on analyzed photo data; and a means for installing the generated story on a robot and showing operating procedures in real time, thereby enabling efficient education and training according to individual needs.

[0830] A "user" is a person who uses the system to input and submit registration information, photo data, and educational content.

[0831] "Server" refers to a computer system that receives, stores, analyzes, and processes registration information, photo data, and educational content sent by users.

[0832] A "terminal" is a device that a user operates to input and display data, and has the function of receiving and displaying data from a server.

[0833] "Registration information" refers to personal information such as name, email address, and password that a user enters when accessing the system for the first time.

[0834] "Photo data" refers to image files that record training and operating procedures and are uploaded by users to the system.

[0835] "Educational content" refers to information that is the subject of education, such as procedures, instructions, lessons, etc. that users should learn.

[0836] "Stories" are a series of educational stories generated by the server based on photo data and educational content.

[0837] "Illustrations" are visual diagrams or pictures based on the story that are generated by the server.

[0838] "Factory equipment" refers to various operating devices and machines used in factories.

[0839] An "operating procedure" is a set of procedures or methods for safely and efficiently operating equipment in a factory.

[0840] "Safety instructions" are instructions and guidelines for ensuring safety when using equipment in a factory.

[0841] "Real-time feedback" refers to immediate user reactions and data obtained during education and training.

[0842] A "robot" is a programmable machine that executes and presents specified operating procedures and safety instructions to the user.

[0843] The present invention provides a personalized training system that allows users to efficiently learn specific operating procedures and safety instructions within a factory. This system functions in cooperation with users, terminals, a server, and robots.

[0844] First, a user accesses the system via a terminal and enters their registration information. The terminal then sends the registration information (name, email address, password, etc.) entered by the user to the server. The server receives this information and stores it in a database.

[0845] After registration, the user uploads photographs of the equipment in the factory to the terminal, which then sends the photographs to the server, where they are stored.

[0846] Next, the user inputs the desired operational procedures and safety instructions into the device and sends the training content to the server. The server then uses a generative AI model based on the photo data and training content to automatically generate a story based on the specific operational procedures and safety instructions. The server then creates illustrations that correspond to the story.

[0847] The generated story and illustrations are sent from the server to the device, where they are displayed, allowing the user to learn the necessary steps and instructions.

[0848] Furthermore, the system is equipped with a real-time feedback function. The device uses a camera and microphone to record the user's reactions and transmits the data to the server. The server analyzes the recorded reaction data and reflects it in the next story generation.

[0849] For example, if a user wants to learn the safety inspection procedures for conveyor belts in a factory, they can send a prompt like this to the server:

[0850] "Generate a story based on the operating procedure of the equipment shown in the photo and the specified lesson. Lesson: 'Review safety procedures'. OCR text: 'Start operation by pressing the green button and ensure the emergency stop is accessible.'"

[0851] The generated story is then installed in a robot, which then provides practical training to the user by showing them operational procedures in real time. For example, the robot can assist the user in actually operating a conveyor belt.

[0852] This system enables efficient education and training tailored to individual needs, ensuring safety within the factory and helping employees learn efficient work procedures.

[0853] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0854] Step 1:

[0855] The user operates the terminal and enters registration information into the system. Specifically, the user enters their name, email address, and password, and presses the send button. The entered data is sent from the terminal to the server. The server receives this registration information and stores it in a database. It then notifies the terminal that registration is complete.

[0856] Step 2:

[0857] The user logs in via the device. The user enters their email address and password and presses the authentication button. The entered information is sent from the device to the server. The server performs authentication, and if successful, sends the home screen to the device; if unsuccessful, an error message is displayed.

[0858] Step 3:

[0859] A user uploads a photo of equipment in a factory. The user selects "Upload photo" from the menu on the terminal, selects a photo, and presses the send button. The terminal sends the photo data to the server. The server stores the sent photo data in a database.

[0860] Step 4:

[0861] The user specifies the operating procedures and safety instructions they want to learn. The user inputs the specific lessons and training content into the terminal and presses the send button. The input data is sent from the terminal to the server. The server stores this information in a database.

[0862] Step 5:

[0863] The server generates a story based on the photo data and educational content. First, the server performs OCR analysis of the uploaded photo data and extracts text information. Next, the generative AI model receives a prompt as input, which automatically generates a story and illustrations. Example prompt: "Generate a story based on the operating procedure for the equipment shown in the photo and the specified lesson. Lesson: 'Review safety procedures'. OCR text: 'Start operation by pressing the green button and ensure the emergency stop is accessible.'"

[0864] Step 6:

[0865] The generated story and illustrations are sent to the device. The server aggregates the generated data and sends it to the device. The device receives it and displays it to the user. The user can learn the educational content through the displayed story and illustrations.

[0866] Step 7:

[0867] Get real-time feedback. The device uses a camera and microphone to record the user's reactions during training. The recorded data is sent from the device to a server, which receives this reaction data and stores it in a database.

[0868] Step 8:

[0869] The server analyzes the response data and reflects it in the next story generation. The server uses a data analysis tool to analyze the user's response data. Based on the analysis results, the prompt sentences of the generative AI model are adjusted and applied to the next story generation.

[0870] Step 9:

[0871] The generated story is installed on the robot, and operating procedures are shown in real time. The server sends the generated story to the robot, and the robot uses the received data to show operating procedures and safety instructions to the user in real time. This allows the user to receive practical training.

[0872] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0873] The system of the present invention generates a personalized educational picture book through cooperation between a user, a terminal, a server, and an emotion engine. A specific embodiment of the system will be described below.

[0874] User Registration and Login

[0875] 1. User Registration

[0876] The user downloads the app to their device and launches it.

[0877] The device displays a registration form to the user, which includes fields for entering name, email address, password, etc.

[0878] The user enters the required information and presses the registration button.

[0879] The terminal transmits the entered registration information to the server.

[0880] The server stores the received registration information in a database and notifies the terminal that registration has been completed.

[0881] The terminal displays a message to the user indicating that registration is complete.

[0882] 2. Log in

[0883] The user launches the app again and goes to the login screen.

[0884] The terminal prompts the user to enter an email address and password.

[0885] The user enters the information and clicks the login button.

[0886] The terminal sends the input information to the server for authentication.

[0887] If the server is successful in authenticating the user, it sends the user's home screen to the device. If authentication fails, it displays an error message.

[0888] Upload photos and assign lessons

[0889] 1. Upload a photo

[0890] The user selects "Upload Photo" from the menu.

[0891] The device prompts the user to select a family photo to upload.

[0892] The user selects a photo and presses the upload button.

[0893] The device sends the photo data to the server.

[0894] The server receives the photo data and stores it in a database.

[0895] 2. Designation of lessons learned

[0896] The user is taken to the "Specify Lessons Learned" screen.

[0897] The device prompts the user to enter more information about the lesson or story they want to learn.

[0898] The user inputs a lesson such as "Greetings are important."

[0899] The terminal transmits the input lesson information to the server.

[0900] The server receives the lesson information and stores it in a database.

[0901] Creating and viewing stories

[0902] 1. Story Generation

[0903] The server starts generating a story based on the photo data and the lesson information.

[0904] The server uses conversational AI to automatically generate a story that follows the specified lesson.

[0905] The server generates illustrations of the characters based on the uploaded photos.

[0906] 2. Integration and transmission of picture book data

[0907] The server combines the generated story and illustrations to create picture book data.

[0908] The server sends the completed picture book data to the terminal.

[0909] 3. Display of picture books

[0910] The terminal displays the received picture book data to the user.

[0911] Users can read picture books together with their children and spend time learning.

[0912] Reaction monitoring and emotion recognition (optional)

[0913] 1. Activating the Emotional Engine

[0914] The device activates the camera and microphone and starts the emotion engine according to the user's selection.

[0915] 2. Reaction recording and emotion recognition

[0916] The device uses a camera and microphone to record the child's reactions and uses an emotion engine to recognize emotions from those reactions.

[0917] The recognized emotional information is processed in real time.

[0918] 3. Sending Emotional Data

[0919] The device transmits the recognized emotion data to the server.

[0920] The server receives the emotion data and stores it in a database.

[0921] 4. Emotional Data Analysis

[0922] The server analyzes the stored emotional data and uses it to generate the next story.

[0923] Gathering feedback and incorporating it into next time

[0924] 1. Enter your feedback

[0925] After finishing reading the picture book, the device displays a feedback form to the user.

[0926] The user inputs customization requests for the story and illustrations.

[0927] 2. Sending and storing feedback data

[0928] The terminal transmits the input feedback data to the server.

[0929] The server stores the feedback in a database and reflects it in the next story generation.

[0930] The above is a specific embodiment of the present invention. This system allows parents to easily create educational picture books tailored to their children and provide learning that is appropriate for their children. Furthermore, by using an emotion engine, more personalized content can be provided based on the child's emotions.

[0931] The processing flow will be explained below.

[0932] Step 1: User Registration and Login

[0933] 1. Install the app

[0934] The user downloads the app to their device and launches it.

[0935] 2. Displaying the user registration form

[0936] The terminal displays a registration form to the user.

[0937] 3. Enter your registration information

[0938] The user enters their name, email address, and password.

[0939] 4. Submission of Registration Information

[0940] The terminal transmits the input information to the server.

[0941] 5. Saving your registration information

[0942] The server verifies the received registration information and stores it in a database.

[0943] 6. Notification of registration completion

[0944] The server sends a registration completion message to the terminal.

[0945] The terminal displays a registration complete message to the user.

[0946] 7. Display the login screen

[0947] The device displays the login screen.

[0948] 8. Enter your authentication information

[0949] The user enters their email address and password.

[0950] 9. Submitting authentication information

[0951] The terminal transmits the entered authentication information to the server.

[0952] 10. Credential Verification

[0953] The server verifies the authentication information and sends the result to the device.

[0954] 11. Notification of Authentication Completion

[0955] If the server is successful in the authentication, it sends the user's home screen data to the terminal.

[0956] The device displays the user's home screen and completes the login.

[0957] Step 2: Upload photos and specify lessons

[0958] 1. Display the photo selection screen

[0959] The terminal displays a screen that allows the user to select a family photo.

[0960] 2. Select a photo

[0961] The user selects a family photo to upload.

[0962] 3. Sending photo data

[0963] The terminal transmits the selected photo data to the server.

[0964] 4. Saving photo data

[0965] The server stores the received photo data in a database.

[0966] 5. Display the lesson specification screen

[0967] The terminal displays a screen that prompts the user to input the lesson they want to learn.

[0968] 6. Input of lessons learned

[0969] The user inputs a lesson such as "Greetings are important."

[0970] 7. Submission of lessons learned data

[0971] The terminal transmits the input lesson data to the server.

[0972] 8. Lessons Learned Data Storage

[0973] The server stores the received lesson data in a database.

[0974] Step 3: Generate and view stories

[0975] 1. Start creating a story

[0976] The server retrieves the photo data and lesson data from the database.

[0977] 2. Story Generation

[0978] Based on the data acquired by the server, a story is generated using conversational AI.

[0979] 3. Illustration Generation

[0980] The server generates illustrations of the characters based on the uploaded photos.

[0981] 4. Picture book data integration

[0982] The server combines the generated story and illustrations to create picture book data.

[0983] 5. Sending picture book data

[0984] The server sends the completed picture book data to the terminal.

[0985] 6. Display of picture books

[0986] The terminal displays the received picture book data to the user.

[0987] 7. Picture book reading

[0988] The user reads a picture book with the child.

[0989] Step 4: Response Monitoring and Emotion Recognition (Optional)

[0990] 1. Activating the Emotional Engine

[0991] The device activates the camera and microphone and starts the emotion engine according to the user's selection.

[0992] 2. Recording the reaction

[0993] The device uses a camera and microphone to record the child's responses.

[0994] 3. Emotional Recognition

[0995] The device analyzes the recorded responses using an emotion engine to recognize emotions.

[0996] 4. Sending Emotional Data

[0997] The device transmits the recognized emotion data to the server.

[0998] The server stores the received emotion data in a database.

[0999] 5. Emotional Data Analysis

[1000] The server analyzes the stored emotional data and uses it to generate the next story.

[1001] Step 5: Gather feedback and incorporate it into next steps

[1002] 1. Display the feedback form

[1003] The device displays a feedback form after the picture book has finished reading.

[1004] 2. Enter your customization requests

[1005] The user inputs customization requests for the story and illustrations.

[1006] 3. Sending feedback data

[1007] The terminal transmits the input feedback data to the server.

[1008] 4. Feedback Data Storage

[1009] The server stores the received feedback in a database.

[1010] 5. Reflection in next story generation

[1011] The server analyzes the feedback data and reflects it in the next story generation.

[1012] Example 2

[1013] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1014] Conventional educational picture books have had difficulty generating stories that fit each child's interests and learning needs. Furthermore, they were unable to incorporate children's reactions as feedback in real time, making it difficult to change the content of a picture book once it was created. Furthermore, they were unable to provide content that responded to children's emotions.

[1015] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1016] In this invention, the server includes means for a user to input registration information and transmit it to the server, means for the server to store the registration information in a database, means for a user to input image data and educational content and transmit it to the server, means for the server to generate a story and create illustrations based on the image data and educational content, and means for the server to transmit the generated story and illustrations to the terminal, thereby enabling the generation of a personalized educational picture book in response to a user's request.

[1017] "User" refers to an individual or organization who enters registration information to use this system and receives services.

[1018] "Server" refers to a central computer system that receives information sent by users, stores it in a database, and performs any necessary processing.

[1019] "Registration information" refers to personal information such as name, email address, and password that a user enters to begin using the system.

[1020] "Database" refers to a collection of various data managed and stored by a server, such as registration information, image data, and educational content.

[1021] "Image data" refers to data containing visual information, such as photographs and pictures, uploaded by users.

[1022] "Educational content" refers to detailed information entered by the user about the lessons or stories they want their children to learn.

[1023] "Stories" refer to sentences and scenarios generated based on educational content using generative AI models.

[1024] "Illustration" refers to a visual picture or diagram generated based on uploaded image data or a story.

[1025] "Terminal" refers to a device such as a computer or smartphone that a user uses to operate the system.

[1026] "Reaction" refers to the child's reactions, including their behavior, facial expressions, and voice while reading an educational picture book.

[1027] "Customization requests" refer to improvements or specific requests that users input to be reflected in the next story generation or illustration creation.

[1028] A "generative AI model" refers to artificial intelligence technology that automatically generates stories based on educational content specified by the user.

[1029] MODE FOR CARRYING OUT THE INVENTION

[1030] The system of the present invention generates personalized educational picture books through cooperation between users, terminals, a server, and an emotion engine. The specific configuration and operation of the system are described below.

[1031] System configuration

[1032] This system consists of a terminal used by the user, a central management server, and an emotion engine that is used as needed.

[1033] Device: Computer, smartphone, tablet, etc.

[1034] Server: A high-performance computer that contains a database, generative AI models, and a network interface for communicating with devices over the internet.

[1035] Emotion Engine: A hardware device with a camera and microphone attached, optionally used by the user.

[1036] User Registration and Login

[1037] The user first downloads the app to their device and launches it. The device then displays a registration form to the user. When the user enters registration information such as their name, email address, and password and presses the registration button, the device sends this information to the server. The server stores the received registration information in a database and notifies the device that registration is complete. The user then launches the app again, enters their email address and password on the login screen, and sends them to the server for authentication.

[1038] Upload photos and assign lessons

[1039] The user selects "Upload Photos" from the menu, selects a family photo, and sends it to the server. The server saves the received photo data in a database. The user then moves to the "Specify Lesson" screen and inputs a lesson, such as "Greetings are important." The device sends this lesson information to the server, which saves it in a database.

[1040] Creating and viewing stories

[1041] The server automatically generates a story using a generative AI model (such as GPT-4) based on the received photo data and lesson information. It also generates illustrations of the characters based on the uploaded photos. The server then integrates the generated story and illustrations to create picture book data and sends it to the device. The user receives the picture book data on their device and reads it together with their child.

[1042] Reaction monitoring and emotion recognition (optional)

[1043] Depending on the user's selection, the device's camera and microphone are activated and the emotion engine is used. While the child is reading the picture book, the device uses the camera and microphone to record the child's reactions and sends the data to the server. The server analyzes the recorded data and reflects it in the next story generation.

[1044] Gathering feedback and incorporating it into next time

[1045] After reading the picture book, the device displays a feedback form to the user. The user enters their requests for customization of the story and illustrations and sends them from the device to the server. The server stores the feedback data in a database and reflects it in the next story generation.

[1046] Specific examples and examples of prompts for generative AI models

[1047] Examples:

[1048] The user specifies the lesson, "Greetings are important," and uploads a family photo. The server follows the following process: receives the photo data and lesson information ⇒ generates a story using a generative AI model (e.g., GPT-4) ⇒ generates illustrations of the characters ⇒ integrates the picture book data ⇒ sends the data to the device. The user opens the generated picture book and teaches their child the importance of greetings such as "good morning" and "thank you."

[1049] Example prompt sentence:

[1050] text

[1051] Lesson: Greetings are important

[1052] Photo: Family photo

[1053] Story content: Describe how greetings are important throughout a family's day. Include scenes such as "Good morning" in the morning, "Itadakimasu" at mealtimes, "I'm home" when you get home, and "Good night" before going to bed.

[1054] This system allows parents to easily create educational picture books tailored to their children and provide learning that is appropriate for them. Furthermore, by using an emotion engine, more personalized content can be provided based on the child's emotions.

[1055] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1056] Step 1:

[1057] User Registration

[1058] The user downloads the application to their device and launches it. The device displays a registration form that prompts the user to enter their name, email address, password, etc. The user enters the required information and presses the registration button. The entered information is sent from the device to the server. The server stores the received registration information in a database and notifies the device that registration is complete. The device displays a message to the user that registration is complete.

[1059] Input: Name, email address, password

[1060] Output: Registration complete message

[1061] Step 2:

[1062] Log in

[1063] The user launches the application and goes to the login screen. The device prompts the user to enter their email address and password. The user enters the information and presses the login button. The entered login information is sent from the device to the server, and the server performs authentication. If authentication is successful, the server sends the user's home screen data to the device, which displays it. If authentication fails, the server sends an error message to the device, which displays it.

[1064] Input: Email address, password

[1065] Output: Home screen data or error message

[1066] Step 3:

[1067] Upload a photo

[1068] The user selects "Upload Photos" from the menu. The device displays a screen prompting the user to select a family photo. When the user selects a photo and presses the upload button, the photo data is sent from the device to the server. The server stores the received photo data in a database.

[1069] Input: Photo data

[1070] Output: Confirmation of saved photo data

[1071] Step 4:

[1072] Designation of lessons learned

[1073] The user moves to the "Specify Lesson" screen and enters detailed information about the lesson or story they want to learn. The device sends this information to the server, which then stores the received lesson information in a database.

[1074] Input: Detailed information about the lesson or story (e.g., greetings are important)

[1075] Output: Confirmation of saved lesson information

[1076] Step 5:

[1077] Story Generation

[1078] The server generates a story using a generative AI model (e.g., GPT-4) based on the received photo data and lesson information, and also generates illustrations of the characters based on the uploaded photos.

[1079] Input: Photo data, lesson information

[1080] Output: Generated story and illustrations

[1081] Step 6:

[1082] Integration and transmission of picture book data

[1083] The server integrates the generated story and illustrations to create a picture book, and the completed picture book data is sent from the server to the device.

[1084] Input: Story, illustration

[1085] Output: Picture book data

[1086] Step 7:

[1087] Picture book display

[1088] The device displays the received picture book data to the user, who then reads the picture book together with their child, spending time learning.

[1089] Input: Picture book data

[1090] Output: The displayed picture book

[1091] Step 8:

[1092] Reaction monitoring and emotion recognition (optional)

[1093] If the user selects it, the device activates the camera and microphone to record the child's reactions. The emotion engine analyzes this data and transmits the recognized emotion data from the device to the server. The server receives the emotion data and stores it in a database.

[1094] Input: Child's reaction data via camera and microphone

[1095] Output: Analyzed sentiment data

[1096] Step 9:

[1097] Gathering feedback and incorporating it into next time

[1098] After reading the picture book, the device displays a feedback form to the user. The user enters their requests for customization of the story and illustrations and sends them from the device to the server. The server stores the feedback data in a database and reflects it in the next story generation.

[1099] Input: Customization request

[1100] Output: A confirmation of saved feedback data

[1101] (Application example 2)

[1102] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1103] Conventional educational picture book systems have issues with the quality and personalization of story generation, even when users enter registration information and provide photos and lesson information. It is also difficult to grasp children's reactions in real time and reflect them in the next story. This makes it impossible to provide a truly personalized educational experience that meets the needs of each user and child.

[1104] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input registration information and transmit it to the server; a means for the server to store the registration information in a database; a means for a user to input photo data and educational content and transmit it to the server; a means for the server to generate a story and create illustrations based on the photo data and educational content; a means for the server to transmit the generated story and illustration to the terminal; a means for the terminal to display the received story and illustration; a means for a user to input feedback and customization requests for the generated story and illustration and transmit it to the server; a means for the server to store the received feedback and customization requests in a database and reflect them in the next story generation; a means for the terminal to record the child's responses using a camera or microphone and transmit it to the server; a means for the server to analyze the recorded child's responses using a generative AI model and reflect them in the next story generation; and a means for an application to run on a smartphone or other available terminal. This makes it possible to provide a personalized educational experience tailored to the needs of each user and child.

[1105] A "user" is a person who uses the personalized educational picture book system.

[1106] "Server" means the central processing unit that receives and processes information from users, stores data, and transmits generated stories and illustrations.

[1107] A "terminal" is a device used by a user, such as a smartphone or tablet, that is used to display stories and illustrations received from the server.

[1108] "Photo data" refers to image files that users upload to the system and are used as characters and backgrounds for educational picture books.

[1109] "Educational content" refers to the lesson or learning material specified by the user, and is the main input information for story generation.

[1110] "Story generation" is the process in which the server automatically creates a story using conversational AI based on the input photo data and educational content.

[1111] "Illustration creation" is the process in which the server draws the characters and backgrounds in the story based on the uploaded photo data.

[1112] "Feedback" refers to the evaluation and customization requests provided by users for the generated stories and illustrations.

[1113] "Customization Request" means a specific change or addition you would like to make provided as part of your Feedback.

[1114] "Response recording" is the process by which the device collects the child's responses using a camera and microphone.

[1115] A "generative AI model" refers to an artificial intelligence technology that performs story and sentiment analysis based on input data.

[1116] "Prompts" refer to instructions or questions used when performing story or sentiment analysis on a generative AI model.

[1117] MODE FOR CARRYING OUT THE INVENTION

[1118] This invention is a system that generates educational picture books using a user interface, where users use devices such as smartphones and tablets. At the heart of the system is a server that processes and stores input from users and uses a generative AI model to create personalized stories and illustrations.

[1119] User Registration and Login

[1120] 1. The user installs and launches a dedicated application on their device.

[1121] 2. The device presents the user with a registration form and asks them to enter registration information such as their name, email address, and password.

[1122] 3. The user enters the required information and sends it to the server.

[1123] 4. The server saves the registration information in its database and completes the user registration.

[1124] Upload photos and assign lessons

[1125] 1. The user selects "Upload Photo" from the menu, selects a family photo, and sends it to the server.

[1126] 2. The server receives the photo data and stores it in a database.

[1127] 3. The user inputs the lesson or learning they want to teach and sends it to the server.

[1128] 4. The server receives the lesson information and stores it in a database.

[1129] Creating and viewing stories

[1130] 1. Based on the photo data and lesson information received by the server, a generative AI model is used to automatically generate a story and illustrations.

[1131] 2. The server combines the generated story and illustrations to create picture book data.

[1132] 3. The server sends the completed picture book data to the terminal.

[1133] 4. The device displays the received picture book data to the user.

[1134] Emotion Recognition and Feedback

[1135] 1. The device uses a camera and microphone to record the child's responses.

[1136] 2. The device sends the recorded data to the server and executes the emotion engine.

[1137] 3. The server uses an emotion engine to analyze the child's reaction data and reflect it in the next story generation.

[1138] 4. The user enters feedback and customization requests from the device and sends them to the server.

[1139] 5. The server stores the feedback and customization requests in a database and reflects them in the next story generation.

[1140] Hardware and software used

[1141] Device: The smartphone or tablet used by the user.

[1142] Server: Central processing unit that stores and processes registration information, photo data, lesson information, emotion data, and feedback.

[1143] Generative AI models: Artificial intelligence techniques used for story and illustration generation, as well as sentiment analysis.

[1144] Open source libraries: HTTP communication using the requests library, conversational AI models using the transformers library, image and video analysis using opencv.

[1145] Examples of concrete examples and prompts

[1146] As a concrete example, if a parent wants to teach their child the importance of greetings, the steps would be as follows:

[1147] 1. A parent uploads a family photo on their smartphone and enters a lesson about the importance of greetings.

[1148] 2. The server uses the generative AI model to generate a story using the prompt: "Create a story about a family going hiking with their parents. Include the lesson that greetings are important. Use the family photo below as the characters."

[1149] 3. The child's reaction when reading the picture book is recorded with a camera and sent to the server. The emotional data recorded at that time is reflected in the next story generation.

[1150] In this way, the system of the present invention can provide personalized education that meets the needs of each individual user and encourage effective learning for children.

[1151] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1152] Step 1:

[1153] The user installs and launches a dedicated application on their device. The device then provides the user with a registration form, prompting them to enter registration information such as their name, email address, and password. The user enters the required information and sends it to the server. The server saves the registration information in a database, completing the user registration. The input is the user's personal information, and the output is the registration information saved in the database.

[1154] Step 2:

[1155] The user selects "Upload Photo" from the menu, selects a family photo, and sends it to the server. The server receives the photo data and stores it in a database. The user then enters the lesson or learning content they want to teach and sends it to the server. The server receives the lesson information and stores it in the database. The input is the photo data and lesson information, and the output is the photo data and lesson information stored in the database.

[1156] Step 3:

[1157] Based on the photo data and lesson information received by the server, a generative AI model is used to automatically generate a story and illustrations. The prompt text "Write a story about a family going hiking with their parents. Include the lesson that greetings are important. Also, use the family photos below as characters." is input into the generative AI model. The server combines the generated story and illustrations to create picture book data. The input is photo data, lesson information, and the prompt text, and the output is the generated story and illustrations.

[1158] Step 4:

[1159] The server sends the completed picture book data to the terminal. The terminal displays the received picture book data to the user. The user reads the picture book together with the child and spends time learning. The input is the picture book data, and the output is the picture book displayed on the terminal.

[1160] Step 5:

[1161] The device uses a camera and microphone to record the child's reactions. The recorded data includes facial expressions and tone of voice. The device sends the recorded data to the server, which runs the emotion engine. The server uses the emotion engine to analyze the child's reaction data. The input is the child's reaction data, and the output is the analyzed emotion data.

[1162] Step 6:

[1163] The user inputs feedback and customization requests from the terminal and sends them to the server. The server stores the feedback and customization requests in a database and reflects them in the next story generation. The input is the feedback and customization requests, and the output is the feedback and customization requests stored in the database.

[1164] Step 7:

[1165] The next time a story is generated, the server uses the customization requests and emotional data stored in the database to input prompts into the generative AI model. This generates a story optimized for each user or child. The input is past customization requests and emotional data, and the output is a new story optimized for the user.

[1166] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1167] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1168] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1169] [Third embodiment]

[1170] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1171] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1172] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1173] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1174] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1175] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1176] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1177] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1178] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1179] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1180] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1181] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1182] The system of the present invention generates a personalized educational picture book through the mutual cooperation of a user, a terminal, and a server. A specific embodiment of the system will be described below.

[1183] User Registration and Login

[1184] 1. User Registration

[1185] The user downloads the app to their device and launches it.

[1186] The device displays a registration form to the user, which includes fields for entering name, email address, password, etc.

[1187] The user enters the required information and presses the registration button.

[1188] The terminal transmits the entered registration information to the server.

[1189] The server stores the received registration information in a database and notifies the terminal that registration has been completed.

[1190] The terminal displays a message to the user indicating that registration is complete.

[1191] 2. Log in

[1192] The user launches the app again and goes to the login screen.

[1193] The terminal prompts the user to enter an email address and password.

[1194] The user enters the information and clicks the login button.

[1195] The terminal sends the input information to the server for authentication.

[1196] If the server is successful in authenticating the user, it sends the user's home screen to the device. If authentication fails, it displays an error message.

[1197] Upload photos and assign lessons

[1198] 1. Upload a photo

[1199] The user selects "Upload Photo" from the menu.

[1200] The device prompts the user to select a family photo to upload.

[1201] The user selects a photo and presses the upload button.

[1202] The device sends the photo data to the server.

[1203] The server receives the photo data and stores it in a database.

[1204] 2. Designation of lessons learned

[1205] The user is taken to the "Specify Lessons Learned" screen.

[1206] The device prompts the user to enter more information about the lesson or story they want to learn.

[1207] The user inputs a lesson such as "Greetings are important."

[1208] The terminal transmits the input lesson information to the server.

[1209] The server receives the lesson information and stores it in a database.

[1210] Creating and viewing stories

[1211] 1. Story generation

[1212] The server starts generating a story based on the photo data and the lesson information.

[1213] The server uses conversational AI to automatically generate a story that follows the specified lesson.

[1214] The server generates illustrations of the characters based on the uploaded photos.

[1215] 2. Integration and transmission of picture book data

[1216] The server combines the generated story and illustrations to create picture book data.

[1217] The server sends the completed picture book data to the terminal.

[1218] 3. Display of picture books

[1219] The terminal displays the received picture book data to the user.

[1220] Users can read picture books together with their children and spend time learning.

[1221] Reaction monitoring (optional)

[1222] 1. Use of camera and microphone

[1223] The device will use its camera and microphone to give you the option to record your child's responses.

[1224] If the user selects this option, the device will start recording.

[1225] 2. Submitting and analyzing reaction data

[1226] The terminal transmits the recorded reaction data to the server.

[1227] The server analyzes the response data and uses it to generate the next story.

[1228] Gathering feedback and incorporating it into next time

[1229] 1. Enter your feedback

[1230] After finishing reading the picture book, the device displays a feedback form to the user.

[1231] The user inputs customization requests for the story and illustrations.

[1232] 2. Sending and storing feedback data

[1233] The terminal transmits the input feedback data to the server.

[1234] The server stores the feedback in a database and reflects it in the next story generation.

[1235] The above is a specific embodiment of the present invention. This system allows parents to easily create educational picture books tailored to their children and provide them with learning that is appropriate for their children.

[1236] The processing flow will be explained below.

[1237] Step 1: User Registration and Login

[1238] 1. Install the app

[1239] The user downloads the app to their device and launches it.

[1240] 2. Displaying the user registration form

[1241] The terminal displays a registration form to the user.

[1242] 3. Enter your registration information

[1243] The user enters their name, email address, and password.

[1244] 4. Submission of Registration Information

[1245] The terminal transmits the input information to the server.

[1246] 5. Saving your registration information

[1247] The server verifies the received registration information and stores it in a database.

[1248] 6. Notification of registration completion

[1249] The server sends a registration completion message to the terminal.

[1250] The terminal displays a registration complete message to the user.

[1251] 7. Display the login screen

[1252] The device displays the login screen.

[1253] 8. Enter your authentication information

[1254] The user enters their email address and password.

[1255] 9. Submitting authentication information

[1256] The terminal transmits the entered authentication information to the server.

[1257] 10. Credential Verification

[1258] The server verifies the authentication information and sends the result to the device.

[1259] 11. Notification of Authentication Completion

[1260] If the server is successful in the authentication, it sends the user's home screen data to the terminal.

[1261] The device displays the user's home screen and completes the login.

[1262] Step 2: Upload photos and specify lessons

[1263] 1. Display the photo selection screen

[1264] The terminal displays a screen that allows the user to select a family photo.

[1265] 2. Select a photo

[1266] The user selects a family photo to upload.

[1267] 3. Sending photo data

[1268] The terminal transmits the selected photo data to the server.

[1269] 4. Saving photo data

[1270] The server stores the received photo data in a database.

[1271] 5. Display the lesson specification screen

[1272] The terminal displays a screen that prompts the user to input the lesson they want to learn.

[1273] 6. Input of lessons learned

[1274] The user inputs a lesson such as "Greetings are important."

[1275] 7. Submission of lessons learned data

[1276] The terminal transmits the input lesson data to the server.

[1277] 8. Lessons Learned Data Storage

[1278] The server stores the received lesson data in a database.

[1279] Step 3: Generate and view stories

[1280] 1. Start creating a story

[1281] The server retrieves the photo data and lesson data from the database.

[1282] 2. Story Generation

[1283] Based on the data acquired by the server, a story is generated using conversational AI.

[1284] 3. Illustration Generation

[1285] The server generates illustrations of the characters based on the uploaded photos.

[1286] 4. Picture book data integration

[1287] The server combines the generated story and illustrations to create picture book data.

[1288] 5. Sending picture book data

[1289] The server sends the completed picture book data to the terminal.

[1290] 6. Display of picture books

[1291] The terminal displays the received picture book data to the user.

[1292] 7. Picture book reading

[1293] The user reads a picture book with the child.

[1294] Step 4: Monitoring the reaction (optional)

[1295] 1. Turn on the camera and microphone

[1296] The device will activate the camera and microphone according to the user's selection.

[1297] 2. Recording the reaction

[1298] The device records the child's responses.

[1299] 3. Submitting reaction data

[1300] The terminal transmits the recorded reaction data to the server.

[1301] 4. Saving and analyzing reaction data

[1302] The server stores the received response data and uses it for the next story generation.

[1303] Step 5: Gather feedback

[1304] 1. Display the feedback form

[1305] The device displays a feedback form after the picture book has finished reading.

[1306] 2. Enter your customization requests

[1307] The user inputs customization requests for the story and illustrations.

[1308] 3. Sending feedback data

[1309] The terminal transmits the input feedback data to the server.

[1310] 4. Feedback Data Storage

[1311] The server stores the received feedback in a database.

[1312] 5. Reflection in next story generation

[1313] The server analyzes the feedback data and reflects it in the next story generation.

[1314] Example 1

[1315] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1316] In modern education, there is a demand for personalized educational materials tailored to each child, but existing systems have difficulty in providing such personalized education. Furthermore, because the content and format of educational materials are fixed, it is difficult to provide education that reflects the interests and reactions of children. Therefore, a system is needed that can effectively provide individual educational experiences tailored to each child.

[1317] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1318] In this invention, the server includes means for a user to input registration information and transmit it to the server, means for the server to store the registration information in a database, means for a user to input photo data and educational information and transmit it to the server, means for the server to automatically generate a story and create illustrations using a generative AI model based on the photo data and educational information, means for the server to combine the generated story and illustrations to create picture book data and transmit it to the terminal, and means for the terminal to display the picture book data received. This enables users to easily create personalized educational picture books and provide learning content tailored to their children.

[1319] A "user" is an individual or group that uses this system and is the entity that creates and uses educational picture books.

[1320] A "terminal" is a device used by a user, such as a computer, smartphone, or tablet, that provides an interface with the system.

[1321] A "server" is the central part of a system and is a computer system that stores, processes, and manages data.

[1322] "Registration information" refers to information such as name, email address, and password required for a user to access the system.

[1323] The "database" is a data storage system for storing and managing user registration information, photo data, educational information, generated picture book data, and the like.

[1324] "Photo data" refers to image files uploaded by users and used for characters and backgrounds in picture books.

[1325] "Educational information" is detailed information about the lessons and stories that the user wants to learn as the content of the picture book.

[1326] A "generative AI model" is an artificial intelligence that automatically generates text and images based on data, such as ChatGPT.

[1327] A "story" is a narrative created by a generative AI model based on educational information.

[1328] "Illustrations" are images of characters and backgrounds generated based on uploaded photo data.

[1329] "Picture book data" is data in the form of an e-book created by combining the generated story and illustrations.

[1330] "Customization requests" are specific requests for changes or improvements that users input regarding stories or illustrations.

[1331] "Reaction data" refers to audio and video data recorded with a camera or microphone of a child's reaction when a picture book is read to them.

[1332] The system of the present invention generates a personalized educational picture book through the mutual cooperation of a user, a terminal, and a server. A specific embodiment of the system will be described below.

[1333] User Registration and Login Implementation

[1334] First, the user downloads the application to their device and launches it. The device displays a registration form to the user, and the user enters registration information such as their name, email address, and password. The entered registration information is sent from the device to the server, and the server stores the information in a database. The server then notifies the device that registration is complete, and the device displays a message to the user that registration is complete. When the user launches the application again, the user enters their email address and password on the login screen and sends them to the server via the device. If the server is successful in its authentication, the user's home screen is sent to the device.

[1335] Uploading photos and assigning lessons

[1336] When the user selects "Upload Photos" from the menu, the device prompts the user to select a family photo to upload. When the user selects a photo and presses the upload button, the photo data is sent from the device to the server. The server receives the photo data and stores it in a database. Next, the user moves to the "Specify Lesson" screen and enters educational information, such as "Greetings are important." The entered educational information is sent to the server via the device, and the server stores the educational information in a database.

[1337] Story Generation and Display Embodiments

[1338] The server begins generating a story based on the photo data and educational information. It uses generative AI models such as ChatGPT to automatically generate a story that follows the specified moral. It also generates illustrations of the characters based on the uploaded photos. The server then combines the generated story and illustrations to create picture book data and sends it to the device. The device then displays the received picture book data to the user, who can then read the book together with their child.

[1339] Reaction Monitoring Embodiments (Optional)

[1340] The device provides an option to record the child's reactions using the camera and microphone. When the user selects this option, the device starts recording. The recorded reaction data is sent from the device to a server, which analyzes the reaction data and uses it to generate the next story. This analysis includes emotion analysis using a voice recognition API.

[1341] How to collect feedback and incorporate it into future projects

[1342] After reading the picture book, the device displays a feedback form to the user. The user enters any customization requests for the story or illustrations and sends them from the device to the server. The server stores the customization requests in a database and reflects them in the next story generation.

[1343] Specific examples

[1344] Users download the app, enter their registration information, and send it to the server. Next, they upload a family photo and specify a lesson, such as "greetings are important." The server uses a generative AI model such as ChatGPT to generate a story and creates illustrations of the characters based on the photos. The completed picture book data is sent to the device, and users can provide learning opportunities by reading the book together with their children.

[1345] An example prompt is "Generate a children's story with the moral that saying hello is important. The story will feature a little bear, and include illustrations based on uploaded family photos."

[1346] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1347] Step 1:

[1348] A user downloads the app and enters their registration information.

[1349] The user downloads the app to their device and launches it. The device displays a form for the user to enter their name, email address, and password. The user enters the required information and presses the "Register" button. The entered information is temporarily stored in the device's memory.

[1350] Input: Name, email address, password

[1351] Output: The registration information is temporarily stored in the device's memory.

[1352] Step 2:

[1353] The terminal sends the registration information to the server, which stores it in a database.

[1354] The terminal sends the registration information entered by the user to the server. The server receives the registration information and stores it in a database using an SQL query. The server sends a notification of registration completion to the terminal. The terminal displays a message to the user notifying them that registration was successful.

[1355] Input: Registration information (name, email address, password)

[1356] Output: The registration information is saved in the database and a message indicating successful registration is displayed.

[1357] Step 3:

[1358] The user logs in again and authenticates.

[1359] The user launches the app again and goes to the login screen. The device prompts the user to enter their email address and password. The user enters the information and presses the "Login" button. The device sends the entered information to the server, which then references the database and performs authentication. If authentication is successful, the server sends the user's home screen to the device.

[1360] Input: Email address, password

[1361] Output: If authentication is successful, the home screen is displayed.

[1362] Step 4:

[1363] The user uploads a photo.

[1364] The user selects "Upload photo" from the menu. The device displays a file selection dialog, and the user selects the photo to upload. The user selects a photo and presses the "Upload" button. The device sends the photo data to the server, and the server saves the received photo data in a database.

[1365] Input: Photo data

[1366] Output: The photo data is saved in the database.

[1367] Step 5:

[1368] The user inputs the lesson information.

[1369] The user moves to the "Specify Lesson" screen and inputs the lesson information. The terminal displays an input form for educational information, and the user inputs a lesson such as "Greetings are important." The user inputs the information and presses the "Send" button. The terminal sends the input educational information to the server, which stores it in a database.

[1370] Input: Lesson learned information

[1371] Output: Lessons learned information is stored in a database.

[1372] Step 6:

[1373] The server uses a generative AI model to generate a story and create illustrations.

[1374] The server starts generating a story based on the photo data and lesson information. It inputs a prompt (e.g., "Greetings are important") into a generative AI model such as ChatGPT to generate a story. The server uses an image processing library to generate illustrations of people based on the uploaded photos.

[1375] Input: Photo data, lesson information

[1376] Output: Generated stories and illustrations

[1377] Step 7:

[1378] The server integrates the generated story and illustrations into picture book data and sends it to the terminal.

[1379] The server combines the generated story and illustrations to create picture book data. The server uses a PDF generation library to combine the text and illustrations into PDF format. The completed picture book data is sent to the device.

[1380] Input: Generated story and illustrations

[1381] Output: Picture book data is sent to the terminal.

[1382] Step 8:

[1383] The device displays the picture book, and the user reads it together with the child.

[1384] The device displays the received picture book data. Using the device's built-in PDF viewer, the user can read the book together with their child.

[1385] Input: Picture book data

[1386] Output: The picture book is displayed on the device.

[1387] Step 9:

[1388] Optionally, the device records the child's responses and transmits them to a server.

[1389] The device provides an option to record the child's reactions using the camera and microphone. When the user selects this option, the device starts recording and sends the reaction data to the server. The server receives the reaction data and performs sentiment analysis using a speech recognition API. The results are used to generate the next story.

[1390] Input: Child's reaction (audio and video data)

[1391] Output: The analysis results are reflected in the next story generation.

[1392] Step 10:

[1393] The user inputs customization requests, which the server reflects in the next story generation.

[1394] After reading the picture book, the device displays a feedback form to the user. The user enters and submits any requests for customization of the story or illustrations. The device then sends the feedback data to the server, which stores it in a database and reflects it in the next story generation.

[1395] Input: Customization request

[1396] Output: Customization requests are reflected in the next story generation.

[1397] (Application example 1)

[1398] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1399] There is a demand for efficient training in specific operating procedures and safety instructions in educational settings and factories. However, traditional training methods have had issues in meeting individual needs, making it difficult to achieve efficient training. It has also been difficult to provide real-time feedback on the reactions of employees and operators and reflect them in training content.

[1400] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1401] In this invention, the server includes: a means for a user to input registration information and transmit it to the server; a means for the server to store the registration information in a database; a means for a user to input photo data and training content and transmit it to the server; a means for the server to generate a story and create illustrations based on the photo data and training content; a means for the server to transmit the generated story and illustrations to a terminal; a means for the terminal to display the received story and illustrations; a means for inputting training content including operating procedures and safety instructions for equipment installed in a factory and transmitting it to the server; a means for the server to automatically generate a story regarding operating procedures and safety instructions based on analyzed photo data; and a means for installing the generated story on a robot and showing operating procedures in real time, thereby enabling efficient education and training according to individual needs.

[1402] A "user" is a person who uses the system to input and submit registration information, photo data, and educational content.

[1403] "Server" refers to a computer system that receives, stores, analyzes, and processes registration information, photo data, and educational content sent by users.

[1404] A "terminal" is a device that a user operates to input and display data, and has the function of receiving and displaying data from a server.

[1405] "Registration information" refers to personal information such as name, email address, and password that a user enters when accessing the system for the first time.

[1406] "Photo data" refers to image files that record training and operating procedures and are uploaded by users to the system.

[1407] "Educational content" refers to information that is the subject of education, such as procedures, instructions, lessons, etc. that users should learn.

[1408] "Stories" are a series of educational stories generated by the server based on photo data and educational content.

[1409] "Illustrations" are visual diagrams or pictures based on the story that are generated by the server.

[1410] "Factory equipment" refers to various operating devices and machines used in factories.

[1411] An "operating procedure" is a set of procedures or methods for safely and efficiently operating equipment in a factory.

[1412] "Safety instructions" are instructions and guidelines for ensuring safety when using equipment in a factory.

[1413] "Real-time feedback" refers to immediate user reactions and data obtained during education and training.

[1414] A "robot" is a programmable machine that executes and presents specified operating procedures and safety instructions to the user.

[1415] The present invention provides a personalized training system that allows users to efficiently learn specific operating procedures and safety instructions within a factory. This system functions in cooperation with users, terminals, a server, and robots.

[1416] First, a user accesses the system via a terminal and enters their registration information. The terminal then sends the registration information (name, email address, password, etc.) entered by the user to the server. The server receives this information and stores it in a database.

[1417] After registration, the user uploads photographs of the equipment in the factory to the terminal, which then sends the photographs to the server, where they are stored.

[1418] Next, the user inputs the desired operational procedures and safety instructions into the device and sends the training content to the server. The server then uses a generative AI model based on the photo data and training content to automatically generate a story based on the specific operational procedures and safety instructions. The server then creates illustrations that correspond to the story.

[1419] The generated story and illustrations are sent from the server to the device, where they are displayed, allowing the user to learn the necessary steps and instructions.

[1420] Furthermore, the system is equipped with a real-time feedback function. The device uses a camera and microphone to record the user's reactions and transmits the data to the server. The server analyzes the recorded reaction data and reflects it in the next story generation.

[1421] For example, if a user wants to learn the safety inspection procedures for conveyor belts in a factory, they can send a prompt like this to the server:

[1422] "Generate a story based on the operating procedure of the equipment shown in the photo and the specified lesson. Lesson: 'Review safety procedures'. OCR text: 'Start operation by pressing the green button and ensure the emergency stop is accessible.'"

[1423] The generated story is then installed in a robot, which then provides practical training to the user by showing them operational procedures in real time. For example, the robot can assist the user in actually operating a conveyor belt.

[1424] This system enables efficient education and training tailored to individual needs, ensuring safety within the factory and helping employees learn efficient work procedures.

[1425] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1426] Step 1:

[1427] The user operates the terminal and enters registration information into the system. Specifically, the user enters their name, email address, and password, and presses the send button. The entered data is sent from the terminal to the server. The server receives this registration information and stores it in a database. It then notifies the terminal that registration is complete.

[1428] Step 2:

[1429] The user logs in via the device. The user enters their email address and password and presses the authentication button. The entered information is sent from the device to the server. The server performs authentication, and if successful, sends the home screen to the device; if unsuccessful, an error message is displayed.

[1430] Step 3:

[1431] A user uploads a photo of equipment in a factory. The user selects "Upload photo" from the menu on the terminal, selects a photo, and presses the send button. The terminal sends the photo data to the server. The server stores the sent photo data in a database.

[1432] Step 4:

[1433] The user specifies the operating procedures and safety instructions they want to learn. The user inputs the specific lessons and training content into the terminal and presses the send button. The input data is sent from the terminal to the server. The server stores this information in a database.

[1434] Step 5:

[1435] The server generates a story based on the photo data and educational content. First, the server performs OCR analysis of the uploaded photo data and extracts text information. Next, the generative AI model receives a prompt as input, which automatically generates a story and illustrations. Example prompt: "Generate a story based on the operating procedure for the equipment shown in the photo and the specified lesson. Lesson: 'Review safety procedures'. OCR text: 'Start operation by pressing the green button and ensure the emergency stop is accessible.'"

[1436] Step 6:

[1437] The generated story and illustrations are sent to the device. The server aggregates the generated data and sends it to the device. The device receives it and displays it to the user. The user can learn the educational content through the displayed story and illustrations.

[1438] Step 7:

[1439] Get real-time feedback. The device uses a camera and microphone to record the user's reactions during training. The recorded data is sent from the device to a server, which receives this reaction data and stores it in a database.

[1440] Step 8:

[1441] The server analyzes the response data and reflects it in the next story generation. The server uses a data analysis tool to analyze the user's response data. Based on the analysis results, the prompt sentences of the generative AI model are adjusted and applied to the next story generation.

[1442] Step 9:

[1443] The generated story is installed on the robot, and operating procedures are shown in real time. The server sends the generated story to the robot, and the robot uses the received data to show operating procedures and safety instructions to the user in real time. This allows the user to receive practical training.

[1444] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1445] The system of the present invention generates a personalized educational picture book through cooperation between a user, a terminal, a server, and an emotion engine. A specific embodiment of the system will be described below.

[1446] User Registration and Login

[1447] 1. User Registration

[1448] The user downloads the app to their device and launches it.

[1449] The device displays a registration form to the user, which includes fields for entering name, email address, password, etc.

[1450] The user enters the required information and presses the registration button.

[1451] The terminal transmits the entered registration information to the server.

[1452] The server stores the received registration information in a database and notifies the terminal that registration has been completed.

[1453] The terminal displays a message to the user indicating that registration is complete.

[1454] 2. Log in

[1455] The user launches the app again and goes to the login screen.

[1456] The terminal prompts the user to enter an email address and password.

[1457] The user enters the information and clicks the login button.

[1458] The terminal sends the input information to the server for authentication.

[1459] If the server is successful in authenticating the user, it sends the user's home screen to the device. If authentication fails, it displays an error message.

[1460] Upload photos and assign lessons

[1461] 1. Upload a photo

[1462] The user selects "Upload Photo" from the menu.

[1463] The device prompts the user to select a family photo to upload.

[1464] The user selects a photo and presses the upload button.

[1465] The device sends the photo data to the server.

[1466] The server receives the photo data and stores it in a database.

[1467] 2. Designation of lessons learned

[1468] The user is taken to the "Specify Lessons Learned" screen.

[1469] The device prompts the user to enter more information about the lesson or story they want to learn.

[1470] The user inputs a lesson such as "Greetings are important."

[1471] The terminal transmits the input lesson information to the server.

[1472] The server receives the lesson information and stores it in a database.

[1473] Creating and viewing stories

[1474] 1. Story Generation

[1475] The server starts generating a story based on the photo data and the lesson information.

[1476] The server uses conversational AI to automatically generate a story that follows the specified lesson.

[1477] The server generates illustrations of the characters based on the uploaded photos.

[1478] 2. Integration and transmission of picture book data

[1479] The server combines the generated story and illustrations to create picture book data.

[1480] The server sends the completed picture book data to the terminal.

[1481] 3. Display of picture books

[1482] The terminal displays the received picture book data to the user.

[1483] Users can read picture books together with their children and spend time learning.

[1484] Reaction monitoring and emotion recognition (optional)

[1485] 1. Activating the Emotional Engine

[1486] The device activates the camera and microphone and starts the emotion engine according to the user's selection.

[1487] 2. Reaction recording and emotion recognition

[1488] The device uses a camera and microphone to record the child's reactions and uses an emotion engine to recognize emotions from those reactions.

[1489] The recognized emotional information is processed in real time.

[1490] 3. Sending Emotional Data

[1491] The device transmits the recognized emotion data to the server.

[1492] The server receives the emotion data and stores it in a database.

[1493] 4. Emotional Data Analysis

[1494] The server analyzes the stored emotional data and uses it to generate the next story.

[1495] Gathering feedback and incorporating it into next time

[1496] 1. Enter your feedback

[1497] After finishing reading the picture book, the device displays a feedback form to the user.

[1498] The user inputs customization requests for the story and illustrations.

[1499] 2. Sending and storing feedback data

[1500] The terminal transmits the input feedback data to the server.

[1501] The server stores the feedback in a database and reflects it in the next story generation.

[1502] The above is a specific embodiment of the present invention. This system allows parents to easily create educational picture books tailored to their children and provide learning that is appropriate for their children. Furthermore, by using an emotion engine, more personalized content can be provided based on the child's emotions.

[1503] The processing flow will be explained below.

[1504] Step 1: User Registration and Login

[1505] 1. Install the app

[1506] The user downloads the app to their device and launches it.

[1507] 2. Displaying the user registration form

[1508] The terminal displays a registration form to the user.

[1509] 3. Enter your registration information

[1510] The user enters their name, email address, and password.

[1511] 4. Submission of Registration Information

[1512] The terminal transmits the input information to the server.

[1513] 5. Saving your registration information

[1514] The server verifies the received registration information and stores it in a database.

[1515] 6. Notification of registration completion

[1516] The server sends a registration completion message to the terminal.

[1517] The terminal displays a registration complete message to the user.

[1518] 7. Display the login screen

[1519] The device displays the login screen.

[1520] 8. Enter your authentication information

[1521] The user enters their email address and password.

[1522] 9. Submitting authentication information

[1523] The terminal transmits the entered authentication information to the server.

[1524] 10. Credential Verification

[1525] The server verifies the authentication information and sends the result to the device.

[1526] 11. Notification of Authentication Completion

[1527] If the server is successful in the authentication, it sends the user's home screen data to the terminal.

[1528] The device displays the user's home screen and completes the login.

[1529] Step 2: Upload photos and specify lessons

[1530] 1. Display the photo selection screen

[1531] The terminal displays a screen that allows the user to select a family photo.

[1532] 2. Select a photo

[1533] The user selects a family photo to upload.

[1534] 3. Sending photo data

[1535] The terminal transmits the selected photo data to the server.

[1536] 4. Saving photo data

[1537] The server stores the received photo data in a database.

[1538] 5. Display the lesson specification screen

[1539] The terminal displays a screen that prompts the user to input the lesson they want to learn.

[1540] 6. Input of lessons learned

[1541] The user inputs a lesson such as "Greetings are important."

[1542] 7. Submission of lessons learned data

[1543] The terminal transmits the input lesson data to the server.

[1544] 8. Lessons Learned Data Storage

[1545] The server stores the received lesson data in a database.

[1546] Step 3: Generate and view stories

[1547] 1. Start creating a story

[1548] The server retrieves the photo data and lesson data from the database.

[1549] 2. Story Generation

[1550] Based on the data acquired by the server, a story is generated using conversational AI.

[1551] 3. Illustration Generation

[1552] The server generates illustrations of the characters based on the uploaded photos.

[1553] 4. Picture book data integration

[1554] The server combines the generated story and illustrations to create picture book data.

[1555] 5. Sending picture book data

[1556] The server sends the completed picture book data to the terminal.

[1557] 6. Display of picture books

[1558] The terminal displays the received picture book data to the user.

[1559] 7. Picture book reading

[1560] The user reads a picture book with the child.

[1561] Step 4: Response Monitoring and Emotion Recognition (Optional)

[1562] 1. Activating the Emotional Engine

[1563] The device activates the camera and microphone and starts the emotion engine according to the user's selection.

[1564] 2. Recording the reaction

[1565] The device uses a camera and microphone to record the child's responses.

[1566] 3. Emotional Recognition

[1567] The device analyzes the recorded responses using an emotion engine to recognize emotions.

[1568] 4. Sending Emotional Data

[1569] The device transmits the recognized emotion data to the server.

[1570] The server stores the received emotion data in a database.

[1571] 5. Emotional Data Analysis

[1572] The server analyzes the stored emotional data and uses it to generate the next story.

[1573] Step 5: Gather feedback and incorporate it into next steps

[1574] 1. Display the feedback form

[1575] The device displays a feedback form after the picture book has finished reading.

[1576] 2. Enter your customization requests

[1577] The user inputs customization requests for the story and illustrations.

[1578] 3. Sending feedback data

[1579] The terminal transmits the input feedback data to the server.

[1580] 4. Feedback Data Storage

[1581] The server stores the received feedback in a database.

[1582] 5. Reflection in next story generation

[1583] The server analyzes the feedback data and reflects it in the next story generation.

[1584] Example 2

[1585] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1586] Conventional educational picture books have had difficulty generating stories that fit each child's interests and learning needs. Furthermore, they were unable to incorporate children's reactions as feedback in real time, making it difficult to change the content of a picture book once it was created. Furthermore, they were unable to provide content that responded to children's emotions.

[1587] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1588] In this invention, the server includes means for a user to input registration information and transmit it to the server, means for the server to store the registration information in a database, means for a user to input image data and educational content and transmit it to the server, means for the server to generate a story and create illustrations based on the image data and educational content, and means for the server to transmit the generated story and illustrations to the terminal, thereby enabling the generation of a personalized educational picture book in response to a user's request.

[1589] "User" refers to an individual or organization who enters registration information to use this system and receives services.

[1590] "Server" refers to a central computer system that receives information sent by users, stores it in a database, and performs any necessary processing.

[1591] "Registration information" refers to personal information such as name, email address, and password that a user enters to begin using the system.

[1592] "Database" refers to a collection of various data managed and stored by a server, such as registration information, image data, and educational content.

[1593] "Image data" refers to data containing visual information, such as photographs and pictures, uploaded by users.

[1594] "Educational content" refers to detailed information entered by the user about the lessons or stories they want their children to learn.

[1595] "Stories" refer to sentences and scenarios generated based on educational content using generative AI models.

[1596] "Illustration" refers to a visual picture or diagram generated based on uploaded image data or a story.

[1597] "Terminal" refers to a device such as a computer or smartphone that a user uses to operate the system.

[1598] "Reaction" refers to the child's reactions, including their behavior, facial expressions, and voice while reading an educational picture book.

[1599] "Customization requests" refer to improvements or specific requests that users input to be reflected in the next story generation or illustration creation.

[1600] A "generative AI model" refers to artificial intelligence technology that automatically generates stories based on educational content specified by the user.

[1601] MODE FOR CARRYING OUT THE INVENTION

[1602] The system of the present invention generates personalized educational picture books through cooperation between users, terminals, a server, and an emotion engine. The specific configuration and operation of the system are described below.

[1603] System configuration

[1604] This system consists of a terminal used by the user, a central management server, and an emotion engine that is used as needed.

[1605] Device: Computer, smartphone, tablet, etc.

[1606] Server: A high-performance computer that contains a database, generative AI models, and a network interface for communicating with devices over the internet.

[1607] Emotion Engine: A hardware device with a camera and microphone attached, optionally used by the user.

[1608] User Registration and Login

[1609] The user first downloads the app to their device and launches it. The device then displays a registration form to the user. When the user enters registration information such as their name, email address, and password and presses the registration button, the device sends this information to the server. The server stores the received registration information in a database and notifies the device that registration is complete. The user then launches the app again, enters their email address and password on the login screen, and sends them to the server for authentication.

[1610] Upload photos and assign lessons

[1611] The user selects "Upload Photos" from the menu, selects a family photo, and sends it to the server. The server saves the received photo data in a database. The user then moves to the "Specify Lesson" screen and inputs a lesson, such as "Greetings are important." The device sends this lesson information to the server, which saves it in a database.

[1612] Creating and viewing stories

[1613] The server automatically generates a story using a generative AI model (such as GPT-4) based on the received photo data and lesson information. It also generates illustrations of the characters based on the uploaded photos. The server then integrates the generated story and illustrations to create picture book data and sends it to the device. The user receives the picture book data on their device and reads it together with their child.

[1614] Reaction monitoring and emotion recognition (optional)

[1615] Depending on the user's selection, the device's camera and microphone are activated and the emotion engine is used. While the child is reading the picture book, the device uses the camera and microphone to record the child's reactions and sends the data to the server. The server analyzes the recorded data and reflects it in the next story generation.

[1616] Gathering feedback and incorporating it into next time

[1617] After reading the picture book, the device displays a feedback form to the user. The user enters their requests for customization of the story and illustrations and sends them from the device to the server. The server stores the feedback data in a database and reflects it in the next story generation.

[1618] Specific examples and examples of prompts for generative AI models

[1619] Examples:

[1620] The user specifies the lesson, "Greetings are important," and uploads a family photo. The server follows the following process: receives the photo data and lesson information ⇒ generates a story using a generative AI model (e.g., GPT-4) ⇒ generates illustrations of the characters ⇒ integrates the picture book data ⇒ sends the data to the device. The user opens the generated picture book and teaches their child the importance of greetings such as "good morning" and "thank you."

[1621] Example prompt sentence:

[1622] text

[1623] Lesson: Greetings are important

[1624] Photo: Family photo

[1625] Story content: Describe how greetings are important throughout a family's day. Include scenes such as "Good morning" in the morning, "Itadakimasu" at mealtimes, "I'm home" when you get home, and "Good night" before going to bed.

[1626] This system allows parents to easily create educational picture books tailored to their children and provide learning that is appropriate for them. Furthermore, by using an emotion engine, more personalized content can be provided based on the child's emotions.

[1627] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1628] Step 1:

[1629] User Registration

[1630] The user downloads the application to their device and launches it. The device displays a registration form that prompts the user to enter their name, email address, password, etc. The user enters the required information and presses the registration button. The entered information is sent from the device to the server. The server stores the received registration information in a database and notifies the device that registration is complete. The device displays a message to the user that registration is complete.

[1631] Input: Name, email address, password

[1632] Output: Registration complete message

[1633] Step 2:

[1634] Log in

[1635] The user launches the application and goes to the login screen. The device prompts the user to enter their email address and password. The user enters the information and presses the login button. The entered login information is sent from the device to the server, and the server performs authentication. If authentication is successful, the server sends the user's home screen data to the device, which displays it. If authentication fails, the server sends an error message to the device, which displays it.

[1636] Input: Email address, password

[1637] Output: Home screen data or error message

[1638] Step 3:

[1639] Upload a photo

[1640] The user selects "Upload Photos" from the menu. The device displays a screen prompting the user to select a family photo. When the user selects a photo and presses the upload button, the photo data is sent from the device to the server. The server stores the received photo data in a database.

[1641] Input: Photo data

[1642] Output: Confirmation of saved photo data

[1643] Step 4:

[1644] Designation of lessons learned

[1645] The user moves to the "Specify Lesson" screen and enters detailed information about the lesson or story they want to learn. The device sends this information to the server, which then stores the received lesson information in a database.

[1646] Input: Detailed information about the lesson or story (e.g., greetings are important)

[1647] Output: Confirmation of saved lesson information

[1648] Step 5:

[1649] Story Generation

[1650] The server generates a story using a generative AI model (e.g., GPT-4) based on the received photo data and lesson information, and also generates illustrations of the characters based on the uploaded photos.

[1651] Input: Photo data, lesson information

[1652] Output: Generated story and illustrations

[1653] Step 6:

[1654] Integration and transmission of picture book data

[1655] The server integrates the generated story and illustrations to create a picture book, and the completed picture book data is sent from the server to the device.

[1656] Input: Story, illustration

[1657] Output: Picture book data

[1658] Step 7:

[1659] Picture book display

[1660] The device displays the received picture book data to the user, who then reads the picture book together with their child, spending time learning.

[1661] Input: Picture book data

[1662] Output: The displayed picture book

[1663] Step 8:

[1664] Reaction monitoring and emotion recognition (optional)

[1665] If the user selects it, the device activates the camera and microphone to record the child's reactions. The emotion engine analyzes this data and transmits the recognized emotion data from the device to the server. The server receives the emotion data and stores it in a database.

[1666] Input: Child's reaction data via camera and microphone

[1667] Output: Analyzed sentiment data

[1668] Step 9:

[1669] Gathering feedback and incorporating it into next time

[1670] After reading the picture book, the device displays a feedback form to the user. The user enters their requests for customization of the story and illustrations and sends them from the device to the server. The server stores the feedback data in a database and reflects it in the next story generation.

[1671] Input: Customization request

[1672] Output: A confirmation of saved feedback data

[1673] (Application example 2)

[1674] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1675] Conventional educational picture book systems have issues with the quality and personalization of story generation, even when users enter registration information and provide photos and lesson information. It is also difficult to grasp children's reactions in real time and reflect them in the next story. This makes it impossible to provide a truly personalized educational experience that meets the needs of each user and child.

[1676] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input registration information and transmit it to the server; a means for the server to store the registration information in a database; a means for a user to input photo data and educational content and transmit it to the server; a means for the server to generate a story and create illustrations based on the photo data and educational content; a means for the server to transmit the generated story and illustration to the terminal; a means for the terminal to display the received story and illustration; a means for a user to input feedback and customization requests for the generated story and illustration and transmit it to the server; a means for the server to store the received feedback and customization requests in a database and reflect them in the next story generation; a means for the terminal to record the child's responses using a camera or microphone and transmit it to the server; a means for the server to analyze the recorded child's responses using a generative AI model and reflect them in the next story generation; and a means for an application to run on a smartphone or other available terminal. This makes it possible to provide a personalized educational experience tailored to the needs of each user and child.

[1677] A "user" is a person who uses the personalized educational picture book system.

[1678] "Server" means the central processing unit that receives and processes information from users, stores data, and transmits generated stories and illustrations.

[1679] A "terminal" is a device used by a user, such as a smartphone or tablet, that is used to display stories and illustrations received from the server.

[1680] "Photo data" refers to image files that users upload to the system and are used as characters and backgrounds for educational picture books.

[1681] "Educational content" refers to the lesson or learning material specified by the user, and is the main input information for story generation.

[1682] "Story generation" is the process in which the server automatically creates a story using conversational AI based on the input photo data and educational content.

[1683] "Illustration creation" is the process in which the server draws the characters and backgrounds in the story based on the uploaded photo data.

[1684] "Feedback" refers to the evaluation and customization requests provided by users for the generated stories and illustrations.

[1685] "Customization Request" means a specific change or addition you would like to make provided as part of your Feedback.

[1686] "Response recording" is the process by which the device collects the child's responses using a camera and microphone.

[1687] A "generative AI model" refers to an artificial intelligence technology that performs story and sentiment analysis based on input data.

[1688] "Prompts" refer to instructions or questions used when performing story or sentiment analysis on a generative AI model.

[1689] MODE FOR CARRYING OUT THE INVENTION

[1690] This invention is a system that generates educational picture books using a user interface, where users use devices such as smartphones and tablets. At the heart of the system is a server that processes and stores input from users and uses a generative AI model to create personalized stories and illustrations.

[1691] User Registration and Login

[1692] 1. The user installs and launches a dedicated application on their device.

[1693] 2. The device presents the user with a registration form and asks them to enter registration information such as their name, email address, and password.

[1694] 3. The user enters the required information and sends it to the server.

[1695] 4. The server saves the registration information in its database and completes the user registration.

[1696] Upload photos and assign lessons

[1697] 1. The user selects "Upload Photo" from the menu, selects a family photo, and sends it to the server.

[1698] 2. The server receives the photo data and stores it in a database.

[1699] 3. The user inputs the lesson or learning they want to teach and sends it to the server.

[1700] 4. The server receives the lesson information and stores it in a database.

[1701] Creating and viewing stories

[1702] 1. Based on the photo data and lesson information received by the server, a generative AI model is used to automatically generate a story and illustrations.

[1703] 2. The server combines the generated story and illustrations to create picture book data.

[1704] 3. The server sends the completed picture book data to the terminal.

[1705] 4. The device displays the received picture book data to the user.

[1706] Emotion Recognition and Feedback

[1707] 1. The device uses a camera and microphone to record the child's responses.

[1708] 2. The device sends the recorded data to the server and executes the emotion engine.

[1709] 3. The server uses an emotion engine to analyze the child's reaction data and reflect it in the next story generation.

[1710] 4. The user enters feedback and customization requests from the device and sends them to the server.

[1711] 5. The server stores the feedback and customization requests in a database and reflects them in the next story generation.

[1712] Hardware and software used

[1713] Device: The smartphone or tablet used by the user.

[1714] Server: Central processing unit that stores and processes registration information, photo data, lesson information, emotion data, and feedback.

[1715] Generative AI models: Artificial intelligence techniques used for story and illustration generation, as well as sentiment analysis.

[1716] Open source libraries: HTTP communication using the requests library, conversational AI models using the transformers library, image and video analysis using opencv.

[1717] Examples of concrete examples and prompts

[1718] As a concrete example, if a parent wants to teach their child the importance of greetings, the steps would be as follows:

[1719] 1. A parent uploads a family photo on their smartphone and enters a lesson about the importance of greetings.

[1720] 2. The server uses the generative AI model to generate a story using the prompt: "Create a story about a family going hiking with their parents. Include the lesson that greetings are important. Use the family photo below as the characters."

[1721] 3. The child's reaction when reading the picture book is recorded with a camera and sent to the server. The emotional data recorded at that time is reflected in the next story generation.

[1722] In this way, the system of the present invention can provide personalized education that meets the needs of each individual user and encourage effective learning for children.

[1723] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1724] Step 1:

[1725] The user installs and launches a dedicated application on their device. The device then provides the user with a registration form, prompting them to enter registration information such as their name, email address, and password. The user enters the required information and sends it to the server. The server saves the registration information in a database, completing the user registration. The input is the user's personal information, and the output is the registration information saved in the database.

[1726] Step 2:

[1727] The user selects "Upload Photo" from the menu, selects a family photo, and sends it to the server. The server receives the photo data and stores it in a database. The user then enters the lesson or learning content they want to teach and sends it to the server. The server receives the lesson information and stores it in the database. The input is the photo data and lesson information, and the output is the photo data and lesson information stored in the database.

[1728] Step 3:

[1729] Based on the photo data and lesson information received by the server, a generative AI model is used to automatically generate a story and illustrations. The prompt text "Write a story about a family going hiking with their parents. Include the lesson that greetings are important. Also, use the family photos below as characters." is input into the generative AI model. The server combines the generated story and illustrations to create picture book data. The input is photo data, lesson information, and the prompt text, and the output is the generated story and illustrations.

[1730] Step 4:

[1731] The server sends the completed picture book data to the terminal. The terminal displays the received picture book data to the user. The user reads the picture book together with the child and spends time learning. The input is the picture book data, and the output is the picture book displayed on the terminal.

[1732] Step 5:

[1733] The device uses a camera and microphone to record the child's reactions. The recorded data includes facial expressions and tone of voice. The device sends the recorded data to the server, which runs the emotion engine. The server uses the emotion engine to analyze the child's reaction data. The input is the child's reaction data, and the output is the analyzed emotion data.

[1734] Step 6:

[1735] The user inputs feedback and customization requests from the terminal and sends them to the server. The server stores the feedback and customization requests in a database and reflects them in the next story generation. The input is the feedback and customization requests, and the output is the feedback and customization requests stored in the database.

[1736] Step 7:

[1737] The next time a story is generated, the server uses the customization requests and emotional data stored in the database to input prompts into the generative AI model. This generates a story optimized for each user or child. The input is past customization requests and emotional data, and the output is a new story optimized for the user.

[1738] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1739] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1740] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1741] [Fourth embodiment]

[1742] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1743] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1744] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1745] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1746] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1747] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1748] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1749] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1750] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1751] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1752] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1753] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1754] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1755] The system of the present invention generates a personalized educational picture book through the mutual cooperation of a user, a terminal, and a server. A specific embodiment of the system will be described below.

[1756] User Registration and Login

[1757] 1. User Registration

[1758] The user downloads the app to their device and launches it.

[1759] The device displays a registration form to the user, which includes fields for entering name, email address, password, etc.

[1760] The user enters the required information and presses the registration button.

[1761] The terminal transmits the entered registration information to the server.

[1762] The server stores the received registration information in a database and notifies the terminal that registration has been completed.

[1763] The terminal displays a message to the user indicating that registration is complete.

[1764] 2. Log in

[1765] The user launches the app again and goes to the login screen.

[1766] The terminal prompts the user to enter an email address and password.

[1767] The user enters the information and clicks the login button.

[1768] The terminal sends the input information to the server for authentication.

[1769] If the server is successful in authenticating the user, it sends the user's home screen to the device. If authentication fails, it displays an error message.

[1770] Upload photos and assign lessons

[1771] 1. Upload a photo

[1772] The user selects "Upload Photo" from the menu.

[1773] The device prompts the user to select a family photo to upload.

[1774] The user selects a photo and presses the upload button.

[1775] The device sends the photo data to the server.

[1776] The server receives the photo data and stores it in a database.

[1777] 2. Designation of lessons learned

[1778] The user is taken to the "Specify Lessons Learned" screen.

[1779] The device prompts the user to enter more information about the lesson or story they want to learn.

[1780] The user inputs a lesson such as "Greetings are important."

[1781] The terminal transmits the input lesson information to the server.

[1782] The server receives the lesson information and stores it in a database.

[1783] Creating and viewing stories

[1784] 1. Story generation

[1785] The server starts generating a story based on the photo data and the lesson information.

[1786] The server uses conversational AI to automatically generate a story that follows the specified lesson.

[1787] The server generates illustrations of the characters based on the uploaded photos.

[1788] 2. Integration and transmission of picture book data

[1789] The server combines the generated story and illustrations to create picture book data.

[1790] The server sends the completed picture book data to the terminal.

[1791] 3. Display of picture books

[1792] The terminal displays the received picture book data to the user.

[1793] Users can read picture books together with their children and spend time learning.

[1794] Reaction monitoring (optional)

[1795] 1. Use of camera and microphone

[1796] The device will use its camera and microphone to give you the option to record your child's responses.

[1797] If the user selects this option, the device will start recording.

[1798] 2. Submitting and analyzing reaction data

[1799] The terminal transmits the recorded reaction data to the server.

[1800] The server analyzes the response data and uses it to generate the next story.

[1801] Gathering feedback and incorporating it into next time

[1802] 1. Enter your feedback

[1803] After finishing reading the picture book, the device displays a feedback form to the user.

[1804] The user inputs customization requests for the story and illustrations.

[1805] 2. Sending and storing feedback data

[1806] The terminal transmits the input feedback data to the server.

[1807] The server stores the feedback in a database and reflects it in the next story generation.

[1808] The above is a specific embodiment of the present invention. This system allows parents to easily create educational picture books tailored to their children and provide them with learning that is appropriate for their children.

[1809] The processing flow will be explained below.

[1810] Step 1: User Registration and Login

[1811] 1. Install the app

[1812] The user downloads the app to their device and launches it.

[1813] 2. Displaying the user registration form

[1814] The terminal displays a registration form to the user.

[1815] 3. Enter your registration information

[1816] The user enters their name, email address, and password.

[1817] 4. Submission of Registration Information

[1818] The terminal transmits the input information to the server.

[1819] 5. Saving your registration information

[1820] The server verifies the received registration information and stores it in a database.

[1821] 6. Notification of registration completion

[1822] The server sends a registration completion message to the terminal.

[1823] The terminal displays a registration complete message to the user.

[1824] 7. Display the login screen

[1825] The device displays the login screen.

[1826] 8. Enter your authentication information

[1827] The user enters their email address and password.

[1828] 9. Submitting authentication information

[1829] The terminal transmits the entered authentication information to the server.

[1830] 10. Credential Verification

[1831] The server verifies the authentication information and sends the result to the device.

[1832] 11. Notification of Authentication Completion

[1833] If the server is successful in the authentication, it sends the user's home screen data to the terminal.

[1834] The device displays the user's home screen and completes the login.

[1835] Step 2: Upload photos and specify lessons

[1836] 1. Display the photo selection screen

[1837] The terminal displays a screen that allows the user to select a family photo.

[1838] 2. Select a photo

[1839] The user selects a family photo to upload.

[1840] 3. Sending photo data

[1841] The terminal transmits the selected photo data to the server.

[1842] 4. Saving photo data

[1843] The server stores the received photo data in a database.

[1844] 5. Display the lesson specification screen

[1845] The terminal displays a screen that prompts the user to input the lesson they want to learn.

[1846] 6. Input of lessons learned

[1847] The user inputs a lesson such as "Greetings are important."

[1848] 7. Submission of lessons learned data

[1849] The terminal transmits the input lesson data to the server.

[1850] 8. Lessons Learned Data Storage

[1851] The server stores the received lesson data in a database.

[1852] Step 3: Generate and view stories

[1853] 1. Start creating a story

[1854] The server retrieves the photo data and lesson data from the database.

[1855] 2. Story Generation

[1856] Based on the data acquired by the server, a story is generated using conversational AI.

[1857] 3. Illustration Generation

[1858] The server generates illustrations of the characters based on the uploaded photos.

[1859] 4. Picture book data integration

[1860] The server combines the generated story and illustrations to create picture book data.

[1861] 5. Sending picture book data

[1862] The server sends the completed picture book data to the terminal.

[1863] 6. Display of picture books

[1864] The terminal displays the received picture book data to the user.

[1865] 7. Picture book reading

[1866] The user reads a picture book with the child.

[1867] Step 4: Monitoring the reaction (optional)

[1868] 1. Turn on the camera and microphone

[1869] The device will activate the camera and microphone according to the user's selection.

[1870] 2. Recording the reaction

[1871] The device records the child's responses.

[1872] 3. Submitting reaction data

[1873] The terminal transmits the recorded reaction data to the server.

[1874] 4. Saving and analyzing reaction data

[1875] The server stores the received response data and uses it for the next story generation.

[1876] Step 5: Gather feedback

[1877] 1. Display the feedback form

[1878] The device displays a feedback form after the picture book has finished reading.

[1879] 2. Enter your customization requests

[1880] The user inputs customization requests for the story and illustrations.

[1881] 3. Sending feedback data

[1882] The terminal transmits the input feedback data to the server.

[1883] 4. Feedback Data Storage

[1884] The server stores the received feedback in a database.

[1885] 5. Reflection in next story generation

[1886] The server analyzes the feedback data and reflects it in the next story generation.

[1887] Example 1

[1888] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1889] In modern education, there is a demand for personalized educational materials tailored to each child, but existing systems have difficulty in providing such personalized education. Furthermore, because the content and format of educational materials are fixed, it is difficult to provide education that reflects the interests and reactions of children. Therefore, a system is needed that can effectively provide individual educational experiences tailored to each child.

[1890] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1891] In this invention, the server includes means for a user to input registration information and transmit it to the server, means for the server to store the registration information in a database, means for a user to input photo data and educational information and transmit it to the server, means for the server to automatically generate a story and create illustrations using a generative AI model based on the photo data and educational information, means for the server to combine the generated story and illustrations to create picture book data and transmit it to the terminal, and means for the terminal to display the picture book data received. This enables users to easily create personalized educational picture books and provide learning content tailored to their children.

[1892] A "user" is an individual or group that uses this system and is the entity that creates and uses educational picture books.

[1893] A "terminal" is a device used by a user, such as a computer, smartphone, or tablet, that provides an interface with the system.

[1894] A "server" is the central part of a system and is a computer system that stores, processes, and manages data.

[1895] "Registration information" refers to information such as name, email address, and password required for a user to access the system.

[1896] The "database" is a data storage system for storing and managing user registration information, photo data, educational information, generated picture book data, and the like.

[1897] "Photo data" refers to image files uploaded by users and used for characters and backgrounds in picture books.

[1898] "Educational information" is detailed information about the lessons and stories that the user wants to learn as the content of the picture book.

[1899] A "generative AI model" is an artificial intelligence that automatically generates text and images based on data, such as ChatGPT.

[1900] A "story" is a narrative created by a generative AI model based on educational information.

[1901] "Illustrations" are images of characters and backgrounds generated based on uploaded photo data.

[1902] "Picture book data" is data in the form of an e-book created by combining the generated story and illustrations.

[1903] "Customization requests" are specific requests for changes or improvements that users input regarding stories or illustrations.

[1904] "Reaction data" refers to audio and video data recorded with a camera or microphone of a child's reaction when a picture book is read to them.

[1905] The system of the present invention generates a personalized educational picture book through the mutual cooperation of a user, a terminal, and a server. A specific embodiment of the system will be described below.

[1906] User Registration and Login Implementation

[1907] First, the user downloads the application to their device and launches it. The device displays a registration form to the user, and the user enters registration information such as their name, email address, and password. The entered registration information is sent from the device to the server, and the server stores the information in a database. The server then notifies the device that registration is complete, and the device displays a message to the user that registration is complete. When the user launches the application again, the user enters their email address and password on the login screen and sends them to the server via the device. If the server is successful in its authentication, the user's home screen is sent to the device.

[1908] Uploading photos and assigning lessons

[1909] When the user selects "Upload Photos" from the menu, the device prompts the user to select a family photo to upload. When the user selects a photo and presses the upload button, the photo data is sent from the device to the server. The server receives the photo data and stores it in a database. Next, the user moves to the "Specify Lesson" screen and enters educational information, such as "Greetings are important." The entered educational information is sent to the server via the device, and the server stores the educational information in a database.

[1910] Story Generation and Display Embodiments

[1911] The server begins generating a story based on the photo data and educational information. It uses generative AI models such as ChatGPT to automatically generate a story that follows the specified moral. It also generates illustrations of the characters based on the uploaded photos. The server then combines the generated story and illustrations to create picture book data and sends it to the device. The device then displays the received picture book data to the user, who can then read the book together with their child.

[1912] Reaction Monitoring Embodiments (Optional)

[1913] The device provides an option to record the child's reactions using the camera and microphone. When the user selects this option, the device starts recording. The recorded reaction data is sent from the device to a server, which analyzes the reaction data and uses it to generate the next story. This analysis includes emotion analysis using a voice recognition API.

[1914] How to collect feedback and incorporate it into future projects

[1915] After reading the picture book, the device displays a feedback form to the user. The user enters any customization requests for the story or illustrations and sends them from the device to the server. The server stores the customization requests in a database and reflects them in the next story generation.

[1916] Specific examples

[1917] Users download the app, enter their registration information, and send it to the server. Next, they upload a family photo and specify a lesson, such as "greetings are important." The server uses a generative AI model such as ChatGPT to generate a story and creates illustrations of the characters based on the photos. The completed picture book data is sent to the device, and users can provide learning opportunities by reading the book together with their children.

[1918] An example prompt is "Generate a children's story with the moral that saying hello is important. The story will feature a little bear, and include illustrations based on uploaded family photos."

[1919] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1920] Step 1:

[1921] A user downloads the app and enters their registration information.

[1922] The user downloads the app to their device and launches it. The device displays a form for the user to enter their name, email address, and password. The user enters the required information and presses the "Register" button. The entered information is temporarily stored in the device's memory.

[1923] Input: Name, email address, password

[1924] Output: The registration information is temporarily stored in the device's memory.

[1925] Step 2:

[1926] The terminal sends the registration information to the server, which stores it in a database.

[1927] The terminal sends the registration information entered by the user to the server. The server receives the registration information and stores it in a database using an SQL query. The server sends a notification of registration completion to the terminal. The terminal displays a message to the user notifying them that registration was successful.

[1928] Input: Registration information (name, email address, password)

[1929] Output: The registration information is saved in the database and a message indicating successful registration is displayed.

[1930] Step 3:

[1931] The user logs in again and authenticates.

[1932] The user launches the app again and goes to the login screen. The device prompts the user to enter their email address and password. The user enters the information and presses the "Login" button. The device sends the entered information to the server, which then references the database and performs authentication. If authentication is successful, the server sends the user's home screen to the device.

[1933] Input: Email address, password

[1934] Output: If authentication is successful, the home screen is displayed.

[1935] Step 4:

[1936] The user uploads a photo.

[1937] The user selects "Upload photo" from the menu. The device displays a file selection dialog, and the user selects the photo to upload. The user selects a photo and presses the "Upload" button. The device sends the photo data to the server, and the server saves the received photo data in a database.

[1938] Input: Photo data

[1939] Output: The photo data is saved in the database.

[1940] Step 5:

[1941] The user inputs the lesson information.

[1942] The user moves to the "Specify Lesson" screen and inputs the lesson information. The terminal displays an input form for educational information, and the user inputs a lesson such as "Greetings are important." The user inputs the information and presses the "Send" button. The terminal sends the input educational information to the server, which stores it in a database.

[1943] Input: Lesson learned information

[1944] Output: Lessons learned information is stored in a database.

[1945] Step 6:

[1946] The server uses a generative AI model to generate a story and create illustrations.

[1947] The server starts generating a story based on the photo data and lesson information. It inputs a prompt (e.g., "Greetings are important") into a generative AI model such as ChatGPT to generate a story. The server uses an image processing library to generate illustrations of people based on the uploaded photos.

[1948] Input: Photo data, lesson information

[1949] Output: Generated stories and illustrations

[1950] Step 7:

[1951] The server integrates the generated story and illustrations into picture book data and sends it to the terminal.

[1952] The server combines the generated story and illustrations to create picture book data. The server uses a PDF generation library to combine the text and illustrations into PDF format. The completed picture book data is sent to the device.

[1953] Input: Generated story and illustrations

[1954] Output: Picture book data is sent to the terminal.

[1955] Step 8:

[1956] The device displays the picture book, and the user reads it together with the child.

[1957] The device displays the received picture book data. Using the device's built-in PDF viewer, the user can read the book together with their child.

[1958] Input: Picture book data

[1959] Output: The picture book is displayed on the device.

[1960] Step 9:

[1961] Optionally, the device records the child's responses and transmits them to a server.

[1962] The device provides an option to record the child's reactions using the camera and microphone. When the user selects this option, the device starts recording and sends the reaction data to the server. The server receives the reaction data and performs sentiment analysis using a speech recognition API. The results are used to generate the next story.

[1963] Input: Child's reaction (audio and video data)

[1964] Output: The analysis results are reflected in the next story generation.

[1965] Step 10:

[1966] The user inputs customization requests, which the server reflects in the next story generation.

[1967] After reading the picture book, the device displays a feedback form to the user. The user enters and submits any requests for customization of the story or illustrations. The device then sends the feedback data to the server, which stores it in a database and reflects it in the next story generation.

[1968] Input: Customization request

[1969] Output: Customization requests are reflected in the next story generation.

[1970] (Application example 1)

[1971] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1972] There is a demand for efficient training in specific operating procedures and safety instructions in educational settings and factories. However, traditional training methods have had issues in meeting individual needs, making it difficult to achieve efficient training. It has also been difficult to provide real-time feedback on the reactions of employees and operators and reflect them in training content.

[1973] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1974] In this invention, the server includes: a means for a user to input registration information and transmit it to the server; a means for the server to store the registration information in a database; a means for a user to input photo data and training content and transmit it to the server; a means for the server to generate a story and create illustrations based on the photo data and training content; a means for the server to transmit the generated story and illustrations to a terminal; a means for the terminal to display the received story and illustrations; a means for inputting training content including operating procedures and safety instructions for equipment installed in a factory and transmitting it to the server; a means for the server to automatically generate a story regarding operating procedures and safety instructions based on analyzed photo data; and a means for installing the generated story on a robot and showing operating procedures in real time, thereby enabling efficient education and training according to individual needs.

[1975] A "user" is a person who uses the system to input and submit registration information, photo data, and educational content.

[1976] "Server" refers to a computer system that receives, stores, analyzes, and processes registration information, photo data, and educational content sent by users.

[1977] A "terminal" is a device that a user operates to input and display data, and has the function of receiving and displaying data from a server.

[1978] "Registration information" refers to personal information such as name, email address, and password that a user enters when accessing the system for the first time.

[1979] "Photo data" refers to image files that record training and operating procedures and are uploaded by users to the system.

[1980] "Educational content" refers to information that is the subject of education, such as procedures, instructions, lessons, etc. that users should learn.

[1981] "Stories" are a series of educational stories generated by the server based on photo data and educational content.

[1982] "Illustrations" are visual diagrams or pictures based on the story that are generated by the server.

[1983] "Factory equipment" refers to various operating devices and machines used in factories.

[1984] An "operating procedure" is a set of procedures or methods for safely and efficiently operating equipment in a factory.

[1985] "Safety instructions" are instructions and guidelines for ensuring safety when using equipment in a factory.

[1986] "Real-time feedback" refers to immediate user reactions and data obtained during education and training.

[1987] A "robot" is a programmable machine that executes and presents specified operating procedures and safety instructions to the user.

[1988] The present invention provides a personalized training system that allows users to efficiently learn specific operating procedures and safety instructions within a factory. This system functions in cooperation with users, terminals, a server, and robots.

[1989] First, a user accesses the system via a terminal and enters their registration information. The terminal then sends the registration information (name, email address, password, etc.) entered by the user to the server. The server receives this information and stores it in a database.

[1990] After registration, the user uploads photographs of the equipment in the factory to the terminal, which then sends the photographs to the server, where they are stored.

[1991] Next, the user inputs the desired operational procedures and safety instructions into the device and sends the training content to the server. The server then uses a generative AI model based on the photo data and training content to automatically generate a story based on the specific operational procedures and safety instructions. The server then creates illustrations that correspond to the story.

[1992] The generated story and illustrations are sent from the server to the device, where they are displayed, allowing the user to learn the necessary steps and instructions.

[1993] Furthermore, the system is equipped with a real-time feedback function. The device uses a camera and microphone to record the user's reactions and transmits the data to the server. The server analyzes the recorded reaction data and reflects it in the next story generation.

[1994] For example, if a user wants to learn the safety inspection procedures for conveyor belts in a factory, they can send a prompt like this to the server:

[1995] "Generate a story based on the operating procedure of the equipment shown in the photo and the specified lesson. Lesson: 'Review safety procedures'. OCR text: 'Start operation by pressing the green button and ensure the emergency stop is accessible.'"

[1996] The generated story is then installed in a robot, which then provides practical training to the user by showing them operational procedures in real time. For example, the robot can assist the user in actually operating a conveyor belt.

[1997] This system enables efficient education and training tailored to individual needs, ensuring safety within the factory and helping employees learn efficient work procedures.

[1998] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1999] Step 1:

[2000] The user operates the terminal and enters registration information into the system. Specifically, the user enters their name, email address, and password, and presses the send button. The entered data is sent from the terminal to the server. The server receives this registration information and stores it in a database. It then notifies the terminal that registration is complete.

[2001] Step 2:

[2002] The user logs in via the device. The user enters their email address and password and presses the authentication button. The entered information is sent from the device to the server. The server performs authentication, and if successful, sends the home screen to the device; if unsuccessful, an error message is displayed.

[2003] Step 3:

[2004] A user uploads a photo of equipment in a factory. The user selects "Upload photo" from the menu on the terminal, selects a photo, and presses the send button. The terminal sends the photo data to the server. The server stores the sent photo data in a database.

[2005] Step 4:

[2006] The user specifies the operating procedures and safety instructions they want to learn. The user inputs the specific lessons and training content into the terminal and presses the send button. The input data is sent from the terminal to the server. The server stores this information in a database.

[2007] Step 5:

[2008] The server generates a story based on the photo data and educational content. First, the server performs OCR analysis of the uploaded photo data and extracts text information. Next, the generative AI model receives a prompt as input, which automatically generates a story and illustrations. Example prompt: "Generate a story based on the operating procedure for the equipment shown in the photo and the specified lesson. Lesson: 'Review safety procedures'. OCR text: 'Start operation by pressing the green button and ensure the emergency stop is accessible.'"

[2009] Step 6:

[2010] The generated story and illustrations are sent to the device. The server aggregates the generated data and sends it to the device. The device receives it and displays it to the user. The user can learn the educational content through the displayed story and illustrations.

[2011] Step 7:

[2012] Get real-time feedback. The device uses a camera and microphone to record the user's reactions during training. The recorded data is sent from the device to a server, which receives this reaction data and stores it in a database.

[2013] Step 8:

[2014] The server analyzes the response data and reflects it in the next story generation. The server uses a data analysis tool to analyze the user's response data. Based on the analysis results, the prompt sentences of the generative AI model are adjusted and applied to the next story generation.

[2015] Step 9:

[2016] The generated story is installed on the robot, and operating procedures are shown in real time. The server sends the generated story to the robot, and the robot uses the received data to show operating procedures and safety instructions to the user in real time. This allows the user to receive practical training.

[2017] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2018] The system of the present invention generates a personalized educational picture book through cooperation between a user, a terminal, a server, and an emotion engine. A specific embodiment of the system will be described below.

[2019] User Registration and Login

[2020] 1. User Registration

[2021] The user downloads the app to their device and launches it.

[2022] The device displays a registration form to the user, which includes fields for entering name, email address, password, etc.

[2023] The user enters the required information and presses the registration button.

[2024] The terminal transmits the entered registration information to the server.

[2025] The server stores the received registration information in a database and notifies the terminal that registration has been completed.

[2026] The terminal displays a message to the user indicating that registration is complete.

[2027] 2. Log in

[2028] The user launches the app again and goes to the login screen.

[2029] The terminal prompts the user to enter an email address and password.

[2030] The user enters the information and clicks the login button.

[2031] The terminal sends the input information to the server for authentication.

[2032] If the server is successful in authenticating the user, it sends the user's home screen to the device. If authentication fails, it displays an error message.

[2033] Upload photos and assign lessons

[2034] 1. Upload a photo

[2035] The user selects "Upload Photo" from the menu.

[2036] The device prompts the user to select a family photo to upload.

[2037] The user selects a photo and presses the upload button.

[2038] The device sends the photo data to the server.

[2039] The server receives the photo data and stores it in a database.

[2040] 2. Designation of lessons learned

[2041] The user is taken to the "Specify Lessons Learned" screen.

[2042] The device prompts the user to enter more information about the lesson or story they want to learn.

[2043] The user inputs a lesson such as "Greetings are important."

[2044] The terminal transmits the input lesson information to the server.

[2045] The server receives the lesson information and stores it in a database.

[2046] Creating and viewing stories

[2047] 1. Story Generation

[2048] The server starts generating a story based on the photo data and the lesson information.

[2049] The server uses conversational AI to automatically generate a story that follows the specified lesson.

[2050] The server generates illustrations of the characters based on the uploaded photos.

[2051] 2. Integration and transmission of picture book data

[2052] The server combines the generated story and illustrations to create picture book data.

[2053] The server sends the completed picture book data to the terminal.

[2054] 3. Display of picture books

[2055] The terminal displays the received picture book data to the user.

[2056] Users can read picture books together with their children and spend time learning.

[2057] Reaction monitoring and emotion recognition (optional)

[2058] 1. Activating the Emotional Engine

[2059] The device activates the camera and microphone and starts the emotion engine according to the user's selection.

[2060] 2. Reaction recording and emotion recognition

[2061] The device uses a camera and microphone to record the child's reactions and uses an emotion engine to recognize emotions from those reactions.

[2062] The recognized emotional information is processed in real time.

[2063] 3. Sending Emotional Data

[2064] The device transmits the recognized emotion data to the server.

[2065] The server receives the emotion data and stores it in a database.

[2066] 4. Emotional Data Analysis

[2067] The server analyzes the stored emotional data and uses it to generate the next story.

[2068] Gathering feedback and incorporating it into next time

[2069] 1. Enter your feedback

[2070] After finishing reading the picture book, the device displays a feedback form to the user.

[2071] The user inputs customization requests for the story and illustrations.

[2072] 2. Sending and storing feedback data

[2073] The terminal transmits the input feedback data to the server.

[2074] The server stores the feedback in a database and reflects it in the next story generation.

[2075] The above is a specific embodiment of the present invention. This system allows parents to easily create educational picture books tailored to their children and provide learning that is appropriate for their children. Furthermore, by using an emotion engine, more personalized content can be provided based on the child's emotions.

[2076] The processing flow will be explained below.

[2077] Step 1: User Registration and Login

[2078] 1. Install the app

[2079] The user downloads the app to their device and launches it.

[2080] 2. Displaying the user registration form

[2081] The terminal displays a registration form to the user.

[2082] 3. Enter your registration information

[2083] The user enters their name, email address, and password.

[2084] 4. Submission of Registration Information

[2085] The terminal transmits the input information to the server.

[2086] 5. Saving your registration information

[2087] The server verifies the received registration information and stores it in a database.

[2088] 6. Notification of registration completion

[2089] The server sends a registration completion message to the terminal.

[2090] The terminal displays a registration complete message to the user.

[2091] 7. Display the login screen

[2092] The device displays the login screen.

[2093] 8. Enter your authentication information

[2094] The user enters their email address and password.

[2095] 9. Submitting authentication information

[2096] The terminal transmits the entered authentication information to the server.

[2097] 10. Credential Verification

[2098] The server verifies the authentication information and sends the result to the device.

[2099] 11. Notification of Authentication Completion

[2100] If the server is successful in the authentication, it sends the user's home screen data to the terminal.

[2101] The device displays the user's home screen and completes the login.

[2102] Step 2: Upload photos and specify lessons

[2103] 1. Display the photo selection screen

[2104] The terminal displays a screen that allows the user to select a family photo.

[2105] 2. Select a photo

[2106] The user selects a family photo to upload.

[2107] 3. Sending photo data

[2108] The terminal transmits the selected photo data to the server.

[2109] 4. Saving photo data

[2110] The server stores the received photo data in a database.

[2111] 5. Display the lesson specification screen

[2112] The terminal displays a screen that prompts the user to input the lesson they want to learn.

[2113] 6. Input of lessons learned

[2114] The user inputs a lesson such as "Greetings are important."

[2115] 7. Submission of lessons learned data

[2116] The terminal transmits the input lesson data to the server.

[2117] 8. Lessons Learned Data Storage

[2118] The server stores the received lesson data in a database.

[2119] Step 3: Generate and view stories

[2120] 1. Start creating a story

[2121] The server retrieves the photo data and lesson data from the database.

[2122] 2. Story Generation

[2123] Based on the data acquired by the server, a story is generated using conversational AI.

[2124] 3. Illustration Generation

[2125] The server generates illustrations of the characters based on the uploaded photos.

[2126] 4. Picture book data integration

[2127] The server combines the generated story and illustrations to create picture book data.

[2128] 5. Sending picture book data

[2129] The server sends the completed picture book data to the terminal.

[2130] 6. Display of picture books

[2131] The terminal displays the received picture book data to the user.

[2132] 7. Picture book reading

[2133] The user reads a picture book with the child.

[2134] Step 4: Response Monitoring and Emotion Recognition (Optional)

[2135] 1. Activating the Emotional Engine

[2136] The device activates the camera and microphone and starts the emotion engine according to the user's selection.

[2137] 2. Recording the reaction

[2138] The device uses a camera and microphone to record the child's responses.

[2139] 3. Emotional Recognition

[2140] The device analyzes the recorded responses using an emotion engine to recognize emotions.

[2141] 4. Sending Emotional Data

[2142] The device transmits the recognized emotion data to the server.

[2143] The server stores the received emotion data in a database.

[2144] 5. Emotional Data Analysis

[2145] The server analyzes the stored emotional data and uses it to generate the next story.

[2146] Step 5: Gather feedback and incorporate it into next steps

[2147] 1. Display the feedback form

[2148] The device displays a feedback form after the picture book has finished reading.

[2149] 2. Enter your customization requests

[2150] The user inputs customization requests for the story and illustrations.

[2151] 3. Sending feedback data

[2152] The terminal transmits the input feedback data to the server.

[2153] 4. Feedback Data Storage

[2154] The server stores the received feedback in a database.

[2155] 5. Reflection in next story generation

[2156] The server analyzes the feedback data and reflects it in the next story generation.

[2157] Example 2

[2158] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2159] Conventional educational picture books have had difficulty generating stories that fit each child's interests and learning needs. Furthermore, they were unable to incorporate children's reactions as feedback in real time, making it difficult to change the content of a picture book once it was created. Furthermore, they were unable to provide content that responded to children's emotions.

[2160] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2161] In this invention, the server includes means for a user to input registration information and transmit it to the server, means for the server to store the registration information in a database, means for a user to input image data and educational content and transmit it to the server, means for the server to generate a story and create illustrations based on the image data and educational content, and means for the server to transmit the generated story and illustrations to the terminal, thereby enabling the generation of a personalized educational picture book in response to a user's request.

[2162] "User" refers to an individual or organization who enters registration information to use this system and receives services.

[2163] "Server" refers to a central computer system that receives information sent by users, stores it in a database, and performs any necessary processing.

[2164] "Registration information" refers to personal information such as name, email address, and password that a user enters to begin using the system.

[2165] "Database" refers to a collection of various data managed and stored by a server, such as registration information, image data, and educational content.

[2166] "Image data" refers to data containing visual information, such as photographs and pictures, uploaded by users.

[2167] "Educational content" refers to detailed information entered by the user about the lessons or stories they want their children to learn.

[2168] "Stories" refer to sentences and scenarios generated based on educational content using generative AI models.

[2169] "Illustration" refers to a visual picture or diagram generated based on uploaded image data or a story.

[2170] "Terminal" refers to a device such as a computer or smartphone that a user uses to operate the system.

[2171] "Reaction" refers to the child's reactions, including their behavior, facial expressions, and voice while reading an educational picture book.

[2172] "Customization requests" refer to improvements or specific requests that users input to be reflected in the next story generation or illustration creation.

[2173] A "generative AI model" refers to artificial intelligence technology that automatically generates stories based on educational content specified by the user.

[2174] MODE FOR CARRYING OUT THE INVENTION

[2175] The system of the present invention generates personalized educational picture books through cooperation between users, terminals, a server, and an emotion engine. The specific configuration and operation of the system are described below.

[2176] System configuration

[2177] This system consists of a terminal used by the user, a central management server, and an emotion engine that is used as needed.

[2178] Device: Computer, smartphone, tablet, etc.

[2179] Server: A high-performance computer that contains a database, generative AI models, and a network interface for communicating with devices over the internet.

[2180] Emotion Engine: A hardware device with a camera and microphone attached, optionally used by the user.

[2181] User Registration and Login

[2182] The user first downloads the app to their device and launches it. The device then displays a registration form to the user. When the user enters registration information such as their name, email address, and password and presses the registration button, the device sends this information to the server. The server stores the received registration information in a database and notifies the device that registration is complete. The user then launches the app again, enters their email address and password on the login screen, and sends them to the server for authentication.

[2183] Upload photos and assign lessons

[2184] The user selects "Upload Photos" from the menu, selects a family photo, and sends it to the server. The server saves the received photo data in a database. The user then moves to the "Specify Lesson" screen and inputs a lesson, such as "Greetings are important." The device sends this lesson information to the server, which saves it in a database.

[2185] Creating and viewing stories

[2186] The server automatically generates a story using a generative AI model (such as GPT-4) based on the received photo data and lesson information. It also generates illustrations of the characters based on the uploaded photos. The server then integrates the generated story and illustrations to create picture book data and sends it to the device. The user receives the picture book data on their device and reads it together with their child.

[2187] Reaction monitoring and emotion recognition (optional)

[2188] Depending on the user's selection, the device's camera and microphone are activated and the emotion engine is used. While the child is reading the picture book, the device uses the camera and microphone to record the child's reactions and sends the data to the server. The server analyzes the recorded data and reflects it in the next story generation.

[2189] Gathering feedback and incorporating it into next time

[2190] After reading the picture book, the device displays a feedback form to the user. The user enters their requests for customization of the story and illustrations and sends them from the device to the server. The server stores the feedback data in a database and reflects it in the next story generation.

[2191] Specific examples and examples of prompts for generative AI models

[2192] Examples:

[2193] The user specifies the lesson, "Greetings are important," and uploads a family photo. The server follows the following process: receives the photo data and lesson information ⇒ generates a story using a generative AI model (e.g., GPT-4) ⇒ generates illustrations of the characters ⇒ integrates the picture book data ⇒ sends the data to the device. The user opens the generated picture book and teaches their child the importance of greetings such as "good morning" and "thank you."

[2194] Example prompt sentence:

[2195] text

[2196] Lesson: Greetings are important

[2197] Photo: Family photo

[2198] Story content: Describe how greetings are important throughout a family's day. Include scenes such as "Good morning" in the morning, "Itadakimasu" at mealtimes, "I'm home" when you get home, and "Good night" before going to bed.

[2199] This system allows parents to easily create educational picture books tailored to their children and provide learning that is appropriate for them. Furthermore, by using an emotion engine, more personalized content can be provided based on the child's emotions.

[2200] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2201] Step 1:

[2202] User Registration

[2203] The user downloads the application to their device and launches it. The device displays a registration form that prompts the user to enter their name, email address, password, etc. The user enters the required information and presses the registration button. The entered information is sent from the device to the server. The server stores the received registration information in a database and notifies the device that registration is complete. The device displays a message to the user that registration is complete.

[2204] Input: Name, email address, password

[2205] Output: Registration complete message

[2206] Step 2:

[2207] Log in

[2208] The user launches the application and goes to the login screen. The device prompts the user to enter their email address and password. The user enters the information and presses the login button. The entered login information is sent from the device to the server, and the server performs authentication. If authentication is successful, the server sends the user's home screen data to the device, which displays it. If authentication fails, the server sends an error message to the device, which displays it.

[2209] Input: Email address, password

[2210] Output: Home screen data or error message

[2211] Step 3:

[2212] Upload a photo

[2213] The user selects "Upload Photos" from the menu. The device displays a screen prompting the user to select a family photo. When the user selects a photo and presses the upload button, the photo data is sent from the device to the server. The server stores the received photo data in a database.

[2214] Input: Photo data

[2215] Output: Confirmation of saved photo data

[2216] Step 4:

[2217] Designation of lessons learned

[2218] The user moves to the "Specify Lesson" screen and enters detailed information about the lesson or story they want to learn. The device sends this information to the server, which then stores the received lesson information in a database.

[2219] Input: Detailed information about the lesson or story (e.g., greetings are important)

[2220] Output: Confirmation of saved lesson information

[2221] Step 5:

[2222] Story Generation

[2223] The server generates a story using a generative AI model (e.g., GPT-4) based on the received photo data and lesson information, and also generates illustrations of the characters based on the uploaded photos.

[2224] Input: Photo data, lesson information

[2225] Output: Generated story and illustrations

[2226] Step 6:

[2227] Integration and transmission of picture book data

[2228] The server integrates the generated story and illustrations to create a picture book, and the completed picture book data is sent from the server to the device.

[2229] Input: Story, illustration

[2230] Output: Picture book data

[2231] Step 7:

[2232] Picture book display

[2233] The device displays the received picture book data to the user, who then reads the picture book together with their child, spending time learning.

[2234] Input: Picture book data

[2235] Output: The displayed picture book

[2236] Step 8:

[2237] Reaction monitoring and emotion recognition (optional)

[2238] If the user selects it, the device activates the camera and microphone to record the child's reactions. The emotion engine analyzes this data and transmits the recognized emotion data from the device to the server. The server receives the emotion data and stores it in a database.

[2239] Input: Child's reaction data via camera and microphone

[2240] Output: Analyzed sentiment data

[2241] Step 9:

[2242] Gathering feedback and incorporating it into next time

[2243] After reading the picture book, the device displays a feedback form to the user. The user enters their requests for customization of the story and illustrations and sends them from the device to the server. The server stores the feedback data in a database and reflects it in the next story generation.

[2244] Input: Customization request

[2245] Output: A confirmation of saved feedback data

[2246] (Application example 2)

[2247] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2248] Conventional educational picture book systems have issues with the quality and personalization of story generation, even when users enter registration information and provide photos and lesson information. It is also difficult to grasp children's reactions in real time and reflect them in the next story. This makes it impossible to provide a truly personalized educational experience that meets the needs of each user and child.

[2249] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input registration information and transmit it to the server; a means for the server to store the registration information in a database; a means for a user to input photo data and educational content and transmit it to the server; a means for the server to generate a story and create illustrations based on the photo data and educational content; a means for the server to transmit the generated story and illustration to the terminal; a means for the terminal to display the received story and illustration; a means for a user to input feedback and customization requests for the generated story and illustration and transmit it to the server; a means for the server to store the received feedback and customization requests in a database and reflect them in the next story generation; a means for the terminal to record the child's responses using a camera or microphone and transmit it to the server; a means for the server to analyze the recorded child's responses using a generative AI model and reflect them in the next story generation; and a means for an application to run on a smartphone or other available terminal. This makes it possible to provide a personalized educational experience tailored to the needs of each user and child.

[2250] A "user" is a person who uses the personalized educational picture book system.

[2251] "Server" means the central processing unit that receives and processes information from users, stores data, and transmits generated stories and illustrations.

[2252] A "terminal" is a device used by a user, such as a smartphone or tablet, that is used to display stories and illustrations received from the server.

[2253] "Photo data" refers to image files that users upload to the system and are used as characters and backgrounds for educational picture books.

[2254] "Educational content" refers to the lesson or learning material specified by the user, and is the main input information for story generation.

[2255] "Story generation" is the process in which the server automatically creates a story using conversational AI based on the input photo data and educational content.

[2256] "Illustration creation" is the process in which the server draws the characters and backgrounds in the story based on the uploaded photo data.

[2257] "Feedback" refers to the evaluation and customization requests provided by users for the generated stories and illustrations.

[2258] "Customization Request" means a specific change or addition you would like to make provided as part of your Feedback.

[2259] "Response recording" is the process by which the device collects the child's responses using a camera and microphone.

[2260] A "generative AI model" refers to an artificial intelligence technology that performs story and sentiment analysis based on input data.

[2261] "Prompts" refer to instructions or questions used when performing story or sentiment analysis on a generative AI model.

[2262] MODE FOR CARRYING OUT THE INVENTION

[2263] This invention is a system that generates educational picture books using a user interface, where users use devices such as smartphones and tablets. At the heart of the system is a server that processes and stores input from users and uses a generative AI model to create personalized stories and illustrations.

[2264] User Registration and Login

[2265] 1. The user installs and launches a dedicated application on their device.

[2266] 2. The device presents the user with a registration form and asks them to enter registration information such as their name, email address, and password.

[2267] 3. The user enters the required information and sends it to the server.

[2268] 4. The server saves the registration information in its database and completes the user registration.

[2269] Upload photos and assign lessons

[2270] 1. The user selects "Upload Photo" from the menu, selects a family photo, and sends it to the server.

[2271] 2. The server receives the photo data and stores it in a database.

[2272] 3. The user inputs the lesson or learning they want to teach and sends it to the server.

[2273] 4. The server receives the lesson information and stores it in a database.

[2274] Creating and viewing stories

[2275] 1. Based on the photo data and lesson information received by the server, a generative AI model is used to automatically generate a story and illustrations.

[2276] 2. The server combines the generated story and illustrations to create picture book data.

[2277] 3. The server sends the completed picture book data to the terminal.

[2278] 4. The device displays the received picture book data to the user.

[2279] Emotion Recognition and Feedback

[2280] 1. The device uses a camera and microphone to record the child's responses.

[2281] 2. The device sends the recorded data to the server and executes the emotion engine.

[2282] 3. The server uses an emotion engine to analyze the child's reaction data and reflect it in the next story generation.

[2283] 4. The user enters feedback and customization requests from the device and sends them to the server.

[2284] 5. The server stores the feedback and customization requests in a database and reflects them in the next story generation.

[2285] Hardware and software used

[2286] Device: The smartphone or tablet used by the user.

[2287] Server: Central processing unit that stores and processes registration information, photo data, lesson information, emotion data, and feedback.

[2288] Generative AI models: Artificial intelligence techniques used for story and illustration generation, as well as sentiment analysis.

[2289] Open source libraries: HTTP communication using the requests library, conversational AI models using the transformers library, image and video analysis using opencv.

[2290] Examples of concrete examples and prompts

[2291] As a concrete example, if a parent wants to teach their child the importance of greetings, the steps would be as follows:

[2292] 1. A parent uploads a family photo on their smartphone and enters a lesson about the importance of greetings.

[2293] 2. The server uses the generative AI model to generate a story using the prompt: "Create a story about a family going hiking with their parents. Include the lesson that greetings are important. Use the family photo below as the characters."

[2294] 3. The child's reaction when reading the picture book is recorded with a camera and sent to the server. The emotional data recorded at that time is reflected in the next story generation.

[2295] In this way, the system of the present invention can provide personalized education that meets the needs of each individual user and encourage effective learning for children.

[2296] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2297] Step 1:

[2298] The user installs and launches a dedicated application on their device. The device then provides the user with a registration form, prompting them to enter registration information such as their name, email address, and password. The user enters the required information and sends it to the server. The server saves the registration information in a database, completing the user registration. The input is the user's personal information, and the output is the registration information saved in the database.

[2299] Step 2:

[2300] The user selects "Upload Photo" from the menu, selects a family photo, and sends it to the server. The server receives the photo data and stores it in a database. The user then enters the lesson or learning content they want to teach and sends it to the server. The server receives the lesson information and stores it in the database. The input is the photo data and lesson information, and the output is the photo data and lesson information stored in the database.

[2301] Step 3:

[2302] Based on the photo data and lesson information received by the server, a generative AI model is used to automatically generate a story and illustrations. The prompt text "Write a story about a family going hiking with their parents. Include the lesson that greetings are important. Also, use the family photos below as characters." is input into the generative AI model. The server combines the generated story and illustrations to create picture book data. The input is photo data, lesson information, and the prompt text, and the output is the generated story and illustrations.

[2303] Step 4:

[2304] The server sends the completed picture book data to the terminal. The terminal displays the received picture book data to the user. The user reads the picture book together with the child and spends time learning. The input is the picture book data, and the output is the picture book displayed on the terminal.

[2305] Step 5:

[2306] The device uses a camera and microphone to record the child's reactions. The recorded data includes facial expressions and tone of voice. The device sends the recorded data to the server, which runs the emotion engine. The server uses the emotion engine to analyze the child's reaction data. The input is the child's reaction data, and the output is the analyzed emotion data.

[2307] Step 6:

[2308] The user inputs feedback and customization requests from the terminal and sends them to the server. The server stores the feedback and customization requests in a database and reflects them in the next story generation. The input is the feedback and customization requests, and the output is the feedback and customization requests stored in the database.

[2309] Step 7:

[2310] The next time a story is generated, the server uses the customization requests and emotional data stored in the database to input prompts into the generative AI model. This generates a story optimized for each user or child. The input is past customization requests and emotional data, and the output is a new story optimized for the user.

[2311] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2312] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2313] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2314] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2315] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2316] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2317] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2318] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a ...

Claims

1. A means for a user to input registration information and transmit it to a server; a means by which the server stores the registration information in a database; A means for a user to input photo data and educational content and transmit them to a server; A server generates a story based on the photo data and educational content, and creates illustrations; A means for the server to transmit the generated story and illustrations to the terminal; a means for displaying the received story and illustrations on the device; A system including:

2. The device uses a camera and microphone to record the child's reactions and transmit them to a server. The server analyzes the recorded responses of the children and reflects them in the next story generation. The system of claim 1 , comprising:

3. A means for users to input customization requests for stories and illustrations and send them to a server; The server stores the customization requests in a database and reflects them in the next story generation. The system of claim 1 , comprising:

Images