System

Abstract

A system is provided.SOLUTION: A system comprising means for generating the generated teaching material on a server based on the understanding level of each student, accepting questions until the student is satisfied, providing visual and verbal support, means for analyzing the progress and the understanding level of the generated teaching material to provide feedback, and means for recording the learning progress of the student and proposing the next appropriate learning content.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In early elementary school education, there are significant disparities in learning progress and comprehension between students due to differences in age, which can affect students' self-esteem, their final educational attainment, and even their future careers. The current school education system lacks individualized attention to each student, and does not provide an environment where students who feel they do not understand can ask repeated questions. This problem needs to be resolved to provide an educational environment where all students can learn at their own pace and experience personal growth. [Means for solving the problem]

[0005] This invention provides a multimodal self-learning educational tool that utilizes generative AI. This generates learning materials at an appropriate level based on each student's level of understanding, allowing them to repeatedly ask questions until they are satisfied. It also includes a means for deepening students' understanding through visual and linguistic support. It also includes a means for analyzing progress and comprehension of the generated learning materials and providing feedback, as well as a means for recording students' learning progress and suggesting appropriate next learning content. This provides an environment that accommodates each student's learning pace and allows them to learn comfortably while enhancing their self-esteem.

[0006] "Generated teaching materials" is a general term for educational content created by generative AI according to each student's level of understanding and vocabulary.

[0007] A "server" is a central computer system that handles various processes such as managing user profile data, generating educational materials, answering questions, and providing feedback.

[0008] The term "students" refers to children in the lower grades of elementary school who study using the system of the present invention.

[0009] "Level of understanding" is an index of the extent to which students have grasped and understood the content of the teaching materials presented.

[0010] A "question" is an action in which a student sends a question that arises while studying the material to the server in text or voice to seek a solution.

[0011] "Visual support" is a supplementary means of promoting student understanding using visual media such as images, videos, and diagrams.

[0012] "Linguistic support" is a supplementary means of using text and audio to help students understand.

[0013] "Feedback" refers to advice or suggestions provided based on a student's learning progress or test results.

[0014] "Progress" is an indicator that shows the progress and level of achievement of students as they progress through their studies.

[0015] "Appropriate learning content" refers to learning materials that are designed to best suit a student's current level of understanding and progress, and to enable them to learn effectively. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] Summary of the Invention

[0038] This invention is a multimodal self-learning educational tool that utilizes generative AI for early elementary school students. It generates individually tailored learning materials based on the student's level of comprehension and vocabulary, allowing students to ask questions until they are satisfied. Learning content is presented using text, images, and in some cases audio and video, deepening students' understanding through visual and linguistic support. It also records learning progress and provides feedback, enhancing self-esteem.

[0039] Program processing

[0040] Log in and get your profile

[0041] When a user logs in on a device, the server receives the authentication information and retrieves and sends the user's profile information from the database. The profile includes information such as learning history, grades, and vocabulary.

[0042] Initial setup and preparation

[0043] The device analyzes the profile information received from the server and sets the user's learning level and personalized elements (such as name and favorite character). Once the settings are complete, a screen is displayed notifying the user that they are ready to start learning.

[0044] Request and generate learning content

[0045] The user selects a subject or topic to study. For example, if they select "Edo Period History," the request is sent to the server. The server generates learning content at the appropriate level based on the user's profile. The generative AI model creates learning materials by combining text, images, and in some cases audio and video, and sends them to the device.

[0046] View and progress through learning content

[0047] The device analyzes the learning materials received from the server and displays them in a format that combines text and images. The user can then read through the materials. If they do not understand the material properly or have questions, they can enter them in a question box on the learning page.

[0048] Receiving questions and generating answers

[0049] The device sends the user's question to the server, which analyzes the question and generates an appropriate answer. The generative AI model generates an image related to the text and sends it to the device as an answer, which the device displays to the user.

[0050] Checking understanding and providing feedback

[0051] When a user wants to check their level of understanding, they make a request for a comprehension check. The server generates a quiz or mini-test to check the user's level of understanding and sends it to the device. The user answers it, and the answer results are sent back to the server. The server analyzes the answers, generates feedback, and sends it back to the device. The device displays the feedback to the user.

[0052] Learning progress management and recording

[0053] The server records the user's learning progress in real time and saves it in a database. It also prepares to suggest new learning content and review questions suitable for the next study. The user can check the progress and select the next learning topic to continue learning.

[0054] Specific examples

[0055] Example 1: A student is studying a history lesson.

[0056] 1. The user selects "History Lesson" on the device and sends a request to the server.

[0057] 2. The server references the user's profile and generates the appropriate level of historical content.

[0058] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[0059] 4. The user reads through the learning material and enters the question, "What is Ooku?"

[0060] 5. The server generates a description of "Ooku" in text and images and returns it to the terminal.

[0061] 6. The user reads the answer and deepens their understanding.

[0062] Example 2: A student solves a math problem

[0063] 1. The user selects "Math Practice Problems" on the device and sends a request to the server.

[0064] 2. The server generates problems appropriate to the user's level of understanding, such as simple addition and subtraction problems.

[0065] 3. The question is displayed on the terminal and the user enters the answer.

[0066] 4. If the user is unsure of the answer, ask, "Is this correct?"

[0067] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[0068] 6. Users receive feedback and improve their understanding.

[0069] In this way, the self-learning educational tool of the present invention accommodates the learning pace of each student and provides an environment in which students can learn at their own pace while enhancing their self-esteem.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The user opens the login screen on the device and enters authentication information (user ID and password).

[0073] Step 2:

[0074] The terminal generates a request to send the input authentication information to the server, and sends it to the authentication endpoint of the server.

[0075] Step 3:

[0076] The server checks the received authentication information against the database to confirm that the user is registered, and returns the check result to the terminal.

[0077] Step 4:

[0078] The terminal receives the authentication result from the server, and if the authentication is successful, sends a request to the server to obtain the user's profile information.

[0079] Step 5:

[0080] The server retrieves the user's profile information (learning history, grades, vocabulary, etc.) from the database and returns it to the terminal in JSON format.

[0081] Step 6:

[0082] The device analyzes the received profile information and sets the user's learning level and personalized elements (name, favorite character, etc.). After the settings are complete, the learning start screen is displayed to the user.

[0083] Step 7:

[0084] Users select the subject or topic they wish to study (e.g., the Edo period in history) on their device and press the "Start Studying" button.

[0085] Step 8:

[0086] The terminal sends the user's selection as a request to the server.

[0087] Step 9:

[0088] The server references the user's profile information and the request to generate learning content at the appropriate level, using a generative AI model to generate learning materials that combine text, images, and in some cases video and audio.

[0089] Step 10:

[0090] The server sends the generated teaching materials to the terminal in JSON format.

[0091] Step 11:

[0092] The device analyzes the received educational material and displays it to the user in a format that combines text and images.

[0093] Step 12:

[0094] Users can read through the displayed learning content to deepen their understanding, and if they have any questions, they can enter them in the question box provided on the learning page.

[0095] Step 13:

[0096] The terminal generates a request to send the user's question to the server and sends it to the server's question endpoint.

[0097] Step 14:

[0098] The server analyzes the received question and collects data to generate an appropriate answer. It uses a generative AI model to generate an answer to the question and provides relevant text and images.

[0099] Step 15:

[0100] The server sends the generated response in JSON format to the terminal.

[0101] Step 16:

[0102] The terminal displays the response received from the server to the user.

[0103] Step 17:

[0104] Users can read the answers to deepen their understanding and ask again if they have further questions.

[0105] Step 18:

[0106] If the user wants to check their level of understanding, they press the "Check understanding" button.

[0107] Step 19:

[0108] The device sends a request for a comprehension check to the server.

[0109] Step 20:

[0110] The server generates quizzes and mini-tests to check the user's level of understanding, using a generative AI model to select questions at the appropriate level and sending them to the device in JSON format.

[0111] Step 21:

[0112] The terminal provides an interface for displaying quizzes and mini-tests to the user and receiving answers.

[0113] Step 22:

[0114] Users answer quizzes and mini-tests and send their answers to their terminals.

[0115] Step 23:

[0116] The terminal generates a request to send the user's answer to the server and sends it to the server's analysis endpoint.

[0117] Step 24:

[0118] The server analyzes the received answers, determines whether they are correct, and generates feedback based on the analysis results, sending it to the device in JSON format.

[0119] Step 25:

[0120] The terminal displays the feedback received from the server to the user.

[0121] Step 26:

[0122] The server records the user's learning progress in real time and stores the progress in a database.

[0123] Step 27:

[0124] The server prepares data to suggest the user's next study topic and review questions.

[0125] Step 28:

[0126] Users can check their learning progress and select the next learning topic to continue their learning.

[0127] Example 1

[0128] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0129] Conventional learning systems make it difficult to provide individualized instruction tailored to each student's level of understanding and progress, and students often cannot receive immediate answers when they have questions. It is also difficult to grasp learning progress and comprehension and provide appropriate feedback based on that information. Furthermore, personalization of learning content is insufficient, making it impossible to provide optimal learning materials to each student. This can result in a decline in students' self-esteem and motivation to learn, so these issues must be resolved.

[0130] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0131] In this invention, the server includes: means for a user to log in using a terminal, the server receiving authentication information, and retrieving and transmitting the user's profile information from a database; means for the terminal analyzing the received profile information and setting the learner's learning level and personalized elements; means for the learner to select a subject or topic to study and sending a request to the server; means for the server to generate learning content at an appropriate level based on the user's profile information and send it to the terminal; means for the terminal to analyze the learning materials received from the server and display them in a format combining text and images; means for the learner to input a question into a question box and the terminal sending it to the server; means for the server to analyze the question, generate an appropriate answer, and send it to the terminal; and means for the server to record the learner's learning progress in real time and suggest new learning content and review questions appropriate for the next lesson. This makes it possible to provide an optimal learning experience for each student, improving learning efficiency and self-esteem.

[0132] A "terminal" is an electronic device that a user operates, and includes computers, tablets, smartphones, and the like.

[0133] A "server" is a computer system that communicates with terminals over a network and processes, stores, and manages data.

[0134] "Student" refers to a student or user who uses the system to study.

[0135] "Profile information" refers to individual data about a learner, including learning history, grades, vocabulary information, and personalized elements.

[0136] A "database" is a system for centrally managing and storing information, and is accessed by a server to search for and store the necessary data.

[0137] "Authentication information" is information used to identify a particular user and grant that user system access rights, and typically includes a username and password.

[0138] "Learning content" refers to the learning materials and workbooks that learners use for learning, and consists of text, images, audio, video, etc.

[0139] A "generative AI model" is an algorithm that uses artificial intelligence technology to generate learning content and answers.

[0140] "Feedback" refers to the process of providing evaluation and advice on a learner's answers or actions.

[0141] "Progress recording" refers to the quantitative or qualitative recording of the results and progress of a learner's learning activities.

[0142] A "question box" refers to an interface where learners can enter their doubts or queries.

[0143] "Level of understanding" is a measure of how well a learner understands the given teaching materials and tasks.

[0144] "Personalized elements" refer to elements of the learning experience that are customized based on the learner's individual preferences and characteristics.

[0145] This invention is a multimodal self-learning educational system that utilizes generative AI models for early elementary school students. This system provides optimal learning content based on the learner's level of understanding and progress, accepts questions until the learner is satisfied, and provides visual and linguistic support. It also aims to enhance learners' self-esteem by recording their learning progress and providing appropriate feedback.

[0146] System Configuration

[0147] This system is realized mainly using the following hardware and software.

[0148] Device: Computer, tablet, smartphone, etc.

[0149] Server: A computer system that processes and stores data over a network.

[0150] Database: A system that manages information accessed by a server (e.g., MySQL (registered trademark), PostgreSQL).

[0151] Generative AI model: An artificial intelligence algorithm that generates learning content and answers (e.g., GPT-4 (registered trademark)).

[0152] Program implementation technologies: Python, Django, Redis, HTML / CSS, JavaScript (registered trademark), etc.

[0153] Program processing

[0154] The system operates as follows:

[0155] 1. Log in and get your profile

[0156] The user enters their username and password on the terminal and presses the login button. The server receives the authentication information, retrieves the user's profile information from the database, and sends it to the terminal.

[0157] 2. Initial Setup and Preparation

[0158] The device analyzes the received profile information and initializes the learner's learning level and personalized elements. Once the setup is complete, a screen will appear informing the learner that they are ready to begin learning.

[0159] 3. Request and generate learning content

[0160] The user selects the subject or topic to study and sends the request to the server, which uses a generative AI model based on the user's profile information to generate learning content at the appropriate level and send it to the device.

[0161] 4. View and progress through learning content

[0162] The device analyzes the learning materials received from the server and displays them in a format that combines text and images. The user can read the materials and, if they do not understand them or have questions, they can enter their questions in the question box.

[0163] 5. Receiving questions and generating answers

[0164] The device sends the user's question to the server, which analyzes the question and generates an appropriate answer using a generative AI model, which then sends it to the device, which then displays it to the user.

[0165] 6. Checking understanding and providing feedback

[0166] When a user wants to check their level of understanding, they make a request to check their level of understanding. The server generates a quiz or mini-test and sends it to the device. The server analyzes the user's answers, generates feedback, sends it back to the device, and the device displays the feedback.

[0167] 7. Learning progress management and recording

[0168] The server records the user's learning progress in real time and suggests new learning content and review questions suitable for the next study. The user can check the progress and select the next learning topic to continue learning.

[0169] Specific examples

[0170] Example 1: Learning a history lesson

[0171] 1. The user selects "History Lesson" on the device and sends a request to the server.

[0172] 2. The server references the user's profile and generates the appropriate level of historical content.

[0173] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[0174] 4. The user types in the question, "What is Ooku?"

[0175] 5. The server generates a description of the "Ooku" and sends it to the terminal.

[0176] 6. The user reads the answer and deepens their understanding.

[0177] Example prompt: "Please explain about the Ooku in the Edo period."

[0178] Example 2: Solving a math problem

[0179] 1. The user selects "Math Practice Problems" on the device and sends a request to the server.

[0180] 2. The server generates questions appropriate to the user's level of understanding.

[0181] 3. The question is displayed on the terminal and the user enters the answer.

[0182] 4. The user asks, "Is this correct?"

[0183] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[0184] 6. Users receive feedback and improve their understanding.

[0185] Example prompt: "What is 2+2?"

[0186] In this way, the system of the present invention responds to the learning pace of each student and provides an environment in which students can learn at their own pace while enhancing their self-esteem.

[0187] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0188] System processing steps

[0189] Step 1:

[0190] The user enters their username and password on the terminal and presses the login button, which sends the authentication information to the server.

[0191] Input: Username, Password

[0192] What it does: Creates an HTTP POST request and sends it to the server

[0193] Output: Authentication information arrives at the server

[0194] Step 2:

[0195] The server receives the login request and authenticates it by checking the database. If authentication is successful, it retrieves the user's profile information from the database and sends it to the terminal.

[0196] Input: Credentials

[0197] Data processing: Execute SQL queries to obtain profile information

[0198] Output: Profile information (learning history, grades, vocabulary information)

[0199] Step 3:

[0200] The device analyzes the received profile information and sets the learner's learning level and personalized elements (name and favorite character).

[0201] Input: Profile Information

[0202] Data processing: Analyzing JSON data and reflecting it in the screen layout

[0203] Output: Notification screen displayed when initial setup is complete

[0204] Step 4:

[0205] The user selects a subject or topic to study and sends the request to the server, for example, selecting "Edo Period in History."

[0206] Input: Select a subject or topic

[0207] What it does: Creates an HTTP POST request and sends it to the server

[0208] Output: Request sent to server successfully

[0209] Step 5:

[0210] The server receives the request and generates learning content at the appropriate level based on the user's profile, sending prompts to a generative AI model to create the learning materials.

[0211] Input: Request details, profile information

[0212] Data processing: Send prompts to the generative AI model and receive generated content

[0213] Output: Learning content (text, images, and possibly audio and video)

[0214] Step 6:

[0215] The server transmits the generated learning content to the terminal.

[0216] Input: Learning content

[0217] Data processing: Convert content into JSON format

[0218] Output: Sending content to a device

[0219] Step 7:

[0220] The device analyzes the teaching materials received from the server and displays them in a format that combines text and images.

[0221] Input: Learning content

[0222] Data processing: JSON data analysis and display

[0223] Output: Display of teaching materials

[0224] Step 8:

[0225] As the user reads through the learning materials, if they feel they do not understand the material properly or have questions, they can enter their questions in the question box and press the send button.

[0226] Input: Question

[0227] What it does: Enter a question into the question box, generate an HTTP POST request, and send it to the server

[0228] Output: Question sent to server successfully

[0229] Step 9:

[0230] The server receives the question, analyzes the question using a generative AI model, and generates an appropriate answer.

[0231] Input: Question

[0232] Data processing: Send the question to a generative AI model to generate an answer

[0233] Output: Answer (text and associated image)

[0234] Step 10:

[0235] The server sends the generated response to the terminal.

[0236] Input: Response data

[0237] Data processing: Convert the response data into JSON format

[0238] Output: Sends the answer to the terminal

[0239] Step 11:

[0240] The terminal displays the received response to the user.

[0241] Input: Response data

[0242] Data processing: JSON data analysis and display

[0243] Output: Display the answer

[0244] Step 12:

[0245] If a user wants to check their level of understanding, they can request a comprehension check.

[0246] Input: Comprehension Check Request

[0247] What it does: Creates an HTTP POST request and sends it to the server

[0248] Output: Comprehension check request sent to server

[0249] Step 13:

[0250] The server generates quizzes and mini-tests to check comprehension and sends them to the device.

[0251] Input: Comprehension Check Request

[0252] Data processing: generating quizzes and mini-tests

[0253] Output: Quiz and mini-test (question data)

[0254] Step 14:

[0255] The device displays the received quizzes and mini-tests to the learner, who answers them. The answers are then sent back to the server.

[0256] Input: Quizzes and mini-tests

[0257] Data manipulation: Problem presentation

[0258] Output: Answer submitted by the learner

[0259] Step 15:

[0260] The server analyzes the response and generates feedback that is sent back to the device.

[0261] Input: Learner's answer

[0262] Data processing: analyzing responses and generating feedback

[0263] Output: Feedback data

[0264] Step 16:

[0265] The device displays the feedback to the learner.

[0266] Input: Feedback data

[0267] Data processing: Displaying feedback

[0268] Output: Display feedback

[0269] Step 17:

[0270] The server records the learner's learning progress in real time and stores it in a database. It also suggests new learning content and review questions suitable for the next lesson.

[0271] Input: Learning progress information

[0272] Data processing: Recording progress information and generating next learning content

[0273] Output: Learning progress data and new learning content suggestions

[0274] This allows us to provide learners with a personalized educational environment and support their independent learning.

[0275] (Application example 1)

[0276] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0277] The purpose of this invention is to provide personalized education for early elementary school students based on their individual learning progress and level of understanding. In particular, to create an environment where students can continue learning outside of school, we have identified a challenge in providing interesting content in real time using smart devices to increase their motivation to learn. Furthermore, there is a need to develop a system that uses a generative AI model to quickly provide appropriate answers.

[0278] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0279] In this invention, the server generates learning materials based on each student's level of understanding on the server, accepts questions until the student is satisfied, and includes means for providing visual and linguistic support, means for analyzing the progress and comprehension of the generated learning materials and providing feedback, means for recording the student's learning progress and suggesting the next appropriate learning content, means for displaying information in real time using a smart device and providing interesting content, and means for generating and presenting answers to user questions using a generative AI model. This allows students to receive appropriate support according to their individual learning needs, and enables them to progress in their studies while obtaining interesting information in real time in a physical store even in a learning environment outside of school.

[0280] "Generated teaching materials" are educational contents that are dynamically created on the server based on the individual student's level of understanding.

[0281] "Visual and verbal supports" are text, images, audio and video that complement and deepen students' understanding of their learning.

[0282] "Feedback" is information provided by analyzing the progress of the created learning materials and the student's level of understanding, and providing information on areas for improvement in the learning content and the next learning steps.

[0283] "Learning progress" is data that records how far a student has progressed in the learning process or their level of understanding.

[0284] "Appropriate learning content" is teaching materials that include what students should learn next based on their current level of understanding and learning progress.

[0285] A "smart device" is an electronic device that displays information in real time and provides interesting content.

[0286] A "generative AI model" is an algorithm that uses artificial intelligence technology to generate content such as text, images, and audio.

[0287] "Profile information" is individual data including a student's learning history, grades, vocabulary, etc.

[0288] "Accepting questions" means that the system receives any doubts or follow-up questions that arise while students are studying and provides appropriate answers to them.

[0289] "Displaying information in real time" means providing content instantly so that students can instantly obtain interesting information through their smart devices.

[0290] In the present invention, an educational tool for elementary school students is realized using a smart device. Specific embodiments will be described below.

[0291] Hardware and software used

[0292] Hardware:

[0293] Smart glasses: Common smart devices include Google® Glass® and Microsoft® HoloLens®.

[0294] Server: A high-performance cloud server (e.g., AWS (registered trademark) EC2) is used.

[0295] Camera: Uses the camera built into the smart glasses.

[0296] Microphone: Use the microphone built into the smart glasses.

[0297] software:

[0298] User authentication: Use OAuth 2.0.

[0299] Database: MySQL database is used.

[0300] Generative AI model: Uses OpenAI (registered trademark) GPT-3 (registered trademark).

[0301] Image recognition: Uses TENSORFLOW (registered trademark) and OpenCV.

[0302] Processing Description

[0303] 1. User authentication and profile acquisition:

[0304] The user puts on the smart glasses and scans the QR code (registered trademark) to log in. The server verifies the user's authentication information using OAuth 2.0 and retrieves the student's profile information from the MySQL database. The profile information includes the student's learning history, grades, and vocabulary.

[0305] 2. Initial setup and preparation:

[0306] The smart glasses will then use the profile information to set the student's name and favorite character, creating a personalized learning environment. Once setup is complete, the smart glasses will notify the student that they are ready to begin learning.

[0307] 3. Creation and display of teaching materials:

[0308] When a student looks at a product or exhibit through the smart glasses, the camera captures the image and sends it to a server. The server analyzes this data using TensorFlow and OpenCV and generates tailored learning content using a generative AI model (GPT-3). Based on the student's profile, the generative AI model creates optimal learning materials and sends them to the smart glasses. The learning materials are displayed in the form of text, images, audio, etc.

[0309] 4. Question and Answer Generation:

[0310] If a student has a question while reading the material, they can input it using the microphone in the smart glasses. The question is then sent to the server, where the generative AI model generates an appropriate answer, which is then displayed in text and audio on the smart glasses.

[0311] 5. Check for understanding and provide feedback:

[0312] When a student wants to check their understanding, they switch to test or quiz mode. The server generates an appropriate quiz and sends it to the smart glasses, and the student answers it. The answers are sent to the server, and feedback is generated and sent to the smart glasses.

[0313] Specific examples

[0314] Examples from the history section:

[0315] When a student gazes at a sword in the history exhibit, the camera in the smart glasses recognizes the sword. The generative AI then provides historical background information in text and images, such as "A sword used during the Sengoku period. Used by a Sengoku warlord..." When a student asks, "What is a Sengoku warlord?" the generative AI model explains, "It refers to a samurai who was active during the Sengoku period."

[0316] Prompt Sentence Examples

[0317] This is an educational tool for early elementary school students. Please explain in kind words the following questions:

[0318] Student Information:

[0319] Age: 7

[0320] Interests: History

[0321] Vocabulary level: Beginner

[0322] Question: What is a Sengoku warlord?

[0323] In this way, by linking the server and smart glasses, students can learn in real time and get their questions answered immediately, which improves the effectiveness of education and provides a learning environment that is tailored to each student.

[0324] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0325] Step 1:

[0326] User logs in and gets profile

[0327] Input: User scans the QR code with smart glasses.

[0328] Processing: The smart glasses (terminal) sends the login information obtained from the QR code to the server and performs user authentication using OAuth 2.0.

[0329] Output: The server retrieves the user's profile information (such as learning history, grades, vocabulary, etc.) from the MySQL database and sends it to the smart glasses.

[0330] Step 2:

[0331] Initial setup and preparation

[0332] Input: User profile information retrieved from the server.

[0333] Processing: The smart glasses (terminal) analyzes the profile information and sets the user's name, favorite character, and learning level.

[0334] Output: The smart glasses notify the user that the initial setup is complete.

[0335] Step 3:

[0336] Creation of teaching materials and preparation for display

[0337] Input: Images of products or exhibits that the user sees through the smart glasses.

[0338] Processing: The smart glasses' camera captures images and sends them to the server, which then analyzes them using TensorFlow and OpenCV and generates learning content based on the user profile using a generative AI model (GPT-3).

[0339] Output: The generated teaching materials (text, images, audio, etc.) are sent from the server to the smart glasses and displayed.

[0340] Step 4:

[0341] Accepting questions and generating answers

[0342] Input: A question that the user types through the microphone in the smart glasses.

[0343] Processing: The smart glasses send the question to the server, which uses a generative AI model to analyze the question and generate an appropriate answer.

[0344] Output: The answer (text, associated images, and audio) is sent from the server to the smart glasses and displayed.

[0345] Step 5:

[0346] Checking understanding and providing feedback

[0347] Input: A request for a quiz or test to test the user's understanding.

[0348] Processing: The server generates quizzes and mini-tests based on the user's profile information and sends them to the smart glasses. The user answers the quiz and sends the results to the server. The server analyzes the results and generates feedback.

[0349] Output: Feedback (level of understanding, suggestions for next study, etc.) is sent from the server to the smart glasses and displayed.

[0350] Step 6:

[0351] Record your learning progress

[0352] Input: The results of the user studying the material and answering the quiz.

[0353] Processing: The server records the user's learning progress in real time and stores it in a database. At the same time, it prepares the next learning content proposal.

[0354] Output: The server provides the user with a progress report and suggestions for the next learning content.

[0355] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0356] Summary of the Invention

[0357] This invention is a multimodal self-learning educational tool for early elementary school students that utilizes generative AI and an emotion engine. It generates individually tailored learning materials based on the student's level of comprehension, vocabulary, and even emotional state, allowing students to ask questions until they are satisfied. Learning content is presented using text, images, audio, and video, deepening students' understanding through visual and linguistic support. It also recognizes the student's emotional state and adjusts the presentation and content of the learning materials accordingly, further enhancing the learning experience. It also has the ability to record learning progress and provide feedback.

[0358] Program processing

[0359] Log in and get your profile

[0360] The user opens the login screen on the device and enters authentication information (user ID and password). The device sends this information to the server. The server compares the authentication information with the database, verifies that the user is registered, and returns the result to the device. If authentication is successful, the device sends a request to the server to obtain profile information. The server obtains profile information such as the user's learning history, grades, and vocabulary, and sends it in JSON format to the device.

[0361] Initial setup and preparation

[0362] The device analyzes the profile information received from the server and sets the user's learning level and personalized elements (name, favorite character, etc.). After the settings are complete, the learning start screen is displayed to the user.

[0363] Request and generate learning content

[0364] The user selects a subject or topic to study. For example, if the user selects "Edo Period History," the device sends the request to the server. The server references the user's profile information and the request and generates learning content at the appropriate level. Utilizing a generative AI model, the server generates learning materials that combine text, images, and in some cases audio and video, and sends them to the device.

[0365] View and progress through learning content

[0366] The device analyzes the learning materials received from the server and displays them to the user in a format that combines text and images. The user can then read through the displayed learning content to deepen their understanding. If they have any questions, they can enter them in the question box provided on the learning page.

[0367] Receiving questions and generating answers

[0368] The device sends the user's question to the server. The server analyzes the question and collects data to generate an appropriate answer. It uses a generative AI model to generate an answer to the question and prepares the associated text and images. The server sends the generated answer to the device. The device displays the answer received from the server to the user.

[0369] Checking understanding and providing feedback

[0370] When a user wants to check their level of understanding, they press the "Check Understanding" button. The device sends this request to the server. The server generates a quiz or mini-test and sends it to the device. The device displays the quiz or mini-test, and the user answers it. The server analyzes the user's answers, generates feedback, and sends it to the device. The device displays the feedback to the user.

[0371] Recognizing and responding to emotional states

[0372] The device captures the user's facial expressions and voice while they are studying and sends them to the emotion engine. The emotion engine analyzes this data to identify the user's emotional state. The server receives this emotional information and generates an appropriate response. For example, if the user is feeling confused or stressed, the device may adjust the difficulty of the study content or suggest relaxation content. The device then flexibly displays the study content based on this information.

[0373] Learning progress management and recording

[0374] The server records the user's learning progress in real time and stores it in a database. It also prepares to suggest new learning content and review questions suitable for the next study session. The user can check their learning progress and select the next learning topic to continue their study.

[0375] Specific examples

[0376] Example 1: A student is studying a history lesson.

[0377] 1. The user selects "History Lesson" and sends a request to the server.

[0378] 2. The server references the user's profile and generates the appropriate level of historical content.

[0379] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[0380] 4. The user types in the question, "What is Ooku?"

[0381] 5. The server generates a description of "Ooku" in text and images and returns it to the terminal.

[0382] 6. The user reads the answer and deepens their understanding.

[0383] 7. During the learning process, the device captures the user's facial expressions, which are analyzed by the emotion engine. If the user is determined to be confused, the server adjusts the learning content and sends it to the device.

[0384] Example 2: A student solves a math problem

[0385] 1. The user selects "Math Exercises" and sends a request to the server.

[0386] 2. The server generates questions according to the user's level of understanding.

[0387] 3. The question is displayed on the terminal and the user enters the answer.

[0388] 4. The user asks, "Is this correct?"

[0389] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[0390] 6. Users receive feedback and improve their understanding.

[0391] 7. During learning, the device captures audio, the emotion engine detects stress levels, and the server proposes relaxation content and sends it to the device.

[0392] In this way, the self-learning educational tool of the present invention provides a more personalized learning experience by taking into account not only the student's level of understanding and learning progress but also their emotional state, allowing them to learn at their own pace and progress smoothly while increasing their self-esteem.

[0393] The processing flow will be explained below.

[0394] MODE FOR CARRYING OUT THE INVENTION

[0395] Log in and get your profile

[0396] Step 1:

[0397] The user opens the login screen on the device and enters their user ID and password.

[0398] Step 2:

[0399] The terminal generates a request to send the input authentication information to the server, and sends it to the authentication endpoint of the server.

[0400] Step 3:

[0401] The server checks the received authentication information against the database to confirm that the user is registered, and returns the confirmation result to the terminal.

[0402] Initial setup and preparation

[0403] Step 4:

[0404] The terminal receives the authentication result from the server, and if the authentication is successful, sends a request to the server to obtain the user's profile information.

[0405] Step 5:

[0406] The server retrieves the user's profile information (learning history, grades, vocabulary, etc.) from the database and sends it to the terminal in JSON format.

[0407] Step 6:

[0408] The device analyzes the received profile information and sets the user's learning level and personalized elements (name, favorite character, etc.). After the settings are complete, the learning start screen is displayed.

[0409] Request and generate learning content

[0410] Step 7:

[0411] The user selects the subject or topic to study (e.g., the Edo period in history) and presses the "Start Study" button.

[0412] Step 8:

[0413] The terminal sends the user's selection as a request to the server.

[0414] Step 9:

[0415] The server references the user's profile information and the selected subjects and topics and begins collecting data to generate learning content at an appropriate level.

[0416] Step 10:

[0417] The server uses a generative AI model to generate learning materials that combine text, images, and in some cases audio and video, and sends them to the device in JSON format.

[0418] View and progress through learning content

[0419] Step 11:

[0420] The device analyzes the received educational material and displays it to the user in a format that combines text and images.

[0421] Step 12:

[0422] Users can read through the displayed learning content to deepen their understanding, and if they have any questions, they can enter them in the question box on the learning page.

[0423] Receiving questions and generating answers

[0424] Step 13:

[0425] The terminal generates a request to send the user's question to the server and sends it to the server's question endpoint.

[0426] Step 14:

[0427] The server analyzes the received question and begins collecting data to generate an appropriate answer.

[0428] Step 15:

[0429] The server uses a generative AI model to generate answers to questions and provide associated text and images.

[0430] Step 16:

[0431] The server sends the generated response to the device in JSON format.

[0432] Step 17:

[0433] The terminal displays the response received from the server to the user.

[0434] Step 18:

[0435] Users can read the answers to deepen their understanding, and if they have further questions, they can re-enter their question.

[0436] Checking understanding and providing feedback

[0437] Step 19:

[0438] If the user wants to check their level of understanding, they press the "Check understanding" button.

[0439] Step 20:

[0440] The device sends a request for a comprehension check to the server.

[0441] Step 21:

[0442] The server generates quizzes and mini-tests to check the user's understanding and sends them to the terminal in JSON format.

[0443] Step 22:

[0444] The terminal provides an interface for displaying quizzes and mini-tests to the user and receiving answers.

[0445] Step 23:

[0446] Users answer quizzes and mini-tests and send their answers to their terminals.

[0447] Step 24:

[0448] The terminal transmits the user's answer to the server.

[0449] Step 25:

[0450] The server analyzes the received answers, determines whether they are correct, and generates feedback based on the results, sending it to the device in JSON format.

[0451] Step 26:

[0452] The terminal displays the feedback received from the server to the user.

[0453] Recognizing and responding to emotional states

[0454] Step 27:

[0455] The device captures the user's facial expressions and voice during training and sends them to the emotion engine.

[0456] Step 28:

[0457] The emotion engine analyzes the received data to determine the user's emotional state.

[0458] Step 29:

[0459] The server queries the emotion information received from the emotion engine and adjusts the learning content and presentation method based on the user's emotional state.

[0460] Step 30:

[0461] For example, if the user is confused, the server generates content with a lower level of difficulty and sends it to the terminal.If the user is feeling stressed, the server prepares data to suggest relaxation content.

[0462] Step 31:

[0463] The device follows instructions from the server and displays learning and relaxation content.

[0464] Learning progress management and recording

[0465] Step 32:

[0466] The server records the user's learning progress in real time and stores it in a database.

[0467] Step 33:

[0468] The server prepares the data to suggest the user's next study topic and review questions.

[0469] Step 34:

[0470] Users can check their learning progress and select the next learning topic to continue their learning.

[0471] Example 2

[0472] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0473] Conventional educational tools have struggled to provide personalized education tailored to each student's level of understanding and emotional state. Students in the early grades of elementary school, in particular, require visual and linguistic support, and it is necessary to record each student's level of understanding and progress in real time and provide appropriate feedback. Another challenge is providing prompt and accurate answers to students' questions and motivating them to learn. To address these challenges, the present invention proposes a system that provides a personalized learning experience based on each student's level of understanding and emotional state.

[0474] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for generating learning materials on the server based on the individual student's level of understanding; means for accepting questions until the student is satisfied and providing visual and linguistic support; means for analyzing the progress and comprehension of the generated learning materials and providing feedback; means for recording the student's learning progress and suggesting appropriate next learning content; means for capturing the student's facial expressions and voice while studying, identifying the student's emotional state, and adjusting the learning content based on the identified emotional state; and means for setting the user's learning level and personalized elements based on profile information. This allows for an optimized learning experience for each student, enabling effective learning.

[0475] "Generated teaching materials" are educational content including text, images, audio, video, etc. that are dynamically generated based on students' understanding and requests.

[0476] A "server" is a computer system that serves as the core of an educational tool, and is a device that processes data, generates teaching materials, manages user profiles, and so on.

[0477] "Student" refers to a user who uses this educational tool to study, and is primarily intended for students in the lower grades of elementary school.

[0478] "Level of understanding" is an indicator of how well a student has understood the content they have learned, and the difficulty and content of the teaching materials are adjusted based on this.

[0479] A "question" refers to a question or uncertainty that a student has while studying, and is information used to generate an answer to that question.

[0480] "Visual and verbal support" refers to the use of images, text, audio, etc. to supplement learning content and help students understand.

[0481] "Feedback" refers to evaluations and advice provided based on students' learning progress and level of understanding, and is information intended to improve learning outcomes.

[0482] "Study progress" is information that indicates how far a student has progressed in their studies, and the next learning content is suggested based on this.

[0483] "Emotional state" refers to the psychological and emotional state of students, which is analyzed from their facial expressions and voice, and the learning content and methods are adjusted based on this.

[0484] "Profile Information" refers to personal information about a student, such as their learning history, grades, and vocabulary, that is used to provide a personalized learning experience.

[0485] This invention is a multimodal self-learning educational tool designed for early elementary school students. It utilizes a generative AI model and an emotion engine to generate and provide individually tailored learning materials based on each student's level of comprehension, vocabulary, and emotional state. The following describes the system's main hardware and software components and their specific functions.

[0486] Hardware and Software Configuration

[0487] Server: The server runs key software components such as the generative AI model and database. The generative AI model is used to generate personalized learning materials based on user requests and profile information. The database stores and manages user profile information such as learning history, grades, and vocabulary. The emotion engine also analyzes the user's emotional state and provides appropriate feedback based on that information.

[0488] Terminal: A terminal is a device that students directly operate, typically a tablet or PC. The terminal provides a user interface and communicates student operations to the server. The terminal also displays the learning materials and feedback received from the server.

[0489] Users: The users are students who use this system, and are primarily children in the lower grades of elementary school. They select learning content, enter questions, and check their understanding through their devices. Their emotional state is also captured through the device's camera and microphone.

[0490] System Operation

[0491] The system starts when a user logs in on a device and the server checks the user's profile information. If the login is successful, the server sends information such as the user's learning history, grades, and vocabulary to the device, which then analyzes this information to set learning levels and personalized elements.

[0492] The user then selects the subject or topic they want to study, such as "Edo Period History" or "Math Exercises." This request is sent to a server, which uses a generative AI model to generate learning materials at the appropriate level. The generated learning materials, which include text, images, audio, and video, are sent to the device.

[0493] During the learning process, if the user has any questions, they can input them through the device. The questions are sent to the server, and the generative AI model generates an appropriate answer. The answer, which consists of text and images, is sent back to the device and displayed to the user.

[0494] In addition, the user's facial expressions and voice while studying are captured by the device's camera and microphone and sent to the emotion engine. The emotion engine analyzes this, and if the user is confused, the server will provide content adjusted to an appropriate level of difficulty or relaxation content.

[0495] The user's learning progress is recorded on the server in real time, and new learning materials are suggested based on the user's progress and next learning content. For example, review questions are suggested for topics where the user's understanding has deepened.

[0496] Specific examples

[0497] In the case of history classes

[0498] 1. The user selects "History Lesson" and sends a request to the server.

[0499] 2. The server references the user's profile information and generates the appropriate level of history content.

[0500] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[0501] 4. The user types in the question, "What is Ooku?"

[0502] 5. The server generates a description of "Ooku" in text and images and returns it to the terminal.

[0503] 6. The user reads the answer and deepens their understanding.

[0504] For math problems

[0505] 1. The user selects "Math Practice Problems" and sends a request to the server.

[0506] 2. The server generates questions according to the user's level of understanding.

[0507] 3. The question is displayed on the terminal and the user enters the answer.

[0508] 4. The user asks, "Is this correct?"

[0509] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[0510] 6. Users receive feedback and improve their understanding.

[0511] Prompt Sentence Examples

[0512] About History Classes

[0513] User ID: 12345

[0514] Subject: History

[0515] Topic: Edo period

[0516] Question: What is Ooku?

[0517] About Mathematics Problems

[0518] User ID: 67890

[0519] Subject: Mathematics

[0520] Question: Is this correct?

[0521] Problem: Solve the equation x + 5 = 10.

[0522] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0523] Step 1: Log in and get your profile

[0524] The user opens the login screen on their device and enters their user ID and password. The device sends this authentication information to the server. The server accesses the database and verifies the entered authentication information. If the login is successful, the server sends profile information associated with the user's ID (learning history, grades, vocabulary, etc.) in JSON format to the device.

[0525] Input: User ID, Password

[0526] Output: Authentication results, profile information (JSON format)

[0527] Step 2: Initial setup and preparation

[0528] The device analyzes the profile information received from the server and sets personalized elements such as the user's learning level, name, favorite character, etc. Once the settings are complete, the device displays the learning start screen to the user.

[0529] Input: Profile information (JSON format)

[0530] Output: Learning start screen

[0531] Step 3: Request and generate learning content

[0532] The user selects the subject or topic they want to study. For example, they select "Edo Period History." The device sends this request to the server. The server generates a prompt based on the user's profile information and the request, and sends it to the generative AI model. The generative AI model then generates learning materials that combine text, images, and in some cases audio and video, and sends them back to the server.

[0533] Input: User's subject and topic selection

[0534] Output: Generated teaching materials (text, images, etc.)

[0535] Step 4: View and progress through learning content

[0536] The device analyzes the learning materials received from the server and displays them in a format appropriate for the user, such as a combination of text, images, or video. The user can then read through the displayed learning content and progress as needed.

[0537] Input: Generated teaching materials (text, images, etc.)

[0538] Output: Displayed learning content

[0539] Step 5: Receiving questions and generating answers

[0540] If a user has a question while learning, they can enter it into a question box installed on the device. For example, they could enter "What is Ooku?" The device then sends this question to the server. The server analyzes the question, collects information from related documents and databases, and sends it to the generative AI model. The generative AI model then generates an appropriate answer and sends the relevant text and images back to the server.

[0541] Input: User question

[0542] Output: Generated answer (text, image, etc.)

[0543] Step 6: Check for understanding and provide feedback

[0544] If a user wants to check their level of understanding during a study, they press the "Check Understanding" button. The device sends a request to the server. The server generates a quiz or mini-test based on the user's learning content and sends it to the device. The device displays the quiz or mini-test, and the user answers. The server analyzes the answers, generates feedback, and sends it to the device.

[0545] Input: Comprehension check request, user answer

[0546] Output: Feedback (rating, advice)

[0547] Step 7: Recognizing and responding to emotional states

[0548] The device captures the user's facial expressions and voice while they are learning. This data is sent to the emotion engine for analysis. The emotion engine identifies the user's emotional state and sends that information to the server. The server uses this information to adjust the difficulty level and suggest relaxation content. The device displays this and adjusts the learning experience.

[0549] Input: Captured facial and voice data

[0550] Output: Tailored learning content based on emotional state

[0551] Step 8: Track and record your learning progress

[0552] The server records the user's learning progress in real time and stores it in a database. This information is used to suggest new learning content and review questions suitable for the next learning session. The user can check their learning progress on their device, select new topics, and review.

[0553] Input: Learning progress data

[0554] Output: Updated learning progress, suggestions for next learning content

[0555] (Application example 2)

[0556] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0557] Conventional educational and sales systems have had difficulty providing appropriate support based on the level of understanding and emotional state of individual students and customers. Furthermore, they were unable to adequately record students' progress in real time, reflect it in the next lesson, or suggest products based on customer emotion analysis. As a result, there were problems with reduced learning efficiency and customer satisfaction.

[0558] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0559] In this invention, the server includes: a means for generating learning materials on the server based on each student's level of comprehension; a means for accepting questions until the student is satisfied and providing visual and linguistic support; a means for analyzing the progress and comprehension of the generated learning materials and providing feedback; a means for recording the student's learning progress and suggesting the next appropriate learning content; a means for acquiring profile information of customers visiting the store and suggesting products suitable for the target individual; a means for visually and linguistically displaying product information using smart glasses and explaining it to customers in real time; a means for analyzing the customer's emotional state and adjusting the product explanation and suggestion method based on that information; and a recording means for collecting customer feedback and utilizing it in future suggestions. This allows students to learn at their own pace, promote self-esteem, and make product suggestions based on the individual customer's needs and emotions.

[0560] "Generated learning materials" refers to learning content generated on the server based on each student's level of understanding.

[0561] "Individual student understanding" refers to the level of knowledge and skills each student currently possesses, and is an important evaluation criterion that is taken into consideration to optimize learning efficiency.

[0562] "Visual and linguistic support" refers to supplementary information provided to learners using text, images, video, audio, etc., and is used to aid comprehension.

[0563] "Means for analyzing progress and understanding and providing feedback" is a function that analyzes data obtained while students are studying the material and provides the next learning step or additional explanation based on that data.

[0564] "Means for recording learning progress and suggesting appropriate next learning content" is a function that records how far a student has progressed in their studies and determines the next learning content based on that data.

[0565] "Means for obtaining profile information of customers visiting the store" is a function for collecting basic information such as the age, interests, and favorite characters of customers who visit the store.

[0566] The "means for proposing products suitable for the target individual" is a function that allows the server to select and propose the most suitable products to the customer based on the collected profile information.

[0567] "Means for visually and verbally displaying product information using smart glasses" refers to a function that displays product information to sales staff via smart glasses and communicates it to customers visually and audibly.

[0568] "Real-time customer explanations" refers to the ability to use smart glasses or other devices to instantly explain products based on current information and context.

[0569] "Means for analyzing the emotional state of customers" refers to technology for reading and analyzing emotions from customers' facial expressions and voices.

[0570] The "means for adjusting the product description and proposal method" is a function for appropriately changing the content and method of product presentation and proposal based on the emotional state of the customer.

[0571] "A means of recording customer feedback and using it in future proposals" is a function that records customer reactions and opinions and uses that information to optimize future proposals.

[0572] This invention uses smart glasses to suggest optimal products to customers and provide services tailored to the needs and emotional state of each individual customer. This system mainly consists of three elements: a server, a terminal (smart glasses), and a user.

[0573] First, when a customer visits the store, the server obtains the customer's profile information entered by the salesperson through the smart glasses. This profile information includes the customer's age, interests, favorite characters, etc. Using this information, the server uses a generative AI model to generate product suggestions that are best suited to the customer and displays them on the smart glasses. This generative AI model generates individually suited products based on past data and each customer's profile.

[0574] A salesperson wearing smart glasses checks the product information sent from the server and explains the product to the customer in real time based on that information. This product information is provided visually and verbally, combining text, images, audio, video, etc.

[0575] The server then uses the smart glasses to capture the customer's facial expressions and voice and transmits them to an emotion analysis engine. The emotion analysis engine then analyzes this data to identify the customer's emotional state. The server then adjusts the product explanation and recommendations in real time based on this emotional information and provides feedback to the salesperson via the smart glasses. For example, if the customer appears confused or stressed, the server can simplify the product explanation or suggest alternative products.

[0576] Finally, the server collects customer feedback and records it for future recommendations, which leads to more personalized recommendations and improved customer satisfaction.

[0577] For example, if a salesperson visits a store and a 10-year-old child enters their profile stating that they are interested in dinosaurs, the server will use the generative AI model to suggest educational toys related to dinosaurs. Information about moving dinosaur figures is displayed on the smart glasses, and the salesperson explains the details. If the child's facial expression shows curiosity during the explanation, the emotion analysis engine will recognize it and suggest a more detailed demonstration. The salesperson will then collect feedback, which will be used to improve the next recommendation.

[0578] Examples of prompts include:

[0579] "My 10-year-old child is interested in dinosaurs. Can you recommend some educational toys?"

[0580] This is the prompt that is input to the generative AI model in the server, and the following data is sent to the sentiment analysis engine:

[0581] "Analyze the child's image and audio data. Provide real-time feedback based on their emotional state."

[0582] Thus, the present invention is a system that enhances the shopping experience by dynamically suggesting products and providing personalized service based on a customer's profile information and emotional state.

[0583] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0584] Step 1:

[0585] The user (salesperson) enters the profile information (age, interests, favorite characters, etc.) of the customer who visits the store through the smart glasses. The input data is sent to the server.

[0586] Step 2:

[0587] The server generates optimal product candidates using a generative AI model based on the received customer profile information. The generated product candidate list is sent to the smart glasses as data including visual and linguistic information (text, images, videos, etc.).

[0588] Step 3:

[0589] The user (salesperson) wearing the smart glasses checks the product candidate list received from the server and explains the product to the customer. Here, detailed product information is displayed in real time and a presentation is made to the customer.

[0590] Step 4:

[0591] The device (smart glasses) captures the customer's facial expressions and voice and transmits them to an emotion analysis engine in real time, where the data is used to identify their emotional state through facial expression analysis and voice emotion analysis algorithms.

[0592] Step 5:

[0593] The server adjusts product explanations and recommendations in real time based on the emotional information obtained from the emotion analysis engine. For example, if the customer is confused, it will simplify the explanation or suggest a different product. The adjusted information is sent as feedback to the smart glasses.

[0594] Step 6:

[0595] Based on the feedback received through the smart glasses, the user (salesperson) responds appropriately to the customer and continues to explain and suggest products.

[0596] Step 7:

[0597] After the sales negotiation, the user (salesperson) collects customer feedback and sends it to the server using the smart glasses. The collected feedback is stored in a database and used for the next proposal.

[0598] Step 8:

[0599] The server processes the data based on the collected feedback and uses it to improve future recommendation algorithms, which will result in more personalized product recommendations for future customers.

[0600] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0601] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0602] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0603] [Second embodiment]

[0604] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0605] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0606] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0607] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0608] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0609] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0610] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0611] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0612] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0613] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0614] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0615] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0616] Summary of the Invention

[0617] This invention is a multimodal self-learning educational tool that utilizes generative AI for early elementary school students. It generates individually tailored learning materials based on the student's level of comprehension and vocabulary, allowing students to ask questions until they are satisfied. Learning content is presented using text, images, and in some cases audio and video, deepening students' understanding through visual and linguistic support. It also records learning progress and provides feedback, enhancing self-esteem.

[0618] Program processing

[0619] Log in and get your profile

[0620] When a user logs in on a device, the server receives the authentication information and retrieves and sends the user's profile information from the database. The profile includes information such as learning history, grades, and vocabulary.

[0621] Initial setup and preparation

[0622] The device analyzes the profile information received from the server and sets the user's learning level and personalized elements (such as name and favorite character). Once the settings are complete, a screen is displayed notifying the user that they are ready to start learning.

[0623] Request and generate learning content

[0624] The user selects a subject or topic to study. For example, if they select "Edo Period History," the request is sent to the server. The server generates learning content at the appropriate level based on the user's profile. The generative AI model creates learning materials by combining text, images, and in some cases audio and video, and sends them to the device.

[0625] View and progress through learning content

[0626] The device analyzes the learning materials received from the server and displays them in a format that combines text and images. The user can then read through the materials. If they do not understand the material properly or have questions, they can enter them in a question box on the learning page.

[0627] Receiving questions and generating answers

[0628] The device sends the user's question to the server, which analyzes the question and generates an appropriate answer. The generative AI model generates an image related to the text and sends it to the device as an answer, which the device displays to the user.

[0629] Checking understanding and providing feedback

[0630] When a user wants to check their level of understanding, they make a request for a comprehension check. The server generates a quiz or mini-test to check the user's level of understanding and sends it to the device. The user answers it, and the answer results are sent back to the server. The server analyzes the answers, generates feedback, and sends it back to the device. The device displays the feedback to the user.

[0631] Learning progress management and recording

[0632] The server records the user's learning progress in real time and saves it in a database. It also prepares to suggest new learning content and review questions suitable for the next study. The user can check the progress and select the next learning topic to continue learning.

[0633] Specific examples

[0634] Example 1: A student is studying a history lesson.

[0635] 1. The user selects "History Lesson" on the device and sends a request to the server.

[0636] 2. The server references the user's profile and generates the appropriate level of historical content.

[0637] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[0638] 4. The user reads through the learning material and enters the question, "What is Ooku?"

[0639] 5. The server generates a description of "Ooku" in text and images and returns it to the terminal.

[0640] 6. The user reads the answer and deepens their understanding.

[0641] Example 2: A student solves a math problem

[0642] 1. The user selects "Math Practice Problems" on the device and sends a request to the server.

[0643] 2. The server generates problems appropriate to the user's level of understanding, such as simple addition and subtraction problems.

[0644] 3. The question is displayed on the terminal and the user enters the answer.

[0645] 4. If the user is unsure of the answer, ask, "Is this correct?"

[0646] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[0647] 6. Users receive feedback and improve their understanding.

[0648] In this way, the self-learning educational tool of the present invention accommodates the learning pace of each student and provides an environment in which students can learn at their own pace while enhancing their self-esteem.

[0649] The processing flow will be explained below.

[0650] Step 1:

[0651] The user opens the login screen on the device and enters authentication information (user ID and password).

[0652] Step 2:

[0653] The terminal generates a request to send the input authentication information to the server, and sends it to the authentication endpoint of the server.

[0654] Step 3:

[0655] The server checks the received authentication information against the database to confirm that the user is registered, and returns the check result to the terminal.

[0656] Step 4:

[0657] The terminal receives the authentication result from the server, and if the authentication is successful, sends a request to the server to obtain the user's profile information.

[0658] Step 5:

[0659] The server retrieves the user's profile information (learning history, grades, vocabulary, etc.) from the database and returns it to the terminal in JSON format.

[0660] Step 6:

[0661] The device analyzes the received profile information and sets the user's learning level and personalized elements (name, favorite character, etc.). After the settings are complete, the learning start screen is displayed to the user.

[0662] Step 7:

[0663] Users select the subject or topic they wish to study (e.g., the Edo period in history) on their device and press the "Start Studying" button.

[0664] Step 8:

[0665] The terminal sends the user's selection as a request to the server.

[0666] Step 9:

[0667] The server references the user's profile information and the request to generate learning content at the appropriate level, using a generative AI model to generate learning materials that combine text, images, and in some cases video and audio.

[0668] Step 10:

[0669] The server sends the generated teaching materials to the terminal in JSON format.

[0670] Step 11:

[0671] The device analyzes the received educational material and displays it to the user in a format that combines text and images.

[0672] Step 12:

[0673] Users can read through the displayed learning content to deepen their understanding, and if they have any questions, they can enter them in the question box provided on the learning page.

[0674] Step 13:

[0675] The terminal generates a request to send the user's question to the server and sends it to the server's question endpoint.

[0676] Step 14:

[0677] The server analyzes the received question and collects data to generate an appropriate answer. It uses a generative AI model to generate an answer to the question and provides relevant text and images.

[0678] Step 15:

[0679] The server sends the generated response in JSON format to the terminal.

[0680] Step 16:

[0681] The terminal displays the response received from the server to the user.

[0682] Step 17:

[0683] Users can read the answers to deepen their understanding and ask again if they have further questions.

[0684] Step 18:

[0685] If the user wants to check their level of understanding, they press the "Check understanding" button.

[0686] Step 19:

[0687] The device sends a request for a comprehension check to the server.

[0688] Step 20:

[0689] The server generates quizzes and mini-tests to check the user's level of understanding, using a generative AI model to select questions at the appropriate level and sending them to the device in JSON format.

[0690] Step 21:

[0691] The terminal provides an interface for displaying quizzes and mini-tests to the user and receiving answers.

[0692] Step 22:

[0693] Users answer quizzes and mini-tests and send their answers to their terminals.

[0694] Step 23:

[0695] The terminal generates a request to send the user's answer to the server and sends it to the server's analysis endpoint.

[0696] Step 24:

[0697] The server analyzes the received answers, determines whether they are correct, and generates feedback based on the analysis results, sending it to the device in JSON format.

[0698] Step 25:

[0699] The terminal displays the feedback received from the server to the user.

[0700] Step 26:

[0701] The server records the user's learning progress in real time and stores the progress in a database.

[0702] Step 27:

[0703] The server prepares data to suggest the user's next study topic and review questions.

[0704] Step 28:

[0705] Users can check their learning progress and select the next learning topic to continue their learning.

[0706] Example 1

[0707] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0708] Conventional learning systems make it difficult to provide individualized instruction tailored to each student's level of understanding and progress, and students often cannot receive immediate answers when they have questions. It is also difficult to grasp learning progress and comprehension and provide appropriate feedback based on that information. Furthermore, personalization of learning content is insufficient, making it impossible to provide optimal learning materials to each student. This can result in a decline in students' self-esteem and motivation to learn, so these issues must be resolved.

[0709] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0710] In this invention, the server includes: means for a user to log in using a terminal, the server receiving authentication information, and retrieving and transmitting the user's profile information from a database; means for the terminal analyzing the received profile information and setting the learner's learning level and personalized elements; means for the learner to select a subject or topic to study and sending a request to the server; means for the server to generate learning content at an appropriate level based on the user's profile information and send it to the terminal; means for the terminal to analyze the learning materials received from the server and display them in a format combining text and images; means for the learner to input a question into a question box and the terminal sending it to the server; means for the server to analyze the question, generate an appropriate answer, and send it to the terminal; and means for the server to record the learner's learning progress in real time and suggest new learning content and review questions appropriate for the next lesson. This makes it possible to provide an optimal learning experience for each student, improving learning efficiency and self-esteem.

[0711] A "terminal" is an electronic device that a user operates, and includes computers, tablets, smartphones, and the like.

[0712] A "server" is a computer system that communicates with terminals over a network and processes, stores, and manages data.

[0713] "Student" refers to a student or user who uses the system to study.

[0714] "Profile information" refers to individual data about a learner, including learning history, grades, vocabulary information, and personalized elements.

[0715] A "database" is a system for centrally managing and storing information, and is accessed by a server to search for and store the necessary data.

[0716] "Authentication information" is information used to identify a particular user and grant that user system access rights, and typically includes a username and password.

[0717] "Learning content" refers to the learning materials and workbooks that learners use for learning, and consists of text, images, audio, video, etc.

[0718] A "generative AI model" is an algorithm that uses artificial intelligence technology to generate learning content and answers.

[0719] "Feedback" refers to the process of providing evaluation and advice on a learner's answers or actions.

[0720] "Progress recording" refers to the quantitative or qualitative recording of the results and progress of a learner's learning activities.

[0721] A "question box" refers to an interface where learners can enter their doubts or queries.

[0722] "Level of understanding" is a measure of how well a learner understands the given teaching materials and tasks.

[0723] "Personalized elements" refer to elements of the learning experience that are customized based on the learner's individual preferences and characteristics.

[0724] This invention is a multimodal self-learning educational system that utilizes generative AI models for early elementary school students. This system provides optimal learning content based on the learner's level of understanding and progress, accepts questions until the learner is satisfied, and provides visual and linguistic support. It also aims to enhance learners' self-esteem by recording their learning progress and providing appropriate feedback.

[0725] System Configuration

[0726] This system is realized mainly using the following hardware and software.

[0727] Device: Computer, tablet, smartphone, etc.

[0728] Server: A computer system that processes and stores data over a network.

[0729] Database: A system that manages the information accessed by a server (e.g., MySQL, PostgreSQL).

[0730] Generative AI models: Artificial intelligence algorithms that generate learning content and answers (e.g., GPT-4).

[0731] Program implementation technologies: Python, Django, Redis, HTML / CSS, JavaScript, etc.

[0732] Program processing

[0733] The system operates as follows:

[0734] 1. Log in and get your profile

[0735] The user enters their username and password on the terminal and presses the login button. The server receives the authentication information, retrieves the user's profile information from the database, and sends it to the terminal.

[0736] 2. Initial Setup and Preparation

[0737] The device analyzes the received profile information and initializes the learner's learning level and personalized elements. Once the setup is complete, a screen will appear informing the learner that they are ready to begin learning.

[0738] 3. Request and generate learning content

[0739] The user selects the subject or topic to study and sends the request to the server, which uses a generative AI model based on the user's profile information to generate learning content at the appropriate level and send it to the device.

[0740] 4. View and progress through learning content

[0741] The device analyzes the learning materials received from the server and displays them in a format that combines text and images. The user can read the materials and, if they do not understand them or have questions, they can enter their questions in the question box.

[0742] 5. Receiving questions and generating answers

[0743] The device sends the user's question to the server, which analyzes the question and generates an appropriate answer using a generative AI model, which then sends it to the device, which then displays it to the user.

[0744] 6. Checking understanding and providing feedback

[0745] When a user wants to check their level of understanding, they make a request to check their level of understanding. The server generates a quiz or mini-test and sends it to the device. The server analyzes the user's answers, generates feedback, sends it back to the device, and the device displays the feedback.

[0746] 7. Learning progress management and recording

[0747] The server records the user's learning progress in real time and suggests new learning content and review questions suitable for the next study. The user can check the progress and select the next learning topic to continue learning.

[0748] Specific examples

[0749] Example 1: Learning a history lesson

[0750] 1. The user selects "History Lesson" on the device and sends a request to the server.

[0751] 2. The server references the user's profile and generates the appropriate level of historical content.

[0752] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[0753] 4. The user types in the question, "What is Ooku?"

[0754] 5. The server generates a description of the "Ooku" and sends it to the terminal.

[0755] 6. The user reads the answer and deepens their understanding.

[0756] Example prompt: "Please explain about the Ooku in the Edo period."

[0757] Example 2: Solving a math problem

[0758] 1. The user selects "Math Practice Problems" on the device and sends a request to the server.

[0759] 2. The server generates questions appropriate to the user's level of understanding.

[0760] 3. The question is displayed on the terminal and the user enters the answer.

[0761] 4. The user asks, "Is this correct?"

[0762] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[0763] 6. Users receive feedback and improve their understanding.

[0764] Example prompt: "What is 2+2?"

[0765] In this way, the system of the present invention responds to the learning pace of each student and provides an environment in which students can learn at their own pace while enhancing their self-esteem.

[0766] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0767] System processing steps

[0768] Step 1:

[0769] The user enters their username and password on the terminal and presses the login button, which sends the authentication information to the server.

[0770] Input: Username, Password

[0771] What it does: Creates an HTTP POST request and sends it to the server

[0772] Output: Authentication information arrives at the server

[0773] Step 2:

[0774] The server receives the login request and authenticates it by checking the database. If authentication is successful, it retrieves the user's profile information from the database and sends it to the terminal.

[0775] Input: Credentials

[0776] Data processing: Execute SQL queries to obtain profile information

[0777] Output: Profile information (learning history, grades, vocabulary information)

[0778] Step 3:

[0779] The device analyzes the received profile information and sets the learner's learning level and personalized elements (name and favorite character).

[0780] Input: Profile Information

[0781] Data processing: Analyzing JSON data and reflecting it in the screen layout

[0782] Output: Notification screen displayed when initial setup is complete

[0783] Step 4:

[0784] The user selects a subject or topic to study and sends the request to the server, for example, selecting "Edo Period in History."

[0785] Input: Select a subject or topic

[0786] What it does: Creates an HTTP POST request and sends it to the server

[0787] Output: Request sent to server successfully

[0788] Step 5:

[0789] The server receives the request and generates learning content at the appropriate level based on the user's profile, sending prompts to a generative AI model to create the learning materials.

[0790] Input: Request details, profile information

[0791] Data processing: Send prompts to the generative AI model and receive generated content

[0792] Output: Learning content (text, images, and possibly audio and video)

[0793] Step 6:

[0794] The server transmits the generated learning content to the terminal.

[0795] Input: Learning content

[0796] Data processing: Convert content into JSON format

[0797] Output: Sending content to a device

[0798] Step 7:

[0799] The device analyzes the teaching materials received from the server and displays them in a format that combines text and images.

[0800] Input: Learning content

[0801] Data processing: JSON data analysis and display

[0802] Output: Display of teaching materials

[0803] Step 8:

[0804] As the user reads through the learning materials, if they feel they do not understand the material properly or have questions, they can enter their questions in the question box and press the send button.

[0805] Input: Question

[0806] What it does: Enter a question into the question box, generate an HTTP POST request, and send it to the server

[0807] Output: Question sent to server successfully

[0808] Step 9:

[0809] The server receives the question, analyzes the question using a generative AI model, and generates an appropriate answer.

[0810] Input: Question

[0811] Data processing: Send the question to a generative AI model to generate an answer

[0812] Output: Answer (text and associated image)

[0813] Step 10:

[0814] The server sends the generated response to the terminal.

[0815] Input: Response data

[0816] Data processing: Convert the response data into JSON format

[0817] Output: Sends the answer to the terminal

[0818] Step 11:

[0819] The terminal displays the received response to the user.

[0820] Input: Response data

[0821] Data processing: JSON data analysis and display

[0822] Output: Display the answer

[0823] Step 12:

[0824] If a user wants to check their level of understanding, they can request a comprehension check.

[0825] Input: Comprehension Check Request

[0826] What it does: Creates an HTTP POST request and sends it to the server

[0827] Output: Comprehension check request sent to server

[0828] Step 13:

[0829] The server generates quizzes and mini-tests to check comprehension and sends them to the device.

[0830] Input: Comprehension Check Request

[0831] Data processing: generating quizzes and mini-tests

[0832] Output: Quiz and mini-test (question data)

[0833] Step 14:

[0834] The device displays the received quizzes and mini-tests to the learner, who answers them. The answers are then sent back to the server.

[0835] Input: Quizzes and mini-tests

[0836] Data manipulation: Problem presentation

[0837] Output: Answer submitted by the learner

[0838] Step 15:

[0839] The server analyzes the response and generates feedback that is sent back to the device.

[0840] Input: Learner's answer

[0841] Data processing: analyzing responses and generating feedback

[0842] Output: Feedback data

[0843] Step 16:

[0844] The device displays the feedback to the learner.

[0845] Input: Feedback data

[0846] Data processing: Displaying feedback

[0847] Output: Display feedback

[0848] Step 17:

[0849] The server records the learner's learning progress in real time and stores it in a database. It also suggests new learning content and review questions suitable for the next lesson.

[0850] Input: Learning progress information

[0851] Data processing: Recording progress information and generating next learning content

[0852] Output: Learning progress data and new learning content suggestions

[0853] This allows us to provide learners with a personalized educational environment and support their independent learning.

[0854] (Application example 1)

[0855] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0856] The purpose of this invention is to provide personalized education for early elementary school students based on their individual learning progress and level of understanding. In particular, to create an environment where students can continue learning outside of school, we have identified a challenge in providing interesting content in real time using smart devices to increase their motivation to learn. Furthermore, there is a need to develop a system that uses a generative AI model to quickly provide appropriate answers.

[0857] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0858] In this invention, the server generates learning materials based on each student's level of understanding on the server, accepts questions until the student is satisfied, and includes means for providing visual and linguistic support, means for analyzing the progress and comprehension of the generated learning materials and providing feedback, means for recording the student's learning progress and suggesting the next appropriate learning content, means for displaying information in real time using a smart device and providing interesting content, and means for generating and presenting answers to user questions using a generative AI model. This allows students to receive appropriate support according to their individual learning needs, and enables them to progress in their studies while obtaining interesting information in real time in a physical store even in a learning environment outside of school.

[0859] "Generated teaching materials" are educational contents that are dynamically created on the server based on the individual student's level of understanding.

[0860] "Visual and verbal supports" are text, images, audio and video that complement and deepen students' understanding of their learning.

[0861] "Feedback" is information provided by analyzing the progress of the created learning materials and the student's level of understanding, and providing information on areas for improvement in the learning content and the next learning steps.

[0862] "Learning progress" is data that records how far a student has progressed in the learning process or their level of understanding.

[0863] "Appropriate learning content" is teaching materials that include what students should learn next based on their current level of understanding and learning progress.

[0864] A "smart device" is an electronic device that displays information in real time and provides interesting content.

[0865] A "generative AI model" is an algorithm that uses artificial intelligence technology to generate content such as text, images, and audio.

[0866] "Profile information" is individual data including a student's learning history, grades, vocabulary, etc.

[0867] "Accepting questions" means that the system receives any doubts or follow-up questions that arise while students are studying and provides appropriate answers to them.

[0868] "Displaying information in real time" means providing content instantly so that students can instantly obtain interesting information through their smart devices.

[0869] In the present invention, an educational tool for elementary school students is realized using a smart device. Specific embodiments will be described below.

[0870] Hardware and software used

[0871] Hardware:

[0872] Smart glasses: Use Google Glass or Microsoft HoloLens as common smart devices.

[0873] Server: Use a high-performance cloud server (e.g., AWS EC2).

[0874] Camera: Uses the camera built into the smart glasses.

[0875] Microphone: Use the microphone built into the smart glasses.

[0876] software:

[0877] User authentication: Use OAuth 2.0.

[0878] Database: MySQL database is used.

[0879] Generative AI model: Uses OpenAI GPT-3.

[0880] Image Recognition: Uses TensorFlow and OpenCV.

[0881] Processing Description

[0882] 1. User authentication and profile acquisition:

[0883] A user puts on the smart glasses and scans a QR code to log in. The server verifies the user's authentication information using OAuth 2.0 and retrieves the student's profile information from a MySQL database, including the student's learning history, grades, and vocabulary.

[0884] 2. Initial setup and preparation:

[0885] The smart glasses will then use the profile information to set the student's name and favorite character, creating a personalized learning environment. Once setup is complete, the smart glasses will notify the student that they are ready to begin learning.

[0886] 3. Creation and display of teaching materials:

[0887] When a student looks at a product or exhibit through the smart glasses, the camera captures the image and sends it to a server. The server analyzes this data using TensorFlow and OpenCV and generates tailored learning content using a generative AI model (GPT-3). Based on the student's profile, the generative AI model creates optimal learning materials and sends them to the smart glasses. The learning materials are displayed in the form of text, images, audio, etc.

[0888] 4. Question and Answer Generation:

[0889] If a student has a question while reading the material, they can input it using the microphone in the smart glasses. The question is then sent to the server, where the generative AI model generates an appropriate answer, which is then displayed in text and audio on the smart glasses.

[0890] 5. Check for understanding and provide feedback:

[0891] When a student wants to check their understanding, they switch to test or quiz mode. The server generates an appropriate quiz and sends it to the smart glasses, and the student answers it. The answers are sent to the server, and feedback is generated and sent to the smart glasses.

[0892] Specific examples

[0893] Examples from the history section:

[0894] When a student gazes at a sword in the history exhibit, the camera in the smart glasses recognizes the sword. The generative AI then provides historical background information in text and images, such as "A sword used during the Sengoku period. Used by a Sengoku warlord..." When a student asks, "What is a Sengoku warlord?" the generative AI model explains, "It refers to a samurai who was active during the Sengoku period."

[0895] Prompt Sentence Examples

[0896] This is an educational tool for early elementary school students. Please explain in kind words the following questions:

[0897] Student Information:

[0898] Age: 7

[0899] Interests: History

[0900] Vocabulary level: Beginner

[0901] Question: What is a Sengoku warlord?

[0902] In this way, by linking the server and smart glasses, students can learn in real time and get their questions answered immediately, which improves the effectiveness of education and provides a learning environment that is tailored to each student.

[0903] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0904] Step 1:

[0905] User logs in and gets profile

[0906] Input: User scans the QR code with smart glasses.

[0907] Processing: The smart glasses (terminal) sends the login information obtained from the QR code to the server and performs user authentication using OAuth 2.0.

[0908] Output: The server retrieves the user's profile information (such as learning history, grades, vocabulary, etc.) from the MySQL database and sends it to the smart glasses.

[0909] Step 2:

[0910] Initial setup and preparation

[0911] Input: User profile information retrieved from the server.

[0912] Processing: The smart glasses (terminal) analyzes the profile information and sets the user's name, favorite character, and learning level.

[0913] Output: The smart glasses notify the user that the initial setup is complete.

[0914] Step 3:

[0915] Creation of teaching materials and preparation for display

[0916] Input: Images of products or exhibits that the user sees through the smart glasses.

[0917] Processing: The smart glasses' camera captures images and sends them to the server, which then analyzes them using TensorFlow and OpenCV and generates learning content based on the user profile using a generative AI model (GPT-3).

[0918] Output: The generated teaching materials (text, images, audio, etc.) are sent from the server to the smart glasses and displayed.

[0919] Step 4:

[0920] Accepting questions and generating answers

[0921] Input: A question that the user types through the microphone in the smart glasses.

[0922] Processing: The smart glasses send the question to the server, which uses a generative AI model to analyze the question and generate an appropriate answer.

[0923] Output: The answer (text, associated images, and audio) is sent from the server to the smart glasses and displayed.

[0924] Step 5:

[0925] Checking understanding and providing feedback

[0926] Input: A request for a quiz or test to test the user's understanding.

[0927] Processing: The server generates quizzes and mini-tests based on the user's profile information and sends them to the smart glasses. The user answers the quiz and sends the results to the server. The server analyzes the results and generates feedback.

[0928] Output: Feedback (level of understanding, suggestions for next study, etc.) is sent from the server to the smart glasses and displayed.

[0929] Step 6:

[0930] Record your learning progress

[0931] Input: The results of the user studying the material and answering the quiz.

[0932] Processing: The server records the user's learning progress in real time and stores it in a database. At the same time, it prepares the next learning content proposal.

[0933] Output: The server provides the user with a progress report and suggestions for the next learning content.

[0934] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0935] Summary of the Invention

[0936] This invention is a multimodal self-learning educational tool for early elementary school students that utilizes generative AI and an emotion engine. It generates individually tailored learning materials based on the student's level of comprehension, vocabulary, and even emotional state, allowing students to ask questions until they are satisfied. Learning content is presented using text, images, audio, and video, deepening students' understanding through visual and linguistic support. It also recognizes the student's emotional state and adjusts the presentation and content of the learning materials accordingly, further enhancing the learning experience. It also has the ability to record learning progress and provide feedback.

[0937] Program processing

[0938] Log in and get your profile

[0939] The user opens the login screen on the device and enters authentication information (user ID and password). The device sends this information to the server. The server compares the authentication information with the database, verifies that the user is registered, and returns the result to the device. If authentication is successful, the device sends a request to the server to obtain profile information. The server obtains profile information such as the user's learning history, grades, and vocabulary, and sends it in JSON format to the device.

[0940] Initial setup and preparation

[0941] The device analyzes the profile information received from the server and sets the user's learning level and personalized elements (name, favorite character, etc.). After the settings are complete, the learning start screen is displayed to the user.

[0942] Request and generate learning content

[0943] The user selects a subject or topic to study. For example, if the user selects "Edo Period History," the device sends the request to the server. The server references the user's profile information and the request and generates learning content at the appropriate level. Utilizing a generative AI model, the server generates learning materials that combine text, images, and in some cases audio and video, and sends them to the device.

[0944] View and progress through learning content

[0945] The device analyzes the learning materials received from the server and displays them to the user in a format that combines text and images. The user can then read through the displayed learning content to deepen their understanding. If they have any questions, they can enter them in the question box provided on the learning page.

[0946] Receiving questions and generating answers

[0947] The device sends the user's question to the server. The server analyzes the question and collects data to generate an appropriate answer. It uses a generative AI model to generate an answer to the question and prepares the associated text and images. The server sends the generated answer to the device. The device displays the answer received from the server to the user.

[0948] Checking understanding and providing feedback

[0949] When a user wants to check their level of understanding, they press the "Check Understanding" button. The device sends this request to the server. The server generates a quiz or mini-test and sends it to the device. The device displays the quiz or mini-test, and the user answers it. The server analyzes the user's answers, generates feedback, and sends it to the device. The device displays the feedback to the user.

[0950] Recognizing and responding to emotional states

[0951] The device captures the user's facial expressions and voice while they are studying and sends them to the emotion engine. The emotion engine analyzes this data to identify the user's emotional state. The server receives this emotional information and generates an appropriate response. For example, if the user is feeling confused or stressed, the device may adjust the difficulty of the study content or suggest relaxation content. The device then flexibly displays the study content based on this information.

[0952] Learning progress management and recording

[0953] The server records the user's learning progress in real time and stores it in a database. It also prepares to suggest new learning content and review questions suitable for the next study session. The user can check their learning progress and select the next learning topic to continue their study.

[0954] Specific examples

[0955] Example 1: A student is studying a history lesson.

[0956] 1. The user selects "History Lesson" and sends a request to the server.

[0957] 2. The server references the user's profile and generates the appropriate level of historical content.

[0958] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[0959] 4. The user types in the question, "What is Ooku?"

[0960] 5. The server generates a description of "Ooku" in text and images and returns it to the terminal.

[0961] 6. The user reads the answer and deepens their understanding.

[0962] 7. During the learning process, the device captures the user's facial expressions, which are analyzed by the emotion engine. If the user is determined to be confused, the server adjusts the learning content and sends it to the device.

[0963] Example 2: A student solves a math problem

[0964] 1. The user selects "Math Exercises" and sends a request to the server.

[0965] 2. The server generates questions according to the user's level of understanding.

[0966] 3. The question is displayed on the terminal and the user enters the answer.

[0967] 4. The user asks, "Is this correct?"

[0968] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[0969] 6. Users receive feedback and improve their understanding.

[0970] 7. During learning, the device captures audio, the emotion engine detects stress levels, and the server proposes relaxation content and sends it to the device.

[0971] In this way, the self-learning educational tool of the present invention provides a more personalized learning experience by taking into account not only the student's level of understanding and learning progress but also their emotional state, allowing them to learn at their own pace and progress smoothly while increasing their self-esteem.

[0972] The processing flow will be explained below.

[0973] MODE FOR CARRYING OUT THE INVENTION

[0974] Log in and get your profile

[0975] Step 1:

[0976] The user opens the login screen on the device and enters their user ID and password.

[0977] Step 2:

[0978] The terminal generates a request to send the input authentication information to the server, and sends it to the authentication endpoint of the server.

[0979] Step 3:

[0980] The server checks the received authentication information against the database to confirm that the user is registered, and returns the confirmation result to the terminal.

[0981] Initial setup and preparation

[0982] Step 4:

[0983] The terminal receives the authentication result from the server, and if the authentication is successful, sends a request to the server to obtain the user's profile information.

[0984] Step 5:

[0985] The server retrieves the user's profile information (learning history, grades, vocabulary, etc.) from the database and sends it to the terminal in JSON format.

[0986] Step 6:

[0987] The device analyzes the received profile information and sets the user's learning level and personalized elements (name, favorite character, etc.). After the settings are complete, the learning start screen is displayed.

[0988] Request and generate learning content

[0989] Step 7:

[0990] The user selects the subject or topic to study (e.g., the Edo period in history) and presses the "Start Study" button.

[0991] Step 8:

[0992] The terminal sends the user's selection as a request to the server.

[0993] Step 9:

[0994] The server references the user's profile information and the selected subjects and topics and begins collecting data to generate learning content at an appropriate level.

[0995] Step 10:

[0996] The server uses a generative AI model to generate learning materials that combine text, images, and in some cases audio and video, and sends them to the device in JSON format.

[0997] View and progress through learning content

[0998] Step 11:

[0999] The device analyzes the received educational material and displays it to the user in a format that combines text and images.

[1000] Step 12:

[1001] Users can read through the displayed learning content to deepen their understanding, and if they have any questions, they can enter them in the question box on the learning page.

[1002] Receiving questions and generating answers

[1003] Step 13:

[1004] The terminal generates a request to send the user's question to the server and sends it to the server's question endpoint.

[1005] Step 14:

[1006] The server analyzes the received question and begins collecting data to generate an appropriate answer.

[1007] Step 15:

[1008] The server uses a generative AI model to generate answers to questions and provide associated text and images.

[1009] Step 16:

[1010] The server sends the generated response to the device in JSON format.

[1011] Step 17:

[1012] The terminal displays the response received from the server to the user.

[1013] Step 18:

[1014] Users can read the answers to deepen their understanding, and if they have further questions, they can re-enter their question.

[1015] Checking understanding and providing feedback

[1016] Step 19:

[1017] If the user wants to check their level of understanding, they press the "Check understanding" button.

[1018] Step 20:

[1019] The device sends a request for a comprehension check to the server.

[1020] Step 21:

[1021] The server generates quizzes and mini-tests to check the user's understanding and sends them to the terminal in JSON format.

[1022] Step 22:

[1023] The terminal provides an interface for displaying quizzes and mini-tests to the user and receiving answers.

[1024] Step 23:

[1025] Users answer quizzes and mini-tests and send their answers to their terminals.

[1026] Step 24:

[1027] The terminal transmits the user's answer to the server.

[1028] Step 25:

[1029] The server analyzes the received answers, determines whether they are correct, and generates feedback based on the results, sending it to the device in JSON format.

[1030] Step 26:

[1031] The terminal displays the feedback received from the server to the user.

[1032] Recognizing and responding to emotional states

[1033] Step 27:

[1034] The device captures the user's facial expressions and voice during training and sends them to the emotion engine.

[1035] Step 28:

[1036] The emotion engine analyzes the received data to determine the user's emotional state.

[1037] Step 29:

[1038] The server queries the emotion information received from the emotion engine and adjusts the learning content and presentation method based on the user's emotional state.

[1039] Step 30:

[1040] For example, if the user is confused, the server generates content with a lower level of difficulty and sends it to the terminal.If the user is feeling stressed, the server prepares data to suggest relaxation content.

[1041] Step 31:

[1042] The device follows instructions from the server and displays learning and relaxation content.

[1043] Learning progress management and recording

[1044] Step 32:

[1045] The server records the user's learning progress in real time and stores it in a database.

[1046] Step 33:

[1047] The server prepares the data to suggest the user's next study topic and review questions.

[1048] Step 34:

[1049] Users can check their learning progress and select the next learning topic to continue their learning.

[1050] Example 2

[1051] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1052] Conventional educational tools have struggled to provide personalized education tailored to each student's level of understanding and emotional state. Students in the early grades of elementary school, in particular, require visual and linguistic support, and it is necessary to record each student's level of understanding and progress in real time and provide appropriate feedback. Another challenge is providing prompt and accurate answers to students' questions and motivating them to learn. To address these challenges, the present invention proposes a system that provides a personalized learning experience based on each student's level of understanding and emotional state.

[1053] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for generating learning materials on the server based on the individual student's level of understanding; means for accepting questions until the student is satisfied and providing visual and linguistic support; means for analyzing the progress and comprehension of the generated learning materials and providing feedback; means for recording the student's learning progress and suggesting appropriate next learning content; means for capturing the student's facial expressions and voice while studying, identifying the student's emotional state, and adjusting the learning content based on the identified emotional state; and means for setting the user's learning level and personalized elements based on profile information. This allows for an optimized learning experience for each student, enabling effective learning.

[1054] "Generated teaching materials" are educational content including text, images, audio, video, etc. that are dynamically generated based on students' understanding and requests.

[1055] A "server" is a computer system that serves as the core of an educational tool, and is a device that processes data, generates teaching materials, manages user profiles, and so on.

[1056] "Student" refers to a user who uses this educational tool to study, and is primarily intended for students in the lower grades of elementary school.

[1057] "Level of understanding" is an indicator of how well a student has understood the content they have learned, and the difficulty and content of the teaching materials are adjusted based on this.

[1058] A "question" refers to a question or uncertainty that a student has while studying, and is information used to generate an answer to that question.

[1059] "Visual and verbal support" refers to the use of images, text, audio, etc. to supplement learning content and help students understand.

[1060] "Feedback" refers to evaluations and advice provided based on students' learning progress and level of understanding, and is information intended to improve learning outcomes.

[1061] "Study progress" is information that indicates how far a student has progressed in their studies, and the next learning content is suggested based on this.

[1062] "Emotional state" refers to the psychological and emotional state of students, which is analyzed from their facial expressions and voice, and the learning content and methods are adjusted based on this.

[1063] "Profile Information" refers to personal information about a student, such as their learning history, grades, and vocabulary, that is used to provide a personalized learning experience.

[1064] This invention is a multimodal self-learning educational tool designed for early elementary school students. It utilizes a generative AI model and an emotion engine to generate and provide individually tailored learning materials based on each student's level of comprehension, vocabulary, and emotional state. The following describes the system's main hardware and software components and their specific functions.

[1065] Hardware and Software Configuration

[1066] Server: The server runs key software components such as the generative AI model and database. The generative AI model is used to generate personalized learning materials based on user requests and profile information. The database stores and manages user profile information such as learning history, grades, and vocabulary. The emotion engine also analyzes the user's emotional state and provides appropriate feedback based on that information.

[1067] Terminal: A terminal is a device that students directly operate, typically a tablet or PC. The terminal provides a user interface and communicates student operations to the server. The terminal also displays the learning materials and feedback received from the server.

[1068] Users: The users are students who use this system, and are primarily children in the lower grades of elementary school. They select learning content, enter questions, and check their understanding through their devices. Their emotional state is also captured through the device's camera and microphone.

[1069] System Operation

[1070] The system starts when a user logs in on a device and the server checks the user's profile information. If the login is successful, the server sends information such as the user's learning history, grades, and vocabulary to the device, which then analyzes this information to set learning levels and personalized elements.

[1071] The user then selects the subject or topic they want to study, such as "Edo Period History" or "Math Exercises." This request is sent to a server, which uses a generative AI model to generate learning materials at the appropriate level. The generated learning materials, which include text, images, audio, and video, are sent to the device.

[1072] During the learning process, if the user has any questions, they can input them through the device. The questions are sent to the server, and the generative AI model generates an appropriate answer. The answer, which consists of text and images, is sent back to the device and displayed to the user.

[1073] In addition, the user's facial expressions and voice while studying are captured by the device's camera and microphone and sent to the emotion engine. The emotion engine analyzes this, and if the user is confused, the server will provide content adjusted to an appropriate level of difficulty or relaxation content.

[1074] The user's learning progress is recorded on the server in real time, and new learning materials are suggested based on the user's progress and next learning content. For example, review questions are suggested for topics where the user's understanding has deepened.

[1075] Specific examples

[1076] In the case of history classes

[1077] 1. The user selects "History Lesson" and sends a request to the server.

[1078] 2. The server references the user's profile information and generates the appropriate level of history content.

[1079] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[1080] 4. The user types in the question, "What is Ooku?"

[1081] 5. The server generates a description of "Ooku" in text and images and returns it to the terminal.

[1082] 6. The user reads the answer and deepens their understanding.

[1083] For math problems

[1084] 1. The user selects "Math Practice Problems" and sends a request to the server.

[1085] 2. The server generates questions according to the user's level of understanding.

[1086] 3. The question is displayed on the terminal and the user enters the answer.

[1087] 4. The user asks, "Is this correct?"

[1088] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[1089] 6. Users receive feedback and improve their understanding.

[1090] Prompt Sentence Examples

[1091] About History Classes

[1092] User ID: 12345

[1093] Subject: History

[1094] Topic: Edo period

[1095] Question: What is Ooku?

[1096] About Mathematics Problems

[1097] User ID: 67890

[1098] Subject: Mathematics

[1099] Question: Is this correct?

[1100] Problem: Solve the equation x + 5 = 10.

[1101] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1102] Step 1: Log in and get your profile

[1103] The user opens the login screen on their device and enters their user ID and password. The device sends this authentication information to the server. The server accesses the database and verifies the entered authentication information. If the login is successful, the server sends profile information associated with the user's ID (learning history, grades, vocabulary, etc.) in JSON format to the device.

[1104] Input: User ID, Password

[1105] Output: Authentication results, profile information (JSON format)

[1106] Step 2: Initial setup and preparation

[1107] The device analyzes the profile information received from the server and sets personalized elements such as the user's learning level, name, favorite character, etc. Once the settings are complete, the device displays the learning start screen to the user.

[1108] Input: Profile information (JSON format)

[1109] Output: Learning start screen

[1110] Step 3: Request and generate learning content

[1111] The user selects the subject or topic they want to study. For example, they select "Edo Period History." The device sends this request to the server. The server generates a prompt based on the user's profile information and the request, and sends it to the generative AI model. The generative AI model then generates learning materials that combine text, images, and in some cases audio and video, and sends them back to the server.

[1112] Input: User's subject and topic selection

[1113] Output: Generated teaching materials (text, images, etc.)

[1114] Step 4: View and progress through learning content

[1115] The device analyzes the learning materials received from the server and displays them in a format appropriate for the user, such as a combination of text, images, or video. The user can then read through the displayed learning content and progress as needed.

[1116] Input: Generated teaching materials (text, images, etc.)

[1117] Output: Displayed learning content

[1118] Step 5: Receiving questions and generating answers

[1119] If a user has a question while learning, they can enter it into a question box installed on the device. For example, they could enter "What is Ooku?" The device then sends this question to the server. The server analyzes the question, collects information from related documents and databases, and sends it to the generative AI model. The generative AI model then generates an appropriate answer and sends the relevant text and images back to the server.

[1120] Input: User question

[1121] Output: Generated answer (text, image, etc.)

[1122] Step 6: Check for understanding and provide feedback

[1123] If a user wants to check their level of understanding during a study, they press the "Check Understanding" button. The device sends a request to the server. The server generates a quiz or mini-test based on the user's learning content and sends it to the device. The device displays the quiz or mini-test, and the user answers. The server analyzes the answers, generates feedback, and sends it to the device.

[1124] Input: Comprehension check request, user answer

[1125] Output: Feedback (rating, advice)

[1126] Step 7: Recognizing and responding to emotional states

[1127] The device captures the user's facial expressions and voice while they are learning. This data is sent to the emotion engine for analysis. The emotion engine identifies the user's emotional state and sends that information to the server. The server uses this information to adjust the difficulty level and suggest relaxation content. The device displays this and adjusts the learning experience.

[1128] Input: Captured facial and voice data

[1129] Output: Tailored learning content based on emotional state

[1130] Step 8: Track and record your learning progress

[1131] The server records the user's learning progress in real time and stores it in a database. This information is used to suggest new learning content and review questions suitable for the next learning session. The user can check their learning progress on their device, select new topics, and review.

[1132] Input: Learning progress data

[1133] Output: Updated learning progress, suggestions for next learning content

[1134] (Application example 2)

[1135] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1136] Conventional educational and sales systems have had difficulty providing appropriate support based on the level of understanding and emotional state of individual students and customers. Furthermore, they were unable to adequately record students' progress in real time, reflect it in the next lesson, or suggest products based on customer emotion analysis. As a result, there were problems with reduced learning efficiency and customer satisfaction.

[1137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1138] In this invention, the server includes: a means for generating learning materials on the server based on each student's level of comprehension; a means for accepting questions until the student is satisfied and providing visual and linguistic support; a means for analyzing the progress and comprehension of the generated learning materials and providing feedback; a means for recording the student's learning progress and suggesting the next appropriate learning content; a means for acquiring profile information of customers visiting the store and suggesting products suitable for the target individual; a means for visually and linguistically displaying product information using smart glasses and explaining it to customers in real time; a means for analyzing the customer's emotional state and adjusting the product explanation and suggestion method based on that information; and a recording means for collecting customer feedback and utilizing it in future suggestions. This allows students to learn at their own pace, promote self-esteem, and make product suggestions based on the individual customer's needs and emotions.

[1139] "Generated learning materials" refers to learning content generated on the server based on each student's level of understanding.

[1140] "Individual student understanding" refers to the level of knowledge and skills each student currently possesses, and is an important evaluation criterion that is taken into consideration to optimize learning efficiency.

[1141] "Visual and linguistic support" refers to supplementary information provided to learners using text, images, video, audio, etc., and is used to aid comprehension.

[1142] "Means for analyzing progress and understanding and providing feedback" is a function that analyzes data obtained while students are studying the material and provides the next learning step or additional explanation based on that data.

[1143] "Means for recording learning progress and suggesting appropriate next learning content" is a function that records how far a student has progressed in their studies and determines the next learning content based on that data.

[1144] "Means for obtaining profile information of customers visiting the store" is a function for collecting basic information such as the age, interests, and favorite characters of customers who visit the store.

[1145] The "means for proposing products suitable for the target individual" is a function that allows the server to select and propose the most suitable products to the customer based on the collected profile information.

[1146] "Means for visually and verbally displaying product information using smart glasses" refers to a function that displays product information to sales staff via smart glasses and communicates it to customers visually and audibly.

[1147] "Real-time customer explanations" refers to the ability to use smart glasses or other devices to instantly explain products based on current information and context.

[1148] "Means for analyzing the emotional state of customers" refers to technology for reading and analyzing emotions from customers' facial expressions and voices.

[1149] The "means for adjusting the product description and proposal method" is a function for appropriately changing the content and method of product presentation and proposal based on the emotional state of the customer.

[1150] "A means of recording customer feedback and using it in future proposals" is a function that records customer reactions and opinions and uses that information to optimize future proposals.

[1151] This invention uses smart glasses to suggest optimal products to customers and provide services tailored to the needs and emotional state of each individual customer. This system mainly consists of three elements: a server, a terminal (smart glasses), and a user.

[1152] First, when a customer visits the store, the server obtains the customer's profile information entered by the salesperson through the smart glasses. This profile information includes the customer's age, interests, favorite characters, etc. Using this information, the server uses a generative AI model to generate product suggestions that are best suited to the customer and displays them on the smart glasses. This generative AI model generates individually suited products based on past data and each customer's profile.

[1153] A salesperson wearing smart glasses checks the product information sent from the server and explains the product to the customer in real time based on that information. This product information is provided visually and verbally, combining text, images, audio, video, etc.

[1154] The server then uses the smart glasses to capture the customer's facial expressions and voice and transmits them to an emotion analysis engine. The emotion analysis engine then analyzes this data to identify the customer's emotional state. The server then adjusts the product explanation and recommendations in real time based on this emotional information and provides feedback to the salesperson via the smart glasses. For example, if the customer appears confused or stressed, the server can simplify the product explanation or suggest alternative products.

[1155] Finally, the server collects customer feedback and records it for future recommendations, which leads to more personalized recommendations and improved customer satisfaction.

[1156] For example, if a salesperson visits a store and a 10-year-old child enters their profile stating that they are interested in dinosaurs, the server will use the generative AI model to suggest educational toys related to dinosaurs. Information about moving dinosaur figures is displayed on the smart glasses, and the salesperson explains the details. If the child's facial expression shows curiosity during the explanation, the emotion analysis engine will recognize it and suggest a more detailed demonstration. The salesperson will then collect feedback, which will be used to improve the next recommendation.

[1157] Examples of prompts include:

[1158] "My 10-year-old child is interested in dinosaurs. Can you recommend some educational toys?"

[1159] This is the prompt that is input to the generative AI model in the server, and the following data is sent to the sentiment analysis engine:

[1160] "Analyze the child's image and audio data. Provide real-time feedback based on their emotional state."

[1161] Thus, the present invention is a system that enhances the shopping experience by dynamically suggesting products and providing personalized service based on a customer's profile information and emotional state.

[1162] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1163] Step 1:

[1164] The user (salesperson) enters the profile information (age, interests, favorite characters, etc.) of the customer who visits the store through the smart glasses. The input data is sent to the server.

[1165] Step 2:

[1166] The server generates optimal product candidates using a generative AI model based on the received customer profile information. The generated product candidate list is sent to the smart glasses as data including visual and linguistic information (text, images, videos, etc.).

[1167] Step 3:

[1168] The user (salesperson) wearing the smart glasses checks the product candidate list received from the server and explains the product to the customer. Here, detailed product information is displayed in real time and a presentation is made to the customer.

[1169] Step 4:

[1170] The device (smart glasses) captures the customer's facial expressions and voice and transmits them to an emotion analysis engine in real time, where the data is used to identify their emotional state through facial expression analysis and voice emotion analysis algorithms.

[1171] Step 5:

[1172] The server adjusts product explanations and recommendations in real time based on the emotional information obtained from the emotion analysis engine. For example, if the customer is confused, it will simplify the explanation or suggest a different product. The adjusted information is sent as feedback to the smart glasses.

[1173] Step 6:

[1174] Based on the feedback received through the smart glasses, the user (salesperson) responds appropriately to the customer and continues to explain and suggest products.

[1175] Step 7:

[1176] After the sales negotiation, the user (salesperson) collects customer feedback and sends it to the server using the smart glasses. The collected feedback is stored in a database and used for the next proposal.

[1177] Step 8:

[1178] The server processes the data based on the collected feedback and uses it to improve future recommendation algorithms, which will result in more personalized product recommendations for future customers.

[1179] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1180] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1181] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1182] [Third embodiment]

[1183] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1184] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1185] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1186] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1187] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1188] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1189] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1190] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1191] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1192] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1193] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1194] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1195] Summary of the Invention

[1196] This invention is a multimodal self-learning educational tool that utilizes generative AI for early elementary school students. It generates individually tailored learning materials based on the student's level of comprehension and vocabulary, allowing students to ask questions until they are satisfied. Learning content is presented using text, images, and in some cases audio and video, deepening students' understanding through visual and linguistic support. It also records learning progress and provides feedback, enhancing self-esteem.

[1197] Program processing

[1198] Log in and get your profile

[1199] When a user logs in on a device, the server receives the authentication information and retrieves and sends the user's profile information from the database. The profile includes information such as learning history, grades, and vocabulary.

[1200] Initial setup and preparation

[1201] The device analyzes the profile information received from the server and sets the user's learning level and personalized elements (such as name and favorite character). Once the settings are complete, a screen is displayed notifying the user that they are ready to start learning.

[1202] Request and generate learning content

[1203] The user selects a subject or topic to study. For example, if they select "Edo Period History," the request is sent to the server. The server generates learning content at the appropriate level based on the user's profile. The generative AI model creates learning materials by combining text, images, and in some cases audio and video, and sends them to the device.

[1204] View and progress through learning content

[1205] The device analyzes the learning materials received from the server and displays them in a format that combines text and images. The user can then read through the materials. If they do not understand the material properly or have questions, they can enter them in a question box on the learning page.

[1206] Receiving questions and generating answers

[1207] The device sends the user's question to the server, which analyzes the question and generates an appropriate answer. The generative AI model generates an image related to the text and sends it to the device as an answer, which the device displays to the user.

[1208] Checking understanding and providing feedback

[1209] When a user wants to check their level of understanding, they make a request for a comprehension check. The server generates a quiz or mini-test to check the user's level of understanding and sends it to the device. The user answers it, and the answer results are sent back to the server. The server analyzes the answers, generates feedback, and sends it back to the device. The device displays the feedback to the user.

[1210] Learning progress management and recording

[1211] The server records the user's learning progress in real time and saves it in a database. It also prepares to suggest new learning content and review questions suitable for the next study. The user can check the progress and select the next learning topic to continue learning.

[1212] Specific examples

[1213] Example 1: A student is studying a history lesson.

[1214] 1. The user selects "History Lesson" on the device and sends a request to the server.

[1215] 2. The server references the user's profile and generates the appropriate level of historical content.

[1216] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[1217] 4. The user reads through the learning material and enters the question, "What is Ooku?"

[1218] 5. The server generates a description of "Ooku" in text and images and returns it to the terminal.

[1219] 6. The user reads the answer and deepens their understanding.

[1220] Example 2: A student solves a math problem

[1221] 1. The user selects "Math Practice Problems" on the device and sends a request to the server.

[1222] 2. The server generates problems appropriate to the user's level of understanding, such as simple addition and subtraction problems.

[1223] 3. The question is displayed on the terminal and the user enters the answer.

[1224] 4. If the user is unsure of the answer, ask, "Is this correct?"

[1225] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[1226] 6. Users receive feedback and improve their understanding.

[1227] In this way, the self-learning educational tool of the present invention accommodates the learning pace of each student and provides an environment in which students can learn at their own pace while enhancing their self-esteem.

[1228] The processing flow will be explained below.

[1229] Step 1:

[1230] The user opens the login screen on the device and enters authentication information (user ID and password).

[1231] Step 2:

[1232] The terminal generates a request to send the input authentication information to the server, and sends it to the authentication endpoint of the server.

[1233] Step 3:

[1234] The server checks the received authentication information against the database to confirm that the user is registered, and returns the check result to the terminal.

[1235] Step 4:

[1236] The terminal receives the authentication result from the server, and if the authentication is successful, sends a request to the server to obtain the user's profile information.

[1237] Step 5:

[1238] The server retrieves the user's profile information (learning history, grades, vocabulary, etc.) from the database and returns it to the terminal in JSON format.

[1239] Step 6:

[1240] The device analyzes the received profile information and sets the user's learning level and personalized elements (name, favorite character, etc.). After the settings are complete, the learning start screen is displayed to the user.

[1241] Step 7:

[1242] Users select the subject or topic they wish to study (e.g., the Edo period in history) on their device and press the "Start Studying" button.

[1243] Step 8:

[1244] The terminal sends the user's selection as a request to the server.

[1245] Step 9:

[1246] The server references the user's profile information and the request to generate learning content at the appropriate level, using a generative AI model to generate learning materials that combine text, images, and in some cases video and audio.

[1247] Step 10:

[1248] The server sends the generated teaching materials to the terminal in JSON format.

[1249] Step 11:

[1250] The device analyzes the received educational material and displays it to the user in a format that combines text and images.

[1251] Step 12:

[1252] Users can read through the displayed learning content to deepen their understanding, and if they have any questions, they can enter them in the question box provided on the learning page.

[1253] Step 13:

[1254] The terminal generates a request to send the user's question to the server and sends it to the server's question endpoint.

[1255] Step 14:

[1256] The server analyzes the received question and collects data to generate an appropriate answer. It uses a generative AI model to generate an answer to the question and provides relevant text and images.

[1257] Step 15:

[1258] The server sends the generated response in JSON format to the terminal.

[1259] Step 16:

[1260] The terminal displays the response received from the server to the user.

[1261] Step 17:

[1262] Users can read the answers to deepen their understanding and ask again if they have further questions.

[1263] Step 18:

[1264] If the user wants to check their level of understanding, they press the "Check understanding" button.

[1265] Step 19:

[1266] The device sends a request for a comprehension check to the server.

[1267] Step 20:

[1268] The server generates quizzes and mini-tests to check the user's level of understanding, using a generative AI model to select questions at the appropriate level and sending them to the device in JSON format.

[1269] Step 21:

[1270] The terminal provides an interface for displaying quizzes and mini-tests to the user and receiving answers.

[1271] Step 22:

[1272] Users answer quizzes and mini-tests and send their answers to their terminals.

[1273] Step 23:

[1274] The terminal generates a request to send the user's answer to the server and sends it to the server's analysis endpoint.

[1275] Step 24:

[1276] The server analyzes the received answers, determines whether they are correct, and generates feedback based on the analysis results, sending it to the device in JSON format.

[1277] Step 25:

[1278] The terminal displays the feedback received from the server to the user.

[1279] Step 26:

[1280] The server records the user's learning progress in real time and stores the progress in a database.

[1281] Step 27:

[1282] The server prepares data to suggest the user's next study topic and review questions.

[1283] Step 28:

[1284] Users can check their learning progress and select the next learning topic to continue their learning.

[1285] Example 1

[1286] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1287] Conventional learning systems make it difficult to provide individualized instruction tailored to each student's level of understanding and progress, and students often cannot receive immediate answers when they have questions. It is also difficult to grasp learning progress and comprehension and provide appropriate feedback based on that information. Furthermore, personalization of learning content is insufficient, making it impossible to provide optimal learning materials to each student. This can result in a decline in students' self-esteem and motivation to learn, so these issues must be resolved.

[1288] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1289] In this invention, the server includes: means for a user to log in using a terminal, the server receiving authentication information, and retrieving and transmitting the user's profile information from a database; means for the terminal analyzing the received profile information and setting the learner's learning level and personalized elements; means for the learner to select a subject or topic to study and sending a request to the server; means for the server to generate learning content at an appropriate level based on the user's profile information and send it to the terminal; means for the terminal to analyze the learning materials received from the server and display them in a format combining text and images; means for the learner to input a question into a question box and the terminal sending it to the server; means for the server to analyze the question, generate an appropriate answer, and send it to the terminal; and means for the server to record the learner's learning progress in real time and suggest new learning content and review questions appropriate for the next lesson. This makes it possible to provide an optimal learning experience for each student, improving learning efficiency and self-esteem.

[1290] A "terminal" is an electronic device that a user operates, and includes computers, tablets, smartphones, and the like.

[1291] A "server" is a computer system that communicates with terminals over a network and processes, stores, and manages data.

[1292] "Student" refers to a student or user who uses the system to study.

[1293] "Profile information" refers to individual data about a learner, including learning history, grades, vocabulary information, and personalized elements.

[1294] A "database" is a system for centrally managing and storing information, and is accessed by a server to search for and store the necessary data.

[1295] "Authentication information" is information used to identify a particular user and grant that user system access rights, and typically includes a username and password.

[1296] "Learning content" refers to the learning materials and workbooks that learners use for learning, and consists of text, images, audio, video, etc.

[1297] A "generative AI model" is an algorithm that uses artificial intelligence technology to generate learning content and answers.

[1298] "Feedback" refers to the process of providing evaluation and advice on a learner's answers or actions.

[1299] "Progress recording" refers to the quantitative or qualitative recording of the results and progress of a learner's learning activities.

[1300] A "question box" refers to an interface where learners can enter their doubts or queries.

[1301] "Level of understanding" is a measure of how well a learner understands the given teaching materials and tasks.

[1302] "Personalized elements" refer to elements of the learning experience that are customized based on the learner's individual preferences and characteristics.

[1303] This invention is a multimodal self-learning educational system that utilizes generative AI models for early elementary school students. This system provides optimal learning content based on the learner's level of understanding and progress, accepts questions until the learner is satisfied, and provides visual and linguistic support. It also aims to enhance learners' self-esteem by recording their learning progress and providing appropriate feedback.

[1304] System Configuration

[1305] This system is realized mainly using the following hardware and software.

[1306] Device: Computer, tablet, smartphone, etc.

[1307] Server: A computer system that processes and stores data over a network.

[1308] Database: A system that manages the information accessed by a server (e.g., MySQL, PostgreSQL).

[1309] Generative AI models: Artificial intelligence algorithms that generate learning content and answers (e.g., GPT-4).

[1310] Program implementation technologies: Python, Django, Redis, HTML / CSS, JavaScript, etc.

[1311] Program processing

[1312] The system operates as follows:

[1313] 1. Log in and get your profile

[1314] The user enters their username and password on the terminal and presses the login button. The server receives the authentication information, retrieves the user's profile information from the database, and sends it to the terminal.

[1315] 2. Initial Setup and Preparation

[1316] The device analyzes the received profile information and initializes the learner's learning level and personalized elements. Once the setup is complete, a screen will appear informing the learner that they are ready to begin learning.

[1317] 3. Request and generate learning content

[1318] The user selects the subject or topic to study and sends the request to the server, which uses a generative AI model based on the user's profile information to generate learning content at the appropriate level and send it to the device.

[1319] 4. View and progress through learning content

[1320] The device analyzes the learning materials received from the server and displays them in a format that combines text and images. The user can read the materials and, if they do not understand them or have questions, they can enter their questions in the question box.

[1321] 5. Receiving questions and generating answers

[1322] The device sends the user's question to the server, which analyzes the question and generates an appropriate answer using a generative AI model, which then sends it to the device, which then displays it to the user.

[1323] 6. Checking understanding and providing feedback

[1324] When a user wants to check their level of understanding, they make a request to check their level of understanding. The server generates a quiz or mini-test and sends it to the device. The server analyzes the user's answers, generates feedback, sends it back to the device, and the device displays the feedback.

[1325] 7. Learning progress management and recording

[1326] The server records the user's learning progress in real time and suggests new learning content and review questions suitable for the next study. The user can check the progress and select the next learning topic to continue learning.

[1327] Specific examples

[1328] Example 1: Learning a history lesson

[1329] 1. The user selects "History Lesson" on the device and sends a request to the server.

[1330] 2. The server references the user's profile and generates the appropriate level of historical content.

[1331] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[1332] 4. The user types in the question, "What is Ooku?"

[1333] 5. The server generates a description of the "Ooku" and sends it to the terminal.

[1334] 6. The user reads the answer and deepens their understanding.

[1335] Example prompt: "Please explain about the Ooku in the Edo period."

[1336] Example 2: Solving a math problem

[1337] 1. The user selects "Math Practice Problems" on the device and sends a request to the server.

[1338] 2. The server generates questions appropriate to the user's level of understanding.

[1339] 3. The question is displayed on the terminal and the user enters the answer.

[1340] 4. The user asks, "Is this correct?"

[1341] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[1342] 6. Users receive feedback and improve their understanding.

[1343] Example prompt: "What is 2+2?"

[1344] In this way, the system of the present invention responds to the learning pace of each student and provides an environment in which students can learn at their own pace while enhancing their self-esteem.

[1345] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1346] System processing steps

[1347] Step 1:

[1348] The user enters their username and password on the terminal and presses the login button, which sends the authentication information to the server.

[1349] Input: Username, Password

[1350] What it does: Creates an HTTP POST request and sends it to the server

[1351] Output: Authentication information arrives at the server

[1352] Step 2:

[1353] The server receives the login request and authenticates it by checking the database. If authentication is successful, it retrieves the user's profile information from the database and sends it to the terminal.

[1354] Input: Credentials

[1355] Data processing: Execute SQL queries to obtain profile information

[1356] Output: Profile information (learning history, grades, vocabulary information)

[1357] Step 3:

[1358] The device analyzes the received profile information and sets the learner's learning level and personalized elements (name and favorite character).

[1359] Input: Profile Information

[1360] Data processing: Analyzing JSON data and reflecting it in the screen layout

[1361] Output: Notification screen displayed when initial setup is complete

[1362] Step 4:

[1363] The user selects a subject or topic to study and sends the request to the server, for example, selecting "Edo Period in History."

[1364] Input: Select a subject or topic

[1365] What it does: Creates an HTTP POST request and sends it to the server

[1366] Output: Request sent to server successfully

[1367] Step 5:

[1368] The server receives the request and generates learning content at the appropriate level based on the user's profile, sending prompts to a generative AI model to create the learning materials.

[1369] Input: Request details, profile information

[1370] Data processing: Send prompts to the generative AI model and receive generated content

[1371] Output: Learning content (text, images, and possibly audio and video)

[1372] Step 6:

[1373] The server transmits the generated learning content to the terminal.

[1374] Input: Learning content

[1375] Data processing: Convert content into JSON format

[1376] Output: Sending content to a device

[1377] Step 7:

[1378] The device analyzes the teaching materials received from the server and displays them in a format that combines text and images.

[1379] Input: Learning content

[1380] Data processing: JSON data analysis and display

[1381] Output: Display of teaching materials

[1382] Step 8:

[1383] As the user reads through the learning materials, if they feel they do not understand the material properly or have questions, they can enter their questions in the question box and press the send button.

[1384] Input: Question

[1385] What it does: Enter a question into the question box, generate an HTTP POST request, and send it to the server

[1386] Output: Question sent to server successfully

[1387] Step 9:

[1388] The server receives the question, analyzes the question using a generative AI model, and generates an appropriate answer.

[1389] Input: Question

[1390] Data processing: Send the question to a generative AI model to generate an answer

[1391] Output: Answer (text and associated image)

[1392] Step 10:

[1393] The server sends the generated response to the terminal.

[1394] Input: Response data

[1395] Data processing: Convert the response data into JSON format

[1396] Output: Sends the answer to the terminal

[1397] Step 11:

[1398] The terminal displays the received response to the user.

[1399] Input: Response data

[1400] Data processing: JSON data analysis and display

[1401] Output: Display the answer

[1402] Step 12:

[1403] If a user wants to check their level of understanding, they can request a comprehension check.

[1404] Input: Comprehension Check Request

[1405] What it does: Creates an HTTP POST request and sends it to the server

[1406] Output: Comprehension check request sent to server

[1407] Step 13:

[1408] The server generates quizzes and mini-tests to check comprehension and sends them to the device.

[1409] Input: Comprehension Check Request

[1410] Data processing: generating quizzes and mini-tests

[1411] Output: Quiz and mini-test (question data)

[1412] Step 14:

[1413] The device displays the received quizzes and mini-tests to the learner, who answers them. The answers are then sent back to the server.

[1414] Input: Quizzes and mini-tests

[1415] Data manipulation: Problem presentation

[1416] Output: Answer submitted by the learner

[1417] Step 15:

[1418] The server analyzes the response and generates feedback that is sent back to the device.

[1419] Input: Learner's answer

[1420] Data processing: analyzing responses and generating feedback

[1421] Output: Feedback data

[1422] Step 16:

[1423] The device displays the feedback to the learner.

[1424] Input: Feedback data

[1425] Data processing: Displaying feedback

[1426] Output: Display feedback

[1427] Step 17:

[1428] The server records the learner's learning progress in real time and stores it in a database. It also suggests new learning content and review questions suitable for the next lesson.

[1429] Input: Learning progress information

[1430] Data processing: Recording progress information and generating next learning content

[1431] Output: Learning progress data and new learning content suggestions

[1432] This allows us to provide learners with a personalized educational environment and support their independent learning.

[1433] (Application example 1)

[1434] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1435] The purpose of this invention is to provide personalized education for early elementary school students based on their individual learning progress and level of understanding. In particular, to create an environment where students can continue learning outside of school, we have identified a challenge in providing interesting content in real time using smart devices to increase their motivation to learn. Furthermore, there is a need to develop a system that uses a generative AI model to quickly provide appropriate answers.

[1436] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1437] In this invention, the server generates learning materials based on each student's level of understanding on the server, accepts questions until the student is satisfied, and includes means for providing visual and linguistic support, means for analyzing the progress and comprehension of the generated learning materials and providing feedback, means for recording the student's learning progress and suggesting the next appropriate learning content, means for displaying information in real time using a smart device and providing interesting content, and means for generating and presenting answers to user questions using a generative AI model. This allows students to receive appropriate support according to their individual learning needs, and enables them to progress in their studies while obtaining interesting information in real time in a physical store even in a learning environment outside of school.

[1438] "Generated teaching materials" are educational contents that are dynamically created on the server based on the individual student's level of understanding.

[1439] "Visual and verbal supports" are text, images, audio and video that complement and deepen students' understanding of their learning.

[1440] "Feedback" is information provided by analyzing the progress of the created learning materials and the student's level of understanding, and providing information on areas for improvement in the learning content and the next learning steps.

[1441] "Learning progress" is data that records how far a student has progressed in the learning process or their level of understanding.

[1442] "Appropriate learning content" is teaching materials that include what students should learn next based on their current level of understanding and learning progress.

[1443] A "smart device" is an electronic device that displays information in real time and provides interesting content.

[1444] A "generative AI model" is an algorithm that uses artificial intelligence technology to generate content such as text, images, and audio.

[1445] "Profile information" is individual data including a student's learning history, grades, vocabulary, etc.

[1446] "Accepting questions" means that the system receives any doubts or follow-up questions that arise while students are studying and provides appropriate answers to them.

[1447] "Displaying information in real time" means providing content instantly so that students can instantly obtain interesting information through their smart devices.

[1448] In the present invention, an educational tool for elementary school students is realized using a smart device. Specific embodiments will be described below.

[1449] Hardware and software used

[1450] Hardware:

[1451] Smart glasses: Use Google Glass or Microsoft HoloLens as common smart devices.

[1452] Server: Use a high-performance cloud server (e.g., AWS EC2).

[1453] Camera: Uses the camera built into the smart glasses.

[1454] Microphone: Use the microphone built into the smart glasses.

[1455] software:

[1456] User authentication: Use OAuth 2.0.

[1457] Database: MySQL database is used.

[1458] Generative AI model: Uses OpenAI GPT-3.

[1459] Image Recognition: Uses TensorFlow and OpenCV.

[1460] Processing Description

[1461] 1. User authentication and profile acquisition:

[1462] A user puts on the smart glasses and scans a QR code to log in. The server verifies the user's authentication information using OAuth 2.0 and retrieves the student's profile information from a MySQL database, including the student's learning history, grades, and vocabulary.

[1463] 2. Initial setup and preparation:

[1464] The smart glasses will then use the profile information to set the student's name and favorite character, creating a personalized learning environment. Once setup is complete, the smart glasses will notify the student that they are ready to begin learning.

[1465] 3. Creation and display of teaching materials:

[1466] When a student looks at a product or exhibit through the smart glasses, the camera captures the image and sends it to a server. The server analyzes this data using TensorFlow and OpenCV and generates tailored learning content using a generative AI model (GPT-3). Based on the student's profile, the generative AI model creates optimal learning materials and sends them to the smart glasses. The learning materials are displayed in the form of text, images, audio, etc.

[1467] 4. Question and Answer Generation:

[1468] If a student has a question while reading the material, they can input it using the microphone in the smart glasses. The question is then sent to the server, where the generative AI model generates an appropriate answer, which is then displayed in text and audio on the smart glasses.

[1469] 5. Check for understanding and provide feedback:

[1470] When a student wants to check their understanding, they switch to test or quiz mode. The server generates an appropriate quiz and sends it to the smart glasses, and the student answers it. The answers are sent to the server, and feedback is generated and sent to the smart glasses.

[1471] Specific examples

[1472] Examples from the history section:

[1473] When a student gazes at a sword in the history exhibit, the camera in the smart glasses recognizes the sword. The generative AI then provides historical background information in text and images, such as "A sword used during the Sengoku period. Used by a Sengoku warlord..." When a student asks, "What is a Sengoku warlord?" the generative AI model explains, "It refers to a samurai who was active during the Sengoku period."

[1474] Prompt Sentence Examples

[1475] This is an educational tool for early elementary school students. Please explain in kind words the following questions:

[1476] Student Information:

[1477] Age: 7

[1478] Interests: History

[1479] Vocabulary level: Beginner

[1480] Question: What is a Sengoku warlord?

[1481] In this way, by linking the server and smart glasses, students can learn in real time and get their questions answered immediately, which improves the effectiveness of education and provides a learning environment that is tailored to each student.

[1482] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1483] Step 1:

[1484] User logs in and gets profile

[1485] Input: User scans the QR code with smart glasses.

[1486] Processing: The smart glasses (terminal) sends the login information obtained from the QR code to the server and performs user authentication using OAuth 2.0.

[1487] Output: The server retrieves the user's profile information (such as learning history, grades, vocabulary, etc.) from the MySQL database and sends it to the smart glasses.

[1488] Step 2:

[1489] Initial setup and preparation

[1490] Input: User profile information retrieved from the server.

[1491] Processing: The smart glasses (terminal) analyzes the profile information and sets the user's name, favorite character, and learning level.

[1492] Output: The smart glasses notify the user that the initial setup is complete.

[1493] Step 3:

[1494] Creation of teaching materials and preparation for display

[1495] Input: Images of products or exhibits that the user sees through the smart glasses.

[1496] Processing: The smart glasses' camera captures images and sends them to the server, which then analyzes them using TensorFlow and OpenCV and generates learning content based on the user profile using a generative AI model (GPT-3).

[1497] Output: The generated teaching materials (text, images, audio, etc.) are sent from the server to the smart glasses and displayed.

[1498] Step 4:

[1499] Accepting questions and generating answers

[1500] Input: A question that the user types through the microphone in the smart glasses.

[1501] Processing: The smart glasses send the question to the server, which uses a generative AI model to analyze the question and generate an appropriate answer.

[1502] Output: The answer (text, associated images, and audio) is sent from the server to the smart glasses and displayed.

[1503] Step 5:

[1504] Checking understanding and providing feedback

[1505] Input: A request for a quiz or test to test the user's understanding.

[1506] Processing: The server generates quizzes and mini-tests based on the user's profile information and sends them to the smart glasses. The user answers the quiz and sends the results to the server. The server analyzes the results and generates feedback.

[1507] Output: Feedback (level of understanding, suggestions for next study, etc.) is sent from the server to the smart glasses and displayed.

[1508] Step 6:

[1509] Record your learning progress

[1510] Input: The results of the user studying the material and answering the quiz.

[1511] Processing: The server records the user's learning progress in real time and stores it in a database. At the same time, it prepares the next learning content proposal.

[1512] Output: The server provides the user with a progress report and suggestions for the next learning content.

[1513] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1514] Summary of the Invention

[1515] This invention is a multimodal self-learning educational tool for early elementary school students that utilizes generative AI and an emotion engine. It generates individually tailored learning materials based on the student's level of comprehension, vocabulary, and even emotional state, allowing students to ask questions until they are satisfied. Learning content is presented using text, images, audio, and video, deepening students' understanding through visual and linguistic support. It also recognizes the student's emotional state and adjusts the presentation and content of the learning materials accordingly, further enhancing the learning experience. It also has the ability to record learning progress and provide feedback.

[1516] Program processing

[1517] Log in and get your profile

[1518] The user opens the login screen on the device and enters authentication information (user ID and password). The device sends this information to the server. The server compares the authentication information with the database, verifies that the user is registered, and returns the result to the device. If authentication is successful, the device sends a request to the server to obtain profile information. The server obtains profile information such as the user's learning history, grades, and vocabulary, and sends it in JSON format to the device.

[1519] Initial setup and preparation

[1520] The device analyzes the profile information received from the server and sets the user's learning level and personalized elements (name, favorite character, etc.). After the settings are complete, the learning start screen is displayed to the user.

[1521] Request and generate learning content

[1522] The user selects a subject or topic to study. For example, if the user selects "Edo Period History," the device sends the request to the server. The server references the user's profile information and the request and generates learning content at the appropriate level. Utilizing a generative AI model, the server generates learning materials that combine text, images, and in some cases audio and video, and sends them to the device.

[1523] View and progress through learning content

[1524] The device analyzes the learning materials received from the server and displays them to the user in a format that combines text and images. The user can then read through the displayed learning content to deepen their understanding. If they have any questions, they can enter them in the question box provided on the learning page.

[1525] Receiving questions and generating answers

[1526] The device sends the user's question to the server. The server analyzes the question and collects data to generate an appropriate answer. It uses a generative AI model to generate an answer to the question and prepares the associated text and images. The server sends the generated answer to the device. The device displays the answer received from the server to the user.

[1527] Checking understanding and providing feedback

[1528] When a user wants to check their level of understanding, they press the "Check Understanding" button. The device sends this request to the server. The server generates a quiz or mini-test and sends it to the device. The device displays the quiz or mini-test, and the user answers it. The server analyzes the user's answers, generates feedback, and sends it to the device. The device displays the feedback to the user.

[1529] Recognizing and responding to emotional states

[1530] The device captures the user's facial expressions and voice while they are studying and sends them to the emotion engine. The emotion engine analyzes this data to identify the user's emotional state. The server receives this emotional information and generates an appropriate response. For example, if the user is feeling confused or stressed, the device may adjust the difficulty of the study content or suggest relaxation content. The device then flexibly displays the study content based on this information.

[1531] Learning progress management and recording

[1532] The server records the user's learning progress in real time and stores it in a database. It also prepares to suggest new learning content and review questions suitable for the next study session. The user can check their learning progress and select the next learning topic to continue their study.

[1533] Specific examples

[1534] Example 1: A student is studying a history lesson.

[1535] 1. The user selects "History Lesson" and sends a request to the server.

[1536] 2. The server references the user's profile and generates the appropriate level of historical content.

[1537] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[1538] 4. The user types in the question, "What is Ooku?"

[1539] 5. The server generates a description of "Ooku" in text and images and returns it to the terminal.

[1540] 6. The user reads the answer and deepens their understanding.

[1541] 7. During the learning process, the device captures the user's facial expressions, which are analyzed by the emotion engine. If the user is determined to be confused, the server adjusts the learning content and sends it to the device.

[1542] Example 2: A student solves a math problem

[1543] 1. The user selects "Math Exercises" and sends a request to the server.

[1544] 2. The server generates questions according to the user's level of understanding.

[1545] 3. The question is displayed on the terminal and the user enters the answer.

[1546] 4. The user asks, "Is this correct?"

[1547] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[1548] 6. Users receive feedback and improve their understanding.

[1549] 7. During learning, the device captures audio, the emotion engine detects stress levels, and the server proposes relaxation content and sends it to the device.

[1550] In this way, the self-learning educational tool of the present invention provides a more personalized learning experience by taking into account not only the student's level of understanding and learning progress but also their emotional state, allowing them to learn at their own pace and progress smoothly while increasing their self-esteem.

[1551] The processing flow will be explained below.

[1552] MODE FOR CARRYING OUT THE INVENTION

[1553] Log in and get your profile

[1554] Step 1:

[1555] The user opens the login screen on the device and enters their user ID and password.

[1556] Step 2:

[1557] The terminal generates a request to send the input authentication information to the server, and sends it to the authentication endpoint of the server.

[1558] Step 3:

[1559] The server checks the received authentication information against the database to confirm that the user is registered, and returns the confirmation result to the terminal.

[1560] Initial setup and preparation

[1561] Step 4:

[1562] The terminal receives the authentication result from the server, and if the authentication is successful, sends a request to the server to obtain the user's profile information.

[1563] Step 5:

[1564] The server retrieves the user's profile information (learning history, grades, vocabulary, etc.) from the database and sends it to the terminal in JSON format.

[1565] Step 6:

[1566] The device analyzes the received profile information and sets the user's learning level and personalized elements (name, favorite character, etc.). After the settings are complete, the learning start screen is displayed.

[1567] Request and generate learning content

[1568] Step 7:

[1569] The user selects the subject or topic to study (e.g., the Edo period in history) and presses the "Start Study" button.

[1570] Step 8:

[1571] The terminal sends the user's selection as a request to the server.

[1572] Step 9:

[1573] The server references the user's profile information and the selected subjects and topics and begins collecting data to generate learning content at an appropriate level.

[1574] Step 10:

[1575] The server uses a generative AI model to generate learning materials that combine text, images, and in some cases audio and video, and sends them to the device in JSON format.

[1576] View and progress through learning content

[1577] Step 11:

[1578] The device analyzes the received educational material and displays it to the user in a format that combines text and images.

[1579] Step 12:

[1580] Users can read through the displayed learning content to deepen their understanding, and if they have any questions, they can enter them in the question box on the learning page.

[1581] Receiving questions and generating answers

[1582] Step 13:

[1583] The terminal generates a request to send the user's question to the server and sends it to the server's question endpoint.

[1584] Step 14:

[1585] The server analyzes the received question and begins collecting data to generate an appropriate answer.

[1586] Step 15:

[1587] The server uses a generative AI model to generate answers to questions and provide associated text and images.

[1588] Step 16:

[1589] The server sends the generated response to the device in JSON format.

[1590] Step 17:

[1591] The terminal displays the response received from the server to the user.

[1592] Step 18:

[1593] Users can read the answers to deepen their understanding, and if they have further questions, they can re-enter their question.

[1594] Checking understanding and providing feedback

[1595] Step 19:

[1596] If the user wants to check their level of understanding, they press the "Check understanding" button.

[1597] Step 20:

[1598] The device sends a request for a comprehension check to the server.

[1599] Step 21:

[1600] The server generates quizzes and mini-tests to check the user's understanding and sends them to the terminal in JSON format.

[1601] Step 22:

[1602] The terminal provides an interface for displaying quizzes and mini-tests to the user and receiving answers.

[1603] Step 23:

[1604] Users answer quizzes and mini-tests and send their answers to their terminals.

[1605] Step 24:

[1606] The terminal transmits the user's answer to the server.

[1607] Step 25:

[1608] The server analyzes the received answers, determines whether they are correct, and generates feedback based on the results, sending it to the device in JSON format.

[1609] Step 26:

[1610] The terminal displays the feedback received from the server to the user.

[1611] Recognizing and responding to emotional states

[1612] Step 27:

[1613] The device captures the user's facial expressions and voice during training and sends them to the emotion engine.

[1614] Step 28:

[1615] The emotion engine analyzes the received data to determine the user's emotional state.

[1616] Step 29:

[1617] The server queries the emotion information received from the emotion engine and adjusts the learning content and presentation method based on the user's emotional state.

[1618] Step 30:

[1619] For example, if the user is confused, the server generates content with a lower level of difficulty and sends it to the terminal.If the user is feeling stressed, the server prepares data to suggest relaxation content.

[1620] Step 31:

[1621] The device follows instructions from the server and displays learning and relaxation content.

[1622] Learning progress management and recording

[1623] Step 32:

[1624] The server records the user's learning progress in real time and stores it in a database.

[1625] Step 33:

[1626] The server prepares the data to suggest the user's next study topic and review questions.

[1627] Step 34:

[1628] Users can check their learning progress and select the next learning topic to continue their learning.

[1629] Example 2

[1630] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1631] Conventional educational tools have struggled to provide personalized education tailored to each student's level of understanding and emotional state. Students in the early grades of elementary school, in particular, require visual and linguistic support, and it is necessary to record each student's level of understanding and progress in real time and provide appropriate feedback. Another challenge is providing prompt and accurate answers to students' questions and motivating them to learn. To address these challenges, the present invention proposes a system that provides a personalized learning experience based on each student's level of understanding and emotional state.

[1632] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for generating learning materials on the server based on the individual student's level of understanding; means for accepting questions until the student is satisfied and providing visual and linguistic support; means for analyzing the progress and comprehension of the generated learning materials and providing feedback; means for recording the student's learning progress and suggesting appropriate next learning content; means for capturing the student's facial expressions and voice while studying, identifying the student's emotional state, and adjusting the learning content based on the identified emotional state; and means for setting the user's learning level and personalized elements based on profile information. This allows for an optimized learning experience for each student, enabling effective learning.

[1633] "Generated teaching materials" are educational content including text, images, audio, video, etc. that are dynamically generated based on students' understanding and requests.

[1634] A "server" is a computer system that serves as the core of an educational tool, and is a device that processes data, generates teaching materials, manages user profiles, and so on.

[1635] "Student" refers to a user who uses this educational tool to study, and is primarily intended for students in the lower grades of elementary school.

[1636] "Level of understanding" is an indicator of how well a student has understood the content they have learned, and the difficulty and content of the teaching materials are adjusted based on this.

[1637] A "question" refers to a question or uncertainty that a student has while studying, and is information used to generate an answer to that question.

[1638] "Visual and verbal support" refers to the use of images, text, audio, etc. to supplement learning content and help students understand.

[1639] "Feedback" refers to evaluations and advice provided based on students' learning progress and level of understanding, and is information intended to improve learning outcomes.

[1640] "Study progress" is information that indicates how far a student has progressed in their studies, and the next learning content is suggested based on this.

[1641] "Emotional state" refers to the psychological and emotional state of students, which is analyzed from their facial expressions and voice, and the learning content and methods are adjusted based on this.

[1642] "Profile Information" refers to personal information about a student, such as their learning history, grades, and vocabulary, that is used to provide a personalized learning experience.

[1643] This invention is a multimodal self-learning educational tool designed for early elementary school students. It utilizes a generative AI model and an emotion engine to generate and provide individually tailored learning materials based on each student's level of comprehension, vocabulary, and emotional state. The following describes the system's main hardware and software components and their specific functions.

[1644] Hardware and Software Configuration

[1645] Server: The server runs key software components such as the generative AI model and database. The generative AI model is used to generate personalized learning materials based on user requests and profile information. The database stores and manages user profile information such as learning history, grades, and vocabulary. The emotion engine also analyzes the user's emotional state and provides appropriate feedback based on that information.

[1646] Terminal: A terminal is a device that students directly operate, typically a tablet or PC. The terminal provides a user interface and communicates student operations to the server. The terminal also displays the learning materials and feedback received from the server.

[1647] Users: The users are students who use this system, and are primarily children in the lower grades of elementary school. They select learning content, enter questions, and check their understanding through their devices. Their emotional state is also captured through the device's camera and microphone.

[1648] System Operation

[1649] The system starts when a user logs in on a device and the server checks the user's profile information. If the login is successful, the server sends information such as the user's learning history, grades, and vocabulary to the device, which then analyzes this information to set learning levels and personalized elements.

[1650] The user then selects the subject or topic they want to study, such as "Edo Period History" or "Math Exercises." This request is sent to a server, which uses a generative AI model to generate learning materials at the appropriate level. The generated learning materials, which include text, images, audio, and video, are sent to the device.

[1651] During the learning process, if the user has any questions, they can input them through the device. The questions are sent to the server, and the generative AI model generates an appropriate answer. The answer, which consists of text and images, is sent back to the device and displayed to the user.

[1652] In addition, the user's facial expressions and voice while studying are captured by the device's camera and microphone and sent to the emotion engine. The emotion engine analyzes this, and if the user is confused, the server will provide content adjusted to an appropriate level of difficulty or relaxation content.

[1653] The user's learning progress is recorded on the server in real time, and new learning materials are suggested based on the user's progress and next learning content. For example, review questions are suggested for topics where the user's understanding has deepened.

[1654] Specific examples

[1655] In the case of history classes

[1656] 1. The user selects "History Lesson" and sends a request to the server.

[1657] 2. The server references the user's profile information and generates the appropriate level of history content.

[1658] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[1659] 4. The user types in the question, "What is Ooku?"

[1660] 5. The server generates a description of "Ooku" in text and images and returns it to the terminal.

[1661] 6. The user reads the answer and deepens their understanding.

[1662] For math problems

[1663] 1. The user selects "Math Practice Problems" and sends a request to the server.

[1664] 2. The server generates questions according to the user's level of understanding.

[1665] 3. The question is displayed on the terminal and the user enters the answer.

[1666] 4. The user asks, "Is this correct?"

[1667] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[1668] 6. Users receive feedback and improve their understanding.

[1669] Prompt Sentence Examples

[1670] About History Classes

[1671] User ID: 12345

[1672] Subject: History

[1673] Topic: Edo period

[1674] Question: What is Ooku?

[1675] About Mathematics Problems

[1676] User ID: 67890

[1677] Subject: Mathematics

[1678] Question: Is this correct?

[1679] Problem: Solve the equation x + 5 = 10.

[1680] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1681] Step 1: Log in and get your profile

[1682] The user opens the login screen on their device and enters their user ID and password. The device sends this authentication information to the server. The server accesses the database and verifies the entered authentication information. If the login is successful, the server sends profile information associated with the user's ID (learning history, grades, vocabulary, etc.) in JSON format to the device.

[1683] Input: User ID, Password

[1684] Output: Authentication results, profile information (JSON format)

[1685] Step 2: Initial setup and preparation

[1686] The device analyzes the profile information received from the server and sets personalized elements such as the user's learning level, name, favorite character, etc. Once the settings are complete, the device displays the learning start screen to the user.

[1687] Input: Profile information (JSON format)

[1688] Output: Learning start screen

[1689] Step 3: Request and generate learning content

[1690] The user selects the subject or topic they want to study. For example, they select "Edo Period History." The device sends this request to the server. The server generates a prompt based on the user's profile information and the request, and sends it to the generative AI model. The generative AI model then generates learning materials that combine text, images, and in some cases audio and video, and sends them back to the server.

[1691] Input: User's subject and topic selection

[1692] Output: Generated teaching materials (text, images, etc.)

[1693] Step 4: View and progress through learning content

[1694] The device analyzes the learning materials received from the server and displays them in a format appropriate for the user, such as a combination of text, images, or video. The user can then read through the displayed learning content and progress as needed.

[1695] Input: Generated teaching materials (text, images, etc.)

[1696] Output: Displayed learning content

[1697] Step 5: Receiving questions and generating answers

[1698] If a user has a question while learning, they can enter it into a question box installed on the device. For example, they could enter "What is Ooku?" The device then sends this question to the server. The server analyzes the question, collects information from related documents and databases, and sends it to the generative AI model. The generative AI model then generates an appropriate answer and sends the relevant text and images back to the server.

[1699] Input: User question

[1700] Output: Generated answer (text, image, etc.)

[1701] Step 6: Check for understanding and provide feedback

[1702] If a user wants to check their level of understanding during a study, they press the "Check Understanding" button. The device sends a request to the server. The server generates a quiz or mini-test based on the user's learning content and sends it to the device. The device displays the quiz or mini-test, and the user answers. The server analyzes the answers, generates feedback, and sends it to the device.

[1703] Input: Comprehension check request, user answer

[1704] Output: Feedback (rating, advice)

[1705] Step 7: Recognizing and responding to emotional states

[1706] The device captures the user's facial expressions and voice while they are learning. This data is sent to the emotion engine for analysis. The emotion engine identifies the user's emotional state and sends that information to the server. The server uses this information to adjust the difficulty level and suggest relaxation content. The device displays this and adjusts the learning experience.

[1707] Input: Captured facial and voice data

[1708] Output: Tailored learning content based on emotional state

[1709] Step 8: Track and record your learning progress

[1710] The server records the user's learning progress in real time and stores it in a database. This information is used to suggest new learning content and review questions suitable for the next learning session. The user can check their learning progress on their device, select new topics, and review.

[1711] Input: Learning progress data

[1712] Output: Updated learning progress, suggestions for next learning content

[1713] (Application example 2)

[1714] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1715] Conventional educational and sales systems have had difficulty providing appropriate support based on the level of understanding and emotional state of individual students and customers. Furthermore, they were unable to adequately record students' progress in real time, reflect it in the next lesson, or suggest products based on customer emotion analysis. As a result, there were problems with reduced learning efficiency and customer satisfaction.

[1716] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1717] In this invention, the server includes: a means for generating learning materials on the server based on each student's level of comprehension; a means for accepting questions until the student is satisfied and providing visual and linguistic support; a means for analyzing the progress and comprehension of the generated learning materials and providing feedback; a means for recording the student's learning progress and suggesting the next appropriate learning content; a means for acquiring profile information of customers visiting the store and suggesting products suitable for the target individual; a means for visually and linguistically displaying product information using smart glasses and explaining it to customers in real time; a means for analyzing the customer's emotional state and adjusting the product explanation and suggestion method based on that information; and a recording means for collecting customer feedback and utilizing it in future suggestions. This allows students to learn at their own pace, promote self-esteem, and make product suggestions based on the individual customer's needs and emotions.

[1718] "Generated learning materials" refers to learning content generated on the server based on each student's level of understanding.

[1719] "Individual student understanding" refers to the level of knowledge and skills each student currently possesses, and is an important evaluation criterion that is taken into consideration to optimize learning efficiency.

[1720] "Visual and linguistic support" refers to supplementary information provided to learners using text, images, video, audio, etc., and is used to aid comprehension.

[1721] "Means for analyzing progress and understanding and providing feedback" is a function that analyzes data obtained while students are studying the material and provides the next learning step or additional explanation based on that data.

[1722] "Means for recording learning progress and suggesting appropriate next learning content" is a function that records how far a student has progressed in their studies and determines the next learning content based on that data.

[1723] "Means for obtaining profile information of customers visiting the store" is a function for collecting basic information such as the age, interests, and favorite characters of customers who visit the store.

[1724] The "means for proposing products suitable for the target individual" is a function that allows the server to select and propose the most suitable products to the customer based on the collected profile information.

[1725] "Means for visually and verbally displaying product information using smart glasses" refers to a function that displays product information to sales staff via smart glasses and communicates it to customers visually and audibly.

[1726] "Real-time customer explanations" refers to the ability to use smart glasses or other devices to instantly explain products based on current information and context.

[1727] "Means for analyzing the emotional state of customers" refers to technology for reading and analyzing emotions from customers' facial expressions and voices.

[1728] The "means for adjusting the product description and proposal method" is a function for appropriately changing the content and method of product presentation and proposal based on the emotional state of the customer.

[1729] "A means of recording customer feedback and using it in future proposals" is a function that records customer reactions and opinions and uses that information to optimize future proposals.

[1730] This invention uses smart glasses to suggest optimal products to customers and provide services tailored to the needs and emotional state of each individual customer. This system mainly consists of three elements: a server, a terminal (smart glasses), and a user.

[1731] First, when a customer visits the store, the server obtains the customer's profile information entered by the salesperson through the smart glasses. This profile information includes the customer's age, interests, favorite characters, etc. Using this information, the server uses a generative AI model to generate product suggestions that are best suited to the customer and displays them on the smart glasses. This generative AI model generates individually suited products based on past data and each customer's profile.

[1732] A salesperson wearing smart glasses checks the product information sent from the server and explains the product to the customer in real time based on that information. This product information is provided visually and verbally, combining text, images, audio, video, etc.

[1733] The server then uses the smart glasses to capture the customer's facial expressions and voice and transmits them to an emotion analysis engine. The emotion analysis engine then analyzes this data to identify the customer's emotional state. The server then adjusts the product explanation and recommendations in real time based on this emotional information and provides feedback to the salesperson via the smart glasses. For example, if the customer appears confused or stressed, the server can simplify the product explanation or suggest alternative products.

[1734] Finally, the server collects customer feedback and records it for future recommendations, which leads to more personalized recommendations and improved customer satisfaction.

[1735] For example, if a salesperson visits a store and a 10-year-old child enters their profile stating that they are interested in dinosaurs, the server will use the generative AI model to suggest educational toys related to dinosaurs. Information about moving dinosaur figures is displayed on the smart glasses, and the salesperson explains the details. If the child's facial expression shows curiosity during the explanation, the emotion analysis engine will recognize it and suggest a more detailed demonstration. The salesperson will then collect feedback, which will be used to improve the next recommendation.

[1736] Examples of prompts include:

[1737] "My 10-year-old child is interested in dinosaurs. Can you recommend some educational toys?"

[1738] This is the prompt that is input to the generative AI model in the server, and the following data is sent to the sentiment analysis engine:

[1739] "Analyze the child's image and audio data. Provide real-time feedback based on their emotional state."

[1740] Thus, the present invention is a system that enhances the shopping experience by dynamically suggesting products and providing personalized service based on a customer's profile information and emotional state.

[1741] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1742] Step 1:

[1743] The user (salesperson) enters the profile information (age, interests, favorite characters, etc.) of the customer who visits the store through the smart glasses. The input data is sent to the server.

[1744] Step 2:

[1745] The server generates optimal product candidates using a generative AI model based on the received customer profile information. The generated product candidate list is sent to the smart glasses as data including visual and linguistic information (text, images, videos, etc.).

[1746] Step 3:

[1747] The user (salesperson) wearing the smart glasses checks the product candidate list received from the server and explains the product to the customer. Here, detailed product information is displayed in real time and a presentation is made to the customer.

[1748] Step 4:

[1749] The device (smart glasses) captures the customer's facial expressions and voice and transmits them to an emotion analysis engine in real time, where the data is used to identify their emotional state through facial expression analysis and voice emotion analysis algorithms.

[1750] Step 5:

[1751] The server adjusts product explanations and recommendations in real time based on the emotional information obtained from the emotion analysis engine. For example, if the customer is confused, it will simplify the explanation or suggest a different product. The adjusted information is sent as feedback to the smart glasses.

[1752] Step 6:

[1753] Based on the feedback received through the smart glasses, the user (salesperson) responds appropriately to the customer and continues to explain and suggest products.

[1754] Step 7:

[1755] After the sales negotiation, the user (salesperson) collects customer feedback and sends it to the server using the smart glasses. The collected feedback is stored in a database and used for the next proposal.

[1756] Step 8:

[1757] The server processes the data based on the collected feedback and uses it to improve future recommendation algorithms, which will result in more personalized product recommendations for future customers.

[1758] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1759] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1760] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1761] [Fourth embodiment]

[1762] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1763] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1764] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1765] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1766] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1767] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1768] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1769] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1770] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1771] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1772] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1773] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1774] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1775] Summary of the Invention

[1776] This invention is a multimodal self-learning educational tool that utilizes generative AI for early elementary school students. It generates individually tailored learning materials based on the student's level of comprehension and vocabulary, allowing students to ask questions until they are satisfied. Learning content is presented using text, images, and in some cases audio and video, deepening students' understanding through visual and linguistic support. It also records learning progress and provides feedback, enhancing self-esteem.

[1777] Program processing

[1778] Log in and get your profile

[1779] When a user logs in on a device, the server receives the authentication information and retrieves and sends the user's profile information from the database. The profile includes information such as learning history, grades, and vocabulary.

[1780] Initial setup and preparation

[1781] The device analyzes the profile information received from the server and sets the user's learning level and personalized elements (such as name and favorite character). Once the settings are complete, a screen is displayed notifying the user that they are ready to start learning.

[1782] Request and generate learning content

[1783] The user selects a subject or topic to study. For example, if they select "Edo Period History," the request is sent to the server. The server generates learning content at the appropriate level based on the user's profile. The generative AI model creates learning materials by combining text, images, and in some cases audio and video, and sends them to the device.

[1784] View and progress through learning content

[1785] The device analyzes the learning materials received from the server and displays them in a format that combines text and images. The user can then read through the materials. If they do not understand the material properly or have questions, they can enter them in a question box on the learning page.

[1786] Receiving questions and generating answers

[1787] The device sends the user's question to the server, which analyzes the question and generates an appropriate answer. The generative AI model generates an image related to the text and sends it to the device as an answer, which the device displays to the user.

[1788] Checking understanding and providing feedback

[1789] When a user wants to check their level of understanding, they make a request for a comprehension check. The server generates a quiz or mini-test to check the user's level of understanding and sends it to the device. The user answers it, and the answer results are sent back to the server. The server analyzes the answers, generates feedback, and sends it back to the device. The device displays the feedback to the user.

[1790] Learning progress management and recording

[1791] The server records the user's learning progress in real time and saves it in a database. It also prepares to suggest new learning content and review questions suitable for the next study. The user can check the progress and select the next learning topic to continue learning.

[1792] Specific examples

[1793] Example 1: A student is studying a history lesson.

[1794] 1. The user selects "History Lesson" on the device and sends a request to the server.

[1795] 2. The server references the user's profile and generates the appropriate level of historical content.

[1796] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[1797] 4. The user reads through the learning material and enters the question, "What is Ooku?"

[1798] 5. The server generates a description of "Ooku" in text and images and returns it to the terminal.

[1799] 6. The user reads the answer and deepens their understanding.

[1800] Example 2: A student solves a math problem

[1801] 1. The user selects "Math Practice Problems" on the device and sends a request to the server.

[1802] 2. The server generates problems appropriate to the user's level of understanding, such as simple addition and subtraction problems.

[1803] 3. The question is displayed on the terminal and the user enters the answer.

[1804] 4. If the user is unsure of the answer, ask, "Is this correct?"

[1805] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[1806] 6. Users receive feedback and improve their understanding.

[1807] In this way, the self-learning educational tool of the present invention accommodates the learning pace of each student and provides an environment in which students can learn at their own pace while enhancing their self-esteem.

[1808] The processing flow will be explained below.

[1809] Step 1:

[1810] The user opens the login screen on the device and enters authentication information (user ID and password).

[1811] Step 2:

[1812] The terminal generates a request to send the input authentication information to the server, and sends it to the authentication endpoint of the server.

[1813] Step 3:

[1814] The server checks the received authentication information against the database to confirm that the user is registered, and returns the check result to the terminal.

[1815] Step 4:

[1816] The terminal receives the authentication result from the server, and if the authentication is successful, sends a request to the server to obtain the user's profile information.

[1817] Step 5:

[1818] The server retrieves the user's profile information (learning history, grades, vocabulary, etc.) from the database and returns it to the terminal in JSON format.

[1819] Step 6:

[1820] The device analyzes the received profile information and sets the user's learning level and personalized elements (name, favorite character, etc.). After the settings are complete, the learning start screen is displayed to the user.

[1821] Step 7:

[1822] Users select the subject or topic they wish to study (e.g., the Edo period in history) on their device and press the "Start Studying" button.

[1823] Step 8:

[1824] The terminal sends the user's selection as a request to the server.

[1825] Step 9:

[1826] The server references the user's profile information and the request to generate learning content at the appropriate level, using a generative AI model to generate learning materials that combine text, images, and in some cases video and audio.

[1827] Step 10:

[1828] The server sends the generated teaching materials to the terminal in JSON format.

[1829] Step 11:

[1830] The device analyzes the received educational material and displays it to the user in a format that combines text and images.

[1831] Step 12:

[1832] Users can read through the displayed learning content to deepen their understanding, and if they have any questions, they can enter them in the question box provided on the learning page.

[1833] Step 13:

[1834] The terminal generates a request to send the user's question to the server and sends it to the server's question endpoint.

[1835] Step 14:

[1836] The server analyzes the received question and collects data to generate an appropriate answer. It uses a generative AI model to generate an answer to the question and provides relevant text and images.

[1837] Step 15:

[1838] The server sends the generated response in JSON format to the terminal.

[1839] Step 16:

[1840] The terminal displays the response received from the server to the user.

[1841] Step 17:

[1842] Users can read the answers to deepen their understanding and ask again if they have further questions.

[1843] Step 18:

[1844] If the user wants to check their level of understanding, they press the "Check understanding" button.

[1845] Step 19:

[1846] The device sends a request for a comprehension check to the server.

[1847] Step 20:

[1848] The server generates quizzes and mini-tests to check the user's level of understanding, using a generative AI model to select questions at the appropriate level and sending them to the device in JSON format.

[1849] Step 21:

[1850] The terminal provides an interface for displaying quizzes and mini-tests to the user and receiving answers.

[1851] Step 22:

[1852] Users answer quizzes and mini-tests and send their answers to their terminals.

[1853] Step 23:

[1854] The terminal generates a request to send the user's answer to the server and sends it to the server's analysis endpoint.

[1855] Step 24:

[1856] The server analyzes the received answers, determines whether they are correct, and generates feedback based on the analysis results, sending it to the device in JSON format.

[1857] Step 25:

[1858] The terminal displays the feedback received from the server to the user.

[1859] Step 26:

[1860] The server records the user's learning progress in real time and stores the progress in a database.

[1861] Step 27:

[1862] The server prepares data to suggest the user's next study topic and review questions.

[1863] Step 28:

[1864] Users can check their learning progress and select the next learning topic to continue their learning.

[1865] Example 1

[1866] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1867] Conventional learning systems make it difficult to provide individualized instruction tailored to each student's level of understanding and progress, and students often cannot receive immediate answers when they have questions. It is also difficult to grasp learning progress and comprehension and provide appropriate feedback based on that information. Furthermore, personalization of learning content is insufficient, making it impossible to provide optimal learning materials to each student. This can result in a decline in students' self-esteem and motivation to learn, so these issues must be resolved.

[1868] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1869] In this invention, the server includes: means for a user to log in using a terminal, the server receiving authentication information, and retrieving and transmitting the user's profile information from a database; means for the terminal analyzing the received profile information and setting the learner's learning level and personalized elements; means for the learner to select a subject or topic to study and sending a request to the server; means for the server to generate learning content at an appropriate level based on the user's profile information and send it to the terminal; means for the terminal to analyze the learning materials received from the server and display them in a format combining text and images; means for the learner to input a question into a question box and the terminal sending it to the server; means for the server to analyze the question, generate an appropriate answer, and send it to the terminal; and means for the server to record the learner's learning progress in real time and suggest new learning content and review questions appropriate for the next lesson. This makes it possible to provide an optimal learning experience for each student, improving learning efficiency and self-esteem.

[1870] A "terminal" is an electronic device that a user operates, and includes computers, tablets, smartphones, and the like.

[1871] A "server" is a computer system that communicates with terminals over a network and processes, stores, and manages data.

[1872] "Student" refers to a student or user who uses the system to study.

[1873] "Profile information" refers to individual data about a learner, including learning history, grades, vocabulary information, and personalized elements.

[1874] A "database" is a system for centrally managing and storing information, and is accessed by a server to search for and store the necessary data.

[1875] "Authentication information" is information used to identify a particular user and grant that user system access rights, and typically includes a username and password.

[1876] "Learning content" refers to the learning materials and workbooks that learners use for learning, and consists of text, images, audio, video, etc.

[1877] A "generative AI model" is an algorithm that uses artificial intelligence technology to generate learning content and answers.

[1878] "Feedback" refers to the process of providing evaluation and advice on a learner's answers or actions.

[1879] "Progress recording" refers to the quantitative or qualitative recording of the results and progress of a learner's learning activities.

[1880] A "question box" refers to an interface where learners can enter their doubts or queries.

[1881] "Level of understanding" is a measure of how well a learner understands the given teaching materials and tasks.

[1882] "Personalized elements" refer to elements of the learning experience that are customized based on the learner's individual preferences and characteristics.

[1883] This invention is a multimodal self-learning educational system that utilizes generative AI models for early elementary school students. This system provides optimal learning content based on the learner's level of understanding and progress, accepts questions until the learner is satisfied, and provides visual and linguistic support. It also aims to enhance learners' self-esteem by recording their learning progress and providing appropriate feedback.

[1884] System Configuration

[1885] This system is realized mainly using the following hardware and software.

[1886] Device: Computer, tablet, smartphone, etc.

[1887] Server: A computer system that processes and stores data over a network.

[1888] Database: A system that manages the information accessed by a server (e.g., MySQL, PostgreSQL).

[1889] Generative AI models: Artificial intelligence algorithms that generate learning content and answers (e.g., GPT-4).

[1890] Program implementation technologies: Python, Django, Redis, HTML / CSS, JavaScript, etc.

[1891] Program processing

[1892] The system operates as follows:

[1893] 1. Log in and get your profile

[1894] The user enters their username and password on the terminal and presses the login button. The server receives the authentication information, retrieves the user's profile information from the database, and sends it to the terminal.

[1895] 2. Initial Setup and Preparation

[1896] The device analyzes the received profile information and initializes the learner's learning level and personalized elements. Once the setup is complete, a screen will appear informing the learner that they are ready to begin learning.

[1897] 3. Request and generate learning content

[1898] The user selects the subject or topic to study and sends the request to the server, which uses a generative AI model based on the user's profile information to generate learning content at the appropriate level and send it to the device.

[1899] 4. View and progress through learning content

[1900] The device analyzes the learning materials received from the server and displays them in a format that combines text and images. The user can read the materials and, if they do not understand them or have questions, they can enter their questions in the question box.

[1901] 5. Receiving questions and generating answers

[1902] The device sends the user's question to the server, which analyzes the question and generates an appropriate answer using a generative AI model, which then sends it to the device, which then displays it to the user.

[1903] 6. Checking understanding and providing feedback

[1904] When a user wants to check their level of understanding, they make a request to check their level of understanding. The server generates a quiz or mini-test and sends it to the device. The server analyzes the user's answers, generates feedback, sends it back to the device, and the device displays the feedback.

[1905] 7. Learning progress management and recording

[1906] The server records the user's learning progress in real time and suggests new learning content and review questions suitable for the next study. The user can check the progress and select the next learning topic to continue learning.

[1907] Specific examples

[1908] Example 1: Learning a history lesson

[1909] 1. The user selects "History Lesson" on the device and sends a request to the server.

[1910] 2. The server references the user's profile and generates the appropriate level of historical content.

[1911] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[1912] 4. The user types in the question, "What is Ooku?"

[1913] 5. The server generates a description of the "Ooku" and sends it to the terminal.

[1914] 6. The user reads the answer and deepens their understanding.

[1915] Example prompt: "Please explain about the Ooku in the Edo period."

[1916] Example 2: Solving a math problem

[1917] 1. The user selects "Math Practice Problems" on the device and sends a request to the server.

[1918] 2. The server generates questions appropriate to the user's level of understanding.

[1919] 3. The question is displayed on the terminal and the user enters the answer.

[1920] 4. The user asks, "Is this correct?"

[1921] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[1922] 6. Users receive feedback and improve their understanding.

[1923] Example prompt: "What is 2+2?"

[1924] In this way, the system of the present invention responds to the learning pace of each student and provides an environment in which students can learn at their own pace while enhancing their self-esteem.

[1925] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1926] System processing steps

[1927] Step 1:

[1928] The user enters their username and password on the terminal and presses the login button, which sends the authentication information to the server.

[1929] Input: Username, Password

[1930] What it does: Creates an HTTP POST request and sends it to the server

[1931] Output: Authentication information arrives at the server

[1932] Step 2:

[1933] The server receives the login request and authenticates it by checking the database. If authentication is successful, it retrieves the user's profile information from the database and sends it to the terminal.

[1934] Input: Credentials

[1935] Data processing: Execute SQL queries to obtain profile information

[1936] Output: Profile information (learning history, grades, vocabulary information)

[1937] Step 3:

[1938] The device analyzes the received profile information and sets the learner's learning level and personalized elements (name and favorite character).

[1939] Input: Profile Information

[1940] Data processing: Analyzing JSON data and reflecting it in the screen layout

[1941] Output: Notification screen displayed when initial setup is complete

[1942] Step 4:

[1943] The user selects a subject or topic to study and sends the request to the server, for example, selecting "Edo Period in History."

[1944] Input: Select a subject or topic

[1945] What it does: Creates an HTTP POST request and sends it to the server

[1946] Output: Request sent to server successfully

[1947] Step 5:

[1948] The server receives the request and generates learning content at the appropriate level based on the user's profile, sending prompts to a generative AI model to create the learning materials.

[1949] Input: Request details, profile information

[1950] Data processing: Send prompts to the generative AI model and receive generated content

[1951] Output: Learning content (text, images, and possibly audio and video)

[1952] Step 6:

[1953] The server transmits the generated learning content to the terminal.

[1954] Input: Learning content

[1955] Data processing: Convert content into JSON format

[1956] Output: Sending content to a device

[1957] Step 7:

[1958] The device analyzes the teaching materials received from the server and displays them in a format that combines text and images.

[1959] Input: Learning content

[1960] Data processing: JSON data analysis and display

[1961] Output: Display of teaching materials

[1962] Step 8:

[1963] As the user reads through the learning materials, if they feel they do not understand the material properly or have questions, they can enter their questions in the question box and press the send button.

[1964] Input: Question

[1965] What it does: Enter a question into the question box, generate an HTTP POST request, and send it to the server

[1966] Output: Question sent to server successfully

[1967] Step 9:

[1968] The server receives the question, analyzes the question using a generative AI model, and generates an appropriate answer.

[1969] Input: Question

[1970] Data processing: Send the question to a generative AI model to generate an answer

[1971] Output: Answer (text and associated image)

[1972] Step 10:

[1973] The server sends the generated response to the terminal.

[1974] Input: Response data

[1975] Data processing: Convert the response data into JSON format

[1976] Output: Sends the answer to the terminal

[1977] Step 11:

[1978] The terminal displays the received response to the user.

[1979] Input: Response data

[1980] Data processing: JSON data analysis and display

[1981] Output: Display the answer

[1982] Step 12:

[1983] If a user wants to check their level of understanding, they can request a comprehension check.

[1984] Input: Comprehension Check Request

[1985] What it does: Creates an HTTP POST request and sends it to the server

[1986] Output: Comprehension check request sent to server

[1987] Step 13:

[1988] The server generates quizzes and mini-tests to check comprehension and sends them to the device.

[1989] Input: Comprehension Check Request

[1990] Data processing: generating quizzes and mini-tests

[1991] Output: Quiz and mini-test (question data)

[1992] Step 14:

[1993] The device displays the received quizzes and mini-tests to the learner, who answers them. The answers are then sent back to the server.

[1994] Input: Quizzes and mini-tests

[1995] Data manipulation: Problem presentation

[1996] Output: Answer submitted by the learner

[1997] Step 15:

[1998] The server analyzes the response and generates feedback that is sent back to the device.

[1999] Input: Learner's answer

[2000] Data processing: analyzing responses and generating feedback

[2001] Output: Feedback data

[2002] Step 16:

[2003] The device displays the feedback to the learner.

[2004] Input: Feedback data

[2005] Data processing: Displaying feedback

[2006] Output: Display feedback

[2007] Step 17:

[2008] The server records the learner's learning progress in real time and stores it in a database. It also suggests new learning content and review questions suitable for the next lesson.

[2009] Input: Learning progress information

[2010] Data processing: Recording progress information and generating next learning content

[2011] Output: Learning progress data and new learning content suggestions

[2012] This allows us to provide learners with a personalized educational environment and support their independent learning.

[2013] (Application example 1)

[2014] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2015] The purpose of this invention is to provide personalized education for early elementary school students based on their individual learning progress and level of understanding. In particular, to create an environment where students can continue learning outside of school, we have identified a challenge in providing interesting content in real time using smart devices to increase their motivation to learn. Furthermore, there is a need to develop a system that uses a generative AI model to quickly provide appropriate answers.

[2016] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2017] In this invention, the server generates learning materials based on each student's level of understanding on the server, accepts questions until the student is satisfied, and includes means for providing visual and linguistic support, means for analyzing the progress and comprehension of the generated learning materials and providing feedback, means for recording the student's learning progress and suggesting the next appropriate learning content, means for displaying information in real time using a smart device and providing interesting content, and means for generating and presenting answers to user questions using a generative AI model. This allows students to receive appropriate support according to their individual learning needs, and enables them to progress in their studies while obtaining interesting information in real time in a physical store even in a learning environment outside of school.

[2018] "Generated teaching materials" are educational contents that are dynamically created on the server based on the individual student's level of understanding.

[2019] "Visual and verbal supports" are text, images, audio and video that complement and deepen students' understanding of their learning.

[2020] "Feedback" is information provided by analyzing the progress of the created learning materials and the student's level of understanding, and providing information on areas for improvement in the learning content and the next learning steps.

[2021] "Learning progress" is data that records how far a student has progressed in the learning process or their level of understanding.

[2022] "Appropriate learning content" is teaching materials that include what students should learn next based on their current level of understanding and learning progress.

[2023] A "smart device" is an electronic device that displays information in real time and provides interesting content.

[2024] A "generative AI model" is an algorithm that uses artificial intelligence technology to generate content such as text, images, and audio.

[2025] "Profile information" is individual data including a student's learning history, grades, vocabulary, etc.

[2026] "Accepting questions" means that the system receives any doubts or follow-up questions that arise while students are studying and provides appropriate answers to them.

[2027] "Displaying information in real time" means providing content instantly so that students can instantly obtain interesting information through their smart devices.

[2028] In the present invention, an educational tool for elementary school students is realized using a smart device. Specific embodiments will be described below.

[2029] Hardware and software used

[2030] Hardware:

[2031] Smart glasses: Use Google Glass or Microsoft HoloLens as common smart devices.

[2032] Server: Use a high-performance cloud server (e.g., AWS EC2).

[2033] Camera: Uses the camera built into the smart glasses.

[2034] Microphone: Use the microphone built into the smart glasses.

[2035] software:

[2036] User authentication: Use OAuth 2.0.

[2037] Database: MySQL database is used.

[2038] Generative AI model: Uses OpenAI GPT-3.

[2039] Image Recognition: Uses TensorFlow and OpenCV.

[2040] Processing Description

[2041] 1. User authentication and profile acquisition:

[2042] A user puts on the smart glasses and scans a QR code to log in. The server verifies the user's authentication information using OAuth 2.0 and retrieves the student's profile information from a MySQL database, including the student's learning history, grades, and vocabulary.

[2043] 2. Initial setup and preparation:

[2044] The smart glasses will then use the profile information to set the student's name and favorite character, creating a personalized learning environment. Once setup is complete, the smart glasses will notify the student that they are ready to begin learning.

[2045] 3. Creation and display of teaching materials:

[2046] When a student looks at a product or exhibit through the smart glasses, the camera captures the image and sends it to a server. The server analyzes this data using TensorFlow and OpenCV and generates tailored learning content using a generative AI model (GPT-3). Based on the student's profile, the generative AI model creates optimal learning materials and sends them to the smart glasses. The learning materials are displayed in the form of text, images, audio, etc.

[2047] 4. Question and Answer Generation:

[2048] If a student has a question while reading the material, they can input it using the microphone in the smart glasses. The question is then sent to the server, where the generative AI model generates an appropriate answer, which is then displayed in text and audio on the smart glasses.

[2049] 5. Check for understanding and provide feedback:

[2050] When a student wants to check their understanding, they switch to test or quiz mode. The server generates an appropriate quiz and sends it to the smart glasses, and the student answers it. The answers are sent to the server, and feedback is generated and sent to the smart glasses.

[2051] Specific examples

[2052] Examples from the history section:

[2053] When a student gazes at a sword in the history exhibit, the camera in the smart glasses recognizes the sword. The generative AI then provides historical background information in text and images, such as "A sword used during the Sengoku period. Used by a Sengoku warlord..." When a student asks, "What is a Sengoku warlord?" the generative AI model explains, "It refers to a samurai who was active during the Sengoku period."

[2054] Prompt Sentence Examples

[2055] This is an educational tool for early elementary school students. Please explain in kind words the following questions:

[2056] Student Information:

[2057] Age: 7

[2058] Interests: History

[2059] Vocabulary level: Beginner

[2060] Question: What is a Sengoku warlord?

[2061] In this way, by linking the server and smart glasses, students can learn in real time and get their questions answered immediately, which improves the effectiveness of education and provides a learning environment that is tailored to each student.

[2062] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2063] Step 1:

[2064] User logs in and gets profile

[2065] Input: User scans the QR code with smart glasses.

[2066] Processing: The smart glasses (terminal) sends the login information obtained from the QR code to the server and performs user authentication using OAuth 2.0.

[2067] Output: The server retrieves the user's profile information (such as learning history, grades, vocabulary, etc.) from the MySQL database and sends it to the smart glasses.

[2068] Step 2:

[2069] Initial setup and preparation

[2070] Input: User profile information retrieved from the server.

[2071] Processing: The smart glasses (terminal) analyzes the profile information and sets the user's name, favorite character, and learning level.

[2072] Output: The smart glasses notify the user that the initial setup is complete.

[2073] Step 3:

[2074] Creation of teaching materials and preparation for display

[2075] Input: Images of products or exhibits that the user sees through the smart glasses.

[2076] Processing: The smart glasses' camera captures images and sends them to the server, which then analyzes them using TensorFlow and OpenCV and generates learning content based on the user profile using a generative AI model (GPT-3).

[2077] Output: The generated teaching materials (text, images, audio, etc.) are sent from the server to the smart glasses and displayed.

[2078] Step 4:

[2079] Accepting questions and generating answers

[2080] Input: A question that the user types through the microphone in the smart glasses.

[2081] Processing: The smart glasses send the question to the server, which uses a generative AI model to analyze the question and generate an appropriate answer.

[2082] Output: The answer (text, associated images, and audio) is sent from the server to the smart glasses and displayed.

[2083] Step 5:

[2084] Checking understanding and providing feedback

[2085] Input: A request for a quiz or test to test the user's understanding.

[2086] Processing: The server generates quizzes and mini-tests based on the user's profile information and sends them to the smart glasses. The user answers the quiz and sends the results to the server. The server analyzes the results and generates feedback.

[2087] Output: Feedback (level of understanding, suggestions for next study, etc.) is sent from the server to the smart glasses and displayed.

[2088] Step 6:

[2089] Record your learning progress

[2090] Input: The results of the user studying the material and answering the quiz.

[2091] Processing: The server records the user's learning progress in real time and stores it in a database. At the same time, it prepares the next learning content proposal.

[2092] Output: The server provides the user with a progress report and suggestions for the next learning content.

[2093] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2094] Summary of the Invention

[2095] This invention is a multimodal self-learning educational tool for early elementary school students that utilizes generative AI and an emotion engine. It generates individually tailored learning materials based on the student's level of comprehension, vocabulary, and even emotional state, allowing students to ask questions until they are satisfied. Learning content is presented using text, images, audio, and video, deepening students' understanding through visual and linguistic support. It also recognizes the student's emotional state and adjusts the presentation and content of the learning materials accordingly, further enhancing the learning experience. It also has the ability to record learning progress and provide feedback.

[2096] Program processing

[2097] Log in and get your profile

[2098] The user opens the login screen on the device and enters authentication information (user ID and password). The device sends this information to the server. The server compares the authentication information with the database, verifies that the user is registered, and returns the result to the device. If authentication is successful, the device sends a request to the server to obtain profile information. The server obtains profile information such as the user's learning history, grades, and vocabulary, and sends it in JSON format to the device.

[2099] Initial setup and preparation

[2100] The device analyzes the profile information received from the server and sets the user's learning level and personalized elements (name, favorite character, etc.). After the settings are complete, the learning start screen is displayed to the user.

[2101] Request and generate learning content

[2102] The user selects a subject or topic to study. For example, if the user selects "Edo Period History," the device sends the request to the server. The server references the user's profile information and the request and generates learning content at the appropriate level. Utilizing a generative AI model, the server generates learning materials that combine text, images, and in some cases audio and video, and sends them to the device.

[2103] View and progress through learning content

[2104] The device analyzes the learning materials received from the server and displays them to the user in a format that combines text and images. The user can then read through the displayed learning content to deepen their understanding. If they have any questions, they can enter them in the question box provided on the learning page.

[2105] Receiving questions and generating answers

[2106] The device sends the user's question to the server. The server analyzes the question and collects data to generate an appropriate answer. It uses a generative AI model to generate an answer to the question and prepares the associated text and images. The server sends the generated answer to the device. The device displays the answer received from the server to the user.

[2107] Checking understanding and providing feedback

[2108] When a user wants to check their level of understanding, they press the "Check Understanding" button. The device sends this request to the server. The server generates a quiz or mini-test and sends it to the device. The device displays the quiz or mini-test, and the user answers it. The server analyzes the user's answers, generates feedback, and sends it to the device. The device displays the feedback to the user.

[2109] Recognizing and responding to emotional states

[2110] The device captures the user's facial expressions and voice while they are studying and sends them to the emotion engine. The emotion engine analyzes this data to identify the user's emotional state. The server receives this emotional information and generates an appropriate response. For example, if the user is feeling confused or stressed, the device may adjust the difficulty of the study content or suggest relaxation content. The device then flexibly displays the study content based on this information.

[2111] Learning progress management and recording

[2112] The server records the user's learning progress in real time and stores it in a database. It also prepares to suggest new learning content and review questions suitable for the next study session. The user can check their learning progress and select the next learning topic to continue their study.

[2113] Specific examples

[2114] Example 1: A student is studying a history lesson.

[2115] 1. The user selects "History Lesson" and sends a request to the server.

[2116] 2. The server references the user's profile and generates the appropriate level of historical content.

[2117] 3. Teaching materials introducing "Life in the Edo Period" will be displayed on the device.

[2118] 4. The user types in the question, "What is Ooku?"

[2119] 5. The server generates a description of "Ooku" in text and images and returns it to the terminal.

[2120] 6. The user reads the answer and deepens their understanding.

[2121] 7. During the learning process, the device captures the user's facial expressions, which are analyzed by the emotion engine. If the user is determined to be confused, the server adjusts the learning content and sends it to the device.

[2122] Example 2: A student solves a math problem

[2123] 1. The user selects "Math Exercises" and sends a request to the server.

[2124] 2. The server generates questions according to the user's level of understanding.

[2125] 3. The question is displayed on the terminal and the user enters the answer.

[2126] 4. The user asks, "Is this correct?"

[2127] 5. The server checks the accuracy of the answer, generates feedback, and sends it to the device.

[2128] 6. Users receive feedback and improve their understanding.

[2129] 7. During learning, the device captures audio, the emotion engine detects stress levels, and the server proposes relaxation content and sends it to the device.

[2130] In this way, the self-learning educational tool of the present invention provides a more personalized learning experience by taking into account not only the student's level of understanding and learning progress but also their emotional state, allowing them to learn at their own pace and progress smoothly while increasing their self-esteem.

[2131] The processing flow will be explained below.

[2132] MODE FOR CARRYING OUT THE INVENTION

[2133] Log in and get your profile

[2134] Step 1:

[2135] The user opens the login screen on the device and enters their user ID and password.

[2136] Step 2:

[2137] The terminal generates a request to send the input authentication information to the server, and sends it to the authentication endpoint of the server.

[2138] Step 3:

[2139] The server checks the received authentication information against the database to confirm that the user is registered, and returns the confirmation result to the terminal.

[2140] Initial setup and preparation

[2141] Step 4:

[2142] The terminal receives the authentication result from the server, and if the authentication is successful, sends a request to the server to obtain the user's profile information.

[2143] Step 5:

[2144] The server retrieves the user's profile information (learning history, grades, vocabulary, etc.) from the database and sends it to the terminal in JSON format.

[2145] Step 6:

[2146] The device analyzes the received profile information and sets the user's learning level and personalized elements (name, favorite character, etc.). After the settings are complete, the learning start screen is displayed.

[2147] Request and generate learning content

[2148] Step 7:

[2149] The user selects the subject or topic to study (e.g., the Edo period in history) and presses the "Start Study" button.

[2150] Step 8:

[2151] The terminal sends the user's selection as a request to the server.

[2152] Step 9:

[2153] The server references the user's profile information and the selected subjects and topics and begins collecting data to generate learning content at an appropriate level.

[2154] Step 10:

[2155] The server uses a generative AI model to generate learning materials that combine text, images, and in some cases audio and video, and sends them to the device in JSON format.

[2156] View and progress through learning content

[2157] Step 11:

[2158] The device analyzes the received educational material and displays it to the user in a format that combines text and images.

[2159] Step 12:

[2160] Users can read through the displayed learning content to deepen their understanding, and if they have any questions, they can enter them in the question box on the learning page.

[2161] Receiving questions and generating answers

[2162] Step 13:

[2163] The terminal generates a request to send the user's question to the server and sends it to the server's question endpoint.

[2164] Step 14:

[2165] The server analyzes the received question and begins collecting data to generate an appropriate answer.

[2166] Step 15:

[2167] The server uses a generative AI model to generate answers to questions and provide associated text and images.

[2168] Step 16:

[2169] The server sends the generated response to the device in JSON format.

[2170] Step 17:

[2171] The terminal displays the response received from the server to the user.

[2172] Step 18:

[2173] Users can read the answers to deepen their understanding, and if they have further questions, they can re-enter their question.

[2174] Checking understanding and providing feedback

[2175] Step 19:

[2176] If the user wants to check their level of understanding, they press the "Check understanding" button.

[2177] Step 20:

[2178] The device sends a request for a comprehension check to the server.

[2179] Step 21:

[2180] The server generates quizzes and mini-tests to check the user's understanding and sends them to the terminal in JSON format.

[2181] Step 22:

[2182] The terminal provides an interface for displaying quizzes and mini-tests to the user and receiving answers.

[2183] Step 23:

[2184] Users answer quizzes and mini-tests and send their answers to their terminals.

[2185] Step 24:

[2186] The terminal transmits the user's answer to the server.

[2187] Step 25:

[2188] The server analyzes the received answers, determines whether they are correct, and generates feedback based on the results, sending it to the device in JSON format.

[2189] Step 26:

[2190] The terminal displays the feedback received from the server to the user.

[2191] Recognizing and responding to emotional states

[2192] Step 27:

[2193] The device captures the user's facial expressions and voice during training and sends them to the emotion engine.

[2194] Step 28:

[2195] The emotion engine analyzes the received data to determine the user's emotional state.

[2196] Step 29:

[2197] The server queries the emotion information received from the emotion engine and adjusts the learning content and presentation method based on the user's emotional state.

[2198] Step 30:

[2199] For example, if the user is confused, the server generates content with a lower level of difficulty and sends it to the terminal.If the user is feeling stressed, the server prepares data to suggest relaxation content.

[2200] Step 31:

[2201] The device follows instructions from the server and displays learning and relaxation content.

[2202] Learning progress management and recording

[2203] Step 32:

[2204] The server records the user's learning progress in real time and stores it in a database.

[2205] Step 33:

[2206] The server prepares the data to suggest the user's next study topic and review questions.

[2207] Step 34:

[2208] Users can check their learning progress and select the next learning topic to continue their learning.

[2209] Example 2

[2210] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2211] Conventional educational tools have struggled to provide personalized education tailored to each student's level of understanding and emotional state. Students in the early grades of elementary school, in particular, require visual and linguistic support, and it is necessary to record each student's level of understanding and progress in real time and provide appropriate feedback. Another challenge is providing prompt and accurate answers to students' questions and motivating them to learn. To address these challenges, the present invention proposes a system that provides a personalized learning experience based on each student's level of understanding and emotional state.

[2212] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for generating learning materials on the server based on the individual student's level of understanding; means for accepting questions until the student is satisfied and providing visual and linguistic support; means for analyzing the progress and comprehension of the generated learning materials and providing feedback; means for recording the student's learning progress and suggesting appropriate next learning content; means for capturing the student's facial expressions and voice while studying, identifying the student's emotional state, and adjusting the learning content based on the identified emotional state; and means for setting the user's learning level and personalized elements based on profile information. This allows for an optimized learning experience for each student, enabling effective learning.

[2213] "Generated teaching materials" are educational content including text, images, audio, video, etc. that are dynamically generated based on students' understanding and requests.

[2214] A "server" is a computer system that serves as the core of an educational tool, and is a device that processes data, generates teaching materials, manages user profiles, and so on.

[2215] "Student" refers to a user who uses this educational tool to study, and is primarily intended for students in the lower grades of elementary school.

[2216] "Level of understanding" is an indicator of how well a student has understood the content they have learned, and the difficulty and content of the teaching materials are adjusted based on this.

[2217] A "question" refers to a question or uncertainty that a student has while studying, and is information used to generate an answer to that question.

[2218] "Visual and verbal support" refers to the use of images, text, audio, etc. to supplement learning content and help students understand.

[2219] "Feedback" refers to evaluations and advice provided based on students' learning progress and level of understanding, and is information intended to improve learning outcomes.

[2220] "Study progress" is information that indicates how far a student has progressed in their studies, and the next learning content is suggested based on this.

[2221] "Emotional state" refers to the psychological and emotional state of students, which is analyzed from their facial expressions and voice, and the learning content and methods are adjusted based on this.

[2222] "Profile Information" refers to personal information about a student, such as their learning history, grades, and vocabulary, that is used to provide a personalized learning experience.

[2223] This invention is a multimodal self-learning educational tool designed for early elementary school students. It utilizes a generative AI model and an emotion engine to generate and provide individually tailored learning materials based on each student's level of comprehension, vocabulary, and emotional state. The following describes the system's main hardware and software components and their specific functions.

[2224] Hardware and Software Configuration

[2225] Server: The server runs key software components such as the generative AI model and database. The generative AI model is used to generate personalized learning materials based on user requests and profile information. The database stores and manages user profile information such as learning history, grades, and vocabulary. The emotion engine also analyzes the user's emotional state and provides appropriate feedback based on that information.

[2226] Terminal: A terminal is a device that students directly operate, typically a tablet or PC. The terminal provides a user interface and communicates student operations to the server. The terminal also displays the learning materials and feedback received from the server.

[2227] Users: The users are students who use this system, and are primarily children in the lower grades of elementary school. They select learning content, enter questions, and check their understanding through their devices. Their emotional state is also captured through the device's camera and microphone.

[2228] System Operation

[2229] The system starts when a user logs in on a device and the server checks the user's profile information. If the login is successful, the server sends information such as the user's learning history, grades, and vocabulary to the device, which then analyzes this information to set learning levels and personalized elements.

[2230] The user then selects the subject or topic they want to study, such as "Edo Period History" or "Math Exercises." This request is sent to a server, which uses a generative AI model to generate learning materials at the appropriate level. The generated learning materials, which include text, images, audio, and video, are sent to the device.

[2231] During the learning process, if the user has any questions, they can input them through the device. The questions are sent to the server, and the generative AI model generates an appropriate answer. The answer, which consists of text and images, is sent back to the device and displayed to the ...

Claims

1. The generated teaching materials are generated on the server based on the individual students' level of understanding. Students can ask questions until they are satisfied, Providing visual and verbal support Means and Analyze the progress and understanding of the generated teaching materials and provide feedback Means and Track student progress and suggest next appropriate learning content Means and A system including:

2. The system according to claim 1, further comprising means for generating teaching materials using a generation AI.

3. 10. The system of claim 1, further comprising means for presenting educational material using text, images, audio and video.

Images