System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to input a question in the form of text, image, or voice; means for transmitting the input question data to a server; means for analyzing the received question data and transmitting a request to an appropriate multimodal artificial intelligence; means for the multimodal artificial intelligence to analyze the question data and generate an answer; and means for providing the generated answer to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the recent spread of online learning, learning support using app matching functions has become popular among students. However, there is a problem that teachers' availability and skills vary, making it difficult for students to get answers immediately. This situation has a negative impact on learning efficiency and causes a decrease in motivation to learn. Therefore, a system that provides fast and consistent learning support is needed. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means: a system including a means for a user to input a question in the form of text, image, or voice; a means for transmitting the input question data to a server; a means for analyzing the received question data and sending a request to an appropriate multimodal artificial intelligence; a means for the multimodal artificial intelligence to analyze the question data and generate an answer; and a means for providing the generated answer to the user. This system allows users to receive immediate support and consistent support based on learning know-how. Furthermore, by including a means for the user to ask a follow-up question regarding the answer, a means for transmitting the follow-up question to the server, reanalyzing the question, generating a detailed answer, and a means for providing the detailed answer to the user, consistency and detail can be improved. Furthermore, by including a means for the server to store the user's registration information in a database and send a confirmation email, and a means for validating the login information of a user who clicks on a confirmation link, the system can improve its convenience and security.

[0006] A "user" is an entity that uses the system to enter questions and receive answers.

[0007] "Text" refers to information expressed in characters, and is one form in which a user can enter a question.

[0008] An "image" is data containing visual information, and is one of the formats in which a user can enter a question.

[0009] "Speech" refers to information conveyed through the auditory sense, and is one form in which a user can input a question.

[0010] "Means" refers to the methods or processes used to achieve a particular goal.

[0011] A "server" is a computer system that receives data from users, analyzes it, stores it, and sends requests to artificial intelligence.

[0012] "Multimodal AI" is an AI technology that comprehensively analyzes data in multiple formats, such as text, images, and audio, to generate optimal answers.

[0013] "Question data" refers to data including information such as text, images, and audio input by a user through the system.

[0014] "Analysis" is the process of breaking down data into smaller pieces and analyzing and understanding its contents.

[0015] A "request" is an act in which a server requests a multimodal artificial intelligence to analyze data and generate an answer.

[0016] "Answer" refers to the response and explanation provided to a user's question.

[0017] "Providing" refers to the act of giving the answer to the user.

[0018] "Additional questions" refers to the act of a user asking a more detailed explanation or a new question in response to an initial answer.

[0019] A "database" is a system that stores and manages information in an organized manner, and accumulates user registration information and question data. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention is a system for effectively supporting online learning, and can be implemented as follows.

[0042] User Registration and Login

[0043] The first step for users to use the system is to install the app and register as a user when they first start it. On the registration screen displayed on the device, the user enters their name, email address, grade, and subjects of interest. This input data is sent from the device to the server, which stores it in a database. Once registration is complete, the server sends a confirmation email to the user's email address, and the user clicks on the link in the confirmation email to activate their account. After that, the user can log in and use all of the system's functions.

[0044] Post a question and select the input format

[0045] If a user wants learning support, they click the "Ask a Question" button, which takes them to a question input screen. The device then displays options for inputting questions in the form of text, image, or voice. For example, if a user doesn't understand a math problem, they can type "I don't understand this calculus problem" and upload an image of the corresponding page from their textbook.

[0046] Question analysis and answer generation using multimodal AI

[0047] The device sends the question data entered by the user to the server. The server first analyzes the text data and identifies the type of question. Next, the server also analyzes the image data and audio data, and uses this information to seek a comprehensive understanding of the problem. Based on this analyzed data, the server sends a request to the multimodal AI. The multimodal AI comprehensively analyzes the received data, searches for related information from a learning database, and generates the optimal answer.

[0048] Providing answers

[0049] The server sends the answer generated by the multimodal AI to the device. For example, for a calculus problem, the generated answer includes a step-by-step process such as "first find the derivative, then integrate." The device displays the answer to the user and provides visual support information such as diagrams as needed.

[0050] Follow-up questions and real-time feedback

[0051] If the user wants to ask a more detailed question about the answer provided, they can enter the question again. For example, they can enter an additional question such as, "I don't understand the intermediate calculations." The device then sends this additional question to the server, and the server again requests a more detailed answer from the multimodal AI based on the previous answer data and the new question data. The multimodal AI then performs a detailed analysis and generates an additional answer, which the server then sends to the device. The device then provides this detailed answer to the user so that they can review it again.

[0052] Specific examples

[0053] If high school student A wants to solve a calculus problem in mathematics, he or she first registers as a user and logs in. Next, he or she enters the text "I don't understand this calculus problem" on the question input screen and uploads an image of the textbook. The server analyzes the text content and image and sends a request to the multimodal AI, which then generates a step-by-step answer. This answer is then sent to A's device via the server. If A asks a follow-up question, such as "I don't understand the intermediate calculations," the multimodal AI performs another detailed analysis and provides an additional answer.

[0054] This system allows users to receive prompt and consistent learning support, significantly improving learning efficiency. In addition, because it is not dependent on the teacher's availability or skills, users can receive high-quality learning support anytime, anywhere.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] The user installs the app and clicks the "Register" button when launching it for the first time.

[0058] Step 2:

[0059] The terminal displays a screen that prompts the user to enter registration information such as name, email address, grade, and subjects of interest.

[0060] Step 3:

[0061] The user enters their name, email address, grade, and subjects of interest and clicks the "Submit" button.

[0062] Step 4:

[0063] The terminal transmits the entered registration data to the server.

[0064] Step 5:

[0065] The server stores the received data in a database and sends a confirmation email to the user's email address.

[0066] Step 6:

[0067] The user clicks on the link in the confirmation email to activate their account.

[0068] Step 7:

[0069] The server accepts the confirmation link and validates the user's login information.

[0070] Step 8:

[0071] The user logs in to the app and navigates to the home screen.

[0072] Step 9:

[0073] The user clicks the "Ask a Question" button on the home screen.

[0074] Step 10:

[0075] The device will display a question entry screen and provide the option to enter the question in the form of text, image, or voice.

[0076] Step 11:

[0077] A user types in the text "I don't understand this calculus problem" and uploads an image of the corresponding page from the textbook.

[0078] Step 12:

[0079] The terminal sends the entered question data (text, image) to the server.

[0080] Step 13:

[0081] The server analyzes the text data and performs initial filtering.

[0082] Step 14:

[0083] The server sends the image data to an image analysis module to extract relevant information.

[0084] Step 15:

[0085] The server integrates the text and image analysis results and sends an analysis request to the multimodal AI.

[0086] Step 16:

[0087] Multimodal AI integrates and analyzes question data to understand the content of the problem.

[0088] Step 17:

[0089] Multimodal AI collates information from relevant learning databases to generate optimal answers.

[0090] Step 18:

[0091] The server receives the generated answer data and transmits it to the user's terminal.

[0092] Step 19:

[0093] The terminal displays the answer data to the user and, if necessary, displays supplementary information such as diagrams.

[0094] Step 20:

[0095] If the user wants to get more details about the answer, he / she can input and submit an "additional question."

[0096] Step 21:

[0097] The terminal sends a follow-up question to the server.

[0098] Step 22:

[0099] The server again sends an analysis request for additional questions to the multimodal AI.

[0100] Step 23:

[0101] Multimodal AI then performs a detailed analysis again to generate additional detailed answers.

[0102] Step 24:

[0103] The server transmits the generated additional answer to the user's terminal.

[0104] Step 25:

[0105] The device displays additional answers to the user to help them deepen their understanding.

[0106] This allows users to receive prompt and consistent learning support, improving learning efficiency and providing high-quality learning support that is not dependent on the availability or skills of teachers.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] Modern online learning systems allow users to input questions in a variety of formats, including text, images, and audio. However, they lack the technology to effectively analyze and integrate this data to generate answers. Furthermore, they lack a mechanism for efficiently reanalyzing and providing detailed answers when users ask follow-up questions about answers already provided. Furthermore, the user registration and authentication process is cumbersome, and account activation via a confirmation link can sometimes be slow. There is a need to address these issues and provide efficient, high-quality online learning support.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes: a means for a user to input a question in the form of text, image, or voice; a means for transmitting the input question data to the server; a means for analyzing the received question data and sending a request to an appropriate multimodal artificial intelligence; a means for the multimodal artificial intelligence to analyze the question data and generate an answer; a means for accepting additional questions, reanalyzing the data, and generating a detailed answer; a means for the server to store user registration information in a database and send a confirmation email; a means for activating the account of a user who clicks a confirmation link; and a means for the server to provide the generated answer to the user step by step and present the answer including visual auxiliary information. This allows users to input questions in various formats, have the data analyzed in an integrated manner, and receive consistent, high-quality answers. Furthermore, the user registration and authentication process is simplified, improving the efficiency of learning support.

[0112] 1. "User" refers to an individual or organization that uses the online learning system to enter questions and receive answers.

[0113] 2. "Text" refers to a data format consisting of characters or sentences, used by users to enter questions or comments.

[0114] 3. "Image" means a data format containing still images and graphics, used to provide users with visual information.

[0115] 4. "Audio" means a data format containing sounds or spoken words, used by a user to provide information orally.

[0116] 5. "Question Data" refers to the collection of data including all information entered by the User in text, image, and audio format.

[0117] 6. "Server" refers to the computer system that receives and analyzes query data and interacts with the Multimodal Artificial Intelligence.

[0118] 7. “Analysis” refers to the process of understanding the input question data and identifying its meaning and intent.

[0119] 8. "Multimodal AI" refers to AI technology that analyzes multiple data formats, such as text, images, and audio, and generates integrated answers.

[0120] 9. "Request" refers to instructions or requests sent by a server to a multimodal artificial intelligence to request analysis and answer generation.

[0121] 10. “Answer” refers to the response or reply generated by a Multimodal Artificial Intelligence to a user’s question.

[0122] 11. "Providing" refers to the process of displaying or notifying the user of a generated answer.

[0123] 12. "Follow-up Question" means a question entered by a User to request further information or clarification on an Initial Answer.

[0124] 13. "Reanalysis" refers to the process of analyzing new data based on additional questions to generate detailed answers.

[0125] 14. "Detailed answer" refers to a more specific and useful answer generated based on reanalysis.

[0126] 15. "User registration information" refers to basic information such as name, email address, grade, and areas of interest that a user enters to use the system.

[0127] 16. "Database" refers to a system for systematically storing and managing a large amount of information, such as user registration information and question data.

[0128] 17. "Verification Email" means the email sent to a User prompting them to activate their Account.

[0129] 18. "Confirmation Link" means the URL included in the confirmation email that the User clicks to activate their account.

[0130] 19. "Account Activation" refers to the process by which a User's account is activated and all its features are made available by clicking on a confirmation link.

[0131] 20. "Step-by-step" refers to a method of explaining a complex problem to a user by breaking it down into a series of simple steps.

[0132] 21. "Visual aids" refers to information containing non-textual visual data, such as charts or images, that is provided to make the answer easier to understand.

[0133] MODE FOR CARRYING OUT THE INVENTION

[0134] The present invention is a system for effectively supporting online learning, and specific embodiments thereof will be described below.

[0135] User Registration and Login

[0136] To use the system, users must first install the application. When the system is launched for the first time, the user registers their name, email address, grade, and subjects of interest. This data is collected by the device, which then sends it to the server. The server stores the received data in a database and sends a confirmation email to the user. When the user clicks the link in the confirmation email, their account is activated. After that, the user can log in by entering their email address and password on the login screen to use all of the system's functions.

[0137] Post a question and select the input format

[0138] If a user has a question about their studies, they click the "Ask a Question" button in the application and are taken to a question input screen. The device then provides the option to input the question in the form of text, image, or voice. For example, if a user wants to ask a question about a math problem, they can input the text "I don't understand this calculus problem" and upload an image of the corresponding page from the textbook. They can also dictate their question using voice input.

[0139] Sending and analyzing question data

[0140] The entered question data is sent from the device to the server. The server passes this data to an analysis module, which analyzes the text data to identify the content of the question. At the same time, it also analyzes image data and audio data to gain a comprehensive understanding of the problem. The server then sends a request to the multimodal AI based on the analysis results.

[0141] Generate and provide answers

[0142] Multimodal AI comprehensively analyzes the data it receives, searches for relevant information from a learning database, and generates the optimal answer. This answer is returned to the server, which then sends it to the device. The device then displays the received answer to the user. For example, a step-by-step solution to a calculus problem might be provided, such as "first find the derivative, then integrate." Diagrams and tables may also be displayed as visual support.

[0143] Providing additional questions and detailed answers

[0144] If the user wants to ask an additional question about the provided answer, they enter the question again, and the device sends that data to the server. The server reanalyzes the new question data and the previous answer data and asks the multimodal AI to generate a detailed answer. The multimodal AI then performs a detailed analysis and generates an additional, detailed answer. The server sends the answer to the device, which then provides it to the user again.

[0145] Examples of concrete examples and prompts

[0146] As a concrete example, consider the case where high school student A is trying to solve a calculus problem. First, A installs the application and enters his / her name, email address, grade, and subjects of interest. Next, A clicks the link in the confirmation email to activate his / her account and logs in. A clicks the "Ask a Question" button, enters the text "I don't understand this calculus problem," and uploads an image of the corresponding page from his / her textbook. The server analyzes the text and image and sends a request to the multimodal AI, which generates a step-by-step answer. This answer is then sent to A's device via the server. If A enters an additional question, such as "I don't understand the intermediate calculations," the multimodal AI performs another detailed analysis and provides an additional answer.

[0147] An example prompt for using a generative AI model is:

[0148] "I don't understand this calculus problem. I've uploaded an image of the relevant page from the textbook. Could you please explain the steps in detail?"

[0149] In this way, this system allows users to receive prompt and consistent learning support, significantly improving learning efficiency. Because it is not dependent on the teacher's availability or skills, high-quality learning support can be received anytime, anywhere.

[0150] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0151] Step 1:

[0152] The user installs the application and enters their name, email address, grade, and subjects of interest when they first launch it. This data is collected by the device and sent to the server. The server stores the received user information in a database and sends a confirmation email to the user. The input data is name, email address, grade, and subjects of interest, and the output data is a confirmation email for account activation.

[0153] Step 2:

[0154] The user clicks the link in the confirmation email to activate the account. The server detects the click on the confirmation link and activates the user's account information. The input data is the click on the confirmation link, and the output data is the activated user account.

[0155] Step 3:

[0156] The user logs in by entering their email address and password on the login screen. The device sends the login information to the server, which then authenticates it. If authentication is successful, the user is redirected to the dashboard screen. The input data is the email address and password, and the output data is the login success status.

[0157] Step 4:

[0158] The user clicks the "Ask a Question" button and is taken to a question input screen. The device displays the option to enter the question in text, image, or audio format. The user enters the question in the required format and uploads the appropriate file. The input data is text, image, or audio file, and the output data is the entered question data.

[0159] Step 5:

[0160] The terminal sends the input question data (text, image, voice) to the server. The server passes the received data to the analysis module. The input data is the question data, and the output data is the data transferred to the analysis module.

[0161] Step 6:

[0162] The server analyzes the text data to identify the content of the question. At the same time, it also analyzes image data and audio data to gain a comprehensive understanding of the problem. Based on the analysis results, it sends a request to the multimodal AI to generate an answer. The input data is the user's question data, and the output data is the analysis results.

[0163] Step 7:

[0164] The multimodal AI analyzes the received data comprehensively, searches for relevant information from the learning database, and generates the optimal answer. This answer is sent back to the server. The input data is the analysis result, and the output data is the generated answer.

[0165] Step 8:

[0166] The server sends the generated answer to the terminal. The terminal displays the answer to the user. The input data is the generated answer, and the output data is the answer displayed to the user.

[0167] Step 9:

[0168] The user inputs a follow-up question for the provided answer. The terminal sends the follow-up question to the server. The input data is the follow-up question, and the output data is the follow-up question data sent to the server.

[0169] Step 10:

[0170] The server requests the multimodal AI to perform reanalysis based on new question data and previous answer data. The multimodal AI performs detailed analysis and generates an additional answer. The server sends the generated answer to the terminal, which then provides it to the user again. The input data is the additional question data and previous answer data, and the output data is a detailed answer.

[0171] The above processing steps allow users to input questions in a variety of formats, and the data is analyzed in an integrated manner to provide consistent, high-quality answers. Furthermore, the user registration and authentication process is simplified, improving the efficiency of learning support.

[0172] (Application example 1)

[0173] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0174] Existing online learning systems mainly provide answers to questions using text, images, and audio, but lack the ability to provide visual answers using 3D models and animations that are easy for users to understand intuitively. Furthermore, there are only a limited number of systems that can respond to detailed follow-up questions in real time. Furthermore, there are limited means of providing learning support within the virtual environment, making it difficult for users to effectively progress through rich interactions.

[0175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0176] In this invention, the server includes: means for a user to input a question in the form of text, image, or voice; means for transmitting the input question data to the server; means for analyzing the received question data and sending a request to an appropriate multimodal AI; means for the multimodal AI to analyze the question data and generate an answer; means for providing the generated answer to the user; means for the user to move around in the virtual environment in real time and access topic rooms specialized for a subject; and means for providing the answer visually using 3D models and animations in the virtual environment, thereby enabling the user to receive detailed learning support with high resolution.

[0177] A "user" is an individual or organization that uses the learning system.

[0178] "Text" is character information entered by the user.

[0179] "Images" are visual content uploaded by users.

[0180] "Audio" refers to sound information that is recorded and transmitted by the user.

[0181] "Question data" refers to the content of a question about learning that is input by the user in the form of text, image, or voice.

[0182] A "server" is a central device that receives question data sent by users, analyzes it, and generates answers.

[0183] "Multimodal artificial intelligence" is an artificial intelligence system that comprehensively analyzes data in multiple formats, such as text, images, and audio, and generates answers.

[0184] An "answer" is information that contains a solution or explanation to a user's question.

[0185] A "virtual environment" is a virtual space in which users can move around in real time and engage in learning experiences.

[0186] "Topic rooms" are study areas dedicated to specific subjects, rooms within a virtual environment staffed with dedicated artificial intelligence assistants.

[0187] A "3D model" is a three-dimensional digital object that visually represents a solution within a virtual environment.

[0188] "Animation" is a moving image effect that dynamically visualizes the solution within the virtual environment.

[0189] A "dedicated artificial intelligence assistant" is an artificial intelligence system that is placed in each topic room and is specialized to provide appropriate answers to users' questions.

[0190] "Registration information" refers to personal information and learning information provided by users in order to use the system.

[0191] "Verification Email" means an email sent to a User to confirm their registration information and activate their Account.

[0192] "Login Information" means the authentication data used by a User to access a System.

[0193] "Motion tracking" refers to tracking a user's position and actions within a virtual environment in real time.

[0194] This invention is a system that effectively supports online learning and is realized mainly through the interaction between a server, a terminal, and a user. Specifically, it includes the following components and the system functions resulting from their interaction:

[0195] User Registration and Login

[0196] First, in order to use the system using a terminal, a user must register when starting up the system for the first time. The user enters information such as their name, email address, grade, and subjects of interest, and this input data is sent from the terminal to the server. The server saves this information in a database and sends a confirmation email to the user's email address. When the user clicks the link in the confirmation email, the server validates the user's login information. After that, the user can log in and use all the system's functions.

[0197] Post a question and select the input format

[0198] If a user wants to receive learning support, they click the "Ask a Question" button on their device and are taken to a question input screen. They can enter their question in the form of text, image, or voice. For example, if they don't understand a math problem, they can enter "I don't understand this calculus problem" and upload an image of the relevant page from their textbook.

[0199] Question analysis and answer generation using multimodal AI

[0200] The device sends the question data entered by the user to the server. The server first analyzes the text data and identifies the type of question. Next, the server also analyzes the image data and audio data, and uses this information to seek a comprehensive understanding of the problem. Based on this analyzed data, the server sends a request to multimodal artificial intelligence (AI). The multimodal AI comprehensively analyzes the received data, searches for related information from a learning database, and generates the optimal answer.

[0201] Providing answers

[0202] The server sends the answer generated by the multimodal AI to the device. For example, for a calculus problem, the generated answer includes a step-by-step process such as "first find the derivative, then integrate." The device displays the answer to the user and provides visual support information such as diagrams as needed.

[0203] Virtual Learning Support Center

[0204] This system provides users with a means to move around in a virtual environment in real time and access topic rooms specialized for specific subjects. When a user enters a topic room related to a specific subject, a dedicated AI assistant is deployed to provide appropriate answers to the user's questions. Answers are visually represented using 3D models and animations, making them easier for users to understand intuitively. Users can also add more detailed questions on the spot and receive detailed answers in real time.

[0205] Examples of concrete examples and prompts

[0206] As a concrete example, when a high school student wants to solve a calculus problem in mathematics, the following process is taken: The user enters a topic room and enters the following prompt sentence into the terminal:

[0207] "I don't understand a calculus problem. Can you give me detailed steps to solve it?"

[0208] "I don't understand this calculation step. Please explain it again in more detail."

[0209] This allows the system to provide users with the information they need in real time, greatly improving the learning experience.

[0210] Hardware and software used

[0211] The following hardware and software are used to realize this system. The hardware includes a server, a user device (a smartphone or personal computer), and a head-mounted display. The software uses Python, Unity (for building the VR environment), and TensorFlow (for the AI ​​model).

[0212] The above configuration allows users to receive learning support efficiently and effectively.

[0213] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0214] Step 1:

[0215] When a user starts up the device and uses it for the first time, they are taken to the user registration screen. The user enters their name, email address, grade, and subjects of interest, and this information is sent from the device to the server. Input data: Name, email address, grade, and subjects of interest. Output data: User registration information saved on the server.

[0216] Step 2:

[0217] The server saves the user registration information in a database and sends a confirmation email to the user's email address. The confirmation email contains a link to activate the account. Input data: User registration information. Output data: Confirmation email.

[0218] Step 3:

[0219] The user clicks the link in the confirmation email and the server validates the user's login information. Input data: Click on the link in the confirmation email. Output data: Validated user login information.

[0220] Step 4:

[0221] The user logs in using a terminal and clicks the "Ask a Question" button to move to the question input screen. The user enters the question in the form of text, image, or voice. Input data: text, image, voice. Output data: question data.

[0222] Step 5:

[0223] The device sends the entered question data to the server. The server receives the question data and first performs text analysis. Next, it analyzes the image and audio data. Input data: Question data. Output data: Analyzed question data.

[0224] Step 6:

[0225] The server sends a request to the appropriate multimodal AI based on the analyzed question data. Input data: Analyzed question data. Output data: Request sent to the AI.

[0226] Step 7:

[0227] Multimodal AI comprehensively analyzes question data, searches for relevant information, and generates optimal answers. Input data: Question data. Output data: Generated answers.

[0228] Step 8:

[0229] The server receives the answer data generated by the multimodal AI and sends it to the terminal. The terminal provides this answer to the user. Input data: Generated answer data. Output data: Answer displayed on the terminal.

[0230] Step 9:

[0231] If the user wants to ask a more detailed question, he or she inputs the additional question through the terminal. The terminal sends the additional question data to the server. Input data: Additional question data. Output data: Additional question data sent to the server.

[0232] Step 10:

[0233] The server re-analyzes the additional question data and sends a request to the multimodal AI again to generate a detailed answer. Input data: Additional question data. Output data: The request sent again to the AI.

[0234] Step 11:

[0235] The multimodal AI generates a detailed answer and sends it to the terminal via the server. The terminal displays this answer to the user. Input data: detailed answer data. Output data: detailed answer displayed on the terminal.

[0236] Step 12:

[0237] Users can move freely around the virtual environment and access topic rooms specific to their subject. A dedicated AI assistant is deployed to provide visual answers to questions using 3D models and animations. Input data: user's location, question. Output data: visual answers using 3D models and animations.

[0238] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0239] The present invention is a system for effectively supporting online learning, which is combined with an emotion engine for recognizing a user's emotions and providing appropriate support in accordance with those emotions. It can be implemented as follows.

[0240] User Registration and Login

[0241] The first step for users to use the system is to install the app and register as a user when they first start it. On the registration screen displayed on the device, the user enters their name, email address, grade, and subjects of interest. This input data is sent from the device to the server, which stores it in a database. Once registration is complete, the server sends a confirmation email to the user's email address, and the user clicks on the link in the confirmation email to activate their account. After that, the user can log in and use all of the system's functions.

[0242] Post a question and select the input format

[0243] If a user wants learning support, they click the "Ask a Question" button, which takes them to a question input screen. The device then displays options for inputting questions in the form of text, image, or voice. For example, if a user doesn't understand a math problem, they can type "I don't understand this calculus problem" and upload an image of the corresponding page from their textbook.

[0244] Question analysis and answer generation using multimodal AI

[0245] The device sends the question data entered by the user to the server. The server first analyzes the text data and identifies the type of question. Next, the server also analyzes the image data and audio data, and uses this information to seek a comprehensive understanding of the problem. Based on this analyzed data, the server sends a request to the multimodal AI. The multimodal AI comprehensively analyzes the received data, searches for related information from a learning database, and generates the optimal answer.

[0246] Emotion recognition and response using emotion engine

[0247] The server not only receives the question data but also sends it to the emotion engine to recognize the user's emotion at the time of question entry. The emotion engine analyzes the user's emotion from text data, image data, and voice data and determines whether it is positive or negative. Based on the results of this emotion recognition, the multimodal AI adjusts the answer it generates. For example, if the user expresses negative emotions, it adds a message of encouragement or support.

[0248] Providing answers

[0249] The server sends the generated answer to the device. For example, for a calculus problem, the generated answer includes a step-by-step process such as "first find the derivative, then integrate." The device displays the answer to the user and provides visual support information such as diagrams as needed. In addition, the device may display an encouraging message based on the user's emotion recognition results.

[0250] Follow-up questions and real-time feedback

[0251] If the user wants to ask a more detailed question about the provided answer, they can enter the question again. For example, they can enter an additional question such as, "I don't understand the calculation in between." The device then sends this additional question to the server, and the server again requests a more detailed answer from the multimodal AI based on the previous answer data and the new question data. The multimodal AI then performs a detailed analysis and generates an additional answer, which the server then sends to the device. The device then provides this detailed answer to the user, helping them to deepen their understanding.

[0252] Specific examples

[0253] If high school student A wants to solve a calculus problem in mathematics, he or she first registers as a user and logs in. Next, he or she enters the text "I don't understand this calculus problem" on the question input screen and uploads an image of the textbook. The server analyzes the text content and image and uses an emotion engine to recognize A's emotions. If A shows negative emotions, the AI ​​generates an answer that includes an encouraging message. The server sends this answer to A's device, where A receives it. If A asks an additional question, such as "I don't understand the calculation in the middle," the multimodal AI performs a more detailed analysis again and provides an additional answer.

[0254] This system allows users to receive prompt and consistent learning support, and by using an emotion engine, it is possible to provide even more personalized support, which further improves learning efficiency and allows users to enjoy a better learning experience.

[0255] The processing flow will be explained below.

[0256] Step 1:

[0257] The user installs the app and clicks the "Register" button when launching it for the first time.

[0258] Step 2:

[0259] The terminal displays a screen that prompts the user to enter registration information such as name, email address, grade, and subjects of interest.

[0260] Step 3:

[0261] The user enters their name, email address, grade, and subjects of interest and clicks the "Submit" button.

[0262] Step 4:

[0263] The terminal transmits the entered registration data to the server.

[0264] Step 5:

[0265] The server stores the received data in a database and sends a confirmation email to the user's email address.

[0266] Step 6:

[0267] The user clicks on the link in the confirmation email to activate their account.

[0268] Step 7:

[0269] The server accepts the confirmation link and validates the user's login information.

[0270] Step 8:

[0271] The user logs in to the app and navigates to the home screen.

[0272] Step 9:

[0273] The user clicks the "Ask a Question" button on the home screen.

[0274] Step 10:

[0275] The device will display a question entry screen and provide the option to enter the question in the form of text, image, or voice.

[0276] Step 11:

[0277] A user types in the text "I don't understand this calculus problem" and uploads an image of the corresponding page from the textbook.

[0278] Step 12:

[0279] The terminal sends the entered question data (text, image) to the server.

[0280] Step 13:

[0281] The server analyzes the text data and performs initial filtering.

[0282] Step 14:

[0283] The server sends the image data to an image analysis module to extract relevant information.

[0284] Step 15:

[0285] The server sends the voice data to a voice analysis module, which analyzes the voice tone and content.

[0286] Step 16:

[0287] The server integrates the text, image, and voice analysis results and sends them to the emotion engine.

[0288] Step 17:

[0289] The emotion engine analyzes the received data and determines the user's emotion. If the emotion is negative, it revisits the specific situation and evaluates whether an encouraging message needs to be added.

[0290] Step 18:

[0291] The emotion engine sends the emotion recognition results back to the server.

[0292] Step 19:

[0293] The server sends an analysis request to the multimodal AI based on the emotion recognition result. If the emotion is negative, the request includes an encouraging message or a special answer format.

[0294] Step 20:

[0295] Multimodal AI integrates and analyzes question data and emotional information to understand the content of the problem.

[0296] Step 21:

[0297] Multimodal AI collates information from relevant learning databases to generate optimal answers, supplemented with emotional messages where appropriate.

[0298] Step 22:

[0299] The server receives the generated answer data and transmits it to the user's terminal.

[0300] Step 23:

[0301] The device displays the answer data to the user, including visual support information such as text, formulas, diagrams, etc. If the emotion is negative, an encouraging message is also displayed.

[0302] Step 24:

[0303] If the user wants to get more details about the answer, he / she can input and submit an "additional question."

[0304] Step 25:

[0305] The terminal sends a follow-up question to the server.

[0306] Step 26:

[0307] The server again sends an analysis request for additional questions to the multimodal AI.

[0308] Step 27:

[0309] Multimodal AI then performs a detailed analysis again to generate additional detailed answers.

[0310] Step 28:

[0311] The server transmits the generated additional answer to the user's terminal.

[0312] Step 29:

[0313] The device displays additional answers to the user to help them deepen their understanding.

[0314] Through these steps, users can receive fast and consistent learning support, and the emotional engine can provide more personalized assistance, improving learning efficiency and providing a more meaningful learning experience for users.

[0315] Example 2

[0316] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0317] Conventional online learning support systems lack the flexibility to respond to the diverse question formats users encounter, and can only accept questions in specific formats. Furthermore, they provide uniform answers without considering changes in the user's motivation or emotions regarding learning, making it impossible to provide support optimized for each individual user. Furthermore, it is difficult to provide quick and detailed real-time feedback on follow-up questions, which hinders users' learning efficiency.

[0318] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for a user to input a question in any one of the formats of text, image, or voice, a means for transmitting the input question data to the server, a means for analyzing the received question data and transmitting a request to an appropriate multimodal AI, a means for the multimodal AI to analyze the question data and generate an answer, a means for providing the generated answer to the user, and a means for analyzing the user's emotions using an emotion recognition engine and adjusting the content of the answer based on the analyzed emotion data. This makes it possible to respond to various types of questions from users and provide optimal answers that take into account each individual's emotional state.

[0319] "User" refers to any person or entity that utilizes the online learning system to enter questions and receive answers.

[0320] "Text" refers to a digital data format that includes character information, and refers to character string data that a user uses to input a question.

[0321] "Image" refers to a digital data format containing visual information, such as a photograph or drawing uploaded by a user to supplement a question.

[0322] "Speech" refers to a digital data format containing auditory information, and refers to voice data used by a user to input a question.

[0323] "Server" refers to a computer system or cloud infrastructure for processing, analyzing, storing, and generating answers from data received from users.

[0324] "Multimodal AI" refers to an AI model that comprehensively analyzes different types of data (text, images, audio) and generates optimal answers.

[0325] An "emotion recognition engine" refers to software or algorithms that analyze emotions from user input data and determine whether they are positive or negative.

[0326] "Answer" refers to output data generated by multimodal artificial intelligence that contains appropriate information and instructions in response to a user's question.

[0327] "Confirmation Email" refers to the email sent from the server at the time of user registration for the purpose of activating the account.

[0328] "Confirmation Link" refers to the URL included in the confirmation email that the User clicks to activate their account.

[0329] "Login Information" refers to the authentication information (e.g., user ID and password) used by a User to access the System.

[0330] The present invention is a system for supporting online learning, in which a user inputs a question in the form of text, image, or voice, and generates an answer to the question using multimodal artificial intelligence and an emotion recognition engine. Specific embodiments of the present invention are described below.

[0331] First, a user installs and launches the online learning application on a device such as a smartphone. In order to use the system, the user must first register. On the registration screen, the user enters their name, email address, grade, and subjects of interest, and this data is sent from the device to the server. The server stores this data in a database (e.g., MySQL database) and sends a confirmation email to the user. The user clicks the link in the confirmation email to activate their account.

[0332] Next, when the user clicks the "Ask a Question" button, the device displays a screen where the user can enter a question in the form of text, image, or voice. For example, the user can enter the text "I don't understand this calculus problem" and upload an image of the corresponding page from the textbook. This input data is sent from the device to the server.

[0333] The server analyzes the received text data using a natural language processing library (e.g., NLTK, SpaCy), and analyzes the image data using an image processing library (e.g., OpenCV). The analyzed data is sent to a multimodal AI, which uses a generative AI model such as GPT-4 to comprehensively analyze the received data and generate the optimal answer.

[0334] The server then sends the received question data to an emotion recognition engine. The emotion recognition engine uses the Emotion API or other tools to analyze the user's emotions from text, image, and audio data and determine whether they are positive or negative. Based on the emotion recognition results, the multimodal AI adjusts the answer it generates. For example, if the user expresses negative emotions, it adds a message of encouragement or support.

[0335] The generated answer is sent from the server to the device, which then displays it to the user. Visual support information (e.g., diagrams) is also displayed as needed. If the user enters additional questions, the multimodal AI again performs a detailed analysis, generating the optimal answer and sending it from the server to the device.

[0336] As a concrete example, if high school student A wants to solve a calculus problem in mathematics, he or she would follow the steps below. First, he or she registers as a user and logs in. Next, he or she enters the text "I don't understand this calculus problem" on the question input screen and uploads an image of the textbook. The server analyzes the text content and image and uses an emotion recognition engine to recognize A's emotions. If A shows negative emotions, the AI ​​generates an answer that includes an encouraging message. The server sends this answer to A's device, where A receives it. If A asks an additional question, such as "I don't understand the calculation in the middle," a more detailed analysis is performed again and the answer is provided to the device.

[0337] An example of a prompt sentence is as follows:

[0338] User: I don't understand this calculus problem. (Image attached) (Image: Calculus problem page in textbook)

[0339] Server: Performs text and image analysis to generate the optimal answer.

[0340] Emotion Recognition Engine: Analyzes the user's emotions and finds that they are negative. Adds encouraging messages.

[0341] Server: Generates the best answer and sends it to the device along with an encouraging message.

[0342] This invention allows users to receive prompt and consistent learning support, and by using an emotion recognition engine, it is possible to provide even more personalized support to users, significantly improving learning efficiency.

[0343] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0344] Step 1:

[0345] The user installs and launches the app.

[0346] A user downloads the app from a digital store and installs it on their device.

[0347] Launch the app and the first screen will appear.

[0348] Input: User downloads and launches the app

[0349] Output: First screen display of the app

[0350] Step 2:

[0351] The terminal displays a user registration screen.

[0352] A registration screen UI is displayed, including fields for the user to enter their name, email address, grade level, and subjects of interest.

[0353] Input: Registration screen displayed after launching the app

[0354] Output: A field for the user to enter input.

[0355] Step 3:

[0356] The user enters the required information and presses the registration button.

[0357] The user enters their name, email address, grade, and subjects of interest and presses the "Register" button.

[0358] Input: Information entered by the user (name, email address, grade, subjects of interest)

[0359] Output: Sending input data by pressing the register button

[0360] Step 4:

[0361] The terminal sends the input data to the server.

[0362] The terminal sends the data entered by the user to the server in JSON format at the REST API endpoint.

[0363] Input: Information entered by the user

[0364] Output: Send data in JSON format

[0365] Step 5:

[0366] The server stores the registration information in a database.

[0367] The server analyzes the received data and stores it in a database (e.g. MySQL).

[0368] Input: JSON format user information data

[0369] Output: User information stored in the database

[0370] Step 6:

[0371] The server sends a confirmation email.

[0372] The server uses a mail server (e.g. Postfix) to send the user a confirmation email containing a link to activate the account.

[0373] Input: User information stored in the database

[0374] Output: Send confirmation email

[0375] Step 7:

[0376] The user clicks the confirmation link.

[0377] The user opens the confirmation email and activates their account by clicking the link.

[0378] Input: Link in confirmation email

[0379] Output: Account enabled

[0380] Step 8:

[0381] The server performs the account activation process.

[0382] The server receives the link and updates the status of the corresponding account to "active."

[0383] Input: The confirmation link the user clicked

[0384] Output: Enabled account status

[0385] Step 9:

[0386] A user logs in.

[0387] Using your user ID and password, you can access the login screen and use all the functions of the system.

[0388] Input: User ID and password

[0389] Output: Successfully logged into the system

[0390] Step 10:

[0391] The user clicks the "Ask a Question" button.

[0392] A user clicks the "Ask a Question" button within the app.

[0393] Input: User clicks "Ask a question" button

[0394] Output: Display of question input screen

[0395] Step 11:

[0396] The device displays a question input screen.

[0397] Displays a question input screen with text, image, and audio input options.

[0398] Input: User clicks "Ask a question" button

[0399] Output: Display of question input screen

[0400] Step 12:

[0401] The user enters a question.

[0402] A user types "I don't understand this calculus problem" into a text box and uploads an image of a page from a textbook.

[0403] Input: Question text entered by the user, and uploaded images

[0404] Output: Question data ready to send

[0405] Step 13:

[0406] The terminal transmits the input data to the server.

[0407] The text data and image data are sent to a server.

[0408] Input: Question data entered by the user (text, images)

[0409] Output: Send data to the server

[0410] Step 14:

[0411] The server parses the text data.

[0412] The server analyzes the text data using a natural language processing library (e.g., NLTK, SpaCy) to identify the question.

[0413] Input: Received text data

[0414] Output: Analysis results (question content identification)

[0415] Step 15:

[0416] The server analyzes the image data.

[0417] The server analyzes the image data using an image processing library (e.g., OpenCV) and extracts relevant information.

[0418] Input: Received image data

[0419] Output: Analysis results (image content analysis)

[0420] Step 16:

[0421] The server sends the analysis data to the multimodal AI.

[0422] The analysis results are integrated and sent to multimodal AI.

[0423] Input: Text and image analysis results

[0424] Output: A request to the multimodal AI

[0425] Step 17:

[0426] Multimodal AI generates optimal answers.

[0427] Generative AI models such as GPT-4 are used to analyze the received data and generate optimal answers.

[0428] Input: Integrated analysis data

[0429] Output: The generated answer

[0430] Step 18:

[0431] The server sends the answer data to the emotion recognition engine.

[0432] The emotion recognition engine analyzes the user's emotions along with the generated answer data.

[0433] Input: Answer data

[0434] Output: User sentiment analysis results

[0435] Step 19:

[0436] An emotion recognition engine analyzes the user's emotions.

[0437] The Emotion API is used to determine the user's emotions from the data sent.

[0438] Input: Answer data

[0439] Output: Sentiment analysis result (positive / negative)

[0440] Step 20:

[0441] The server adjusts the answer based on the emotional data.

[0442] Based on the results of emotion recognition, the multimodal AI is instructed to adjust its answers, for example by adding encouraging messages.

[0443] Input: Sentiment analysis results

[0444] Output: Adjusted answer

[0445] Step 21:

[0446] The server sends the answer data to the terminal.

[0447] The optimized answer is sent to the device.

[0448] Input: Adjusted answer data

[0449] Output: Send answer data to the device

[0450] Step 22:

[0451] The terminal displays the answer to the user.

[0452] The terminal displays the answer and supplementary information to the user.

[0453] Input: Answer data received from the server

[0454] Output: What is displayed to the user

[0455] Step 23:

[0456] The user enters a follow-up question.

[0457] Enter additional detailed questions in text for the answers provided.

[0458] Input: Additional questions from the user

[0459] Output: Additional question data

[0460] Step 24:

[0461] The terminal sends a follow-up question to the server.

[0462] Send additional question data to the server.

[0463] Input: Additional question data by the user

[0464] Output: Send data to the server

[0465] Step 25:

[0466] The server reanalyzes the data and sends the request to the multimodal AI again.

[0467] Send additional question data to the multimodal AI to generate detailed answers.

[0468] Input: Additional question data

[0469] Output: Detailed answer request

[0470] Step 26:

[0471] Multimodal AI generates detailed answers.

[0472] Generate detailed answers based on additional question data.

[0473] Input: Additional question data

[0474] Output: Detailed answer

[0475] Step 27:

[0476] The server sends detailed answer data to the terminal.

[0477] The generated detailed answer is sent to the device.

[0478] Input: Detailed answer data

[0479] Output: Detailed answer data sent to the device

[0480] Step 28:

[0481] The terminal displays the detailed answer to the user.

[0482] The terminal displays the detailed answer content to the user.

[0483] Input: Detailed answer data received from the server

[0484] Output: What is displayed to the user

[0485] In this way, the system can effectively execute a series of processes to receive a user's question, analyze it, and provide the most appropriate answer.

[0486] (Application example 2)

[0487] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0488] Conventional online learning systems can generate answers by analyzing text, images, and audio data provided by users, but they rarely take the user's emotional state into consideration. This can lead to problems such as a decrease in learning motivation and a lack of appropriate feedback, resulting in a decrease in learning effectiveness. Furthermore, when asked follow-up questions about answers, real-time feedback that reflects the user's emotional changes cannot be provided, making it difficult to provide advanced support tailored to individual learners.

[0489] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input a question in the form of text, image, or voice; a means for transmitting the input question data to the server; a means for analyzing the received question data and sending a request to an appropriate multimodal AI; a means for the multimodal AI to analyze the question data and generate an answer; a means for providing the generated answer to the user; and an emotion engine that recognizes the user's emotions and provides a response according to the emotion. This provides appropriate support that takes into account the user's emotional state, thereby improving learning effectiveness. In addition, the server includes a means for a user to ask a follow-up question in response to the answer; a means for transmitting the follow-up question to the server, reanalyzing the question, and generating a detailed answer; a means for providing the detailed answer to the user; and a means for providing real-time feedback according to changes in the user's emotions, thereby realizing advanced support tailored to individual learners. Furthermore, the system includes a means for the server to store the user's registration information in a database and send a confirmation email, a means for validating the login information of a user who clicks on a confirmation link, and a means for initializing the user's emotional data at the time of login and providing support tailored to each individual learner, thereby enabling more personalized learning support to be provided.

[0490] A "user" is an individual who utilizes the system to enter questions and receive answers.

[0491] "Text" refers to the written information that a user inputs into the system.

[0492] "Image" means information in the form of a still image that a user uploads to the system.

[0493] "Speech" refers to the acoustic information provided by the user to the system.

[0494] "Question Data" refers to information entered into the system by a user in the form of text, images, or voice.

[0495] A "server" is a computer system that analyzes question data received from a user and generates an appropriate answer.

[0496] "Multimodal AI" refers to AI that analyzes and integrates multiple data formats, such as text, images, and audio, to generate answers.

[0497] An "emotion engine" is a system that recognizes emotions from user input data and provides a response that corresponds to that emotion.

[0498] An "answer" refers to response information generated based on question data analyzed by multimodal artificial intelligence.

[0499] "Registration Information" refers to personal information such as name, email address, grade, and subjects of interest that a user provides when registering with the system.

[0500] "Verification Email" means an email sent by the System to verify a User's registration information and activate their Account.

[0501] "Login information" refers to the authentication information (user name and password) used by a user to access a system.

[0502] "Emotional state" indicates the psychological state of the user when he or she inputs the question.

[0503] "Real-time feedback" refers to a system that provides immediate responses to users' questions and follow-up questions.

[0504] The present invention provides an online learning support system that combines an emotion engine that recognizes a user's emotions and provides appropriate support in accordance with those emotions. Specific embodiments of this system will be described below.

[0505] User Registration and Login

[0506] When using the system for the first time, the user installs a dedicated app on their smartphone and launches it. When launched for the first time, a user registration screen is displayed. The user enters their name, email address, grade, and subjects of interest. This input data is sent from the smartphone to the server, which stores it in a database. After registration is complete, the server sends a confirmation email to the user's email address, and the user activates their account by clicking the confirmation link. After that, the user can log in and use all of the system's functions.

[0507] Question posting and emotion recognition

[0508] When a user wants to receive learning support, they click the "Ask a Question" button in the app, which takes them to a question input screen. Here, the user can enter their question in the form of text, image, or voice. The entered question data is sent from the smartphone to the server. The server analyzes the received question data and identifies the content of the question. At the same time, the emotion engine analyzes the question data and recognizes the user's emotional state.

[0509] Answer generation and emotion regulation

[0510] The server sends a request to the multimodal AI based on the analyzed data. The multimodal AI comprehensively analyzes the received data and generates an answer. At that time, the answer is adjusted based on the analysis results of the emotion engine according to the user's emotions. For example, if the user shows negative emotions, the AI ​​adds a message of encouragement or support.

[0511] Providing answers

[0512] The generated answer is sent from the server to the smartphone and provided to the user. For example, the generated answer to a math problem may include a step-by-step process such as "first find the derivative, then do the integral." Encouraging messages based on the emotion recognition results may also be displayed.

[0513] Follow-up questions and real-time feedback

[0514] If the user wants to ask a more detailed question about the provided answer, they can enter the question again. For example, they can enter an additional question such as, "I don't understand the calculation in between." The smartphone again sends this additional question to the server, and the server again requests a more detailed answer from the multimodal AI based on the previous answer data and the new question data. The multimodal AI then performs a detailed analysis and generates an additional answer, which the server then sends to the smartphone. The smartphone then provides this detailed answer to the user, helping them to deepen their understanding.

[0515] Hardware and software used

[0516] The hardware used is a smartphone camera, and the software used is image preprocessing using OpenCV, emotion recognition modeling using TensorFlow / Keras, and text emotion analysis using TextBlob.

[0517] Examples and prompts

[0518] For example, when a user types, "Today's test results were poor," the sentiment engine detects negative sentiment from the text, and the AI ​​adds an encouraging message such as, "Failure is the mother of success, so try harder next time!"

[0519] Example prompt sentence:

[0520] Classify the emotion of the given sentence as positive or negative. Specifically, if the sentence contains negative expressions such as "I don't understand," "It's difficult," or "It's hard," output an appropriate encouraging message.

[0521] For example, given the following input:

[0522] Today's test results were bad

[0523] As described above, the system of the present invention provides support according to the user's emotional state, improves learning efficiency, and provides optimal feedback to each individual learner.

[0524] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0525] Step 1:

[0526] A user installs a dedicated app on their smartphone and launches the app. A user registration screen appears, where the user enters their name, email address, grade, and subjects of interest. This input data is sent from the device to the server. The server stores the received registration information in a database and sends a confirmation email to the user's email address. The user activates their account by clicking the confirmation link in the email. The entered data is in text format, and the server stores it in the database. The output is the account activation status.

[0527] Step 2:

[0528] The user logs in and clicks the "Ask a Question" button to move to the question input screen. The user enters the question in the form of text, image, or voice. This data is sent from the terminal to the server. The server processes the received question data using one of the following methods: text analysis, image analysis, or voice analysis. The input is the question data in the respective data format, and the output is the analyzed question content.

[0529] Step 3:

[0530] The emotion engine analyzes the question data and recognizes the user's emotional state. The server uses the emotion engine to identify the user's emotion from text data, image data, and audio data. For example, for text, it uses TextBlob to calculate the emotion score. The input is the question data in each format, and the output is the recognized emotional state (positive or negative).

[0531] Step 4:

[0532] The server sends a request to a multimodal AI based on the analyzed data. The multimodal AI integrates text, image, and audio data to generate an answer. To generate the answer, a generative AI model is used to search and integrate information from relevant knowledge bases. The input is the analyzed data, and the output is the generated answer.

[0533] Step 5:

[0534] The generated answer is sent from the server to the device and provided to the user. Based on the results of question data analysis and the recognition results of the emotion engine, an encouraging message is added as needed. The device displays this information on the screen. The input is the generated answer and encouraging message, and the output is the answer and message displayed to the user.

[0535] Step 6:

[0536] If the user wants to enter a more detailed question in response to the provided answer, the question is entered again and the additional question data is sent from the terminal to the server. The server then re-analyzes the previous answer data and the new question data to generate a detailed answer. The input is the additional question data and the output is the detailed answer.

[0537] Step 7:

[0538] The server sends the detailed answer to the terminal and provides it back to the user. The terminal displays the detailed answer to the user and provides further assistance or feedback as needed. The input is the detailed answer and any required feedback messages, and the output is the detailed answer and feedback displayed to the user.

[0539] Through these steps, users can learn efficiently while receiving appropriate support according to their emotions, thereby improving the effectiveness and satisfaction of their learning.

[0540] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0541] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0542] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0543] [Second embodiment]

[0544] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0545] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0546] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0547] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0548] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0549] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0550] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0551] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0552] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0553] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0554] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0555] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0556] The present invention is a system for effectively supporting online learning, and can be implemented as follows.

[0557] User Registration and Login

[0558] The first step for users to use the system is to install the app and register as a user when they first start it. On the registration screen displayed on the device, the user enters their name, email address, grade, and subjects of interest. This input data is sent from the device to the server, which stores it in a database. Once registration is complete, the server sends a confirmation email to the user's email address, and the user clicks on the link in the confirmation email to activate their account. After that, the user can log in and use all of the system's functions.

[0559] Post a question and select the input format

[0560] If a user wants learning support, they click the "Ask a Question" button, which takes them to a question input screen. The device then displays options for inputting questions in the form of text, image, or voice. For example, if a user doesn't understand a math problem, they can type "I don't understand this calculus problem" and upload an image of the corresponding page from their textbook.

[0561] Question analysis and answer generation using multimodal AI

[0562] The device sends the question data entered by the user to the server. The server first analyzes the text data and identifies the type of question. Next, the server also analyzes the image data and audio data, and uses this information to seek a comprehensive understanding of the problem. Based on this analyzed data, the server sends a request to the multimodal AI. The multimodal AI comprehensively analyzes the received data, searches for related information from a learning database, and generates the optimal answer.

[0563] Providing answers

[0564] The server sends the answer generated by the multimodal AI to the device. For example, for a calculus problem, the generated answer includes a step-by-step process such as "first find the derivative, then integrate." The device displays the answer to the user and provides visual support information such as diagrams as needed.

[0565] Follow-up questions and real-time feedback

[0566] If the user wants to ask a more detailed question about the answer provided, they can enter the question again. For example, they can enter an additional question such as, "I don't understand the intermediate calculations." The device then sends this additional question to the server, and the server again requests a more detailed answer from the multimodal AI based on the previous answer data and the new question data. The multimodal AI then performs a detailed analysis and generates an additional answer, which the server then sends to the device. The device then provides this detailed answer to the user so that they can review it again.

[0567] Specific examples

[0568] If high school student A wants to solve a calculus problem in mathematics, he or she first registers as a user and logs in. Next, he or she enters the text "I don't understand this calculus problem" on the question input screen and uploads an image of the textbook. The server analyzes the text content and image and sends a request to the multimodal AI, which then generates a step-by-step answer. This answer is then sent to A's device via the server. If A asks a follow-up question, such as "I don't understand the intermediate calculations," the multimodal AI performs another detailed analysis and provides an additional answer.

[0569] This system allows users to receive prompt and consistent learning support, significantly improving learning efficiency. In addition, because it is not dependent on the teacher's availability or skills, users can receive high-quality learning support anytime, anywhere.

[0570] The processing flow will be explained below.

[0571] Step 1:

[0572] The user installs the app and clicks the "Register" button when launching it for the first time.

[0573] Step 2:

[0574] The terminal displays a screen that prompts the user to enter registration information such as name, email address, grade, and subjects of interest.

[0575] Step 3:

[0576] The user enters their name, email address, grade, and subjects of interest and clicks the "Submit" button.

[0577] Step 4:

[0578] The terminal transmits the entered registration data to the server.

[0579] Step 5:

[0580] The server stores the received data in a database and sends a confirmation email to the user's email address.

[0581] Step 6:

[0582] The user clicks on the link in the confirmation email to activate their account.

[0583] Step 7:

[0584] The server accepts the confirmation link and validates the user's login information.

[0585] Step 8:

[0586] The user logs in to the app and navigates to the home screen.

[0587] Step 9:

[0588] The user clicks the "Ask a Question" button on the home screen.

[0589] Step 10:

[0590] The device will display a question entry screen and provide the option to enter the question in the form of text, image, or voice.

[0591] Step 11:

[0592] A user types in the text "I don't understand this calculus problem" and uploads an image of the corresponding page from the textbook.

[0593] Step 12:

[0594] The terminal sends the entered question data (text, image) to the server.

[0595] Step 13:

[0596] The server analyzes the text data and performs initial filtering.

[0597] Step 14:

[0598] The server sends the image data to an image analysis module to extract relevant information.

[0599] Step 15:

[0600] The server integrates the text and image analysis results and sends an analysis request to the multimodal AI.

[0601] Step 16:

[0602] Multimodal AI integrates and analyzes question data to understand the content of the problem.

[0603] Step 17:

[0604] Multimodal AI collates information from relevant learning databases to generate optimal answers.

[0605] Step 18:

[0606] The server receives the generated answer data and transmits it to the user's terminal.

[0607] Step 19:

[0608] The terminal displays the answer data to the user and, if necessary, displays supplementary information such as diagrams.

[0609] Step 20:

[0610] If the user wants to get more details about the answer, he / she can input and submit an "additional question."

[0611] Step 21:

[0612] The terminal sends a follow-up question to the server.

[0613] Step 22:

[0614] The server again sends an analysis request for additional questions to the multimodal AI.

[0615] Step 23:

[0616] Multimodal AI then performs a detailed analysis again to generate additional detailed answers.

[0617] Step 24:

[0618] The server transmits the generated additional answer to the user's terminal.

[0619] Step 25:

[0620] The device displays additional answers to the user to help them deepen their understanding.

[0621] This allows users to receive prompt and consistent learning support, improving learning efficiency and providing high-quality learning support that is not dependent on the availability or skills of teachers.

[0622] Example 1

[0623] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0624] Modern online learning systems allow users to input questions in a variety of formats, including text, images, and audio. However, they lack the technology to effectively analyze and integrate this data to generate answers. Furthermore, they lack a mechanism for efficiently reanalyzing and providing detailed answers when users ask follow-up questions about answers already provided. Furthermore, the user registration and authentication process is cumbersome, and account activation via a confirmation link can sometimes be slow. There is a need to address these issues and provide efficient, high-quality online learning support.

[0625] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0626] In this invention, the server includes: a means for a user to input a question in the form of text, image, or voice; a means for transmitting the input question data to the server; a means for analyzing the received question data and sending a request to an appropriate multimodal artificial intelligence; a means for the multimodal artificial intelligence to analyze the question data and generate an answer; a means for accepting additional questions, reanalyzing the data, and generating a detailed answer; a means for the server to store user registration information in a database and send a confirmation email; a means for activating the account of a user who clicks a confirmation link; and a means for the server to provide the generated answer to the user step by step and present the answer including visual auxiliary information. This allows users to input questions in various formats, have the data analyzed in an integrated manner, and receive consistent, high-quality answers. Furthermore, the user registration and authentication process is simplified, improving the efficiency of learning support.

[0627] 1. "User" refers to an individual or organization that uses the online learning system to enter questions and receive answers.

[0628] 2. "Text" refers to a data format consisting of characters or sentences, used by users to enter questions or comments.

[0629] 3. "Image" means a data format containing still images and graphics, used to provide users with visual information.

[0630] 4. "Audio" means a data format containing sounds or spoken words, used by a user to provide information orally.

[0631] 5. "Question Data" refers to the collection of data including all information entered by the User in text, image, and audio format.

[0632] 6. "Server" refers to the computer system that receives and analyzes query data and interacts with the Multimodal Artificial Intelligence.

[0633] 7. “Analysis” refers to the process of understanding the input question data and identifying its meaning and intent.

[0634] 8. "Multimodal AI" refers to AI technology that analyzes multiple data formats, such as text, images, and audio, and generates integrated answers.

[0635] 9. "Request" refers to instructions or requests sent by a server to a multimodal artificial intelligence to request analysis and answer generation.

[0636] 10. “Answer” refers to the response or reply generated by a Multimodal Artificial Intelligence to a user’s question.

[0637] 11. "Providing" refers to the process of displaying or notifying the user of a generated answer.

[0638] 12. "Follow-up Question" means a question entered by a User to request further information or clarification on an Initial Answer.

[0639] 13. "Reanalysis" refers to the process of analyzing new data based on additional questions to generate detailed answers.

[0640] 14. "Detailed answer" refers to a more specific and useful answer generated based on reanalysis.

[0641] 15. "User registration information" refers to basic information such as name, email address, grade, and areas of interest that a user enters to use the system.

[0642] 16. "Database" refers to a system for systematically storing and managing a large amount of information, such as user registration information and question data.

[0643] 17. "Verification Email" means the email sent to a User prompting them to activate their Account.

[0644] 18. "Confirmation Link" means the URL included in the confirmation email that the User clicks to activate their account.

[0645] 19. "Account Activation" refers to the process by which a User's account is activated and all its features are made available by clicking on a confirmation link.

[0646] 20. "Step-by-step" refers to a method of explaining a complex problem to a user by breaking it down into a series of simple steps.

[0647] 21. "Visual aids" refers to information containing non-textual visual data, such as charts or images, that is provided to make the answer easier to understand.

[0648] MODE FOR CARRYING OUT THE INVENTION

[0649] The present invention is a system for effectively supporting online learning, and specific embodiments thereof will be described below.

[0650] User Registration and Login

[0651] To use the system, users must first install the application. When the system is launched for the first time, the user registers their name, email address, grade, and subjects of interest. This data is collected by the device, which then sends it to the server. The server stores the received data in a database and sends a confirmation email to the user. When the user clicks the link in the confirmation email, their account is activated. After that, the user can log in by entering their email address and password on the login screen to use all of the system's functions.

[0652] Post a question and select the input format

[0653] If a user has a question about their studies, they click the "Ask a Question" button in the application and are taken to a question input screen. The device then provides the option to input the question in the form of text, image, or voice. For example, if a user wants to ask a question about a math problem, they can input the text "I don't understand this calculus problem" and upload an image of the corresponding page from the textbook. They can also dictate their question using voice input.

[0654] Sending and analyzing question data

[0655] The entered question data is sent from the device to the server. The server passes this data to an analysis module, which analyzes the text data to identify the content of the question. At the same time, it also analyzes image data and audio data to gain a comprehensive understanding of the problem. The server then sends a request to the multimodal AI based on the analysis results.

[0656] Generate and provide answers

[0657] Multimodal AI comprehensively analyzes the data it receives, searches for relevant information from a learning database, and generates the optimal answer. This answer is returned to the server, which then sends it to the device. The device then displays the received answer to the user. For example, a step-by-step solution to a calculus problem might be provided, such as "first find the derivative, then integrate." Diagrams and tables may also be displayed as visual support.

[0658] Providing additional questions and detailed answers

[0659] If the user wants to ask an additional question about the provided answer, they enter the question again, and the device sends that data to the server. The server reanalyzes the new question data and the previous answer data and asks the multimodal AI to generate a detailed answer. The multimodal AI then performs a detailed analysis and generates an additional, detailed answer. The server sends the answer to the device, which then provides it to the user again.

[0660] Examples of concrete examples and prompts

[0661] As a concrete example, consider the case where high school student A is trying to solve a calculus problem. First, A installs the application and enters his / her name, email address, grade, and subjects of interest. Next, A clicks the link in the confirmation email to activate his / her account and logs in. A clicks the "Ask a Question" button, enters the text "I don't understand this calculus problem," and uploads an image of the corresponding page from his / her textbook. The server analyzes the text and image and sends a request to the multimodal AI, which generates a step-by-step answer. This answer is then sent to A's device via the server. If A enters an additional question, such as "I don't understand the intermediate calculations," the multimodal AI performs another detailed analysis and provides an additional answer.

[0662] An example prompt for using a generative AI model is:

[0663] "I don't understand this calculus problem. I've uploaded an image of the relevant page from the textbook. Could you please explain the steps in detail?"

[0664] In this way, this system allows users to receive prompt and consistent learning support, significantly improving learning efficiency. Because it is not dependent on the teacher's availability or skills, high-quality learning support can be received anytime, anywhere.

[0665] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0666] Step 1:

[0667] The user installs the application and enters their name, email address, grade, and subjects of interest when they first launch it. This data is collected by the device and sent to the server. The server stores the received user information in a database and sends a confirmation email to the user. The input data is name, email address, grade, and subjects of interest, and the output data is a confirmation email for account activation.

[0668] Step 2:

[0669] The user clicks the link in the confirmation email to activate the account. The server detects the click on the confirmation link and activates the user's account information. The input data is the click on the confirmation link, and the output data is the activated user account.

[0670] Step 3:

[0671] The user logs in by entering their email address and password on the login screen. The device sends the login information to the server, which then authenticates it. If authentication is successful, the user is redirected to the dashboard screen. The input data is the email address and password, and the output data is the login success status.

[0672] Step 4:

[0673] The user clicks the "Ask a Question" button and is taken to a question input screen. The device displays the option to enter the question in text, image, or audio format. The user enters the question in the required format and uploads the appropriate file. The input data is text, image, or audio file, and the output data is the entered question data.

[0674] Step 5:

[0675] The terminal sends the input question data (text, image, voice) to the server. The server passes the received data to the analysis module. The input data is the question data, and the output data is the data transferred to the analysis module.

[0676] Step 6:

[0677] The server analyzes the text data to identify the content of the question. At the same time, it also analyzes image data and audio data to gain a comprehensive understanding of the problem. Based on the analysis results, it sends a request to the multimodal AI to generate an answer. The input data is the user's question data, and the output data is the analysis results.

[0678] Step 7:

[0679] The multimodal AI analyzes the received data comprehensively, searches for relevant information from the learning database, and generates the optimal answer. This answer is sent back to the server. The input data is the analysis result, and the output data is the generated answer.

[0680] Step 8:

[0681] The server sends the generated answer to the terminal. The terminal displays the answer to the user. The input data is the generated answer, and the output data is the answer displayed to the user.

[0682] Step 9:

[0683] The user inputs a follow-up question for the provided answer. The terminal sends the follow-up question to the server. The input data is the follow-up question, and the output data is the follow-up question data sent to the server.

[0684] Step 10:

[0685] The server requests the multimodal AI to perform reanalysis based on new question data and previous answer data. The multimodal AI performs detailed analysis and generates an additional answer. The server sends the generated answer to the terminal, which then provides it to the user again. The input data is the additional question data and previous answer data, and the output data is a detailed answer.

[0686] The above processing steps allow users to input questions in a variety of formats, and the data is analyzed in an integrated manner to provide consistent, high-quality answers. Furthermore, the user registration and authentication process is simplified, improving the efficiency of learning support.

[0687] (Application example 1)

[0688] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0689] Existing online learning systems mainly provide answers to questions using text, images, and audio, but lack the ability to provide visual answers using 3D models and animations that are easy for users to understand intuitively. Furthermore, there are only a limited number of systems that can respond to detailed follow-up questions in real time. Furthermore, there are limited means of providing learning support within the virtual environment, making it difficult for users to effectively progress through rich interactions.

[0690] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0691] In this invention, the server includes: means for a user to input a question in the form of text, image, or voice; means for transmitting the input question data to the server; means for analyzing the received question data and sending a request to an appropriate multimodal AI; means for the multimodal AI to analyze the question data and generate an answer; means for providing the generated answer to the user; means for the user to move around in the virtual environment in real time and access topic rooms specialized for a subject; and means for providing the answer visually using 3D models and animations in the virtual environment, thereby enabling the user to receive detailed learning support with high resolution.

[0692] A "user" is an individual or organization that uses the learning system.

[0693] "Text" is character information entered by the user.

[0694] "Images" are visual content uploaded by users.

[0695] "Audio" refers to sound information that is recorded and transmitted by the user.

[0696] "Question data" refers to the content of a question about learning that is input by the user in the form of text, image, or voice.

[0697] A "server" is a central device that receives question data sent by users, analyzes it, and generates answers.

[0698] "Multimodal artificial intelligence" is an artificial intelligence system that comprehensively analyzes data in multiple formats, such as text, images, and audio, and generates answers.

[0699] An "answer" is information that contains a solution or explanation to a user's question.

[0700] A "virtual environment" is a virtual space in which users can move around in real time and engage in learning experiences.

[0701] "Topic rooms" are study areas dedicated to specific subjects, rooms within a virtual environment staffed with dedicated artificial intelligence assistants.

[0702] A "3D model" is a three-dimensional digital object that visually represents a solution within a virtual environment.

[0703] "Animation" is a moving image effect that dynamically visualizes the solution within the virtual environment.

[0704] A "dedicated artificial intelligence assistant" is an artificial intelligence system that is placed in each topic room and is specialized to provide appropriate answers to users' questions.

[0705] "Registration information" refers to personal information and learning information provided by users in order to use the system.

[0706] "Verification Email" means an email sent to a User to confirm their registration information and activate their Account.

[0707] "Login Information" means the authentication data used by a User to access a System.

[0708] "Motion tracking" refers to tracking a user's position and actions within a virtual environment in real time.

[0709] This invention is a system that effectively supports online learning and is realized mainly through the interaction between a server, a terminal, and a user. Specifically, it includes the following components and the system functions resulting from their interaction:

[0710] User Registration and Login

[0711] First, in order to use the system using a terminal, a user must register when starting up the system for the first time. The user enters information such as their name, email address, grade, and subjects of interest, and this input data is sent from the terminal to the server. The server saves this information in a database and sends a confirmation email to the user's email address. When the user clicks the link in the confirmation email, the server validates the user's login information. After that, the user can log in and use all the system's functions.

[0712] Post a question and select the input format

[0713] If a user wants to receive learning support, they click the "Ask a Question" button on their device and are taken to a question input screen. They can enter their question in the form of text, image, or voice. For example, if they don't understand a math problem, they can enter "I don't understand this calculus problem" and upload an image of the relevant page from their textbook.

[0714] Question analysis and answer generation using multimodal AI

[0715] The device sends the question data entered by the user to the server. The server first analyzes the text data and identifies the type of question. Next, the server also analyzes the image data and audio data, and uses this information to seek a comprehensive understanding of the problem. Based on this analyzed data, the server sends a request to multimodal artificial intelligence (AI). The multimodal AI comprehensively analyzes the received data, searches for related information from a learning database, and generates the optimal answer.

[0716] Providing answers

[0717] The server sends the answer generated by the multimodal AI to the device. For example, for a calculus problem, the generated answer includes a step-by-step process such as "first find the derivative, then integrate." The device displays the answer to the user and provides visual support information such as diagrams as needed.

[0718] Virtual Learning Support Center

[0719] This system provides users with a means to move around in a virtual environment in real time and access topic rooms specialized for specific subjects. When a user enters a topic room related to a specific subject, a dedicated AI assistant is deployed to provide appropriate answers to the user's questions. Answers are visually represented using 3D models and animations, making them easier for users to understand intuitively. Users can also add more detailed questions on the spot and receive detailed answers in real time.

[0720] Examples of concrete examples and prompts

[0721] As a concrete example, when a high school student wants to solve a calculus problem in mathematics, the following process is taken: The user enters a topic room and enters the following prompt sentence into the terminal:

[0722] "I don't understand a calculus problem. Can you give me detailed steps to solve it?"

[0723] "I don't understand this calculation step. Please explain it again in more detail."

[0724] This allows the system to provide users with the information they need in real time, greatly improving the learning experience.

[0725] Hardware and software used

[0726] The following hardware and software are used to realize this system. The hardware includes a server, a user device (a smartphone or personal computer), and a head-mounted display. The software uses Python, Unity (for building the VR environment), and TensorFlow (for the AI ​​model).

[0727] The above configuration allows users to receive learning support efficiently and effectively.

[0728] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0729] Step 1:

[0730] When a user starts up the device and uses it for the first time, they are taken to the user registration screen. The user enters their name, email address, grade, and subjects of interest, and this information is sent from the device to the server. Input data: Name, email address, grade, and subjects of interest. Output data: User registration information saved on the server.

[0731] Step 2:

[0732] The server saves the user registration information in a database and sends a confirmation email to the user's email address. The confirmation email contains a link to activate the account. Input data: User registration information. Output data: Confirmation email.

[0733] Step 3:

[0734] The user clicks the link in the confirmation email and the server validates the user's login information. Input data: Click on the link in the confirmation email. Output data: Validated user login information.

[0735] Step 4:

[0736] The user logs in using a terminal and clicks the "Ask a Question" button to move to the question input screen. The user enters the question in the form of text, image, or voice. Input data: text, image, voice. Output data: question data.

[0737] Step 5:

[0738] The device sends the entered question data to the server. The server receives the question data and first performs text analysis. Next, it analyzes the image and audio data. Input data: Question data. Output data: Analyzed question data.

[0739] Step 6:

[0740] The server sends a request to the appropriate multimodal AI based on the analyzed question data. Input data: Analyzed question data. Output data: Request sent to the AI.

[0741] Step 7:

[0742] Multimodal AI comprehensively analyzes question data, searches for relevant information, and generates optimal answers. Input data: Question data. Output data: Generated answers.

[0743] Step 8:

[0744] The server receives the answer data generated by the multimodal AI and sends it to the terminal. The terminal provides this answer to the user. Input data: Generated answer data. Output data: Answer displayed on the terminal.

[0745] Step 9:

[0746] If the user wants to ask a more detailed question, he or she inputs the additional question through the terminal. The terminal sends the additional question data to the server. Input data: Additional question data. Output data: Additional question data sent to the server.

[0747] Step 10:

[0748] The server re-analyzes the additional question data and sends a request to the multimodal AI again to generate a detailed answer. Input data: Additional question data. Output data: The request sent again to the AI.

[0749] Step 11:

[0750] The multimodal AI generates a detailed answer and sends it to the terminal via the server. The terminal displays this answer to the user. Input data: detailed answer data. Output data: detailed answer displayed on the terminal.

[0751] Step 12:

[0752] Users can move freely around the virtual environment and access topic rooms specific to their subject. A dedicated AI assistant is deployed to provide visual answers to questions using 3D models and animations. Input data: user's location, question. Output data: visual answers using 3D models and animations.

[0753] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0754] The present invention is a system for effectively supporting online learning, which is combined with an emotion engine for recognizing a user's emotions and providing appropriate support in accordance with those emotions. It can be implemented as follows.

[0755] User Registration and Login

[0756] The first step for users to use the system is to install the app and register as a user when they first start it. On the registration screen displayed on the device, the user enters their name, email address, grade, and subjects of interest. This input data is sent from the device to the server, which stores it in a database. Once registration is complete, the server sends a confirmation email to the user's email address, and the user clicks on the link in the confirmation email to activate their account. After that, the user can log in and use all of the system's functions.

[0757] Post a question and select the input format

[0758] If a user wants learning support, they click the "Ask a Question" button, which takes them to a question input screen. The device then displays options for inputting questions in the form of text, image, or voice. For example, if a user doesn't understand a math problem, they can type "I don't understand this calculus problem" and upload an image of the corresponding page from their textbook.

[0759] Question analysis and answer generation using multimodal AI

[0760] The device sends the question data entered by the user to the server. The server first analyzes the text data and identifies the type of question. Next, the server also analyzes the image data and audio data, and uses this information to seek a comprehensive understanding of the problem. Based on this analyzed data, the server sends a request to the multimodal AI. The multimodal AI comprehensively analyzes the received data, searches for related information from a learning database, and generates the optimal answer.

[0761] Emotion recognition and response using emotion engine

[0762] The server not only receives the question data but also sends it to the emotion engine to recognize the user's emotion at the time of question entry. The emotion engine analyzes the user's emotion from text data, image data, and voice data and determines whether it is positive or negative. Based on the results of this emotion recognition, the multimodal AI adjusts the answer it generates. For example, if the user expresses negative emotions, it adds a message of encouragement or support.

[0763] Providing answers

[0764] The server sends the generated answer to the device. For example, for a calculus problem, the generated answer includes a step-by-step process such as "first find the derivative, then integrate." The device displays the answer to the user and provides visual support information such as diagrams as needed. In addition, the device may display an encouraging message based on the user's emotion recognition results.

[0765] Follow-up questions and real-time feedback

[0766] If the user wants to ask a more detailed question about the provided answer, they can enter the question again. For example, they can enter an additional question such as, "I don't understand the calculation in between." The device then sends this additional question to the server, and the server again requests a more detailed answer from the multimodal AI based on the previous answer data and the new question data. The multimodal AI then performs a detailed analysis and generates an additional answer, which the server then sends to the device. The device then provides this detailed answer to the user, helping them to deepen their understanding.

[0767] Specific examples

[0768] If high school student A wants to solve a calculus problem in mathematics, he or she first registers as a user and logs in. Next, he or she enters the text "I don't understand this calculus problem" on the question input screen and uploads an image of the textbook. The server analyzes the text content and image and uses an emotion engine to recognize A's emotions. If A shows negative emotions, the AI ​​generates an answer that includes an encouraging message. The server sends this answer to A's device, where A receives it. If A asks an additional question, such as "I don't understand the calculation in the middle," the multimodal AI performs a more detailed analysis again and provides an additional answer.

[0769] This system allows users to receive prompt and consistent learning support, and by using an emotion engine, it is possible to provide even more personalized support, which further improves learning efficiency and allows users to enjoy a better learning experience.

[0770] The processing flow will be explained below.

[0771] Step 1:

[0772] The user installs the app and clicks the "Register" button when launching it for the first time.

[0773] Step 2:

[0774] The terminal displays a screen that prompts the user to enter registration information such as name, email address, grade, and subjects of interest.

[0775] Step 3:

[0776] The user enters their name, email address, grade, and subjects of interest and clicks the "Submit" button.

[0777] Step 4:

[0778] The terminal transmits the entered registration data to the server.

[0779] Step 5:

[0780] The server stores the received data in a database and sends a confirmation email to the user's email address.

[0781] Step 6:

[0782] The user clicks on the link in the confirmation email to activate their account.

[0783] Step 7:

[0784] The server accepts the confirmation link and validates the user's login information.

[0785] Step 8:

[0786] The user logs in to the app and navigates to the home screen.

[0787] Step 9:

[0788] The user clicks the "Ask a Question" button on the home screen.

[0789] Step 10:

[0790] The device will display a question entry screen and provide the option to enter the question in the form of text, image, or voice.

[0791] Step 11:

[0792] A user types in the text "I don't understand this calculus problem" and uploads an image of the corresponding page from the textbook.

[0793] Step 12:

[0794] The terminal sends the entered question data (text, image) to the server.

[0795] Step 13:

[0796] The server analyzes the text data and performs initial filtering.

[0797] Step 14:

[0798] The server sends the image data to an image analysis module to extract relevant information.

[0799] Step 15:

[0800] The server sends the voice data to a voice analysis module, which analyzes the voice tone and content.

[0801] Step 16:

[0802] The server integrates the text, image, and voice analysis results and sends them to the emotion engine.

[0803] Step 17:

[0804] The emotion engine analyzes the received data and determines the user's emotion. If the emotion is negative, it revisits the specific situation and evaluates whether an encouraging message needs to be added.

[0805] Step 18:

[0806] The emotion engine sends the emotion recognition results back to the server.

[0807] Step 19:

[0808] The server sends an analysis request to the multimodal AI based on the emotion recognition result. If the emotion is negative, the request includes an encouraging message or a special answer format.

[0809] Step 20:

[0810] Multimodal AI integrates and analyzes question data and emotional information to understand the content of the problem.

[0811] Step 21:

[0812] Multimodal AI collates information from relevant learning databases to generate optimal answers, supplemented with emotional messages where appropriate.

[0813] Step 22:

[0814] The server receives the generated answer data and transmits it to the user's terminal.

[0815] Step 23:

[0816] The device displays the answer data to the user, including visual support information such as text, formulas, diagrams, etc. If the emotion is negative, an encouraging message is also displayed.

[0817] Step 24:

[0818] If the user wants to get more details about the answer, he / she can input and submit an "additional question."

[0819] Step 25:

[0820] The terminal sends a follow-up question to the server.

[0821] Step 26:

[0822] The server again sends an analysis request for additional questions to the multimodal AI.

[0823] Step 27:

[0824] Multimodal AI then performs a detailed analysis again to generate additional detailed answers.

[0825] Step 28:

[0826] The server transmits the generated additional answer to the user's terminal.

[0827] Step 29:

[0828] The device displays additional answers to the user to help them deepen their understanding.

[0829] Through these steps, users can receive fast and consistent learning support, and the emotional engine can provide more personalized assistance, improving learning efficiency and providing a more meaningful learning experience for users.

[0830] Example 2

[0831] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0832] Conventional online learning support systems lack the flexibility to respond to the diverse question formats users encounter, and can only accept questions in specific formats. Furthermore, they provide uniform answers without considering changes in the user's motivation or emotions regarding learning, making it impossible to provide support optimized for each individual user. Furthermore, it is difficult to provide quick and detailed real-time feedback on follow-up questions, which hinders users' learning efficiency.

[0833] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for a user to input a question in any one of the formats of text, image, or voice, a means for transmitting the input question data to the server, a means for analyzing the received question data and transmitting a request to an appropriate multimodal AI, a means for the multimodal AI to analyze the question data and generate an answer, a means for providing the generated answer to the user, and a means for analyzing the user's emotions using an emotion recognition engine and adjusting the content of the answer based on the analyzed emotion data. This makes it possible to respond to various types of questions from users and provide optimal answers that take into account each individual's emotional state.

[0834] "User" refers to any person or entity that utilizes the online learning system to enter questions and receive answers.

[0835] "Text" refers to a digital data format that includes character information, and refers to character string data that a user uses to input a question.

[0836] "Image" refers to a digital data format containing visual information, such as a photograph or drawing uploaded by a user to supplement a question.

[0837] "Speech" refers to a digital data format containing auditory information, and refers to voice data used by a user to input a question.

[0838] "Server" refers to a computer system or cloud infrastructure for processing, analyzing, storing, and generating answers from data received from users.

[0839] "Multimodal AI" refers to an AI model that comprehensively analyzes different types of data (text, images, audio) and generates optimal answers.

[0840] An "emotion recognition engine" refers to software or algorithms that analyze emotions from user input data and determine whether they are positive or negative.

[0841] "Answer" refers to output data generated by multimodal artificial intelligence that contains appropriate information and instructions in response to a user's question.

[0842] "Confirmation Email" refers to the email sent from the server at the time of user registration for the purpose of activating the account.

[0843] "Confirmation Link" refers to the URL included in the confirmation email that the User clicks to activate their account.

[0844] "Login Information" refers to the authentication information (e.g., user ID and password) used by a User to access the System.

[0845] The present invention is a system for supporting online learning, in which a user inputs a question in the form of text, image, or voice, and generates an answer to the question using multimodal artificial intelligence and an emotion recognition engine. Specific embodiments of the present invention are described below.

[0846] First, a user installs and launches the online learning application on a device such as a smartphone. In order to use the system, the user must first register. On the registration screen, the user enters their name, email address, grade, and subjects of interest, and this data is sent from the device to the server. The server stores this data in a database (e.g., MySQL database) and sends a confirmation email to the user. The user clicks the link in the confirmation email to activate their account.

[0847] Next, when the user clicks the "Ask a Question" button, the device displays a screen where the user can enter a question in the form of text, image, or voice. For example, the user can enter the text "I don't understand this calculus problem" and upload an image of the corresponding page from the textbook. This input data is sent from the device to the server.

[0848] The server analyzes the received text data using a natural language processing library (e.g., NLTK, SpaCy), and analyzes the image data using an image processing library (e.g., OpenCV). The analyzed data is sent to a multimodal AI, which uses a generative AI model such as GPT-4 to comprehensively analyze the received data and generate the optimal answer.

[0849] The server then sends the received question data to an emotion recognition engine. The emotion recognition engine uses the Emotion API or other tools to analyze the user's emotions from text, image, and audio data and determine whether they are positive or negative. Based on the emotion recognition results, the multimodal AI adjusts the answer it generates. For example, if the user expresses negative emotions, it adds a message of encouragement or support.

[0850] The generated answer is sent from the server to the device, which then displays it to the user. Visual support information (e.g., diagrams) is also displayed as needed. If the user enters additional questions, the multimodal AI again performs a detailed analysis, generating the optimal answer and sending it from the server to the device.

[0851] As a concrete example, if high school student A wants to solve a calculus problem in mathematics, he or she would follow the steps below. First, he or she registers as a user and logs in. Next, he or she enters the text "I don't understand this calculus problem" on the question input screen and uploads an image of the textbook. The server analyzes the text content and image and uses an emotion recognition engine to recognize A's emotions. If A shows negative emotions, the AI ​​generates an answer that includes an encouraging message. The server sends this answer to A's device, where A receives it. If A asks an additional question, such as "I don't understand the calculation in the middle," a detailed analysis is performed again and the answer is provided to the device.

[0852] An example of a prompt sentence is as follows:

[0853] User: I don't understand this calculus problem. (Image attached) (Image: Calculus problem page in textbook)

[0854] Server: Performs text and image analysis to generate the optimal answer.

[0855] Emotion Recognition Engine: Analyzes the user's emotions and finds that they are negative. Adds encouraging messages.

[0856] Server: Generates the best answer and sends it to the device along with an encouraging message.

[0857] This invention allows users to receive prompt and consistent learning support, and by using an emotion recognition engine, it is possible to provide even more personalized support to users, significantly improving learning efficiency.

[0858] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0859] Step 1:

[0860] The user installs and launches the app.

[0861] A user downloads the app from a digital store and installs it on their device.

[0862] Launch the app and the first screen will appear.

[0863] Input: User downloads and launches the app

[0864] Output: First screen display of the app

[0865] Step 2:

[0866] The terminal displays a user registration screen.

[0867] A registration screen UI is displayed, including fields for the user to enter their name, email address, grade level, and subjects of interest.

[0868] Input: Registration screen displayed after launching the app

[0869] Output: A field for the user to enter input.

[0870] Step 3:

[0871] The user enters the required information and presses the registration button.

[0872] The user enters their name, email address, grade, and subjects of interest and presses the "Register" button.

[0873] Input: Information entered by the user (name, email address, grade, subjects of interest)

[0874] Output: Sending input data by pressing the register button

[0875] Step 4:

[0876] The terminal sends the input data to the server.

[0877] The terminal sends the data entered by the user to the server in JSON format at the REST API endpoint.

[0878] Input: Information entered by the user

[0879] Output: Send data in JSON format

[0880] Step 5:

[0881] The server stores the registration information in a database.

[0882] The server analyzes the received data and stores it in a database (e.g. MySQL).

[0883] Input: JSON format user information data

[0884] Output: User information stored in the database

[0885] Step 6:

[0886] The server sends a confirmation email.

[0887] The server uses a mail server (e.g. Postfix) to send the user a confirmation email containing a link to activate the account.

[0888] Input: User information stored in the database

[0889] Output: Send confirmation email

[0890] Step 7:

[0891] The user clicks the confirmation link.

[0892] The user opens the confirmation email and activates their account by clicking the link.

[0893] Input: Link in confirmation email

[0894] Output: Account enabled

[0895] Step 8:

[0896] The server performs the account activation process.

[0897] The server receives the link and updates the status of the corresponding account to "active."

[0898] Input: The confirmation link the user clicked

[0899] Output: Enabled account status

[0900] Step 9:

[0901] A user logs in.

[0902] Using your user ID and password, you can access the login screen and use all the functions of the system.

[0903] Input: User ID and password

[0904] Output: Successfully logged into the system

[0905] Step 10:

[0906] The user clicks the "Ask a Question" button.

[0907] A user clicks the "Ask a Question" button within the app.

[0908] Input: User clicks "Ask a question" button

[0909] Output: Display of question input screen

[0910] Step 11:

[0911] The device displays a question input screen.

[0912] Displays a question input screen with text, image, and audio input options.

[0913] Input: User clicks "Ask a question" button

[0914] Output: Display of question input screen

[0915] Step 12:

[0916] The user enters a question.

[0917] A user types "I don't understand this calculus problem" into a text box and uploads an image of a page from a textbook.

[0918] Input: Question text entered by the user, and uploaded images

[0919] Output: Question data ready to send

[0920] Step 13:

[0921] The terminal transmits the input data to the server.

[0922] The text data and image data are sent to a server.

[0923] Input: Question data entered by the user (text, images)

[0924] Output: Send data to the server

[0925] Step 14:

[0926] The server parses the text data.

[0927] The server analyzes the text data using a natural language processing library (e.g., NLTK, SpaCy) to identify the question.

[0928] Input: Received text data

[0929] Output: Analysis results (question content identification)

[0930] Step 15:

[0931] The server analyzes the image data.

[0932] The server analyzes the image data using an image processing library (e.g., OpenCV) and extracts relevant information.

[0933] Input: Received image data

[0934] Output: Analysis results (image content analysis)

[0935] Step 16:

[0936] The server sends the analysis data to the multimodal AI.

[0937] The analysis results are integrated and sent to multimodal AI.

[0938] Input: Text and image analysis results

[0939] Output: A request to the multimodal AI

[0940] Step 17:

[0941] Multimodal AI generates optimal answers.

[0942] Generative AI models such as GPT-4 are used to analyze the received data and generate optimal answers.

[0943] Input: Integrated analysis data

[0944] Output: The generated answer

[0945] Step 18:

[0946] The server sends the answer data to the emotion recognition engine.

[0947] The emotion recognition engine analyzes the user's emotions along with the generated answer data.

[0948] Input: Answer data

[0949] Output: User sentiment analysis results

[0950] Step 19:

[0951] An emotion recognition engine analyzes the user's emotions.

[0952] The Emotion API is used to determine the user's emotions from the data sent.

[0953] Input: Answer data

[0954] Output: Sentiment analysis result (positive / negative)

[0955] Step 20:

[0956] The server adjusts the answer based on the emotional data.

[0957] Based on the results of emotion recognition, the multimodal AI is instructed to adjust its answers, for example by adding encouraging messages.

[0958] Input: Sentiment analysis results

[0959] Output: Adjusted answer

[0960] Step 21:

[0961] The server sends the answer data to the terminal.

[0962] The optimized answer is sent to the device.

[0963] Input: Adjusted answer data

[0964] Output: Send answer data to the device

[0965] Step 22:

[0966] The terminal displays the answer to the user.

[0967] The terminal displays the answer and supplementary information to the user.

[0968] Input: Answer data received from the server

[0969] Output: What is displayed to the user

[0970] Step 23:

[0971] The user enters a follow-up question.

[0972] Enter additional detailed questions in text for the answers provided.

[0973] Input: Additional questions from the user

[0974] Output: Additional question data

[0975] Step 24:

[0976] The terminal sends a follow-up question to the server.

[0977] Send additional question data to the server.

[0978] Input: Additional question data by the user

[0979] Output: Send data to the server

[0980] Step 25:

[0981] The server reanalyzes the data and sends the request to the multimodal AI again.

[0982] Send additional question data to the multimodal AI to generate detailed answers.

[0983] Input: Additional question data

[0984] Output: Detailed answer request

[0985] Step 26:

[0986] Multimodal AI generates detailed answers.

[0987] Generate detailed answers based on additional question data.

[0988] Input: Additional question data

[0989] Output: Detailed answer

[0990] Step 27:

[0991] The server sends detailed answer data to the terminal.

[0992] The generated detailed answer is sent to the device.

[0993] Input: Detailed answer data

[0994] Output: Detailed answer data sent to the device

[0995] Step 28:

[0996] The terminal displays the detailed answer to the user.

[0997] The terminal displays the detailed answer content to the user.

[0998] Input: Detailed answer data received from the server

[0999] Output: What is displayed to the user

[1000] In this way, the system can effectively execute a series of processes to receive a user's question, analyze it, and provide the most appropriate answer.

[1001] (Application example 2)

[1002] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1003] Conventional online learning systems can generate answers by analyzing text, images, and audio data provided by users, but they rarely take the user's emotional state into consideration. This can lead to problems such as a decrease in learning motivation and a lack of appropriate feedback, resulting in a decrease in learning effectiveness. Furthermore, when asked follow-up questions about answers, real-time feedback that reflects the user's emotional changes cannot be provided, making it difficult to provide advanced support tailored to individual learners.

[1004] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input a question in the form of text, image, or voice; a means for transmitting the input question data to the server; a means for analyzing the received question data and sending a request to an appropriate multimodal AI; a means for the multimodal AI to analyze the question data and generate an answer; a means for providing the generated answer to the user; and an emotion engine that recognizes the user's emotions and provides a response according to the emotion. This provides appropriate support that takes into account the user's emotional state, thereby improving learning effectiveness. In addition, the server includes a means for a user to ask a follow-up question in response to the answer; a means for transmitting the follow-up question to the server, reanalyzing the question, and generating a detailed answer; a means for providing the detailed answer to the user; and a means for providing real-time feedback according to changes in the user's emotions, thereby realizing advanced support tailored to individual learners. Furthermore, the system includes a means for the server to store the user's registration information in a database and send a confirmation email, a means for validating the login information of a user who clicks on a confirmation link, and a means for initializing the user's emotional data at the time of login and providing support tailored to each individual learner, thereby enabling more personalized learning support to be provided.

[1005] A "user" is an individual who utilizes the system to enter questions and receive answers.

[1006] "Text" refers to the written information that a user inputs into the system.

[1007] "Image" means information in the form of a still image that a user uploads to the system.

[1008] "Speech" refers to the acoustic information provided by the user to the system.

[1009] "Question Data" refers to information entered into the system by a user in the form of text, images, or voice.

[1010] A "server" is a computer system that analyzes question data received from a user and generates an appropriate answer.

[1011] "Multimodal AI" refers to AI that analyzes and integrates multiple data formats, such as text, images, and audio, to generate answers.

[1012] An "emotion engine" is a system that recognizes emotions from user input data and provides a response that corresponds to that emotion.

[1013] An "answer" refers to response information generated based on question data analyzed by multimodal artificial intelligence.

[1014] "Registration Information" refers to personal information such as name, email address, grade, and subjects of interest that a user provides when registering with the system.

[1015] "Verification Email" means an email sent by the System to verify a User's registration information and activate their Account.

[1016] "Login information" refers to the authentication information (user name and password) used by a user to access a system.

[1017] "Emotional state" indicates the psychological state of the user when he or she inputs the question.

[1018] "Real-time feedback" refers to a system that provides immediate responses to users' questions and follow-up questions.

[1019] The present invention provides an online learning support system that combines an emotion engine that recognizes a user's emotions and provides appropriate support in accordance with those emotions. Specific embodiments of this system will be described below.

[1020] User Registration and Login

[1021] When using the system for the first time, the user installs a dedicated app on their smartphone and launches it. When launched for the first time, a user registration screen is displayed. The user enters their name, email address, grade, and subjects of interest. This input data is sent from the smartphone to the server, which stores it in a database. After registration is complete, the server sends a confirmation email to the user's email address, and the user activates their account by clicking the confirmation link. After that, the user can log in and use all of the system's functions.

[1022] Question posting and emotion recognition

[1023] When a user wants to receive learning support, they click the "Ask a Question" button in the app, which takes them to a question input screen. Here, the user can enter their question in the form of text, image, or voice. The entered question data is sent from the smartphone to the server. The server analyzes the received question data and identifies the content of the question. At the same time, the emotion engine analyzes the question data and recognizes the user's emotional state.

[1024] Answer generation and emotion regulation

[1025] The server sends a request to the multimodal AI based on the analyzed data. The multimodal AI comprehensively analyzes the received data and generates an answer. At that time, the answer is adjusted based on the analysis results of the emotion engine according to the user's emotions. For example, if the user shows negative emotions, the AI ​​adds a message of encouragement or support.

[1026] Providing answers

[1027] The generated answer is sent from the server to the smartphone and provided to the user. For example, the generated answer to a math problem may include a step-by-step process such as "first find the derivative, then do the integral." Encouraging messages based on the emotion recognition results may also be displayed.

[1028] Follow-up questions and real-time feedback

[1029] If the user wants to ask a more detailed question about the provided answer, they can enter the question again. For example, they can enter an additional question such as, "I don't understand the calculation in between." The smartphone again sends this additional question to the server, and the server again requests a more detailed answer from the multimodal AI based on the previous answer data and the new question data. The multimodal AI then performs a detailed analysis and generates an additional answer, which the server then sends to the smartphone. The smartphone then provides this detailed answer to the user, helping them to deepen their understanding.

[1030] Hardware and software used

[1031] The hardware used is a smartphone camera, and the software used is image preprocessing using OpenCV, emotion recognition modeling using TensorFlow / Keras, and text emotion analysis using TextBlob.

[1032] Examples and prompts

[1033] For example, when a user types, "Today's test results were poor," the sentiment engine detects negative sentiment from the text, and the AI ​​adds an encouraging message such as, "Failure is the mother of success, so try harder next time!"

[1034] Example prompt sentence:

[1035] Classify the emotion of the given sentence as positive or negative. Specifically, if the sentence contains negative expressions such as "I don't understand," "It's difficult," or "It's hard," output an appropriate encouraging message.

[1036] For example, given the following input:

[1037] Today's test results were bad

[1038] As described above, the system of the present invention provides support according to the user's emotional state, improves learning efficiency, and provides optimal feedback to each individual learner.

[1039] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1040] Step 1:

[1041] A user installs a dedicated app on their smartphone and launches the app. A user registration screen appears, where the user enters their name, email address, grade, and subjects of interest. This input data is sent from the device to the server. The server stores the received registration information in a database and sends a confirmation email to the user's email address. The user activates their account by clicking the confirmation link in the email. The entered data is in text format, and the server stores it in the database. The output is the account activation status.

[1042] Step 2:

[1043] The user logs in and clicks the "Ask a Question" button to move to the question input screen. The user enters the question in the form of text, image, or voice. This data is sent from the terminal to the server. The server processes the received question data using one of the following methods: text analysis, image analysis, or voice analysis. The input is the question data in the respective data format, and the output is the analyzed question content.

[1044] Step 3:

[1045] The emotion engine analyzes the question data and recognizes the user's emotional state. The server uses the emotion engine to identify the user's emotion from text data, image data, and audio data. For example, for text, it uses TextBlob to calculate the emotion score. The input is the question data in each format, and the output is the recognized emotional state (positive or negative).

[1046] Step 4:

[1047] The server sends a request to a multimodal AI based on the analyzed data. The multimodal AI integrates text, image, and audio data to generate an answer. To generate the answer, a generative AI model is used to search and integrate information from relevant knowledge bases. The input is the analyzed data, and the output is the generated answer.

[1048] Step 5:

[1049] The generated answer is sent from the server to the device and provided to the user. Based on the results of question data analysis and the recognition results of the emotion engine, an encouraging message is added as needed. The device displays this information on the screen. The input is the generated answer and encouraging message, and the output is the answer and message displayed to the user.

[1050] Step 6:

[1051] If the user wants to enter a more detailed question in response to the provided answer, the question is entered again and the additional question data is sent from the terminal to the server. The server then re-analyzes the previous answer data and the new question data to generate a detailed answer. The input is the additional question data and the output is the detailed answer.

[1052] Step 7:

[1053] The server sends the detailed answer to the terminal and provides it back to the user. The terminal displays the detailed answer to the user and provides further assistance or feedback as needed. The input is the detailed answer and any required feedback messages, and the output is the detailed answer and feedback displayed to the user.

[1054] Through these steps, users can learn efficiently while receiving appropriate support according to their emotions, thereby improving the effectiveness and satisfaction of their learning.

[1055] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1056] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1057] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1058] [Third embodiment]

[1059] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1060] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1061] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1062] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1063] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1064] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1065] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1066] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1067] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1068] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1069] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1070] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1071] The present invention is a system for effectively supporting online learning, and can be implemented as follows.

[1072] User Registration and Login

[1073] The first step for users to use the system is to install the app and register as a user when they first start it. On the registration screen displayed on the device, the user enters their name, email address, grade, and subjects of interest. This input data is sent from the device to the server, which stores it in a database. Once registration is complete, the server sends a confirmation email to the user's email address, and the user clicks on the link in the confirmation email to activate their account. After that, the user can log in and use all of the system's functions.

[1074] Post a question and select the input format

[1075] If a user wants learning support, they click the "Ask a Question" button, which takes them to a question input screen. The device then displays options for inputting questions in the form of text, image, or voice. For example, if a user doesn't understand a math problem, they can type "I don't understand this calculus problem" and upload an image of the corresponding page from their textbook.

[1076] Question analysis and answer generation using multimodal AI

[1077] The device sends the question data entered by the user to the server. The server first analyzes the text data and identifies the type of question. Next, the server also analyzes the image data and audio data, and uses this information to seek a comprehensive understanding of the problem. Based on this analyzed data, the server sends a request to the multimodal AI. The multimodal AI comprehensively analyzes the received data, searches for related information from a learning database, and generates the optimal answer.

[1078] Providing answers

[1079] The server sends the answer generated by the multimodal AI to the device. For example, for a calculus problem, the generated answer includes a step-by-step process such as "first find the derivative, then integrate." The device displays the answer to the user and provides visual support information such as diagrams as needed.

[1080] Follow-up questions and real-time feedback

[1081] If the user wants to ask a more detailed question about the answer provided, they can enter the question again. For example, they can enter an additional question such as, "I don't understand the intermediate calculations." The device then sends this additional question to the server, and the server again requests a more detailed answer from the multimodal AI based on the previous answer data and the new question data. The multimodal AI then performs a detailed analysis and generates an additional answer, which the server then sends to the device. The device then provides this detailed answer to the user so that they can review it again.

[1082] Specific examples

[1083] If high school student A wants to solve a calculus problem in mathematics, he or she first registers as a user and logs in. Next, he or she enters the text "I don't understand this calculus problem" on the question input screen and uploads an image of the textbook. The server analyzes the text content and image and sends a request to the multimodal AI, which then generates a step-by-step answer. This answer is then sent to A's device via the server. If A asks a follow-up question, such as "I don't understand the intermediate calculations," the multimodal AI performs another detailed analysis and provides an additional answer.

[1084] This system allows users to receive prompt and consistent learning support, significantly improving learning efficiency. In addition, because it is not dependent on the teacher's availability or skills, users can receive high-quality learning support anytime, anywhere.

[1085] The processing flow will be explained below.

[1086] Step 1:

[1087] The user installs the app and clicks the "Register" button when launching it for the first time.

[1088] Step 2:

[1089] The terminal displays a screen that prompts the user to enter registration information such as name, email address, grade, and subjects of interest.

[1090] Step 3:

[1091] The user enters their name, email address, grade, and subjects of interest and clicks the "Submit" button.

[1092] Step 4:

[1093] The terminal transmits the entered registration data to the server.

[1094] Step 5:

[1095] The server stores the received data in a database and sends a confirmation email to the user's email address.

[1096] Step 6:

[1097] The user clicks on the link in the confirmation email to activate their account.

[1098] Step 7:

[1099] The server accepts the confirmation link and validates the user's login information.

[1100] Step 8:

[1101] The user logs in to the app and navigates to the home screen.

[1102] Step 9:

[1103] The user clicks the "Ask a Question" button on the home screen.

[1104] Step 10:

[1105] The device will display a question entry screen and provide the option to enter the question in the form of text, image, or voice.

[1106] Step 11:

[1107] A user types in the text "I don't understand this calculus problem" and uploads an image of the corresponding page from the textbook.

[1108] Step 12:

[1109] The terminal sends the entered question data (text, image) to the server.

[1110] Step 13:

[1111] The server analyzes the text data and performs initial filtering.

[1112] Step 14:

[1113] The server sends the image data to an image analysis module to extract relevant information.

[1114] Step 15:

[1115] The server integrates the text and image analysis results and sends an analysis request to the multimodal AI.

[1116] Step 16:

[1117] Multimodal AI integrates and analyzes question data to understand the content of the problem.

[1118] Step 17:

[1119] Multimodal AI collates information from relevant learning databases to generate optimal answers.

[1120] Step 18:

[1121] The server receives the generated answer data and transmits it to the user's terminal.

[1122] Step 19:

[1123] The terminal displays the answer data to the user and, if necessary, displays supplementary information such as diagrams.

[1124] Step 20:

[1125] If the user wants to get more details about the answer, he / she can input and submit an "additional question."

[1126] Step 21:

[1127] The terminal sends a follow-up question to the server.

[1128] Step 22:

[1129] The server again sends an analysis request for additional questions to the multimodal AI.

[1130] Step 23:

[1131] Multimodal AI then performs a detailed analysis again to generate additional detailed answers.

[1132] Step 24:

[1133] The server transmits the generated additional answer to the user's terminal.

[1134] Step 25:

[1135] The device displays additional answers to the user to help them deepen their understanding.

[1136] This allows users to receive prompt and consistent learning support, improving learning efficiency and providing high-quality learning support that is not dependent on the availability or skills of teachers.

[1137] Example 1

[1138] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1139] Modern online learning systems allow users to input questions in a variety of formats, including text, images, and audio. However, they lack the technology to effectively analyze and integrate this data to generate answers. Furthermore, they lack a mechanism for efficiently reanalyzing and providing detailed answers when users ask follow-up questions about answers already provided. Furthermore, the user registration and authentication process is cumbersome, and account activation via a confirmation link can sometimes be slow. There is a need to address these issues and provide efficient, high-quality online learning support.

[1140] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1141] In this invention, the server includes: a means for a user to input a question in the form of text, image, or voice; a means for transmitting the input question data to the server; a means for analyzing the received question data and sending a request to an appropriate multimodal artificial intelligence; a means for the multimodal artificial intelligence to analyze the question data and generate an answer; a means for accepting additional questions, reanalyzing the data, and generating a detailed answer; a means for the server to store user registration information in a database and send a confirmation email; a means for activating the account of a user who clicks a confirmation link; and a means for the server to provide the generated answer to the user step by step and present the answer including visual auxiliary information. This allows users to input questions in various formats, have the data analyzed in an integrated manner, and receive consistent, high-quality answers. Furthermore, the user registration and authentication process is simplified, improving the efficiency of learning support.

[1142] 1. "User" refers to an individual or organization that uses the online learning system to enter questions and receive answers.

[1143] 2. "Text" refers to a data format consisting of characters or sentences, used by users to enter questions or comments.

[1144] 3. "Image" means a data format containing still images and graphics, used to provide users with visual information.

[1145] 4. "Audio" means a data format containing sounds or spoken words, used by a user to provide information orally.

[1146] 5. "Question Data" refers to the collection of data including all information entered by the User in text, image, and audio format.

[1147] 6. "Server" refers to the computer system that receives and analyzes query data and interacts with the Multimodal Artificial Intelligence.

[1148] 7. “Analysis” refers to the process of understanding the input question data and identifying its meaning and intent.

[1149] 8. "Multimodal AI" refers to AI technology that analyzes multiple data formats, such as text, images, and audio, and generates integrated answers.

[1150] 9. "Request" refers to instructions or requests sent by a server to a multimodal artificial intelligence to request analysis and answer generation.

[1151] 10. “Answer” refers to the response or reply generated by a Multimodal Artificial Intelligence to a user’s question.

[1152] 11. "Providing" refers to the process of displaying or notifying the user of a generated answer.

[1153] 12. "Follow-up Question" means a question entered by a User to request further information or clarification on an Initial Answer.

[1154] 13. "Reanalysis" refers to the process of analyzing new data based on additional questions to generate detailed answers.

[1155] 14. "Detailed answer" refers to a more specific and useful answer generated based on reanalysis.

[1156] 15. "User registration information" refers to basic information such as name, email address, grade, and areas of interest that a user enters to use the system.

[1157] 16. "Database" refers to a system for systematically storing and managing a large amount of information, such as user registration information and question data.

[1158] 17. "Verification Email" means the email sent to a User prompting them to activate their Account.

[1159] 18. "Confirmation Link" means the URL included in the confirmation email that the User clicks to activate their account.

[1160] 19. "Account Activation" refers to the process by which a User's account is activated and all its features are made available by clicking on a confirmation link.

[1161] 20. "Step-by-step" refers to a method of explaining a complex problem to a user by breaking it down into a series of simple steps.

[1162] 21. "Visual aids" refers to information containing non-textual visual data, such as charts or images, that is provided to make the answer easier to understand.

[1163] MODE FOR CARRYING OUT THE INVENTION

[1164] The present invention is a system for effectively supporting online learning, and specific embodiments thereof will be described below.

[1165] User Registration and Login

[1166] To use the system, users must first install the application. When the system is launched for the first time, the user registers their name, email address, grade, and subjects of interest. This data is collected by the device, which then sends it to the server. The server stores the received data in a database and sends a confirmation email to the user. When the user clicks the link in the confirmation email, their account is activated. After that, the user can log in by entering their email address and password on the login screen to use all of the system's functions.

[1167] Post a question and select the input format

[1168] If a user has a question about their studies, they click the "Ask a Question" button in the application and are taken to a question input screen. The device then provides the option to input the question in the form of text, image, or voice. For example, if a user wants to ask a question about a math problem, they can input the text "I don't understand this calculus problem" and upload an image of the corresponding page from the textbook. They can also dictate their question using voice input.

[1169] Sending and analyzing question data

[1170] The entered question data is sent from the device to the server. The server passes this data to an analysis module, which analyzes the text data to identify the content of the question. At the same time, it also analyzes image data and audio data to gain a comprehensive understanding of the problem. The server then sends a request to the multimodal AI based on the analysis results.

[1171] Generate and provide answers

[1172] Multimodal AI comprehensively analyzes the data it receives, searches for relevant information from a learning database, and generates the optimal answer. This answer is returned to the server, which then sends it to the device. The device then displays the received answer to the user. For example, a step-by-step solution to a calculus problem might be provided, such as "first find the derivative, then integrate." Diagrams and tables may also be displayed as visual support.

[1173] Providing additional questions and detailed answers

[1174] If the user wants to ask an additional question about the provided answer, they enter the question again, and the device sends that data to the server. The server reanalyzes the new question data and the previous answer data and asks the multimodal AI to generate a detailed answer. The multimodal AI then performs a detailed analysis and generates an additional, detailed answer. The server sends the answer to the device, which then provides it to the user again.

[1175] Examples of concrete examples and prompts

[1176] As a concrete example, consider the case where high school student A is trying to solve a calculus problem. First, A installs the application and enters his / her name, email address, grade, and subjects of interest. Next, A clicks the link in the confirmation email to activate his / her account and logs in. A clicks the "Ask a Question" button, enters the text "I don't understand this calculus problem," and uploads an image of the corresponding page from his / her textbook. The server analyzes the text and image and sends a request to the multimodal AI, which generates a step-by-step answer. This answer is then sent to A's device via the server. If A enters an additional question, such as "I don't understand the intermediate calculations," the multimodal AI performs another detailed analysis and provides an additional answer.

[1177] An example prompt for using a generative AI model is:

[1178] "I don't understand this calculus problem. I've uploaded an image of the relevant page from the textbook. Could you please explain the steps in detail?"

[1179] In this way, this system allows users to receive prompt and consistent learning support, significantly improving learning efficiency. Because it is not dependent on the teacher's availability or skills, high-quality learning support can be received anytime, anywhere.

[1180] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1181] Step 1:

[1182] The user installs the application and enters their name, email address, grade, and subjects of interest when they first launch it. This data is collected by the device and sent to the server. The server stores the received user information in a database and sends a confirmation email to the user. The input data is name, email address, grade, and subjects of interest, and the output data is a confirmation email for account activation.

[1183] Step 2:

[1184] The user clicks the link in the confirmation email to activate the account. The server detects the click on the confirmation link and activates the user's account information. The input data is the click on the confirmation link, and the output data is the activated user account.

[1185] Step 3:

[1186] The user logs in by entering their email address and password on the login screen. The device sends the login information to the server, which then authenticates it. If authentication is successful, the user is redirected to the dashboard screen. The input data is the email address and password, and the output data is the login success status.

[1187] Step 4:

[1188] The user clicks the "Ask a Question" button and is taken to a question input screen. The device displays the option to enter the question in text, image, or audio format. The user enters the question in the required format and uploads the appropriate file. The input data is text, image, or audio file, and the output data is the entered question data.

[1189] Step 5:

[1190] The terminal sends the input question data (text, image, voice) to the server. The server passes the received data to the analysis module. The input data is the question data, and the output data is the data transferred to the analysis module.

[1191] Step 6:

[1192] The server analyzes the text data to identify the content of the question. At the same time, it also analyzes image data and audio data to gain a comprehensive understanding of the problem. Based on the analysis results, it sends a request to the multimodal AI to generate an answer. The input data is the user's question data, and the output data is the analysis results.

[1193] Step 7:

[1194] The multimodal AI analyzes the received data comprehensively, searches for relevant information from the learning database, and generates the optimal answer. This answer is sent back to the server. The input data is the analysis result, and the output data is the generated answer.

[1195] Step 8:

[1196] The server sends the generated answer to the terminal. The terminal displays the answer to the user. The input data is the generated answer, and the output data is the answer displayed to the user.

[1197] Step 9:

[1198] The user inputs a follow-up question for the provided answer. The terminal sends the follow-up question to the server. The input data is the follow-up question, and the output data is the follow-up question data sent to the server.

[1199] Step 10:

[1200] The server requests the multimodal AI to perform reanalysis based on new question data and previous answer data. The multimodal AI performs detailed analysis and generates an additional answer. The server sends the generated answer to the terminal, which then provides it to the user again. The input data is the additional question data and previous answer data, and the output data is a detailed answer.

[1201] The above processing steps allow users to input questions in a variety of formats, and the data is analyzed in an integrated manner to provide consistent, high-quality answers. Furthermore, the user registration and authentication process is simplified, improving the efficiency of learning support.

[1202] (Application example 1)

[1203] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1204] Existing online learning systems mainly provide answers to questions using text, images, and audio, but lack the ability to provide visual answers using 3D models and animations that are easy for users to understand intuitively. Furthermore, there are only a limited number of systems that can respond to detailed follow-up questions in real time. Furthermore, there are limited means of providing learning support within the virtual environment, making it difficult for users to effectively progress through rich interactions.

[1205] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1206] In this invention, the server includes: means for a user to input a question in the form of text, image, or voice; means for transmitting the input question data to the server; means for analyzing the received question data and sending a request to an appropriate multimodal AI; means for the multimodal AI to analyze the question data and generate an answer; means for providing the generated answer to the user; means for the user to move around in the virtual environment in real time and access topic rooms specialized for a subject; and means for providing the answer visually using 3D models and animations in the virtual environment, thereby enabling the user to receive detailed learning support with high resolution.

[1207] A "user" is an individual or organization that uses the learning system.

[1208] "Text" is character information entered by the user.

[1209] "Images" are visual content uploaded by users.

[1210] "Audio" refers to sound information that is recorded and transmitted by the user.

[1211] "Question data" refers to the content of a question about learning that is input by the user in the form of text, image, or voice.

[1212] A "server" is a central device that receives question data sent by users, analyzes it, and generates answers.

[1213] "Multimodal artificial intelligence" is an artificial intelligence system that comprehensively analyzes data in multiple formats, such as text, images, and audio, and generates answers.

[1214] An "answer" is information that contains a solution or explanation to a user's question.

[1215] A "virtual environment" is a virtual space in which users can move around in real time and engage in learning experiences.

[1216] "Topic rooms" are study areas dedicated to specific subjects, rooms within a virtual environment staffed with dedicated artificial intelligence assistants.

[1217] A "3D model" is a three-dimensional digital object that visually represents a solution within a virtual environment.

[1218] "Animation" is a moving image effect that dynamically visualizes the solution within the virtual environment.

[1219] A "dedicated artificial intelligence assistant" is an artificial intelligence system that is placed in each topic room and is specialized to provide appropriate answers to users' questions.

[1220] "Registration information" refers to personal information and learning information provided by users in order to use the system.

[1221] "Verification Email" means an email sent to a User to confirm their registration information and activate their Account.

[1222] "Login Information" means the authentication data used by a User to access a System.

[1223] "Motion tracking" refers to tracking a user's position and actions within a virtual environment in real time.

[1224] This invention is a system that effectively supports online learning and is realized mainly through the interaction between a server, a terminal, and a user. Specifically, it includes the following components and the system functions resulting from their interaction:

[1225] User Registration and Login

[1226] First, in order to use the system using a terminal, a user must register when starting up the system for the first time. The user enters information such as their name, email address, grade, and subjects of interest, and this input data is sent from the terminal to the server. The server saves this information in a database and sends a confirmation email to the user's email address. When the user clicks the link in the confirmation email, the server validates the user's login information. After that, the user can log in and use all the system's functions.

[1227] Post a question and select the input format

[1228] If a user wants to receive learning support, they click the "Ask a Question" button on their device and are taken to a question input screen. They can enter their question in the form of text, image, or voice. For example, if they don't understand a math problem, they can enter "I don't understand this calculus problem" and upload an image of the relevant page from their textbook.

[1229] Question analysis and answer generation using multimodal AI

[1230] The device sends the question data entered by the user to the server. The server first analyzes the text data and identifies the type of question. Next, the server also analyzes the image data and audio data, and uses this information to seek a comprehensive understanding of the problem. Based on this analyzed data, the server sends a request to multimodal artificial intelligence (AI). The multimodal AI comprehensively analyzes the received data, searches for related information from a learning database, and generates the optimal answer.

[1231] Providing answers

[1232] The server sends the answer generated by the multimodal AI to the device. For example, for a calculus problem, the generated answer includes a step-by-step process such as "first find the derivative, then integrate." The device displays the answer to the user and provides visual support information such as diagrams as needed.

[1233] Virtual Learning Support Center

[1234] This system provides users with a means to move around in a virtual environment in real time and access topic rooms specialized for specific subjects. When a user enters a topic room related to a specific subject, a dedicated AI assistant is deployed to provide appropriate answers to the user's questions. Answers are visually represented using 3D models and animations, making them easier for users to understand intuitively. Users can also add more detailed questions on the spot and receive detailed answers in real time.

[1235] Examples of concrete examples and prompts

[1236] As a concrete example, when a high school student wants to solve a calculus problem in mathematics, the following process is taken: The user enters a topic room and enters the following prompt sentence into the terminal:

[1237] "I don't understand a calculus problem. Can you give me detailed steps to solve it?"

[1238] "I don't understand this calculation step. Please explain it again in more detail."

[1239] This allows the system to provide users with the information they need in real time, greatly improving the learning experience.

[1240] Hardware and software used

[1241] The following hardware and software are used to realize this system. The hardware includes a server, a user device (a smartphone or personal computer), and a head-mounted display. The software uses Python, Unity (for building the VR environment), and TensorFlow (for the AI ​​model).

[1242] The above configuration allows users to receive learning support efficiently and effectively.

[1243] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1244] Step 1:

[1245] When a user starts up the device and uses it for the first time, they are taken to the user registration screen. The user enters their name, email address, grade, and subjects of interest, and this information is sent from the device to the server. Input data: Name, email address, grade, and subjects of interest. Output data: User registration information saved on the server.

[1246] Step 2:

[1247] The server saves the user registration information in a database and sends a confirmation email to the user's email address. The confirmation email contains a link to activate the account. Input data: User registration information. Output data: Confirmation email.

[1248] Step 3:

[1249] The user clicks the link in the confirmation email and the server validates the user's login information. Input data: Click on the link in the confirmation email. Output data: Validated user login information.

[1250] Step 4:

[1251] The user logs in using a terminal and clicks the "Ask a Question" button to move to the question input screen. The user enters the question in the form of text, image, or voice. Input data: text, image, voice. Output data: question data.

[1252] Step 5:

[1253] The device sends the entered question data to the server. The server receives the question data and first performs text analysis. Next, it analyzes the image and audio data. Input data: Question data. Output data: Analyzed question data.

[1254] Step 6:

[1255] The server sends a request to the appropriate multimodal AI based on the analyzed question data. Input data: Analyzed question data. Output data: Request sent to the AI.

[1256] Step 7:

[1257] Multimodal AI comprehensively analyzes question data, searches for relevant information, and generates optimal answers. Input data: Question data. Output data: Generated answers.

[1258] Step 8:

[1259] The server receives the answer data generated by the multimodal AI and sends it to the terminal. The terminal provides this answer to the user. Input data: Generated answer data. Output data: Answer displayed on the terminal.

[1260] Step 9:

[1261] If the user wants to ask a more detailed question, he or she inputs the additional question through the terminal. The terminal sends the additional question data to the server. Input data: Additional question data. Output data: Additional question data sent to the server.

[1262] Step 10:

[1263] The server re-analyzes the additional question data and sends a request to the multimodal AI again to generate a detailed answer. Input data: Additional question data. Output data: The request sent again to the AI.

[1264] Step 11:

[1265] The multimodal AI generates a detailed answer and sends it to the terminal via the server. The terminal displays this answer to the user. Input data: detailed answer data. Output data: detailed answer displayed on the terminal.

[1266] Step 12:

[1267] Users can move freely around the virtual environment and access topic rooms specific to their subject. A dedicated AI assistant is deployed to provide visual answers to questions using 3D models and animations. Input data: user's location, question. Output data: visual answers using 3D models and animations.

[1268] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1269] The present invention is a system for effectively supporting online learning, which is combined with an emotion engine for recognizing a user's emotions and providing appropriate support in accordance with those emotions. It can be implemented as follows.

[1270] User Registration and Login

[1271] The first step for users to use the system is to install the app and register as a user when they first start it. On the registration screen displayed on the device, the user enters their name, email address, grade, and subjects of interest. This input data is sent from the device to the server, which stores it in a database. Once registration is complete, the server sends a confirmation email to the user's email address, and the user clicks on the link in the confirmation email to activate their account. After that, the user can log in and use all of the system's functions.

[1272] Post a question and select the input format

[1273] If a user wants learning support, they click the "Ask a Question" button, which takes them to a question input screen. The device then displays options for inputting questions in the form of text, image, or voice. For example, if a user doesn't understand a math problem, they can type "I don't understand this calculus problem" and upload an image of the corresponding page from their textbook.

[1274] Question analysis and answer generation using multimodal AI

[1275] The device sends the question data entered by the user to the server. The server first analyzes the text data and identifies the type of question. Next, the server also analyzes the image data and audio data, and uses this information to seek a comprehensive understanding of the problem. Based on this analyzed data, the server sends a request to the multimodal AI. The multimodal AI comprehensively analyzes the received data, searches for related information from a learning database, and generates the optimal answer.

[1276] Emotion recognition and response using emotion engine

[1277] The server not only receives the question data but also sends it to the emotion engine to recognize the user's emotion at the time of question entry. The emotion engine analyzes the user's emotion from text data, image data, and voice data and determines whether it is positive or negative. Based on the results of this emotion recognition, the multimodal AI adjusts the answer it generates. For example, if the user expresses negative emotions, it adds a message of encouragement or support.

[1278] Providing answers

[1279] The server sends the generated answer to the device. For example, for a calculus problem, the generated answer includes a step-by-step process such as "first find the derivative, then integrate." The device displays the answer to the user and provides visual support information such as diagrams as needed. In addition, the device may display an encouraging message based on the user's emotion recognition results.

[1280] Follow-up questions and real-time feedback

[1281] If the user wants to ask a more detailed question about the provided answer, they can enter the question again. For example, they can enter an additional question such as, "I don't understand the calculation in between." The device then sends this additional question to the server, and the server again requests a more detailed answer from the multimodal AI based on the previous answer data and the new question data. The multimodal AI then performs a detailed analysis and generates an additional answer, which the server then sends to the device. The device then provides this detailed answer to the user, helping them to deepen their understanding.

[1282] Specific examples

[1283] If high school student A wants to solve a calculus problem in mathematics, he or she first registers as a user and logs in. Next, he or she enters the text "I don't understand this calculus problem" on the question input screen and uploads an image of the textbook. The server analyzes the text content and image and uses an emotion engine to recognize A's emotions. If A shows negative emotions, the AI ​​generates an answer that includes an encouraging message. The server sends this answer to A's device, where A receives it. If A asks an additional question, such as "I don't understand the calculation in the middle," the multimodal AI performs a more detailed analysis again and provides an additional answer.

[1284] This system allows users to receive prompt and consistent learning support, and by using an emotion engine, it is possible to provide even more personalized support, which further improves learning efficiency and allows users to enjoy a better learning experience.

[1285] The processing flow will be explained below.

[1286] Step 1:

[1287] The user installs the app and clicks the "Register" button when launching it for the first time.

[1288] Step 2:

[1289] The terminal displays a screen that prompts the user to enter registration information such as name, email address, grade, and subjects of interest.

[1290] Step 3:

[1291] The user enters their name, email address, grade, and subjects of interest and clicks the "Submit" button.

[1292] Step 4:

[1293] The terminal transmits the entered registration data to the server.

[1294] Step 5:

[1295] The server stores the received data in a database and sends a confirmation email to the user's email address.

[1296] Step 6:

[1297] The user clicks on the link in the confirmation email to activate their account.

[1298] Step 7:

[1299] The server accepts the confirmation link and validates the user's login information.

[1300] Step 8:

[1301] The user logs in to the app and navigates to the home screen.

[1302] Step 9:

[1303] The user clicks the "Ask a Question" button on the home screen.

[1304] Step 10:

[1305] The device will display a question entry screen and provide the option to enter the question in the form of text, image, or voice.

[1306] Step 11:

[1307] A user types in the text "I don't understand this calculus problem" and uploads an image of the corresponding page from the textbook.

[1308] Step 12:

[1309] The terminal sends the entered question data (text, image) to the server.

[1310] Step 13:

[1311] The server analyzes the text data and performs initial filtering.

[1312] Step 14:

[1313] The server sends the image data to an image analysis module to extract relevant information.

[1314] Step 15:

[1315] The server sends the voice data to a voice analysis module, which analyzes the voice tone and content.

[1316] Step 16:

[1317] The server integrates the text, image, and voice analysis results and sends them to the emotion engine.

[1318] Step 17:

[1319] The emotion engine analyzes the received data and determines the user's emotion. If the emotion is negative, it revisits the specific situation and evaluates whether an encouraging message needs to be added.

[1320] Step 18:

[1321] The emotion engine sends the emotion recognition results back to the server.

[1322] Step 19:

[1323] The server sends an analysis request to the multimodal AI based on the emotion recognition result. If the emotion is negative, the request includes an encouraging message or a special answer format.

[1324] Step 20:

[1325] Multimodal AI integrates and analyzes question data and emotional information to understand the content of the problem.

[1326] Step 21:

[1327] Multimodal AI collates information from relevant learning databases to generate optimal answers, supplemented with emotional messages where appropriate.

[1328] Step 22:

[1329] The server receives the generated answer data and transmits it to the user's terminal.

[1330] Step 23:

[1331] The device displays the answer data to the user, including visual support information such as text, formulas, diagrams, etc. If the emotion is negative, an encouraging message is also displayed.

[1332] Step 24:

[1333] If the user wants to get more details about the answer, he / she can input and submit an "additional question."

[1334] Step 25:

[1335] The terminal sends a follow-up question to the server.

[1336] Step 26:

[1337] The server again sends an analysis request for additional questions to the multimodal AI.

[1338] Step 27:

[1339] Multimodal AI then performs a detailed analysis again to generate additional detailed answers.

[1340] Step 28:

[1341] The server transmits the generated additional answer to the user's terminal.

[1342] Step 29:

[1343] The device displays additional answers to the user to help them deepen their understanding.

[1344] Through these steps, users can receive fast and consistent learning support, and the emotional engine can provide more personalized assistance, improving learning efficiency and providing a more meaningful learning experience for users.

[1345] Example 2

[1346] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1347] Conventional online learning support systems lack the flexibility to respond to the diverse question formats users encounter, and can only accept questions in specific formats. Furthermore, they provide uniform answers without considering changes in the user's motivation or emotions regarding learning, making it impossible to provide support optimized for each individual user. Furthermore, it is difficult to provide quick and detailed real-time feedback on follow-up questions, which hinders users' learning efficiency.

[1348] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for a user to input a question in any one of the formats of text, image, or voice, a means for transmitting the input question data to the server, a means for analyzing the received question data and transmitting a request to an appropriate multimodal AI, a means for the multimodal AI to analyze the question data and generate an answer, a means for providing the generated answer to the user, and a means for analyzing the user's emotions using an emotion recognition engine and adjusting the content of the answer based on the analyzed emotion data. This makes it possible to respond to various types of questions from users and provide optimal answers that take into account each individual's emotional state.

[1349] "User" refers to any person or entity that utilizes the online learning system to enter questions and receive answers.

[1350] "Text" refers to a digital data format that includes character information, and refers to character string data that a user uses to input a question.

[1351] "Image" refers to a digital data format containing visual information, such as a photograph or drawing uploaded by a user to supplement a question.

[1352] "Speech" refers to a digital data format containing auditory information, and refers to voice data used by a user to input a question.

[1353] "Server" refers to a computer system or cloud infrastructure for processing, analyzing, storing, and generating answers from data received from users.

[1354] "Multimodal AI" refers to an AI model that comprehensively analyzes different types of data (text, images, audio) and generates optimal answers.

[1355] An "emotion recognition engine" refers to software or algorithms that analyze emotions from user input data and determine whether they are positive or negative.

[1356] "Answer" refers to output data generated by multimodal artificial intelligence that contains appropriate information and instructions in response to a user's question.

[1357] "Confirmation Email" refers to the email sent from the server at the time of user registration for the purpose of activating the account.

[1358] "Confirmation Link" refers to the URL included in the confirmation email that the User clicks to activate their account.

[1359] "Login Information" refers to the authentication information (e.g., user ID and password) used by a User to access the System.

[1360] The present invention is a system for supporting online learning, in which a user inputs a question in the form of text, image, or voice, and generates an answer to the question using multimodal artificial intelligence and an emotion recognition engine. Specific embodiments of the present invention are described below.

[1361] First, a user installs and launches the online learning application on a device such as a smartphone. In order to use the system, the user must first register. On the registration screen, the user enters their name, email address, grade, and subjects of interest, and this data is sent from the device to the server. The server stores this data in a database (e.g., MySQL database) and sends a confirmation email to the user. The user clicks the link in the confirmation email to activate their account.

[1362] Next, when the user clicks the "Ask a Question" button, the device displays a screen where the user can enter a question in the form of text, image, or voice. For example, the user can enter the text "I don't understand this calculus problem" and upload an image of the corresponding page from the textbook. This input data is sent from the device to the server.

[1363] The server analyzes the received text data using a natural language processing library (e.g., NLTK, SpaCy), and analyzes the image data using an image processing library (e.g., OpenCV). The analyzed data is sent to a multimodal AI, which uses a generative AI model such as GPT-4 to comprehensively analyze the received data and generate the optimal answer.

[1364] The server then sends the received question data to an emotion recognition engine. The emotion recognition engine uses the Emotion API or other tools to analyze the user's emotions from text, image, and audio data and determine whether they are positive or negative. Based on the emotion recognition results, the multimodal AI adjusts the answer it generates. For example, if the user expresses negative emotions, it adds a message of encouragement or support.

[1365] The generated answer is sent from the server to the device, which then displays it to the user. Visual support information (e.g., diagrams) is also displayed as needed. If the user enters additional questions, the multimodal AI again performs a detailed analysis, generating the optimal answer and sending it from the server to the device.

[1366] As a concrete example, if high school student A wants to solve a calculus problem in mathematics, he or she would follow the steps below. First, he or she registers as a user and logs in. Next, he or she enters the text "I don't understand this calculus problem" on the question input screen and uploads an image of the textbook. The server analyzes the text content and image and uses an emotion recognition engine to recognize A's emotions. If A shows negative emotions, the AI ​​generates an answer that includes an encouraging message. The server sends this answer to A's device, where A receives it. If A asks an additional question, such as "I don't understand the calculation in the middle," a more detailed analysis is performed again and the answer is provided to the device.

[1367] An example of a prompt sentence is as follows:

[1368] User: I don't understand this calculus problem. (Image attached) (Image: Calculus problem page in textbook)

[1369] Server: Performs text and image analysis to generate the optimal answer.

[1370] Emotion Recognition Engine: Analyzes the user's emotions and finds that they are negative. Adds encouraging messages.

[1371] Server: Generates the best answer and sends it to the device along with an encouraging message.

[1372] This invention allows users to receive prompt and consistent learning support, and by using an emotion recognition engine, it is possible to provide even more personalized support to users, significantly improving learning efficiency.

[1373] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1374] Step 1:

[1375] The user installs and launches the app.

[1376] A user downloads the app from a digital store and installs it on their device.

[1377] Launch the app and the first screen will appear.

[1378] Input: User downloads and launches the app

[1379] Output: First screen display of the app

[1380] Step 2:

[1381] The terminal displays a user registration screen.

[1382] A registration screen UI is displayed, including fields for the user to enter their name, email address, grade level, and subjects of interest.

[1383] Input: Registration screen displayed after launching the app

[1384] Output: A field for the user to enter input.

[1385] Step 3:

[1386] The user enters the required information and presses the registration button.

[1387] The user enters their name, email address, grade, and subjects of interest and presses the "Register" button.

[1388] Input: Information entered by the user (name, email address, grade, subjects of interest)

[1389] Output: Sending input data by pressing the register button

[1390] Step 4:

[1391] The terminal sends the input data to the server.

[1392] The terminal sends the data entered by the user to the server in JSON format at the REST API endpoint.

[1393] Input: Information entered by the user

[1394] Output: Send data in JSON format

[1395] Step 5:

[1396] The server stores the registration information in a database.

[1397] The server analyzes the received data and stores it in a database (e.g. MySQL).

[1398] Input: JSON format user information data

[1399] Output: User information stored in the database

[1400] Step 6:

[1401] The server sends a confirmation email.

[1402] The server uses a mail server (e.g. Postfix) to send the user a confirmation email containing a link to activate the account.

[1403] Input: User information stored in the database

[1404] Output: Send confirmation email

[1405] Step 7:

[1406] The user clicks the confirmation link.

[1407] The user opens the confirmation email and activates their account by clicking the link.

[1408] Input: Link in confirmation email

[1409] Output: Account enabled

[1410] Step 8:

[1411] The server performs the account activation process.

[1412] The server receives the link and updates the status of the corresponding account to "active."

[1413] Input: The confirmation link the user clicked

[1414] Output: Enabled account status

[1415] Step 9:

[1416] A user logs in.

[1417] Using your user ID and password, you can access the login screen and use all the functions of the system.

[1418] Input: User ID and password

[1419] Output: Successfully logged into the system

[1420] Step 10:

[1421] The user clicks the "Ask a Question" button.

[1422] A user clicks the "Ask a Question" button within the app.

[1423] Input: User clicks "Ask a question" button

[1424] Output: Display of question input screen

[1425] Step 11:

[1426] The device displays a question input screen.

[1427] Displays a question input screen with text, image, and audio input options.

[1428] Input: User clicks "Ask a question" button

[1429] Output: Display of question input screen

[1430] Step 12:

[1431] The user enters a question.

[1432] A user types "I don't understand this calculus problem" into a text box and uploads an image of a page from a textbook.

[1433] Input: Question text entered by the user, and uploaded images

[1434] Output: Question data ready to send

[1435] Step 13:

[1436] The terminal transmits the input data to the server.

[1437] The text data and image data are sent to a server.

[1438] Input: Question data entered by the user (text, images)

[1439] Output: Send data to the server

[1440] Step 14:

[1441] The server parses the text data.

[1442] The server analyzes the text data using a natural language processing library (e.g., NLTK, SpaCy) to identify the question.

[1443] Input: Received text data

[1444] Output: Analysis results (question content identification)

[1445] Step 15:

[1446] The server analyzes the image data.

[1447] The server analyzes the image data using an image processing library (e.g., OpenCV) and extracts relevant information.

[1448] Input: Received image data

[1449] Output: Analysis results (image content analysis)

[1450] Step 16:

[1451] The server sends the analysis data to the multimodal AI.

[1452] The analysis results are integrated and sent to multimodal AI.

[1453] Input: Text and image analysis results

[1454] Output: A request to the multimodal AI

[1455] Step 17:

[1456] Multimodal AI generates optimal answers.

[1457] Generative AI models such as GPT-4 are used to analyze the received data and generate optimal answers.

[1458] Input: Integrated analysis data

[1459] Output: The generated answer

[1460] Step 18:

[1461] The server sends the answer data to the emotion recognition engine.

[1462] The emotion recognition engine analyzes the user's emotions along with the generated answer data.

[1463] Input: Answer data

[1464] Output: User sentiment analysis results

[1465] Step 19:

[1466] An emotion recognition engine analyzes the user's emotions.

[1467] The Emotion API is used to determine the user's emotions from the data sent.

[1468] Input: Answer data

[1469] Output: Sentiment analysis result (positive / negative)

[1470] Step 20:

[1471] The server adjusts the answer based on the emotional data.

[1472] Based on the results of emotion recognition, the multimodal AI is instructed to adjust its answers, for example by adding encouraging messages.

[1473] Input: Sentiment analysis results

[1474] Output: Adjusted answer

[1475] Step 21:

[1476] The server sends the answer data to the terminal.

[1477] The optimized answer is sent to the device.

[1478] Input: Adjusted answer data

[1479] Output: Send answer data to the device

[1480] Step 22:

[1481] The terminal displays the answer to the user.

[1482] The terminal displays the answer and supplementary information to the user.

[1483] Input: Answer data received from the server

[1484] Output: What is displayed to the user

[1485] Step 23:

[1486] The user enters a follow-up question.

[1487] Enter additional detailed questions in text for the answers provided.

[1488] Input: Additional questions from the user

[1489] Output: Additional question data

[1490] Step 24:

[1491] The terminal sends a follow-up question to the server.

[1492] Send additional question data to the server.

[1493] Input: Additional question data by the user

[1494] Output: Send data to the server

[1495] Step 25:

[1496] The server reanalyzes the data and sends the request to the multimodal AI again.

[1497] Send additional question data to the multimodal AI to generate detailed answers.

[1498] Input: Additional question data

[1499] Output: Detailed answer request

[1500] Step 26:

[1501] Multimodal AI generates detailed answers.

[1502] Generate detailed answers based on additional question data.

[1503] Input: Additional question data

[1504] Output: Detailed answer

[1505] Step 27:

[1506] The server sends detailed answer data to the terminal.

[1507] The generated detailed answer is sent to the device.

[1508] Input: Detailed answer data

[1509] Output: Detailed answer data sent to the device

[1510] Step 28:

[1511] The terminal displays the detailed answer to the user.

[1512] The terminal displays the detailed answer content to the user.

[1513] Input: Detailed answer data received from the server

[1514] Output: What is displayed to the user

[1515] In this way, the system can effectively execute a series of processes to receive a user's question, analyze it, and provide the most appropriate answer.

[1516] (Application example 2)

[1517] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1518] Conventional online learning systems can generate answers by analyzing text, images, and audio data provided by users, but they rarely take the user's emotional state into consideration. This can lead to problems such as a decrease in learning motivation and a lack of appropriate feedback, resulting in a decrease in learning effectiveness. Furthermore, when asked follow-up questions about answers, real-time feedback that reflects the user's emotional changes cannot be provided, making it difficult to provide advanced support tailored to individual learners.

[1519] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a means for a user to input a question in the form of text, image, or voice; a means for transmitting the input question data to the server; a means for analyzing the received question data and sending a request to an appropriate multimodal AI; a means for the multimodal AI to analyze the question data and generate an answer; a means for providing the generated answer to the user; and an emotion engine that recognizes the user's emotions and provides a response according to the emotion. This provides appropriate support that takes into account the user's emotional state, thereby improving learning effectiveness. In addition, the server includes a means for a user to ask a follow-up question in response to the answer; a means for transmitting the follow-up question to the server, reanalyzing the question, and generating a detailed answer; a means for providing the detailed answer to the user; and a means for providing real-time feedback according to changes in the user's emotions, thereby realizing advanced support tailored to individual learners. Furthermore, the system includes a means for the server to store the user's registration information in a database and send a confirmation email, a means for validating the login information of a user who clicks on a confirmation link, and a means for initializing the user's emotional data at the time of login and providing support tailored to each individual learner, thereby enabling more personalized learning support to be provided.

[1520] A "user" is an individual who utilizes the system to enter questions and receive answers.

[1521] "Text" refers to the written information that a user inputs into the system.

[1522] "Image" means information in the form of a still image that a user uploads to the system.

[1523] "Speech" refers to the acoustic information provided by the user to the system.

[1524] "Question Data" refers to information entered into the system by a user in the form of text, images, or voice.

[1525] A "server" is a computer system that analyzes question data received from a user and generates an appropriate answer.

[1526] "Multimodal AI" refers to AI that analyzes and integrates multiple data formats, such as text, images, and audio, to generate answers.

[1527] An "emotion engine" is a system that recognizes emotions from user input data and provides a response that corresponds to that emotion.

[1528] An "answer" refers to response information generated based on question data analyzed by multimodal artificial intelligence.

[1529] "Registration Information" refers to personal information such as name, email address, grade, and subjects of interest that a user provides when registering with the system.

[1530] "Verification Email" means an email sent by the System to verify a User's registration information and activate their Account.

[1531] "Login information" refers to the authentication information (user name and password) used by a user to access a system.

[1532] "Emotional state" indicates the psychological state of the user when he or she inputs the question.

[1533] "Real-time feedback" refers to a system that provides immediate responses to users' questions and follow-up questions.

[1534] The present invention provides an online learning support system that combines an emotion engine that recognizes a user's emotions and provides appropriate support in accordance with those emotions. Specific embodiments of this system will be described below.

[1535] User Registration and Login

[1536] When using the system for the first time, the user installs a dedicated app on their smartphone and launches it. When launched for the first time, a user registration screen is displayed. The user enters their name, email address, grade, and subjects of interest. This input data is sent from the smartphone to the server, which stores it in a database. After registration is complete, the server sends a confirmation email to the user's email address, and the user activates their account by clicking the confirmation link. After that, the user can log in and use all of the system's functions.

[1537] Question posting and emotion recognition

[1538] When a user wants to receive learning support, they click the "Ask a Question" button in the app, which takes them to a question input screen. Here, the user can enter their question in the form of text, image, or voice. The entered question data is sent from the smartphone to the server. The server analyzes the received question data and identifies the content of the question. At the same time, the emotion engine analyzes the question data and recognizes the user's emotional state.

[1539] Answer generation and emotion regulation

[1540] The server sends a request to the multimodal AI based on the analyzed data. The multimodal AI comprehensively analyzes the received data and generates an answer. At that time, the answer is adjusted based on the analysis results of the emotion engine according to the user's emotions. For example, if the user shows negative emotions, the AI ​​adds a message of encouragement or support.

[1541] Providing answers

[1542] The generated answer is sent from the server to the smartphone and provided to the user. For example, the generated answer to a math problem may include a step-by-step process such as "first find the derivative, then do the integral." Encouraging messages based on the emotion recognition results may also be displayed.

[1543] Follow-up questions and real-time feedback

[1544] If the user wants to ask a more detailed question about the provided answer, they can enter the question again. For example, they can enter an additional question such as, "I don't understand the calculation in between." The smartphone again sends this additional question to the server, and the server again requests a more detailed answer from the multimodal AI based on the previous answer data and the new question data. The multimodal AI then performs a detailed analysis and generates an additional answer, which the server then sends to the smartphone. The smartphone then provides this detailed answer to the user, helping them to deepen their understanding.

[1545] Hardware and software used

[1546] The hardware used is a smartphone camera, and the software used is image preprocessing with OpenCV, emotion recognition models with TensorFlow / Keras, and text emotion analysis with TextBlob.

[1547] Examples and prompts

[1548] For example, when a user types, "Today's test results were poor," the sentiment engine detects negative sentiment from the text, and the AI ​​adds an encouraging message such as, "Failure is the mother of success, so try harder next time!"

[1549] Example prompt sentence:

[1550] Classify the emotion of the given sentence as positive or negative. Specifically, if the sentence contains negative expressions such as "I don't understand," "It's difficult," or "It's hard," output an appropriate encouraging message.

[1551] For example, given the following input:

[1552] Today's test results were bad

[1553] As described above, the system of the present invention provides support according to the user's emotional state, improves learning efficiency, and provides optimal feedback to each individual learner.

[1554] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1555] Step 1:

[1556] A user installs a dedicated app on their smartphone and launches the app. A user registration screen appears, where the user enters their name, email address, grade, and subjects of interest. This input data is sent from the device to the server. The server stores the received registration information in a database and sends a confirmation email to the user's email address. The user activates their account by clicking the confirmation link in the email. The entered data is in text format, and the server stores it in the database. The output is the account activation status.

[1557] Step 2:

[1558] The user logs in and clicks the "Ask a Question" button to move to the question input screen. The user enters the question in the form of text, image, or voice. This data is sent from the terminal to the server. The server processes the received question data using one of the following methods: text analysis, image analysis, or voice analysis. The input is the question data in the respective data format, and the output is the analyzed question content.

[1559] Step 3:

[1560] The emotion engine analyzes the question data and recognizes the user's emotional state. The server uses the emotion engine to identify the user's emotion from text data, image data, and audio data. For example, for text, it uses TextBlob to calculate the emotion score. The input is the question data in each format, and the output is the recognized emotional state (positive or negative).

[1561] Step 4:

[1562] The server sends a request to a multimodal AI based on the analyzed data. The multimodal AI integrates text, image, and audio data to generate an answer. To generate the answer, a generative AI model is used to search and integrate information from relevant knowledge bases. The input is the analyzed data, and the output is the generated answer.

[1563] Step 5:

[1564] The generated answer is sent from the server to the device and provided to the user. Based on the results of question data analysis and the recognition results of the emotion engine, an encouraging message is added as needed. The device displays this information on the screen. The input is the generated answer and encouraging message, and the output is the answer and message displayed to the user.

[1565] Step 6:

[1566] If the user wants to enter a more detailed question in response to the provided answer, the question is entered again and the additional question data is sent from the terminal to the server. The server then re-analyzes the previous answer data and the new question data to generate a detailed answer. The input is the additional question data and the output is the detailed answer.

[1567] Step 7:

[1568] The server sends the detailed answer to the terminal and provides it back to the user. The terminal displays the detailed answer to the user and provides further assistance or feedback as needed. The input is the detailed answer and any required feedback messages, and the output is the detailed answer and feedback displayed to the user.

[1569] Through these steps, users can learn efficiently while receiving appropriate support according to their emotions, thereby improving the effectiveness and satisfaction of their learning.

[1570] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1571] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1572] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1573] [Fourth embodiment]

[1574] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1575] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1576] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1577] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1578] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1579] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1580] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1581] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1582] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1583] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1584] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1585] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1586] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1587] The present invention is a system for effectively supporting online learning, and can be implemented as follows.

[1588] User Registration and Login

[1589] The first step for users to use the system is to install the app and register as a user when they first start it. On the registration screen displayed on the device, the user enters their name, email address, grade, and subjects of interest. This input data is sent from the device to the server, which stores it in a database. Once registration is complete, the server sends a confirmation email to the user's email address, and the user clicks on the link in the confirmation email to activate their account. After that, the user can log in and use all of the system's functions.

[1590] Post a question and select the input format

[1591] If a user wants learning support, they click the "Ask a Question" button, which takes them to a question input screen. The device then displays options for inputting questions in the form of text, image, or voice. For example, if a user doesn't understand a math problem, they can type "I don't understand this calculus problem" and upload an image of the corresponding page from their textbook.

[1592] Question analysis and answer generation using multimodal AI

[1593] The device sends the question data entered by the user to the server. The server first analyzes the text data and identifies the type of question. Next, the server also analyzes the image data and audio data, and uses this information to seek a comprehensive understanding of the problem. Based on this analyzed data, the server sends a request to the multimodal AI. The multimodal AI comprehensively analyzes the received data, searches for related information from a learning database, and generates the optimal answer.

[1594] Providing answers

[1595] The server sends the answer generated by the multimodal AI to the device. For example, for a calculus problem, the generated answer includes a step-by-step process such as "first find the derivative, then integrate." The device displays the answer to the user and provides visual support information such as diagrams as needed.

[1596] Follow-up questions and real-time feedback

[1597] If the user wants to ask a more detailed question about the answer provided, they can enter the question again. For example, they can enter an additional question such as, "I don't understand the intermediate calculations." The device then sends this additional question to the server, and the server again requests a more detailed answer from the multimodal AI based on the previous answer data and the new question data. The multimodal AI then performs a detailed analysis and generates an additional answer, which the server then sends to the device. The device then provides this detailed answer to the user so that they can review it again.

[1598] Specific examples

[1599] If high school student A wants to solve a calculus problem in mathematics, he or she first registers as a user and logs in. Next, he or she enters the text "I don't understand this calculus problem" on the question input screen and uploads an image of the textbook. The server analyzes the text content and image and sends a request to the multimodal AI, which then generates a step-by-step answer. This answer is then sent to A's device via the server. If A asks a follow-up question, such as "I don't understand the intermediate calculations," the multimodal AI performs another detailed analysis and provides an additional answer.

[1600] This system allows users to receive prompt and consistent learning support, significantly improving learning efficiency. In addition, because it is not dependent on the teacher's availability or skills, users can receive high-quality learning support anytime, anywhere.

[1601] The processing flow will be explained below.

[1602] Step 1:

[1603] The user installs the app and clicks the "Register" button when launching it for the first time.

[1604] Step 2:

[1605] The terminal displays a screen that prompts the user to enter registration information such as name, email address, grade, and subjects of interest.

[1606] Step 3:

[1607] The user enters their name, email address, grade, and subjects of interest and clicks the "Submit" button.

[1608] Step 4:

[1609] The terminal transmits the entered registration data to the server.

[1610] Step 5:

[1611] The server stores the received data in a database and sends a confirmation email to the user's email address.

[1612] Step 6:

[1613] The user clicks on the link in the confirmation email to activate their account.

[1614] Step 7:

[1615] The server accepts the confirmation link and validates the user's login information.

[1616] Step 8:

[1617] The user logs in to the app and navigates to the home screen.

[1618] Step 9:

[1619] The user clicks the "Ask a Question" button on the home screen.

[1620] Step 10:

[1621] The device will display a question entry screen and provide the option to enter the question in the form of text, image, or voice.

[1622] Step 11:

[1623] A user types in the text "I don't understand this calculus problem" and uploads an image of the corresponding page from the textbook.

[1624] Step 12:

[1625] The terminal sends the entered question data (text, image) to the server.

[1626] Step 13:

[1627] The server analyzes the text data and performs initial filtering.

[1628] Step 14:

[1629] The server sends the image data to an image analysis module to extract relevant information.

[1630] Step 15:

[1631] The server integrates the text and image analysis results and sends an analysis request to the multimodal AI.

[1632] Step 16:

[1633] Multimodal AI integrates and analyzes question data to understand the content of the problem.

[1634] Step 17:

[1635] Multimodal AI collates information from relevant learning databases to generate optimal answers.

[1636] Step 18:

[1637] The server receives the generated answer data and transmits it to the user's terminal.

[1638] Step 19:

[1639] The terminal displays the answer data to the user and, if necessary, displays supplementary information such as diagrams.

[1640] Step 20:

[1641] If the user wants to get more details about the answer, he / she can input and submit an "additional question."

[1642] Step 21:

[1643] The terminal sends a follow-up question to the server.

[1644] Step 22:

[1645] The server again sends an analysis request for additional questions to the multimodal AI.

[1646] Step 23:

[1647] Multimodal AI then performs a detailed analysis again to generate additional detailed answers.

[1648] Step 24:

[1649] The server transmits the generated additional answer to the user's terminal.

[1650] Step 25:

[1651] The device displays additional answers to the user to help them deepen their understanding.

[1652] This allows users to receive prompt and consistent learning support, improving learning efficiency and providing high-quality learning support that is not dependent on the availability or skills of teachers.

[1653] Example 1

[1654] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1655] Modern online learning systems allow users to input questions in a variety of formats, including text, images, and audio. However, they lack the technology to effectively analyze and integrate this data to generate answers. Furthermore, they lack a mechanism for efficiently reanalyzing and providing detailed answers when users ask follow-up questions about answers already provided. Furthermore, the user registration and authentication process is cumbersome, and account activation via a confirmation link can sometimes be slow. There is a need to address these issues and provide efficient, high-quality online learning support.

[1656] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1657] In this invention, the server includes: a means for a user to input a question in the form of text, image, or voice; a means for transmitting the input question data to the server; a means for analyzing the received question data and sending a request to an appropriate multimodal artificial intelligence; a means for the multimodal artificial intelligence to analyze the question data and generate an answer; a means for accepting additional questions, reanalyzing the data, and generating a detailed answer; a means for the server to store user registration information in a database and send a confirmation email; a means for activating the account of a user who clicks a confirmation link; and a means for the server to provide the generated answer to the user step by step and present the answer including visual auxiliary information. This allows users to input questions in various formats, have the data analyzed in an integrated manner, and receive consistent, high-quality answers. Furthermore, the user registration and authentication process is simplified, improving the efficiency of learning support.

[1658] 1. "User" refers to an individual or organization that uses the online learning system to enter questions and receive answers.

[1659] 2. "Text" refers to a data format consisting of characters or sentences, used by users to enter questions or comments.

[1660] 3. "Image" means a data format containing still images and graphics, used to provide users with visual information.

[1661] 4. "Audio" means a data format containing sounds or spoken words, used by a user to provide information orally.

[1662] 5. "Question Data" refers to the collection of data including all information entered by the User in text, image, and audio format.

[1663] 6. "Server" refers to the computer system that receives and analyzes query data and interacts with the Multimodal Artificial Intelligence.

[1664] 7. “Analysis” refers to the process of understanding the input question data and identifying its meaning and intent.

[1665] 8. "Multimodal AI" refers to AI technology that analyzes multiple data formats, such as text, images, and audio, and generates integrated answers.

[1666] 9. "Request" refers to instructions or requests sent by a server to a multimodal artificial intelligence to request analysis and answer generation.

[1667] 10. “Answer” refers to the response or reply generated by a Multimodal Artificial Intelligence to a user’s question.

[1668] 11. "Providing" refers to the process of displaying or notifying the user of a generated answer.

[1669] 12. "Follow-up Question" means a question entered by a User to request further information or clarification on an Initial Answer.

[1670] 13. "Reanalysis" refers to the process of analyzing new data based on additional questions to generate detailed answers.

[1671] 14. "Detailed answer" refers to a more specific and useful answer generated based on reanalysis.

[1672] 15. "User registration information" refers to basic information such as name, email address, grade, and areas of interest that a user enters to use the system.

[1673] 16. "Database" refers to a system for systematically storing and managing a large amount of information, such as user registration information and question data.

[1674] 17. "Verification Email" means the email sent to a User prompting them to activate their Account.

[1675] 18. "Confirmation Link" means the URL included in the confirmation email that the User clicks to activate their account.

[1676] 19. "Account Activation" refers to the process by which a User's account is activated and all its features are made available by clicking on a confirmation link.

[1677] 20. "Step-by-step" refers to a method of explaining a complex problem to a user by breaking it down into a series of simple steps.

[1678] 21. "Visual aids" refers to information containing non-textual visual data, such as charts or images, that is provided to make the answer easier to understand.

[1679] MODE FOR CARRYING OUT THE INVENTION

[1680] The present invention is a system for effectively supporting online learning, and specific embodiments thereof will be described below.

[1681] User Registration and Login

[1682] To use the system, users must first install the application. When the system is launched for the first time, the user registers their name, email address, grade, and subjects of interest. This data is collected by the device, which then sends it to the server. The server stores the received data in a database and sends a confirmation email to the user. When the user clicks the link in the confirmation email, their account is activated. After that, the user can log in by entering their email address and password on the login screen to use all of the system's functions.

[1683] Post a question and select the input format

[1684] If a user has a question about their studies, they click the "Ask a Question" button in the application and are taken to a question input screen. The device then provides the option to input the question in the form of text, image, or voice. For example, if a user wants to ask a question about a math problem, they can input the text "I don't understand this calculus problem" and upload an image of the corresponding page from the textbook. They can also dictate their question using voice input.

[1685] Sending and analyzing question data

[1686] The entered question data is sent from the device to the server. The server passes this data to an analysis module, which analyzes the text data to identify the content of the question. At the same time, it also analyzes image data and audio data to gain a comprehensive understanding of the problem. The server then sends a request to the multimodal AI based on the analysis results.

[1687] Generate and provide answers

[1688] Multimodal AI comprehensively analyzes the data it receives, searches for relevant information from a learning database, and generates the optimal answer. This answer is returned to the server, which then sends it to the device. The device then displays the received answer to the user. For example, a step-by-step solution to a calculus problem might be provided, such as "first find the derivative, then integrate." Diagrams and tables may also be displayed as visual support.

[1689] Providing additional questions and detailed answers

[1690] If the user wants to ask an additional question about the provided answer, they enter the question again, and the device sends that data to the server. The server reanalyzes the new question data and the previous answer data and asks the multimodal AI to generate a detailed answer. The multimodal AI then performs a detailed analysis and generates an additional, detailed answer. The server sends the answer to the device, which then provides it to the user again.

[1691] Examples of concrete examples and prompts

[1692] As a concrete example, consider the case where high school student A is trying to solve a calculus problem. First, A installs the application and enters his / her name, email address, grade, and subjects of interest. Next, A clicks the link in the confirmation email to activate his / her account and logs in. A clicks the "Ask a Question" button, enters the text "I don't understand this calculus problem," and uploads an image of the corresponding page from his / her textbook. The server analyzes the text and image and sends a request to the multimodal AI, which generates a step-by-step answer. This answer is then sent to A's device via the server. If A enters an additional question, such as "I don't understand the intermediate calculations," the multimodal AI performs another detailed analysis and provides an additional answer.

[1693] An example prompt for using a generative AI model is:

[1694] "I don't understand this calculus problem. I've uploaded an image of the relevant page from the textbook. Could you please explain the steps in detail?"

[1695] In this way, this system allows users to receive prompt and consistent learning support, significantly improving learning efficiency. Because it is not dependent on the teacher's availability or skills, high-quality learning support can be received anytime, anywhere.

[1696] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1697] Step 1:

[1698] The user installs the application and enters their name, email address, grade, and subjects of interest when they first launch it. This data is collected by the device and sent to the server. The server stores the received user information in a database and sends a confirmation email to the user. The input data is name, email address, grade, and subjects of interest, and the output data is a confirmation email for account activation.

[1699] Step 2:

[1700] The user clicks the link in the confirmation email to activate the account. The server detects the click on the confirmation link and activates the user's account information. The input data is the click on the confirmation link, and the output data is the activated user account.

[1701] Step 3:

[1702] The user logs in by entering their email address and password on the login screen. The device sends the login information to the server, which then authenticates it. If authentication is successful, the user is redirected to the dashboard screen. The input data is the email address and password, and the output data is the login success status.

[1703] Step 4:

[1704] The user clicks the "Ask a Question" button and is taken to a question input screen. The device displays the option to enter the question in text, image, or audio format. The user enters the question in the required format and uploads the appropriate file. The input data is text, image, or audio file, and the output data is the entered question data.

[1705] Step 5:

[1706] The terminal sends the input question data (text, image, voice) to the server. The server passes the received data to the analysis module. The input data is the question data, and the output data is the data transferred to the analysis module.

[1707] Step 6:

[1708] The server analyzes the text data to identify the content of the question. At the same time, it also analyzes image data and audio data to gain a comprehensive understanding of the problem. Based on the analysis results, it sends a request to the multimodal AI to generate an answer. The input data is the user's question data, and the output data is the analysis results.

[1709] Step 7:

[1710] The multimodal AI analyzes the received data comprehensively, searches for relevant information from the learning database, and generates the optimal answer. This answer is sent back to the server. The input data is the analysis result, and the output data is the generated answer.

[1711] Step 8:

[1712] The server sends the generated answer to the terminal. The terminal displays the answer to the user. The input data is the generated answer, and the output data is the answer displayed to the user.

[1713] Step 9:

[1714] The user inputs a follow-up question for the provided answer. The terminal sends the follow-up question to the server. The input data is the follow-up question, and the output data is the follow-up question data sent to the server.

[1715] Step 10:

[1716] The server requests the multimodal AI to perform reanalysis based on new question data and previous answer data. The multimodal AI performs detailed analysis and generates an additional answer. The server sends the generated answer to the terminal, which then provides it to the user again. The input data is the additional question data and previous answer data, and the output data is a detailed answer.

[1717] The above processing steps allow users to input questions in a variety of formats, and the data is analyzed in an integrated manner to provide consistent, high-quality answers. Furthermore, the user registration and authentication process is simplified, improving the efficiency of learning support.

[1718] (Application example 1)

[1719] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1720] Existing online learning systems mainly provide answers to questions using text, images, and audio, but lack the ability to provide visual answers using 3D models and animations that are easy for users to understand intuitively. Furthermore, there are only a limited number of systems that can respond to detailed follow-up questions in real time. Furthermore, there are limited means of providing learning support within the virtual environment, making it difficult for users to effectively progress through rich interactions.

[1721] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1722] In this invention, the server includes: means for a user to input a question in the form of text, image, or voice; means for transmitting the input question data to the server; means for analyzing the received question data and sending a request to an appropriate multimodal AI; means for the multimodal AI to analyze the question data and generate an answer; means for providing the generated answer to the user; means for the user to move around in the virtual environment in real time and access topic rooms specialized for a subject; and means for providing the answer visually using 3D models and animations in the virtual environment, thereby enabling the user to receive detailed learning support with high resolution.

[1723] A "user" is an individual or organization that uses the learning system.

[1724] "Text" is character information entered by the user.

[1725] "Images" are visual content uploaded by users.

[1726] "Audio" refers to sound information that is recorded and transmitted by the user.

[1727] "Question data" refers to the content of a question about learning that is input by the user in the form of text, image, or voice.

[1728] A "server" is a central device that receives question data sent by users, analyzes it, and generates answers.

[1729] "Multimodal artificial intelligence" is an artificial intelligence system that comprehensively analyzes data in multiple formats, such as text, images, and audio, and generates answers.

[1730] An "answer" is information that contains a solution or explanation to a user's question.

[1731] A "virtual environment" is a virtual space in which users can move around in real time and engage in learning experiences.

[1732] "Topic rooms" are study areas dedicated to specific subjects, rooms within a virtual environment staffed with dedicated artificial intelligence assistants.

[1733] A "3D model" is a three-dimensional digital object that visually represents a solution within a virtual environment.

[1734] "Animation" is a moving image effect that dynamically visualizes the solution within the virtual environment.

[1735] A "dedicated artificial intelligence assistant" is an artificial intelligence system that is placed in each topic room and is specialized to provide appropriate answers to users' questions.

[1736] "Registration information" refers to personal information and learning information provided by users in order to use the system.

[1737] "Verification Email" means an email sent to a User to confirm their registration information and activate their Account.

[1738] "Login Information" means the authentication data used by a User to access a System.

[1739] "Motion tracking" refers to tracking a user's position and actions within a virtual environment in real time.

[1740] This invention is a system that effectively supports online learning and is realized mainly through the interaction between a server, a terminal, and a user. Specifically, it includes the following components and the system functions resulting from their interaction:

[1741] User Registration and Login

[1742] First, in order to use the system using a terminal, a user must register when starting up the system for the first time. The user enters information such as their name, email address, grade, and subjects of interest, and this input data is sent from the terminal to the server. The server saves this information in a database and sends a confirmation email to the user's email address. When the user clicks the link in the confirmation email, the server validates the user's login information. After that, the user can log in and use all the system's functions.

[1743] Post a question and select the input format

[1744] If a user wants to receive learning support, they click the "Ask a Question" button on their device and are taken to a question input screen. They can enter their question in the form of text, image, or voice. For example, if they don't understand a math problem, they can enter "I don't understand this calculus problem" and upload an image of the relevant page from their textbook.

[1745] Question analysis and answer generation using multimodal AI

[1746] The device sends the question data entered by the user to the server. The server first analyzes the text data and identifies the type of question. Next, the server also analyzes the image data and audio data, and uses this information to seek a comprehensive understanding of the problem. Based on this analyzed data, the server sends a request to multimodal artificial intelligence (AI). The multimodal AI comprehensively analyzes the received data, searches for related information from a learning database, and generates the optimal answer.

[1747] Providing answers

[1748] The server sends the answer generated by the multimodal AI to the device. For example, for a calculus problem, the generated answer includes a step-by-step process such as "first find the derivative, then integrate." The device displays the answer to the user and provides visual support information such as diagrams as needed.

[1749] Virtual Learning Support Center

[1750] This system provides users with a means to move around in a virtual environment in real time and access topic rooms specialized for specific subjects. When a user enters a topic room related to a specific subject, a dedicated AI assistant is deployed to provide appropriate answers to the user's questions. Answers are visually represented using 3D models and animations, making them easier for users to understand intuitively. Users can also add more detailed questions on the spot and receive detailed answers in real time.

[1751] Examples of concrete examples and prompts

[1752] As a concrete example, when a high school student wants to solve a calculus problem in mathematics, the following process is taken: The user enters a topic room and enters the following prompt sentence into the terminal:

[1753] "I don't understand a calculus problem. Can you give me detailed steps to solve it?"

[1754] "I don't understand this calculation step. Please explain it again in more detail."

[1755] This allows the system to provide users with the information they need in real time, greatly improving the learning experience.

[1756] Hardware and software used

[1757] The following hardware and software are used to realize this system. The hardware includes a server, a user device (a smartphone or personal computer), and a head-mounted display. The software uses Python, Unity (for building the VR environment), and TensorFlow (for the AI ​​model).

[1758] The above configuration allows users to receive learning support efficiently and effectively.

[1759] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1760] Step 1:

[1761] When a user starts up the device and uses it for the first time, they are taken to the user registration screen. The user enters their name, email address, grade, and subjects of interest, and this information is sent from the device to the server. Input data: Name, email address, grade, and subjects of interest. Output data: User registration information saved on the server.

[1762] Step 2:

[1763] The server saves the user registration information in a database and sends a confirmation email to the user's email address. The confirmation email contains a link to activate the account. Input data: User registration information. Output data: Confirmation email.

[1764] Step 3:

[1765] The user clicks the link in the confirmation email and the server validates the user's login information. Input data: Click on the link in the confirmation email. Output data: Validated user login information.

[1766] Step 4:

[1767] The user logs in using a terminal and clicks the "Ask a Question" button to move to the question input screen. The user enters the question in the form of text, image, or voice. Input data: text, image, voice. Output data: question data.

[1768] Step 5:

[1769] The device sends the entered question data to the server. The server receives the question data and first performs text analysis. Next, it analyzes the image and audio data. Input data: Question data. Output data: Analyzed question data.

[1770] Step 6:

[1771] The server sends a request to the appropriate multimodal AI based on the analyzed question data. Input data: Analyzed question data. Output data: Request sent to the AI.

[1772] Step 7:

[1773] Multimodal AI comprehensively analyzes question data, searches for relevant information, and generates optimal answers. Input data: Question data. Output data: Generated answers.

[1774] Step 8:

[1775] The server receives the answer data generated by the multimodal AI and sends it to the terminal. The terminal provides this answer to the user. Input data: Generated answer data. Output data: Answer displayed on the terminal.

[1776] Step 9:

[1777] If the user wants to ask a more detailed question, he or she inputs the additional question through the terminal. The terminal sends the additional question data to the server. Input data: Additional question data. Output data: Additional question data sent to the server.

[1778] Step 10:

[1779] The server re-analyzes the additional question data and sends a request to the multimodal AI again to generate a detailed answer. Input data: Additional question data. Output data: The request sent again to the AI.

[1780] Step 11:

[1781] The multimodal AI generates a detailed answer and sends it to the terminal via the server. The terminal displays this answer to the user. Input data: detailed answer data. Output data: detailed answer displayed on the terminal.

[1782] Step 12:

[1783] Users can move freely around the virtual environment and access topic rooms specific to their subject. A dedicated AI assistant is deployed to provide visual answers to questions using 3D models and animations. Input data: user's location, question. Output data: visual answers using 3D models and animations.

[1784] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1785] The present invention is a system for effectively supporting online learning, which is combined with an emotion engine for recognizing a user's emotions and providing appropriate support in accordance with those emotions. It can be implemented as follows.

[1786] User Registration and Login

[1787] The first step for users to use the system is to install the app and register as a user when they first start it. On the registration screen displayed on the device, the user enters their name, email address, grade, and subjects of interest. This input data is sent from the device to the server, which stores it in a database. Once registration is complete, the server sends a confirmation email to the user's email address, and the user clicks on the link in the confirmation email to activate their account. After that, the user can log in and use all of the system's functions.

[1788] Post a question and select the input format

[1789] If a user wants learning support, they click the "Ask a Question" button, which takes them to a question input screen. The device then displays options for inputting questions in the form of text, image, or voice. For example, if a user doesn't understand a math problem, they can type "I don't understand this calculus problem" and upload an image of the corresponding page from their textbook.

[1790] Question analysis and answer generation using multimodal AI

[1791] The device sends the question data entered by the user to the server. The server first analyzes the text data and identifies the type of question. Next, the server also analyzes the image data and audio data, and uses this information to seek a comprehensive understanding of the problem. Based on this analyzed data, the server sends a request to the multimodal AI. The multimodal AI comprehensively analyzes the received data, searches for related information from a learning database, and generates the optimal answer.

[1792] Emotion recognition and response using emotion engine

[1793] The server not only receives the question data but also sends it to the emotion engine to recognize the user's emotion at the time of question entry. The emotion engine analyzes the user's emotion from text data, image data, and voice data and determines whether it is positive or negative. Based on the results of this emotion recognition, the multimodal AI adjusts the answer it generates. For example, if the user expresses negative emotions, it adds a message of encouragement or support.

[1794] Providing answers

[1795] The server sends the generated answer to the device. For example, for a calculus problem, the generated answer includes a step-by-step process such as "first find the derivative, then integrate." The device displays the answer to the user and provides visual support information such as diagrams as needed. In addition, the device may display an encouraging message based on the user's emotion recognition results.

[1796] Follow-up questions and real-time feedback

[1797] If the user wants to ask a more detailed question about the provided answer, they can enter the question again. For example, they can enter an additional question such as, "I don't understand the calculation in between." The device then sends this additional question to the server, and the server again requests a more detailed answer from the multimodal AI based on the previous answer data and the new question data. The multimodal AI then performs a detailed analysis and generates an additional answer, which the server then sends to the device. The device then provides this detailed answer to the user, helping them to deepen their understanding.

[1798] Specific examples

[1799] If high school student A wants to solve a calculus problem in mathematics, he or she first registers as a user and logs in. Next, he or she enters the text "I don't understand this calculus problem" on the question input screen and uploads an image of the textbook. The server analyzes the text content and image and uses an emotion engine to recognize A's emotions. If A shows negative emotions, the AI ​​generates an answer that includes an encouraging message. The server sends this answer to A's device, where A receives it. If A asks an additional question, such as "I don't understand the calculation in the middle," the multimodal AI performs a more detailed analysis again and provides an additional answer.

[1800] This system allows users to receive prompt and consistent learning support, and by using an emotion engine, it is possible to provide even more personalized support, which further improves learning efficiency and allows users to enjoy a better learning experience.

[1801] The processing flow will be explained below.

[1802] Step 1:

[1803] The user installs the app and clicks the "Register" button when launching it for the first time.

[1804] Step 2:

[1805] The terminal displays a screen that prompts the user to enter registration information such as name, email address, grade, and subjects of interest.

[1806] Step 3:

[1807] The user enters their name, email address, grade, and subjects of interest and clicks the "Submit" button.

[1808] Step 4:

[1809] The terminal transmits the entered registration data to the server.

[1810] Step 5:

[1811] The server stores the received data in a database and sends a confirmation email to the user's email address.

[1812] Step 6:

[1813] The user clicks on the link in the confirmation email to activate their account.

[1814] Step 7:

[1815] The server accepts the confirmation link and validates the user's login information.

[1816] Step 8:

[1817] The user logs in to the app and navigates to the home screen.

[1818] Step 9:

[1819] The user clicks the "Ask a Question" button on the home screen.

[1820] Step 10:

[1821] The device will display a question entry screen and provide the option to enter the question in the form of text, image, or voice.

[1822] Step 11:

[1823] A user types in the text "I don't understand this calculus problem" and uploads an image of the corresponding page from the textbook.

[1824] Step 12:

[1825] The terminal sends the entered question data (text, image) to the server.

[1826] Step 13:

[1827] The server analyzes the text data and performs initial filtering.

[1828] Step 14:

[1829] The server sends the image data to an image analysis module to extract relevant information.

[1830] Step 15:

[1831] The server sends the voice data to a voice analysis module, which analyzes the voice tone and content.

[1832] Step 16:

[1833] The server integrates the text, image, and voice analysis results and sends them to the emotion engine.

[1834] Step 17:

[1835] The emotion engine analyzes the received data and determines the user's emotion. If the emotion is negative, it revisits the specific situation and evaluates whether an encouraging message needs to be added.

[1836] Step 18:

[1837] The emotion engine sends the emotion recognition results back to the server.

[1838] Step 19:

[1839] The server sends an analysis request to the multimodal AI based on the emotion recognition result. If the emotion is negative, the request includes an encouraging message or a special answer format.

[1840] Step 20:

[1841] Multimodal AI integrates and analyzes question data and emotional information to understand the content of the problem.

[1842] Step 21:

[1843] Multimodal AI collates information from relevant learning databases to generate optimal answers, supplemented with emotional messages where appropriate.

[1844] Step 22:

[1845] The server receives the generated answer data and transmits it to the user's terminal.

[1846] Step 23:

[1847] The device displays the answer data to the user, including visual support information such as text, formulas, diagrams, etc. If the emotion is negative, an encouraging message is also displayed.

[1848] Step 24:

[1849] If the user wants to get more details about the answer, he / she can input and submit an "additional question."

[1850] Step 25:

[1851] The terminal sends a follow-up question to the server.

[1852] Step 26:

[1853] The server again sends an analysis request for additional questions to the multimodal AI.

[1854] Step 27:

[1855] Multimodal AI then performs a detailed analysis again to generate additional detailed answers.

[1856] Step 28:

[1857] The server transmits the generated additional answer to the user's terminal.

[1858] Step 29:

[1859] The device displays additional answers to the user to help them deepen their understanding.

[1860] Through these steps, users can receive fast and consistent learning support, and the emotional engine can provide more personalized assistance, improving learning efficiency and providing a more meaningful learning experience for users.

[1861] Example 2

[1862] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1863] Conventional online learning support systems lack the flexibility to respond to the diverse question formats users encounter, and can only accept questions in specific formats. Furthermore, they provide uniform answers without considering changes in the user's motivation or emotions regarding learning, making it impossible to provide support optimized for each individual user. Furthermore, it is difficult to provide quick and detailed real-time feedback on follow-up questions, which hinders users' learning efficiency.

[1864] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for a user to input a question in any one of the formats of text, image, or voice, a means for transmitting the input question data to the server, a means for analyzing the received question data and transmitting a request to an appropriate multimodal AI, a means for the multimodal AI to analyze the question data and generate an answer, a means for providing the generated answer to the user, and a means for analyzing the user's emotions using an emotion recognition engine and adjusting the content of the answer based on the analyzed emotion data. This makes it possible to respond to various types of questions from users and provide optimal answers that take into account each individual's emotional state.

[1865] "User" refers to any person or entity that utilizes the online learning system to enter questions and receive answers.

[1866] "Text" refers to a digital data format that includes character information, and refers to character string data that a user uses to input a question.

[1867] "Image" refers to a digital data format containing visual information, such as a photograph or drawing uploaded by a user to supplement a question.

[1868] "Speech" refers to a digital data format containing auditory information, and refers to voice data used by a user to input a question.

[1869] "Server" refers to a computer system or cloud infrastructure for processing, analyzing, storing, and generating answers from data received from users.

[1870] "Multimodal AI" refers to an AI model that comprehensively analyzes different types of data (text, images, audio) and generates optimal answers.

[1871] An "emotion recognition engine" refers to software or algorithms that analyze emotions from user input data and determine whether they are positive or negative.

[1872] "Answer" refers to output data generated by multimodal artificial intelligence that contains appropriate information and instructions in response to a user's question.

[1873] "Confirmation Email" refers to the email sent from the server at the time of user registration for the purpose of activating the account.

[1874] "Confirmation Link" refers to the URL included in the confirmation email that the User clicks to activate their account.

[1875] "Login Information" refers to the authentication information (e.g., user ID and password) used by a User to access the System.

[1876] The present invention is a system for supporting online learning, in which a user inputs a question in the form of text, image, or voice, and generates an answer to the question using multimodal artificial intelligence and an emotion recognition engine. Specific embodiments of the present invention are described below.

[1877] First, a user installs and launches the online learning application on a device such as a smartphone. In order to use the system, the user must first register. On the registration screen, the user enters their name, email address, grade, and subjects of interest, and this data is sent from the device to the server. The server stores this data in a database (e.g., MySQL database) and sends a confirmation email to the user. The user clicks the link in the confirmation email to activate their account.

[1878] Next, when the user clicks the "Ask a Question" button, the device displays a screen where the user can enter a question in the form of text, image, or voice. For example, the user can enter the text "I don't understand this calculus problem" and upload an image of the corresponding page from the textbook. This input data is sent from the device to the server.

[1879] The server analyzes the received text data using a natural language processing library (e.g., NLTK, SpaCy), and analyzes the image data using an image processing library (e.g., OpenCV). The analyzed data is sent to a multimodal AI, which uses a generative AI model such as GPT-4 to comprehensively analyze the received data and generate the optimal answer.

[1880] The server then sends the received question data to an emotion recognition engine. The emotion recognition engine uses the Emotion API or other tools to analyze the user's emotions from text, image, and audio data and determine whether they are positive or negative. Based on the emotion recognition results, the multimodal AI adjusts the answer it generates. For example, if the user expresses negative emotions, it adds a message of encouragement or support.

[1881] The generated answer is sent from the server to the device, which then displays it to the user. Visual support information (e.g., diagrams) is also displayed as needed. If the user enters additional questions, the multimodal AI again performs a detailed analysis, generating the optimal answer and sending it from the server to the device.

[1882] As a concrete example, if high school student A wants to solve a calculus problem in mathematics, he or she would follow the steps below. First, he or she registers as a user and logs in. Next, he or she enters the text "I don't understand this calculus problem" on the question input screen and uploads an image of the textbook. The server analyzes the text content and image and uses an emotion recognition engine to recognize A's emotions. If A shows negative emotions, the AI ​​generates an answer that includes an encouraging message. The server sends this answer to A's device, where A receives it. If A asks an additional question, such as "I don't understand the calculation in the middle," a more detailed analysis is performed again and the answer is provided to the device.

[1883] An example of a prompt sentence is as follows:

[1884] User: I don't understand this calculus problem. (Image attached) (Image: Calculus problem page in textbook)

[1885] Server: Performs text and image analysis to generate the optimal answer.

[1886] Emotion Recognition Engine: Analyzes the user's emotions and finds that they are negative. Adds encouraging messages.

[1887] Server: Generates the best answer and sends it to the device along with an encouraging message.

[1888] This invention allows users to receive prompt and consistent learning support, and by using an emotion recognition engine, it is possible to provide even more personalized support to users, significantly improving learning efficiency.

[1889] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1890] Step 1:

[1891] The user installs and launches the app.

[1892] A user downloads the app from a digital store and installs it on their device.

[1893] Launch the app and the first screen will appear.

[1894] Input: User downloads and launches the app

[1895] Output: First screen display of the app

[1896] Step 2:

[1897] The terminal displays a user registration screen.

[1898] A registration screen UI is displayed, including fields for the user to enter their name, email address, grade level, and subjects of interest.

[1899] Input: Registration screen displayed after launching the app

[1900] Output: A field for the user to enter input.

[1901] Step 3:

[1902] The user enters the required information and presses the registration button.

[1903] The user enters their name, email address, grade, and subjects of interest and presses the "Register" button.

[1904] Input: Information entered by the user (name, email address, grade, subjects of interest)

[1905] Output: Sending input data by pressing the register button

[1906] Step 4:

[1907] The terminal sends the input data to the server.

[1908] The terminal sends the data entered by the user to the server in JSON format at the REST API endpoint.

[1909] Input: Information entered by the user

[1910] Output: Send data in JSON format

[1911] Step 5:

[1912] The server stores the registration information in a database.

[1913] The server analyzes the received data and stores it in a database (e.g. MySQL).

[1914] Input: JSON format user information data

[1915] Output: User information stored in the database

[1916] Step 6:

[1917] The server sends a confirmation email.

[1918] The server uses a mail server (e.g. Postfix) to send the user a confirmation email containing a link to activate the account.

[1919] Input: User information stored in the database

[1920] Output: Send confirmation email

[1921] Step 7:

[1922] The user clicks the confirmation link.

[1923] The user opens the confirmation email and activates their account by clicking the link.

[1924] Input: Link in confirmation email

[1925] Output: Account enabled

[1926] Step 8:

[1927] The server performs the account activation process.

[1928] The server receives the link and updates the status of the corresponding account to "active."

[1929] Input: The confirmation link the user clicked

[1930] Output: Enabled account status

[1931] Step 9:

[1932] A user logs in.

[1933] Using your user ID and password, you can access the login screen and use all the functions of the system.

[1934] Input: User ID and password

[1935] Output: Successfully logged into the system

[1936] Step 10:

[1937] The user clicks the "Ask a Question" button.

[1938] A user clicks the "Ask a Question" button within the app.

[1939] Input: User clicks "Ask a question" button

[1940] Output: Display of question input screen

[1941] Step 11:

[1942] The device displays a question input screen.

[1943] Displays a question input screen with text, image, and audio input options.

[1944] Input: User clicks "Ask a question" button

[1945] Output: Display of question input screen

[1946] Step 12:

[1947] The user enters a question.

[1948] A user types "I don't understand this calculus problem" into a text box and uploads an image of a page from a textbook.

[1949] Input: Question text entered by the user, and uploaded images

[1950] Output: Question data ready to send

[1951] Step 13:

[1952] The terminal transmits the input data to the server.

[1953] The text data and image data are sent to a server.

[1954] Input: Question data entered by the user (text, images)

[1955] Output: Send data to the server

[1956] Step 14:

[1957] The server parses the text data.

[1958] The server analyzes the text data using a natural language processing library (e.g., NLTK, SpaCy) to identify the question.

[1959] Input: Received text data

[1960] Output: Analysis results (question content identification)

[1961] Step 15:

[1962] The server analyzes the image data.

[1963] The server analyzes the image data using an image processing library (e.g., OpenCV) and extracts relevant information.

[1964] Input: Received image data

[1965] Output: Analysis results (image content analysis)

[1966] Step 16:

[1967] The server sends the analysis data to the multimodal AI.

[1968] The analysis results are integrated and sent to multimodal AI.

[1969] Input: Text and image analysis results

[1970] Output: A request to the multimodal AI

[1971] Step 17:

[1972] Multimodal AI generates optimal answers.

[1973] Generative AI models such as GPT-4 are used to analyze the received data and generate optimal answers.

[1974] Input: Integrated analysis data

[1975] Output: The generated answer

[1976] Step 18:

[1977] The server sends the answer data to the emotion recognition engine.

[1978] The emotion recognition engine analyzes the user's emotions along with the generated answer data.

[1979] Input: Answer data

[1980] Output: User sentiment analysis results

[1981] Step 19:

[1982] An emotion recognition engine analyzes the user's emotions.

[1983] The Emotion API is used to determine the user's emotions from the data sent.

[1984] Input: Answer data

[1985] Output: Sentiment analysis result (positive / negative)

[1986] Step 20:

[1987] The server adjusts the answer based on the emotional data.

[1988] Based on the results of emotion recognition, the multimodal AI is instructed to adjust its answers, for example by adding encouraging messages.

[1989] Input: Sentiment analysis results

[1990] Output: Adjusted answer

[1991] Step 21:

[1992] The server sends the answer data to the terminal.

[1993] The optimized answer is sent to the device.

[1994] Input: Adjusted answer data

[1995] Output: Send answer data to the device

[1996] Step 22:

[1997] The terminal displays the answer to the user.

[1998] The terminal displays the answer and supplementary information to the user.

[1999] Input: Answer data received from the server

[2000] Output: What is displayed to the user

[2001] Step 23:

[2002] The user enters a follow-up question.

[2003] Enter additional detailed questions in text for the answers provided.

[2004] Input: Additional questions from the user

[2005] Output: Additional question data

[2006] Step 24:

[2007] The terminal sends a follow-up question to the server.

[2008] Send additional question data to the server.

[2009] Input: Additional question data by the user

[2010] Output: Send data to the server

[2011] Step 25:

[2012] The server reanalyzes the data and sends the request to the multimodal AI again.

[2013] Send additional question data to the multimodal AI to generate detailed answers.

[2014] Input: Additional question data

[2015] Output: Detailed answer request

[2016] Step 26:

[2017] Multimodal AI generates detailed answers.

[2018] Generate detailed answers based on additional question data.

[2019] Input: Additional question data

[2020] Output: Detailed answer

[2021] Step 27:

[2022] The server sends detailed answer data to the terminal.

[2023] The generated detailed answer is sent to the device.

[2024] Input: Detailed answer data

[2025] Output: Detailed answer data sent to the device

[2026] Step 28:

[2027] The terminal displays the detailed answer to the user.

[2028] The terminal displays the detailed answer content to the user.

[2029] Input: Detailed answer data received from the server

[2030] Output: What is displayed to the user

[2031] In this way, the system can effectively execute a series of processes to receive a user's question, analyze it, and provide the most appropriate answer.

[2032] (Application example 2)

[2033] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2034] Conventional online learning systems can generate answers by analyzing text, images, and audio data provided by users, but they rarely take the user's emotional state into consideration. This can lead to problems such as a decrease in learning motivation and a lack of appropriate feedback, r...

Claims

1. a means for the user to input a question in the form of text, image, or voice; means for transmitting the input question data to a server; means for analyzing the received question data and sending a request to an appropriate multimodal artificial intelligence; A means for multimodal artificial intelligence to analyze question data and generate answers; means for providing the generated answer to a user; A system including:

2. A means for the user to ask follow-up questions about the answers; a means for sending additional questions to the server for re-analysis and generation of detailed answers; a means for providing a detailed answer to the user; The system of claim 1 , comprising:

3. A means for the server to store the user's registration information in a database and send a confirmation email; A means to validate the login information of users who click on the confirmation link; The system of claim 1 , comprising:

Images