system

Abstract

We provide the system. [Solution] A system including a user terminal with an educational application installed and a server, A means of sending question data entered from the user terminal to the server, A method for analyzing question data using a natural language processing model and searching for relevant lecture content, A means of generating the optimal answer pattern based on the searched lecture content, A means for sending the generated response patterns to the user's terminal in text, audio, and video formats, A system that includes means for displaying and playing back response patterns received on a user's terminal.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional learning method, when a student has doubts during learning, it is difficult to obtain an accurate answer immediately. In particular, outside of school or learning institution hours, it is not possible to directly ask a teacher, resulting in a problem of reduced learning efficiency. Also, in online or self-study, there has been a lack of a system for efficiently linking the content of questions with relevant learning resources. To solve these problems, a system capable of efficient and rapid learning support has been demanded.

Means for Solving the Problems

[0005] The present invention provides a system including a user terminal with an educational application installed and a server. Specifically, the system includes means for sending question data entered from the user terminal to the server, means for analyzing the question data using a natural language processing model and searching for relevant lecture content, means for generating an optimal answer pattern based on the searched lecture content, means for sending the generated answer pattern to the user terminal in text, audio, and video formats, and means for displaying or playing back the received answer pattern on the user terminal. This system allows students to resolve questions immediately during learning, improving learning efficiency. Furthermore, the system's adaptability and effectiveness can be further enhanced by including means for storing feedback data sent by the user on the server and using it to improve the accuracy of future answers. Additionally, by including means for synchronizing the latest lecture content data from the server when the user terminal first logs in, the system can always provide the latest learning resources.

[0006] An "educational application" is software used by users as part of their learning process, and it has functions to help them resolve questions and access learning resources.

[0007] A "user terminal" refers to an electronic device, such as a smartphone, tablet, or personal computer, that a user uses to access an educational support system via an application.

[0008] A "server" is a central computer system that hosts databases and AI models, processes requests from user terminals, and provides appropriate data.

[0009] "Question data" refers to information about learning-related questions and inquiries entered by users through the application.

[0010] A "natural language processing model" is an algorithm or machine learning model that analyzes text data to understand its meaning and extract information related to a question.

[0011] "Lecture content" refers to the specific content and structure of learning resources that have been stored in a database in advance for learning support purposes.

[0012] An "answer pattern" is an information format, such as text, audio, or video, generated to provide an answer to a question.

[0013] "Feedback data" refers to information in which users have entered evaluations and opinions on the provided answers, and is used to improve the system.

[0014] "Synchronization" refers to the process of matching data between the user's terminal and the server, and specifically to the operation of updating the user's terminal with the latest lecture content data. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] Displays an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be described.

[0018] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] The system of this invention aims to provide effective learning support using educational applications. This system consists of user terminals, a server, and a communication network connecting them.

[0037] Initial setup

[0038] Settings on the user terminal

[0039] First, users install the educational application on their smartphone, tablet, or computer. After installation, users launch the application and enter the necessary information to create an account. Once account creation and login are complete, the user's device connects to the server and synchronizes with the latest lecture data from the server.

[0040] Enter and submit your question

[0041] If a user has a question while learning, they can select a specific subject or topic through the educational application and enter their question in the text box. Once they have finished entering their question, they press the submit button.

[0042] User terminal operation

[0043] The user terminal sends the entered question data to the server. This question data includes the text entered by the user and other metadata (such as a timestamp and user ID).

[0044] Question analysis and answer generation

[0045] The server receives question data sent from the user's terminal. First, it analyzes the question using a natural language processing (NLP) model. This analysis extracts keywords and intent from the question. For example, in the question "How to solve a quadratic equation," "quadratic equation" and "how to solve" are extracted as important keywords.

[0046] Server operation

[0047] The server searches the database for relevant lecture content based on the analysis results. This search yields potential candidates such as relevant teaching materials, whiteboard notes, and past questions and answers. It then generates the optimal response pattern (text, audio, video) and sends it to the user's terminal.

[0048] Providing answers to users

[0049] The user terminal analyzes the response data received from the server and displays it in an appropriate format. For example, an explanation of how to solve a quadratic equation might be displayed as text, and corresponding audio or video can be played as needed. The user can then review the provided answers and continue their learning.

[0050] Gathering feedback and improving the system

[0051] Users can provide satisfaction ratings and additional feedback on the answers. This feedback data is sent from the user's device to the server, which stores it in a database. This feedback data is used to improve the accuracy of future automated responses and to improve the overall system.

[0052] Specific example

[0053] For example, suppose a user asks a question about quadratic equations in mathematics. The user enters "how to solve a quadratic equation" into a text box and presses the submit button. The user's terminal sends this question data to the server. The server uses an NLP model to extract the important keywords "quadratic equation" and "how to solve," and searches the database for relevant lecture content. From the search results, it generates the optimal answer pattern, for example, explanatory text such as "A quadratic equation is of the form ax^2 + bx + c = 0, and the quadratic formula is...", along with corresponding audio and video data, and sends it to the user's terminal. The user's terminal receives this and displays or plays it for the user.

[0054] This system allows users to efficiently resolve questions and advance their learning anytime, anywhere.

[0055] The following describes the processing flow.

[0056] Step 1:

[0057] Users install the educational application on their smartphone, tablet, or PC and launch the app. They then create an account by entering their email address and password and log in.

[0058] Step 2:

[0059] The device sends login information to the server. The server authenticates the received information and returns the authentication result to the device. If the login is successful, the device displays the home screen.

[0060] Step 3:

[0061] Upon initial login, the device requests the latest lecture data from the server. The server retrieves the lecture data from the database and sends it to the device. The device saves the received lecture data to local storage and displays a "synchronization complete" notification to the user.

[0062] Step 4:

[0063] The user operates the app to select the subject or topic for which they have a question during their studies. After selecting, they enter the specific question in the text box and press the submit button.

[0064] Step 5:

[0065] The terminal sends the user-entered question data to the server. This question data includes the entered text and metadata (e.g., timestamp, user ID).

[0066] Step 6:

[0067] The server passes the received question data to a natural language processing (NLP) model. The NLP model analyzes the question content and extracts important keywords and the intent of the question.

[0068] Step 7:

[0069] Based on the analysis results, the server searches the database for relevant lecture content. The database includes lecture content, whiteboard notes, past questions and answers, etc.

[0070] Step 8:

[0071] The server generates the optimal response pattern based on the searched lecture content. This response pattern is generated in text, audio, and video formats.

[0072] Step 9:

[0073] The server sends the generated response pattern to the terminal. The terminal analyzes the received response data and configures the display settings in the appropriate format.

[0074] Step 10:

[0075] The device displays the answer in text format on the user's screen. For example, it might display, "The quadratic equation is of the form ax^2 + bx + c = 0, and the quadratic formula is..." If necessary, it displays audio or video playback buttons, and playback begins when the user presses them.

[0076] Step 11:

[0077] Users rate the provided answers and provide feedback. Ratings are given in the format of "Good" or "Bad."

[0078] Step 12:

[0079] The device sends user feedback data to the server. The server stores the received feedback data in a database and uses it to improve the accuracy of future responses.

[0080] (Example 1)

[0081] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0082] In today's educational environment, students often lack the means to immediately resolve their questions. Furthermore, the limited availability of teaching materials and lecture content from educational institutions makes it difficult to provide support tailored to the individual needs of learners. Moreover, there are insufficient means of collecting feedback to continuously improve the quality and accuracy of the answers provided. This invention aims to solve these problems and provide a system that enables users to learn effectively.

[0083] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0084] In this invention, the server includes means for transmitting inquiry data entered from a terminal to an information processing device, means for analyzing the inquiry data using a generation AI model and searching for relevant educational content, and means for generating an optimal answer format based on the searched educational content. This makes it possible for users to efficiently resolve their questions anytime, anywhere, improving their satisfaction and learning effectiveness.

[0085] An "educational program" is software intended for use by users to support educational activities.

[0086] A "device" refers to a hardware device used by a user to install and utilize educational programs, such as a smartphone, tablet, or personal computer.

[0087] An "information processing device" is a computer system that analyzes and processes data transmitted from a terminal and provides the necessary data.

[0088] "Inquiry data" refers to the content of questions entered and submitted by users through educational programs, along with related data (such as timestamps and user IDs).

[0089] A "generative AI model" is an artificial intelligence model that uses natural language processing and other machine learning techniques to analyze input text data and extract intent and keywords.

[0090] "Educational content" includes teaching materials, lecture notes, and past questions and answers related to specific subjects or topics.

[0091] "Answer format" refers to the method of explanation provided to the user, and includes formats such as text, audio, and video.

[0092] The system of this invention is intended to support learning using educational programs and consists of a terminal, an information processing device, and a communication network connecting them. The detailed configuration and processing flow of the system are described below.

[0093] Initial setup

[0094] Settings on the user terminal

[0095] Users install the educational program on their smartphones, tablets, PCs, or other devices. After installation, users launch the application and enter the necessary information to create an account. Once account creation and login are complete, the device connects to the information processing unit and synchronizes the latest lecture data from the server. This allows users to access the latest educational content at any time.

[0096] Enter and submit your question

[0097] If a user has a question while learning, they select a specific subject or topic through the educational program and enter their question into the text box. After completing the input, they press the submit button. The terminal then sends the entered inquiry data (including the question text, timestamp, and user ID) to the information processing device.

[0098] Question analysis and answer generation

[0099] The information processing device receives inquiry data sent from the terminal. First, it analyzes the inquiry data using a generative AI model to extract keywords and intent of the question. For example, in the case of the question "How to solve a quadratic equation," "quadratic equation" and "how to solve" are analyzed as important keywords. Next, the information processing device searches a database based on the extracted keywords to find relevant educational content. This database includes teaching materials, lecture notes, and past questions and answers. It then generates the most suitable answer format (text, audio, video) and sends it to the terminal.

[0100] Providing answers to users

[0101] The terminal analyzes the response data received from the information processing device and displays it in an appropriate format. For example, an explanation of how to solve a quadratic equation might be displayed as text, and corresponding audio or video can be played as needed. This allows the user to review the provided answers and continue their learning.

[0102] Gathering feedback and improving the system

[0103] Users can input their satisfaction level and provide additional feedback on the provided answers. The terminal sends this feedback data to the information processing device. The information processing device stores the feedback data in a database and uses it to improve the accuracy of future answers and the overall system. This feedback is crucial for the continuous improvement of the system and the enhancement of the user experience.

[0104] Specific example

[0105] The following are specific examples of how the system can be used.

[0106] For example, if a user asks a question about quadratic equations in mathematics, the user enters "how to solve a quadratic equation" into the text box and presses the submit button. The terminal sends this inquiry data to the information processing device. The information processing device uses a generative AI model to extract the important keywords "quadratic equation" and "how to solve," and searches its database for relevant educational content. From the search results, it generates the optimal answer format, such as "A quadratic equation is in the form ax^2 + bx + c = 0, and the quadratic formula is...", along with corresponding audio and video data, and sends it to the terminal. The terminal receives this and displays or plays it for the user.

[0107] This system allows users to efficiently resolve questions and advance their learning anytime, anywhere.

[0108] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0109] Step 1:

[0110] Initial setup on the user terminal

[0111] Users install the educational program on their smartphones, tablets, or computers. After installation, users launch the application and enter necessary information such as their name, email address, and password to create an account. The entered information is sent from the device to the server, which stores this information in a database. Once authentication is complete, the device synchronizes the latest lecture data from the server and displays it on the device.

[0112] Input: User's personal information (name, email address, password)

[0113] Output: Account creation confirmation and synchronization of the latest lecture data

[0114] Step 2:

[0115] Enter and submit your question

[0116] If a user has a question while learning, they select a specific subject or topic through the educational program and enter their question in a text box. Once this is complete, the user presses the submit button. The device then sends this inquiry data (including the question text, timestamp, and user ID) to the server.

[0117] Input: User's question, timestamp, user ID

[0118] Output: Confirmation of sending query data to the server

[0119] Step 3:

[0120] Question analysis

[0121] The server receives query data sent from the terminal. The server uses a generative AI model to analyze the query data and extract keywords and intent from the question. For example, in the question "How to solve a quadratic equation," "quadratic equation" and "how to solve" are extracted as important keywords.

[0122] Input: Inquiry data (question content, timestamp, user ID)

[0123] Output: Analysis results (keywords and intent)

[0124] Step 4:

[0125] Search for related educational content

[0126] Based on the analysis results, the server searches the database for relevant educational content. This database includes teaching materials, lecture notes, and past questions and answers. The server finds the most suitable educational content that matches the search criteria.

[0127] Input: Analysis results (keywords and intent)

[0128] Output: Related educational content (teaching materials, lecture notes, past questions and answers)

[0129] Step 5:

[0130] Generating an answer format

[0131] The server generates the most suitable response format based on the searched educational content. The response format includes text, audio, and video, and is provided in the format that is easiest for the user to understand.

[0132] Input: Related educational content (textbooks, lecture notes, past questions and answers)

[0133] Output: Optimal response format (text, audio, video)

[0134] Step 6:

[0135] Submit your response

[0136] The server sends the generated response format to the terminal. The transmitted data includes the response text and corresponding audio or video data.

[0137] Input: Optimal response format (text, audio, video)

[0138] Output: Confirmation of sending response data to the terminal

[0139] Step 7:

[0140] Providing answers to users

[0141] The device analyzes the response data received from the server and displays or plays it back to the user in an appropriate format. For example, text may be displayed on the screen, and audio or video may be played. The user can then proceed with their learning based on this information.

[0142] Input: Response data (text, audio, video)

[0143] Output: Display and play back the response to the user.

[0144] Step 8:

[0145] Gathering feedback

[0146] Users can input their satisfaction level and provide additional feedback on the provided answers. The device sends this feedback to the server, which stores it in a database. This helps improve the accuracy of future answers.

[0147] Input: User feedback (satisfaction level, additional comments)

[0148] Output: Confirmation and storage of feedback data sent to the server.

[0149] (Application Example 1)

[0150] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0151] With the increasing prevalence of autonomous vehicles, there is a growing need for ways to ensure passengers have a meaningful experience during long journeys. However, conventional in-car entertainment systems are primarily limited to entertainment and information provision, lacking effective learning tools for education. Therefore, providing interactive educational support systems that passengers can use within autonomous vehicles is a key challenge.

[0152] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0153] In this invention, the server includes means for transmitting question data entered from a user terminal to the server, means for analyzing the question data using a natural language processing model and searching for relevant lecture content, and means for generating an optimal answer pattern based on the searched lecture content. This enables passengers to learn effectively in an autonomous vehicle and spend long journeys meaningfully.

[0154] An "educational application" is software designed for users to learn specific knowledge or skills.

[0155] A "user terminal" is a computer device used by a user to run and operate educational applications.

[0156] A "server" is a central computer system responsible for data processing and storage for educational applications, and for communicating with user terminals.

[0157] A "natural language processing model" is an artificial intelligence technology used to analyze question data and text data entered by users.

[0158] An "infotainment system" is an in-car system that provides information and entertainment to passengers in autonomous vehicles.

[0159] "Question data" refers to information about questions and problems that users have entered into educational applications.

[0160] An "answer pattern" is the format of the explanations and descriptions that an educational application provides in response to a user's question.

[0161] "Feedback data" refers to information in which users have entered evaluations and opinions regarding the provided answers.

[0162] "Synchronization" is the process performed to make data consistent between the user's terminal and the server.

[0163] System Overview

[0164] The system of the present invention is for the effective use of educational applications in autonomous vehicles. It consists of a user terminal, a server, a communication network, and an infotainment system within the autonomous vehicle.

[0165] Program Operation Overview

[0166] The user installs the educational application on the in-car infotainment system and begins the learning process. The system operates in the following steps:

[0167] 1. The user operates the infotainment system and launches an educational application.

[0168] 2. Enter the information needed to create a user account on the infotainment system, create the account, and log in. This will start synchronization with the server.

[0169] 3. If a question arises during learning, the user selects a specific subject or topic within the application and enters the question in the text box.

[0170] 4. The infotainment system sends the question data to the server. This question data includes metadata such as text, timestamp, and user ID.

[0171] 5. The server receives the question data and performs analysis using a natural language processing (NLP) model. This analysis extracts important keywords and intents.

[0172] 6. The server searches the database for relevant lecture content and generates the optimal response pattern (text, audio, video).

[0173] 7. The generated response patterns are sent back to the infotainment system and presented to the user.

[0174] 8. Users review the provided answers and continue learning. Simultaneously, they can provide feedback on the answers. This feedback data is sent to the server and used to improve the accuracy of future answers.

[0175] Hardware and software usage

[0176] Hardware: Infotainment systems for autonomous vehicles, smartphones, tablets

[0177] software:

[0178] Natural Language Processing (NLP) models (e.g., BERT model)

[0179] Communication protocol: Internet Protocol (HTTP, HTTPS)

[0180] Specific example

[0181] For example, if a passenger enters a request to learn "the basics of quantum mechanics" while in the car, the in-car infotainment system automatically sends the question to a server. The server uses an NLP model to extract the keywords "quantum mechanics" and "basics." Based on the keywords, the server searches for the most suitable learning materials (such as introductory videos and text explanations) and sends them to the infotainment system. The infotainment system then displays or plays the received materials for the passenger.

[0182] Examples of prompts to input into a generative AI model

[0183] "The user typed 'Teach me the basics of quantum mechanics.' Please search for and provide appropriate teaching materials."

[0184] This system allows passengers to spend their time productively while efficiently pursuing their studies.

[0185] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0186] Step 1:

[0187] The user launches the educational application through the infotainment system in the autonomous vehicle and enters information to create an account. This information includes username, email address, and password. Once the user enters the information and presses the submit button, the infotainment system sends it to the server. The server receives the input data and creates the user account. This initiates communication between the server and the infotainment system, and the latest lecture content data is synchronized upon the first login.

[0188] Input: Username, email address, password, etc.

[0189] Output: User account created, synchronization with server started

[0190] Step 2:

[0191] When a user has a question, they operate an educational application on the infotainment system, select a specific subject or topic, and enter their question in a text box. For example, they might enter, "Please explain the basics of quantum mechanics." This question data, along with a timestamp and user ID, is sent to the server.

[0192] Input: Question (e.g., "Please explain the basics of quantum mechanics"), timestamp, user ID

[0193] Output: Sending query data to the server

[0194] Step 3:

[0195] The server receives the submitted question data and analyzes the question content using a natural language processing (NLP) model. During the analysis, important keywords and the user's intent are extracted. For example, keywords such as "quantum mechanics" and "fundamentals" may be extracted. Based on these analysis results, the server searches the database for relevant lecture content.

[0196] Input: Question data (text, timestamp, user ID)

[0197] Output: Keyword extraction, search for related lecture content

[0198] Step 4:

[0199] The server generates the most suitable response pattern from the search results. For example, it might select a text explanation or video on "Fundamentals of Quantum Mechanics" for beginners. These responses are generated in various formats such as text, audio, and video, and sent to the infotainment system.

[0200] Input: Analysis results, related lecture content

[0201] Output: Generation of optimal response patterns (text, audio, video)

[0202] Step 5:

[0203] The infotainment system receives the response patterns and presents them to the user. For example, a text explanation may be displayed on the screen, and videos or audio may be played as needed. The user can review the provided answers and continue learning.

[0204] Input: Response patterns (text, audio, video)

[0205] Output: Providing answers to the user (display, play)

[0206] Step 6:

[0207] Users can provide feedback on the provided answers. This feedback data is then sent back to the server via the infotainment system. The server stores this feedback data and uses it to improve the accuracy of future answers.

[0208] Input: User Feedback

[0209] Output: Feedback data is stored on the server and used for future improvements.

[0210] This will enable a system that allows passengers to efficiently learn while inside an autonomous vehicle.

[0211] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0212] This invention provides more effective and personalized learning support by combining an emotion engine with a learning support system that uses educational applications. The system consists of user terminals, a server, and a communication network connecting them.

[0213] Initial setup

[0214] Settings on the user terminal

[0215] First, users install the educational application on their smartphone, tablet, or PC. After installation, users launch the application and enter the necessary information to create an account. Once account creation and login are complete, the user's device connects to the server and synchronizes with the latest lecture data from the server.

[0216] Recognition of emotions and input of questions

[0217] Facial expression and voice analysis

[0218] The application is equipped with a camera and microphone, and includes an emotion engine that uses these to recognize emotions from the user's facial expressions and voice. When the user interacts with the application, the camera and microphone automatically activate and analyze the user's facial expressions and voice tone.

[0219] User question input

[0220] When a user has a question during their studies, they select a specific subject or topic, enter the question into a text box, and press the submit button. At this point, the emotion engine analyzes the user's emotions in real time and adds that data to the question data.

[0221] User terminal operation

[0222] The terminal sends the entered question data and sentiment data to the server. This question data includes text, sentiment data, and metadata (such as a timestamp and user ID).

[0223] Question analysis and answer generation

[0224] Natural language processing and sentiment analysis

[0225] The server receives question data sent from the user's terminal and analyzes the question content using a natural language processing (NLP) model. Furthermore, it also analyzes sentiment data to understand the user's emotional state (e.g., confusion, dissatisfaction, excitement).

[0226] Search related lecture content

[0227] The server searches the database for relevant lecture content based on the analysis results of the NLP model and sentiment data. This search considers not only teaching materials and past answers related to the question, but also answers that are appropriate for the user's emotional state.

[0228] Generating response patterns

[0229] The server generates the optimal response pattern based on the searched lecture content. This response pattern is adjusted according to the user's emotional state and is provided in text, audio, and video formats.

[0230] Providing answers to users

[0231] User terminal operation

[0232] The device analyzes the response data received from the server and sets the display format to an appropriate one. For example, response patterns that include explanations about "how to solve quadratic equations" are displayed in a format that is easy to understand and tailored to the user's level of confusion. Audio and video playback are also provided as needed.

[0233] User feedback

[0234] Users enter their satisfaction level and additional feedback regarding the provided answers. This feedback data includes evaluations of the quality of the answers and responses regarding their feelings.

[0235] Gathering feedback and improving the system

[0236] Server operation

[0237] The device sends user feedback data to the server. The server stores the received feedback data in a database and uses it to improve the accuracy of future responses and enhance the overall system. The feedback data also includes information about emotional states, which allows the system to provide more adaptive learning support.

[0238] Specific example

[0239] For example, suppose a user asks a question about "how to solve a quadratic equation" in mathematics. The user types "how to solve a quadratic equation" into a text box and presses the submit button. The user's terminal sends this question data and emotion data indicating the user's state of confusion to the server. The server uses an NLP model to extract the keywords "quadratic equation" and "how to solve" and searches its database for relevant lecture content. Furthermore, it generates an answer pattern that is tailored to the user's state of confusion, with a detailed and easy-to-understand explanation. For example, it might generate text such as "A quadratic equation is of the form ax^2 + bx + c = 0, and the formula for solving it is...", along with gentle audio and corresponding video data, and send them to the user's terminal. The user's terminal receives this and displays or plays it for the user.

[0240] This system allows users to effectively and quickly resolve their questions and learn while receiving emotionally tailored support.

[0241] The following describes the processing flow.

[0242] Step 1:

[0243] Users install the educational application on their smartphones, tablets, or computers. After installation, users launch the app, enter their email address and password to create an account, and log in.

[0244] Step 2:

[0245] The device sends login information to the server. The server authenticates the received information and returns the authentication result to the device. If the login is successful, the device displays the home screen.

[0246] Step 3:

[0247] Upon initial login, the device requests the latest lecture data from the server. The server retrieves the lecture data from the database and sends it to the device. The device saves the received lecture data to local storage and displays a "synchronization complete" notification to the user.

[0248] Step 4:

[0249] The user operates the app to select a subject or topic they have questions about during their studies. After selecting, they enter their specific question in the text box and press the submit button.

[0250] Step 5:

[0251] The terminal sends the user-entered question data to the server. This question data includes the entered text, sentiment data, and metadata (such as a timestamp and user ID). Sentiment data is generated by an emotion engine that analyzes the user's facial expressions and voice.

[0252] Step 6:

[0253] The server passes the received question data to a natural language processing (NLP) model. The NLP model analyzes the question content and extracts keywords and the intent behind the question.

[0254] Step 7:

[0255] The server receives emotional data analyzed by the emotion engine. This allows it to understand the user's current emotional state (e.g., confused, frustrated, excited).

[0256] Step 8:

[0257] The server searches the database for relevant lecture content based on the analysis results of the NLP model and sentiment data. This search includes teaching materials, whiteboard notes, and past questions and answers related to the question.

[0258] Step 9:

[0259] The server generates the optimal response pattern based on the user's emotional state. For example, for a confused user, it generates text, audio, and video data that includes more detailed and thorough explanations.

[0260] Step 10:

[0261] The server sends the generated response pattern to the terminal. The terminal analyzes the received response data and displays or plays it in the appropriate format. For example, it might display text such as, "A quadratic equation is of the form ax^2 + bx + c = 0, and the formula for solving it is..." and play audio or video as needed.

[0262] Step 11:

[0263] Users rate the provided answers and provide feedback. Ratings are given in the format of "Good" or "Bad."

[0264] Step 12:

[0265] The device sends user feedback data to the server. The server stores the received feedback data in a database and uses it to improve the accuracy of future responses and enhance the overall system. The feedback data also includes information about emotional states, allowing the system to provide more adaptive learning support.

[0266] Specific example:

[0267] 1. User input and emotion recognition

[0268] When a user asks a question about "how to solve a quadratic equation," they type "how to solve a quadratic equation" into the text box and press the submit button. The emotion engine recognizes the user's state of confusion, and the emotion data is sent to the server along with the question data.

[0269] 2. Question analysis and answer generation

[0270] The server uses an NLP model to extract the keywords "quadratic equation" and "solution method" from the question and searches for related lecture content. It generates answer patterns that include detailed and easy-to-understand explanations for confused users and sends them to the terminal.

[0271] 3. Providing responses and feedback to users

[0272] The terminal displays the received response data, showing the text "The quadratic equation is in the form ax^2 + bx + c = 0, and the quadratic formula is..." along with the corresponding audio or video. The user is satisfied with the response and sends a positive rating and feedback such as "very easy to understand." The server saves this feedback data and uses it to improve the response system in the future.

[0273] This system allows users to receive effective learning support tailored to their emotions.

[0274] (Example 2)

[0275] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0276] Traditional educational support systems provided uniform answers without considering the individual emotional states of users, resulting in limited learning effectiveness. Furthermore, they lacked mechanisms to incorporate feedback based on user emotions. This led to problems in quickly and appropriately resolving users' questions and concerns.

[0277] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for transmitting question data and emotion data input from the user terminal to the data server; means for analyzing the question data using a natural language processing model and searching for relevant lecture content; means for analyzing the emotion data and understanding the user's emotional state; means for generating an optimal answer pattern based on the retrieved lecture content and emotional state; means for transmitting the generated answer pattern to the user terminal in the form of text, audio, and video; means for displaying or playing back the received answer pattern on the user terminal; and means for analyzing the user's facial expressions and voice and acquiring emotion data. This makes it possible to provide optimal learning support tailored to the user's individual emotional state.

[0278] An "educational program" is a software application specifically designed for learning support and educational purposes.

[0279] A "user terminal" refers to a digital device, such as a smartphone, tablet, or personal computer, used to install and operate educational programs.

[0280] A "data server" is a server system that stores and manages question data, sentiment data, and lecture content data, and communicates with user terminals.

[0281] "Question data" refers to text data containing questions and doubts that users have while learning.

[0282] "Emotional data" refers to data that indicates the emotional state (e.g., confusion, anxiety, excitement) of a user, analyzed from their facial expressions and voice.

[0283] A "natural language processing model" is a machine learning model used to analyze the meaning of text data and extract keywords and syntax.

[0284] The "answer pattern" refers to the optimal answer format (text, voice, video) for a user's question, which is generated based on question data and sentiment data.

[0285] The "means for analyzing the user's facial expressions and voice" is a combination of software and hardware that uses a camera and a microphone to analyze the user's facial expressions and voice tone to obtain sentiment data.

[0286] "Feedback data" refers to data that includes the satisfaction level, additional opinions, and evaluations input by the user for the provided answer.

[0287] "Lecture content data" refers to digital content such as teaching materials, lecture notes, and video lectures required for learning in an educational program.

[0288] The present invention provides more effective and personalized learning support by combining an emotion engine with a learning support system using an educational program. This system is composed of a user terminal, a data server, and a communication network connecting them.

[0289] First, the user installs an educational program on a terminal such as a smartphone, tablet, or personal computer. The educational program includes various teaching materials and functions related to learning support. After installation, the user starts the program and enters information (name, email address, password, etc.) required to create an account. When account creation and login are completed, the user terminal connects to the data server and synchronizes the latest lecture content data. Specifically, it communicates with the server using technologies such as REST API and WebSocket, and saves new lecture data and update information locally.

[0290] When a user begins learning, the camera and microphone automatically activate to analyze the user's facial expressions and voice tone in real time. OpenCV and TENSORFLOW® are used for facial recognition, and MLKit and speech recognition APIs are used for voice analysis. This allows the user's emotional state (e.g., confusion, frustration, excitement) to be captured. When the user selects a specific subject or topic, enters their questions into a text box, and presses the submit button, the device sends the text data and the real-time emotional data to the data server.

[0291] The data server analyzes the received question data and sentiment data using natural language processing (NLP) models. NLP models such as Transformers and BERT are used to extract the meaning of the questions and important keywords. Furthermore, sentiment data is also analyzed to understand the user's emotional state. Based on the analysis results, the data server searches the database for relevant lecture content. This search includes textbook data, past Q&A, and the user's learning history. Based on the search results and the emotional state, the server generates the optimal response pattern. Responses are provided in text, audio, and video formats.

[0292] The generated answer patterns are sent to the user's device, which receives and displays or plays them. Specifically, HTML, CSS, and JavaScript (registered trademark) are used to provide an appropriate UI / UX and present the answers in a format that is easy for the user to understand. For example, if a user asks about "how to solve a quadratic equation" and is confused, an answer adjusted to be detailed and easy to understand will be provided. For example, an explanation such as "A quadratic equation is in the form ax^2 + bx + c = 0, and the quadratic formula is..." or a corresponding video will be played.

[0293] Users provide satisfaction ratings and additional feedback on the provided answers. This feedback data is then sent back to the data server via the device. The received feedback data is stored in a database and used to improve the accuracy of future answers and enhance the overall system. Because the feedback data also includes information about emotional states, the system can increasingly provide adaptive learning support to users.

[0294] Examples of prompt statements

[0295] For example, the following prompt statements are possible.

[0296] User: I don't know how to solve quadratic equations.

[0297] Emotional state: Confusion

[0298] Based on this prompt, the data server generates an appropriate response pattern and provides optimized support to resolve the user's confusion.

[0299] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0300] Step 1: Initial Setup

[0301] Users install the educational program on their smartphones, tablets, or PCs. After installation, users launch the program and enter the necessary information to create an account (name, email address, password, etc.). An account is generated based on the entered data and saved to the data server. Once logged in, the user's device connects to the data server and synchronizes the latest lecture content data. Specifically, data communication is performed using REST APIs or WebSockets, and the lecture data is saved locally.

[0302] Input: User information (name, email address, password), login information

[0303] Output: Generation of account data, synchronization of lecture content data

[0304] Step 2: Start of facial expression and voice analysis

[0305] When the user starts learning, the camera and microphone of the terminal are automatically activated. The terminal uses these devices to analyze the user's facial expressions and voice tones in real time by an emotion engine. OpenCV or TensorFlow is used for face recognition, and MLKit or voice recognition API is used for voice analysis. At this stage, the user's facial expression and voice tone data are collected and output as emotion data in real time.

[0306] Input: User's facial expression data, voice data

[0307] Output: Emotion data (e.g., confusion, dissatisfaction, excitement)

[0308] Step 3: Input and transmission of questions

[0309] The user selects a specific subject or course and enters the content they are confused about into a text box. When the user presses the send button, the terminal sends the text data and the emotion data obtained in real time to the data server. The transmitted data includes the question content, emotion data, timestamp, and user ID. The transmitted data is received and saved on the server side.

[0310] Input: User's question (text data), emotion data

[0311] Output: Transmission of question data and emotion data to the data server

[0312] Step 4: Analysis of question data

[0313] The data server analyzes the received question data and sentiment data using natural language processing (NLP) models. NLP models such as Transformers and BERT are used to extract the meaning of the question and important keywords. In addition, sentiment data is analyzed to identify the user's emotional state. The analysis results are output as a series of keywords and sentiment state data.

[0314] Input: Question data, sentiment data

[0315] Output: Analysis results (keywords, emotional state)

[0316] Step 5: Search for related lecture content

[0317] The data server searches the database for relevant lecture content based on the analysis results. The database contains textbook data, past Q&A, and user learning history. As a result of the search, the relevant lecture content data is identified and retrieved.

[0318] Input: Analysis results (keywords, emotional state)

[0319] Output: Related lecture content data

[0320] Step 6: Generating response patterns

[0321] The data server generates the optimal response pattern based on the retrieved lecture content data. The response is generated in text, audio, and video formats, depending on the user's emotional state (e.g., detailed and gentle tone if confused). At this stage, the final response pattern is determined and stored on the data server.

[0322] Input: Related lecture content data, emotional state

[0323] Output: Response patterns (text, audio, video)

[0324] Step 7: Provide answer patterns

[0325] The user terminal receives response patterns generated from the data server. Based on the received response patterns, the terminal displays or plays content for the user in an appropriate format. Specifically, it uses HTML, CSS, and JavaScript to provide a user-friendly UI / UX. For example, it plays videos explaining how to solve quadratic equations or providing example problems.

[0326] Input: Response patterns (text, audio, video)

[0327] Output: Learning support for the user (display, playback)

[0328] Step 8: Collecting and submitting feedback

[0329] Users provide satisfaction ratings and additional feedback on the provided answers. This feedback data is sent to a data server via the device. The input feedback data includes evaluations of the quality of the answers and responses to emotions.

[0330] Input: User feedback (satisfaction level, additional comments)

[0331] Output: Sending feedback data to the data server

[0332] Step 9: Feedback Analysis and System Improvement

[0333] The data server stores the received feedback data in a database. The feedback data is analyzed and used to improve future response accuracy and the overall system. Machine learning algorithms are used to analyze the feedback data and improve the system's accuracy and performance.

[0334] Input: Feedback data

[0335] Output: Analysis results, system improvement data

[0336] (Application Example 2)

[0337] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0338] Conventional learning support systems unilaterally provided information without considering the user's emotional state, making it difficult to provide adaptive support tailored to each learner's level of understanding and emotions. Furthermore, in physical stores, customers often felt an emotional distance when asking questions about products, resulting in a lack of interactive customer support.

[0339] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for transmitting question data input from a user terminal to the server, means for analyzing the question data using a natural language processing model and searching for relevant lecture content, means for generating an optimal answer pattern based on the searched lecture content, means for analyzing the user's emotions in real time using an emotion engine with a smart device and adding the analyzed emotion data to the question data, and means for adjusting the generated answer pattern according to the user's emotional state. This makes it possible to provide personalized learning support according to the user's emotional state. Furthermore, even in physical stores, it becomes possible to provide product descriptions and support according to the customer's emotions, thereby improving customer satisfaction.

[0340] An "educational application" is software used by users for learning, providing lecture data and learning materials.

[0341] A "user terminal" refers to a device, such as a smartphone, tablet, or personal computer, on which a user installs and uses educational applications.

[0342] A "server" is a computer system that receives data sent from a user terminal and performs analysis and processing on it.

[0343] "Question data" refers to data that includes the questions that users have and enter during their learning process.

[0344] A "natural language processing model" refers to algorithms and techniques for analyzing natural language and understanding its meaning.

[0345] "Relevant lecture content" refers to appropriate teaching materials and lecture content in response to the question entered by the user.

[0346] An "answer pattern" refers to the format or method used to provide the most appropriate answer to a question.

[0347] An "emotion engine" refers to software and hardware that recognizes and analyzes emotions from a user's facial expressions and voice.

[0348] "Emotional data" refers to data that indicates the user's emotional state, as analyzed by the emotion engine.

[0349] A "smart device" refers to a device that has internet connectivity and is used by a user, such as smart glasses or a smartphone.

[0350] "Feedback data" refers to data that includes evaluations and opinions from users regarding the responses they provide.

[0351] "Synchronization" refers to the process of ensuring that the latest data is matched between the user's terminal and the server.

[0352] "Real-time" means that processing and responses to user actions occur immediately.

[0353] The above are definitions of the important terms included in the scope of the claim.

[0354] This invention applies to a learning support system using educational applications and a customer support system using smart devices in physical stores. User terminals include smartphones, tablets, and personal computers, and the system is built by installing educational applications on these devices. In physical stores, smart devices such as smart glasses are used.

[0355] System Configuration

[0356] This system consists of the following components:

[0357] 1. User terminal: A device equipped with a camera and microphone, and with educational applications installed.

[0358] 2. Server: A computer system that receives data sent from user terminals, performs analysis, and generates responses.

[0359] 3. Emotion Engine: Software that analyzes the user's facial expressions and voice to generate emotion data.

[0360] 4. Natural Language Processing Model (NLP Model): An analytical tool that analyzes user questions and searches for relevant lecture content or product information.

[0361] 5. Database: A storage system that stores lecture content and product information.

[0362] 6. Smart devices: Devices that display information to customers in physical stores, such as smart glasses.

[0363] Flow of operations

[0364] 1. Initial setup of the user terminal:

[0365] After the user installs the application and creates an account upon their first login, they synchronize with the server to retrieve the latest lecture data.

[0366] 2. Emotion recognition and data collection:

[0367] When the application is launched, the camera and microphone are automatically activated, and the emotion engine analyzes the user's facial expressions and voice to generate emotion data. When the user enters a question, the question data and emotion data are sent to the server.

[0368] 3. Question analysis and answer generation:

[0369] The server analyzes the received question data using a natural language processing model and searches the database for relevant lecture content and product information. Based on the search results, it generates the optimal response pattern. This response pattern is adjusted according to the user's emotional state.

[0370] 4. Display the answer:

[0371] The generated response patterns are sent to the user's device in text, audio, or video format. The user's device displays or plays the received responses in the appropriate format.

[0372] For example, if the question is "How do I use this product?" and the user is confused, a detailed and easy-to-understand explanation will be provided.

[0373] 5. Gathering feedback:

[0374] User feedback is sent from the device to the server and stored as feedback data. This data is used to improve the accuracy of the system.

[0375] Specific usage examples

[0376] In the educational application, when a user asks "How to solve a quadratic equation," the emotion engine detects confusion and provides a detailed and easy-to-understand explanation.

[0377] In smart shopping at physical stores, when a customer asks "How do I use this product?" through smart glasses, the store provides appropriate product information and instructions in real time.

[0378] Examples of prompts to input into a generative AI model

[0379] If you ask the "Smart Shopping Assistant" in-store, "How do I use this product?", please generate a detailed explanation of how to use it based on emotion recognition.

[0380] If the customer is confused, please explain gently and in detail.

[0381] If the customer is excited, get straight to the point.

[0382] This system allows users to receive excellent support tailored to their emotions, which is expected to improve learning efficiency and customer satisfaction.

[0383] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0384] Step 1:

[0385] Initial settings:

[0386] Users first install the educational application on their device, such as a smartphone, tablet, or PC. After installation is complete, they launch the application, enter the necessary information to create an account, and then create an account and log in.

[0387] Input: User account information (name, email address, etc.)

[0388] Output: A user account is created on the server and synchronized with the user's terminal.

[0389] Specific operation: When a user enters the required information on the account creation screen and presses the data submission button, that data is sent to the server and registered as a new account.

[0390] Step 2:

[0391] Data synchronization from the server:

[0392] Upon initial login, the user's device connects to the server, and the latest lecture and course material data is synchronized from the server.

[0393] Input: Latest lecture content data stored on the server

[0394] Output: The latest lecture content data is downloaded to the user's terminal.

[0395] Specific operation: After the user presses the login button, a request is sent from the terminal to the server, the server sends the latest lecture data to the terminal, and the data is updated on the terminal.

[0396] Step 3:

[0397] Recognition of emotions:

[0398] When a user begins learning, the camera and microphone on the user's device are automatically activated, and the emotion engine analyzes the user's facial expressions and voice to generate emotion data.

[0399] Input: User's facial expressions and voice data

[0400] Output: Analyzed emotion data (e.g., confusion, excitement, etc.)

[0401] Specific operation: The camera and microphone capture the user's face, and that data is analyzed in real time by an emotion engine to identify the user's emotions.

[0402] Step 4:

[0403] Entering question data:

[0404] When a user has a question while studying, they select a specific subject or topic, enter their question into a text box, and press the submit button.

[0405] Input: Question entered by the user

[0406] Output: Questionnaire data (text format) and corresponding sentiment data

[0407] Specific operation: When a user enters a question in a text box and presses the submit button, the question content and sentiment data are sent to the server.

[0408] Step 5:

[0409] Question analysis on the server:

[0410] The server analyzes the received question data using a natural language processing model and searches the database for relevant lecture content.

[0411] Input: Questionnaire data and sentiment data

[0412] Output: Related lecture content data

[0413] Specific operation: The server analyzes the question data, searches the database for relevant textbooks and lecture content, and retrieves the appropriate information.

[0414] Step 6:

[0415] Generating response patterns:

[0416] The server generates the optimal response pattern based on the search results and adjusts the format of the response according to the user's emotional state.

[0417] Input: Relevant lecture content data, user sentiment data

[0418] Output: Adjusted response patterns (text, audio, video, etc.)

[0419] Specific operation: On the server side, the program decides whether to make the response detailed or concise based on the user's emotions, and then outputs it as text or audio data.

[0420] Step 7:

[0421] Providing responses to user terminals:

[0422] The generated response patterns are sent to the user's device in text, audio, or video format and displayed or played back in the appropriate format.

[0423] Input: Adjusted response patterns (text, audio, video, etc.)

[0424] Output: The answer that the user sees or hears on their device.

[0425] Specific operation: The user terminal analyzes the received response data, displays it in a user-friendly format, and also plays the audio.

[0426] Step 8:

[0427] Gathering feedback:

[0428] Users enter their satisfaction level with the provided answers and any additional feedback, then submit it to the server.

[0429] Input: User feedback data

[0430] Output: Feedback data stored on the server

[0431] Specific operation: When a user enters their opinions and ratings on the feedback screen and presses the submit button, that data is saved to the server.

[0432] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0433] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0434] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0435] [Second Embodiment]

[0436] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0437] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0438] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0439] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0440] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0441] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0442] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0443] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0444] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0445] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0446] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0447] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0448] The system of this invention aims to provide effective learning support using educational applications. This system consists of user terminals, a server, and a communication network connecting them.

[0449] Initial setup

[0450] Settings on the user terminal

[0451] First, users install the educational application on their smartphone, tablet, or computer. After installation, users launch the application and enter the necessary information to create an account. Once account creation and login are complete, the user's device connects to the server and synchronizes with the latest lecture data from the server.

[0452] Enter and submit your question

[0453] If a user has a question while learning, they can select a specific subject or topic through the educational application and enter their question in the text box. Once they have finished entering their question, they press the submit button.

[0454] User terminal operation

[0455] The user terminal sends the entered question data to the server. This question data includes the text entered by the user and other metadata (such as a timestamp and user ID).

[0456] Question analysis and answer generation

[0457] The server receives question data sent from the user's terminal. First, it analyzes the question using a natural language processing (NLP) model. This analysis extracts keywords and intent from the question. For example, in the question "How to solve a quadratic equation," "quadratic equation" and "how to solve" are extracted as important keywords.

[0458] Server operation

[0459] The server searches the database for relevant lecture content based on the analysis results. This search yields potential candidates such as relevant teaching materials, whiteboard notes, and past questions and answers. It then generates the optimal response pattern (text, audio, video) and sends it to the user's terminal.

[0460] Providing answers to users

[0461] The user terminal analyzes the response data received from the server and displays it in an appropriate format. For example, an explanation of how to solve a quadratic equation might be displayed as text, and corresponding audio or video can be played as needed. The user can then review the provided answers and continue their learning.

[0462] Gathering feedback and improving the system

[0463] Users can provide satisfaction ratings and additional feedback on the answers. This feedback data is sent from the user's device to the server, which stores it in a database. This feedback data is used to improve the accuracy of future automated responses and to improve the overall system.

[0464] Specific example

[0465] For example, suppose a user asks a question about quadratic equations in mathematics. The user enters "how to solve a quadratic equation" into a text box and presses the submit button. The user's terminal sends this question data to the server. The server uses an NLP model to extract the important keywords "quadratic equation" and "how to solve," and searches the database for relevant lecture content. From the search results, it generates the optimal answer pattern, for example, explanatory text such as "A quadratic equation is of the form ax^2 + bx + c = 0, and the quadratic formula is...", along with corresponding audio and video data, and sends it to the user's terminal. The user's terminal receives this and displays or plays it for the user.

[0466] This system allows users to efficiently resolve questions and advance their learning anytime, anywhere.

[0467] The following describes the processing flow.

[0468] Step 1:

[0469] Users install the educational application on their smartphone, tablet, or PC and launch the app. They then create an account by entering their email address and password and log in.

[0470] Step 2:

[0471] The device sends login information to the server. The server authenticates the received information and returns the authentication result to the device. If the login is successful, the device displays the home screen.

[0472] Step 3:

[0473] Upon initial login, the device requests the latest lecture data from the server. The server retrieves the lecture data from the database and sends it to the device. The device saves the received lecture data to local storage and displays a "synchronization complete" notification to the user.

[0474] Step 4:

[0475] The user operates the app to select the subject or topic for which they have a question during their studies. After selecting, they enter the specific question in the text box and press the submit button.

[0476] Step 5:

[0477] The terminal sends the user-entered question data to the server. This question data includes the entered text and metadata (e.g., timestamp, user ID).

[0478] Step 6:

[0479] The server passes the received question data to a natural language processing (NLP) model. The NLP model analyzes the question content and extracts important keywords and the intent of the question.

[0480] Step 7:

[0481] Based on the analysis results, the server searches the database for relevant lecture content. The database includes lecture content, whiteboard notes, past questions and answers, etc.

[0482] Step 8:

[0483] The server generates the optimal response pattern based on the searched lecture content. This response pattern is generated in text, audio, and video formats.

[0484] Step 9:

[0485] The server sends the generated response pattern to the terminal. The terminal analyzes the received response data and configures the display settings in the appropriate format.

[0486] Step 10:

[0487] The device displays the answer in text format on the user's screen. For example, it might display, "The quadratic equation is of the form ax^2 + bx + c = 0, and the quadratic formula is..." If necessary, it displays audio or video playback buttons, and playback begins when the user presses them.

[0488] Step 11:

[0489] Users rate the provided answers and provide feedback. Ratings are given in the format of "Good" or "Bad."

[0490] Step 12:

[0491] The device sends user feedback data to the server. The server stores the received feedback data in a database and uses it to improve the accuracy of future responses.

[0492] (Example 1)

[0493] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0494] In today's educational environment, students often lack the means to immediately resolve their questions. Furthermore, the limited availability of teaching materials and lecture content from educational institutions makes it difficult to provide support tailored to the individual needs of learners. Moreover, there are insufficient means of collecting feedback to continuously improve the quality and accuracy of the answers provided. This invention aims to solve these problems and provide a system that enables users to learn effectively.

[0495] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0496] In this invention, the server includes means for transmitting inquiry data entered from a terminal to an information processing device, means for analyzing the inquiry data using a generation AI model and searching for relevant educational content, and means for generating an optimal answer format based on the searched educational content. This makes it possible for users to efficiently resolve their questions anytime, anywhere, improving their satisfaction and learning effectiveness.

[0497] An "educational program" is software intended for use by users to support educational activities.

[0498] A "device" refers to a hardware device used by a user to install and utilize educational programs, such as a smartphone, tablet, or personal computer.

[0499] An "information processing device" is a computer system that analyzes and processes data transmitted from a terminal and provides the necessary data.

[0500] "Inquiry data" refers to the content of questions entered and submitted by users through educational programs, along with related data (such as timestamps and user IDs).

[0501] A "generative AI model" is an artificial intelligence model that uses natural language processing and other machine learning techniques to analyze input text data and extract intent and keywords.

[0502] "Educational content" includes teaching materials, lecture notes, and past questions and answers related to specific subjects or topics.

[0503] "Answer format" refers to the method of explanation provided to the user, and includes formats such as text, audio, and video.

[0504] The system of this invention is intended to support learning using educational programs and consists of a terminal, an information processing device, and a communication network connecting them. The detailed configuration and processing flow of the system are described below.

[0505] Initial setup

[0506] Settings on the user terminal

[0507] Users install the educational program on their smartphones, tablets, PCs, or other devices. After installation, users launch the application and enter the necessary information to create an account. Once account creation and login are complete, the device connects to the information processing unit and synchronizes the latest lecture data from the server. This allows users to access the latest educational content at any time.

[0508] Enter and submit your question

[0509] If a user has a question while learning, they select a specific subject or topic through the educational program and enter their question into the text box. After completing the input, they press the submit button. The terminal then sends the entered inquiry data (including the question text, timestamp, and user ID) to the information processing device.

[0510] Question analysis and answer generation

[0511] The information processing device receives inquiry data sent from the terminal. First, it analyzes the inquiry data using a generative AI model to extract keywords and intent of the question. For example, in the case of the question "How to solve a quadratic equation," "quadratic equation" and "how to solve" are analyzed as important keywords. Next, the information processing device searches a database based on the extracted keywords to find relevant educational content. This database includes teaching materials, lecture notes, and past questions and answers. It then generates the most suitable answer format (text, audio, video) and sends it to the terminal.

[0512] Providing answers to users

[0513] The terminal analyzes the response data received from the information processing device and displays it in an appropriate format. For example, an explanation of how to solve a quadratic equation might be displayed as text, and corresponding audio or video can be played as needed. This allows the user to review the provided answers and continue their learning.

[0514] Gathering feedback and improving the system

[0515] Users can input their satisfaction level and provide additional feedback on the provided answers. The terminal sends this feedback data to the information processing device. The information processing device stores the feedback data in a database and uses it to improve the accuracy of future answers and the overall system. This feedback is crucial for the continuous improvement of the system and the enhancement of the user experience.

[0516] Specific example

[0517] The following are specific examples of how the system can be used.

[0518] For example, if a user asks a question about quadratic equations in mathematics, the user enters "how to solve a quadratic equation" into the text box and presses the submit button. The terminal sends this inquiry data to the information processing device. The information processing device uses a generative AI model to extract the important keywords "quadratic equation" and "how to solve," and searches its database for relevant educational content. From the search results, it generates the optimal answer format, such as "A quadratic equation is in the form ax^2 + bx + c = 0, and the quadratic formula is...", along with corresponding audio and video data, and sends it to the terminal. The terminal receives this and displays or plays it for the user.

[0519] This system allows users to efficiently resolve questions and advance their learning anytime, anywhere.

[0520] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0521] Step 1:

[0522] Initial setup on the user terminal

[0523] Users install the educational program on their smartphones, tablets, or computers. After installation, users launch the application and enter necessary information such as their name, email address, and password to create an account. The entered information is sent from the device to the server, which stores this information in a database. Once authentication is complete, the device synchronizes the latest lecture data from the server and displays it on the device.

[0524] Input: User's personal information (name, email address, password)

[0525] Output: Account creation confirmation and synchronization of the latest lecture data

[0526] Step 2:

[0527] Enter and submit your question

[0528] If a user has a question while learning, they select a specific subject or topic through the educational program and enter their question in a text box. Once this is complete, the user presses the submit button. The device then sends this inquiry data (including the question text, timestamp, and user ID) to the server.

[0529] Input: User's question, timestamp, user ID

[0530] Output: Confirmation of sending query data to the server

[0531] Step 3:

[0532] Question analysis

[0533] The server receives query data sent from the terminal. The server uses a generative AI model to analyze the query data and extract keywords and intent from the question. For example, in the question "How to solve a quadratic equation," "quadratic equation" and "how to solve" are extracted as important keywords.

[0534] Input: Inquiry data (question content, timestamp, user ID)

[0535] Output: Analysis results (keywords and intent)

[0536] Step 4:

[0537] Search for related educational content

[0538] Based on the analysis results, the server searches the database for relevant educational content. This database includes teaching materials, lecture notes, and past questions and answers. The server finds the most suitable educational content that matches the search criteria.

[0539] Input: Analysis results (keywords and intent)

[0540] Output: Related educational content (teaching materials, lecture notes, past questions and answers)

[0541] Step 5:

[0542] Generating an answer format

[0543] The server generates the most suitable response format based on the searched educational content. The response format includes text, audio, and video, and is provided in the format that is easiest for the user to understand.

[0544] Input: Related educational content (textbooks, lecture notes, past questions and answers)

[0545] Output: Optimal response format (text, audio, video)

[0546] Step 6:

[0547] Submit your response

[0548] The server sends the generated response format to the terminal. The transmitted data includes the response text and corresponding audio or video data.

[0549] Input: Optimal response format (text, audio, video)

[0550] Output: Confirmation of sending response data to the terminal

[0551] Step 7:

[0552] Providing answers to users

[0553] The device analyzes the response data received from the server and displays or plays it back to the user in an appropriate format. For example, text may be displayed on the screen, and audio or video may be played. The user can then proceed with their learning based on this information.

[0554] Input: Response data (text, audio, video)

[0555] Output: Display and play back the response to the user.

[0556] Step 8:

[0557] Gathering feedback

[0558] Users can input their satisfaction level and provide additional feedback on the provided answers. The device sends this feedback to the server, which stores it in a database. This helps improve the accuracy of future answers.

[0559] Input: User feedback (satisfaction level, additional comments)

[0560] Output: Confirmation and storage of feedback data sent to the server.

[0561] (Application Example 1)

[0562] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0563] With the increasing prevalence of autonomous vehicles, there is a growing need for ways to ensure passengers have a meaningful experience during long journeys. However, conventional in-car entertainment systems are primarily limited to entertainment and information provision, lacking effective learning tools for education. Therefore, providing interactive educational support systems that passengers can use within autonomous vehicles is a key challenge.

[0564] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0565] In this invention, the server includes means for transmitting question data entered from a user terminal to the server, means for analyzing the question data using a natural language processing model and searching for relevant lecture content, and means for generating an optimal answer pattern based on the searched lecture content. This enables passengers to learn effectively in an autonomous vehicle and spend long journeys meaningfully.

[0566] An "educational application" is software designed for users to learn specific knowledge or skills.

[0567] A "user terminal" is a computer device used by a user to run and operate educational applications.

[0568] A "server" is a central computer system responsible for data processing and storage for educational applications, and for communicating with user terminals.

[0569] A "natural language processing model" is an artificial intelligence technology used to analyze question data and text data entered by users.

[0570] An "infotainment system" is an in-car system that provides information and entertainment to passengers in autonomous vehicles.

[0571] "Question data" refers to information about questions and problems that users have entered into educational applications.

[0572] An "answer pattern" is the format of the explanations and descriptions that an educational application provides in response to a user's question.

[0573] "Feedback data" refers to information in which users have entered evaluations and opinions regarding the provided answers.

[0574] "Synchronization" is the process performed to make data consistent between the user's terminal and the server.

[0575] System Overview

[0576] The system of the present invention is for the effective use of educational applications in autonomous vehicles. It consists of a user terminal, a server, a communication network, and an infotainment system within the autonomous vehicle.

[0577] Program Operation Overview

[0578] The user installs the educational application on the in-car infotainment system and begins the learning process. The system operates in the following steps:

[0579] 1. The user operates the infotainment system and launches an educational application.

[0580] 2. Enter the information needed to create a user account on the infotainment system, create the account, and log in. This will start synchronization with the server.

[0581] 3. If a question arises during learning, the user selects a specific subject or topic within the application and enters the question in the text box.

[0582] 4. The infotainment system sends the question data to the server. This question data includes metadata such as text, timestamp, and user ID.

[0583] 5. The server receives the question data and performs analysis using a natural language processing (NLP) model. This analysis extracts important keywords and intents.

[0584] 6. The server searches the database for relevant lecture content and generates the optimal response pattern (text, audio, video).

[0585] 7. The generated response patterns are sent back to the infotainment system and presented to the user.

[0586] 8. Users review the provided answers and continue learning. Simultaneously, they can provide feedback on the answers. This feedback data is sent to the server and used to improve the accuracy of future answers.

[0587] Hardware and software usage

[0588] Hardware: Infotainment systems for autonomous vehicles, smartphones, tablets

[0589] software:

[0590] Natural Language Processing (NLP) models (e.g., BERT model)

[0591] Communication protocol: Internet Protocol (HTTP, HTTPS)

[0592] Specific example

[0593] For example, if a passenger enters a request to learn "the basics of quantum mechanics" while in the car, the in-car infotainment system automatically sends the question to a server. The server uses an NLP model to extract the keywords "quantum mechanics" and "basics." Based on the keywords, the server searches for the most suitable learning materials (such as introductory videos and text explanations) and sends them to the infotainment system. The infotainment system then displays or plays the received materials for the passenger.

[0594] Examples of prompts to input into a generative AI model

[0595] "The user typed 'Teach me the basics of quantum mechanics.' Please search for and provide appropriate teaching materials."

[0596] This system allows passengers to spend their time productively while efficiently pursuing their studies.

[0597] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0598] Step 1:

[0599] The user launches the educational application through the infotainment system in the autonomous vehicle and enters information to create an account. This information includes username, email address, and password. Once the user enters the information and presses the submit button, the infotainment system sends it to the server. The server receives the input data and creates the user account. This initiates communication between the server and the infotainment system, and the latest lecture content data is synchronized upon the first login.

[0600] Input: Username, email address, password, etc.

[0601] Output: User account created, synchronization with server started

[0602] Step 2:

[0603] When a user has a question, they operate an educational application on the infotainment system, select a specific subject or topic, and enter their question in a text box. For example, they might enter, "Please explain the basics of quantum mechanics." This question data, along with a timestamp and user ID, is sent to the server.

[0604] Input: Question (e.g., "Please explain the basics of quantum mechanics"), timestamp, user ID

[0605] Output: Sending query data to the server

[0606] Step 3:

[0607] The server receives the submitted question data and analyzes the question content using a natural language processing (NLP) model. During the analysis, important keywords and the user's intent are extracted. For example, keywords such as "quantum mechanics" and "fundamentals" may be extracted. Based on these analysis results, the server searches the database for relevant lecture content.

[0608] Input: Question data (text, timestamp, user ID)

[0609] Output: Keyword extraction, search for related lecture content

[0610] Step 4:

[0611] The server generates the most suitable response pattern from the search results. For example, it might select a text explanation or video on "Fundamentals of Quantum Mechanics" for beginners. These responses are generated in various formats such as text, audio, and video, and sent to the infotainment system.

[0612] Input: Analysis results, related lecture content

[0613] Output: Generation of optimal response patterns (text, audio, video)

[0614] Step 5:

[0615] The infotainment system receives the response patterns and presents them to the user. For example, a text explanation may be displayed on the screen, and videos or audio may be played as needed. The user can review the provided answers and continue learning.

[0616] Input: Response patterns (text, audio, video)

[0617] Output: Providing answers to the user (display, play)

[0618] Step 6:

[0619] Users can provide feedback on the provided answers. This feedback data is then sent back to the server via the infotainment system. The server stores this feedback data and uses it to improve the accuracy of future answers.

[0620] Input: User Feedback

[0621] Output: Feedback data is stored on the server and used for future improvements.

[0622] This will enable a system that allows passengers to efficiently learn while inside an autonomous vehicle.

[0623] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0624] This invention provides more effective and personalized learning support by combining an emotion engine with a learning support system that uses educational applications. The system consists of user terminals, a server, and a communication network connecting them.

[0625] Initial setup

[0626] Settings on the user terminal

[0627] First, users install the educational application on their smartphone, tablet, or PC. After installation, users launch the application and enter the necessary information to create an account. Once account creation and login are complete, the user's device connects to the server and synchronizes with the latest lecture data from the server.

[0628] Recognition of emotions and input of questions

[0629] Facial expression and voice analysis

[0630] The application is equipped with a camera and microphone, and includes an emotion engine that uses these to recognize emotions from the user's facial expressions and voice. When the user interacts with the application, the camera and microphone automatically activate and analyze the user's facial expressions and voice tone.

[0631] User question input

[0632] When a user has a question during their studies, they select a specific subject or topic, enter the question into a text box, and press the submit button. At this point, the emotion engine analyzes the user's emotions in real time and adds that data to the question data.

[0633] User terminal operation

[0634] The terminal sends the entered question data and sentiment data to the server. This question data includes text, sentiment data, and metadata (such as a timestamp and user ID).

[0635] Question analysis and answer generation

[0636] Natural language processing and sentiment analysis

[0637] The server receives question data sent from the user's terminal and analyzes the question content using a natural language processing (NLP) model. Furthermore, it also analyzes sentiment data to understand the user's emotional state (e.g., confusion, dissatisfaction, excitement).

[0638] Search related lecture content

[0639] The server searches the database for relevant lecture content based on the analysis results of the NLP model and sentiment data. This search considers not only teaching materials and past answers related to the question, but also answers that are appropriate for the user's emotional state.

[0640] Generating response patterns

[0641] The server generates the optimal response pattern based on the searched lecture content. This response pattern is adjusted according to the user's emotional state and is provided in text, audio, and video formats.

[0642] Providing answers to users

[0643] User terminal operation

[0644] The device analyzes the response data received from the server and sets the display format to an appropriate one. For example, response patterns that include explanations about "how to solve quadratic equations" are displayed in a format that is easy to understand and tailored to the user's level of confusion. Audio and video playback are also provided as needed.

[0645] User feedback

[0646] Users enter their satisfaction level and additional feedback regarding the provided answers. This feedback data includes evaluations of the quality of the answers and responses regarding their feelings.

[0647] Gathering feedback and improving the system

[0648] Server operation

[0649] The device sends user feedback data to the server. The server stores the received feedback data in a database and uses it to improve the accuracy of future responses and enhance the overall system. The feedback data also includes information about emotional states, which allows the system to provide more adaptive learning support.

[0650] Specific example

[0651] For example, suppose a user asks a question about "how to solve a quadratic equation" in mathematics. The user types "how to solve a quadratic equation" into a text box and presses the submit button. The user's terminal sends this question data and emotion data indicating the user's state of confusion to the server. The server uses an NLP model to extract the keywords "quadratic equation" and "how to solve" and searches its database for relevant lecture content. Furthermore, it generates an answer pattern that is tailored to the user's state of confusion, with a detailed and easy-to-understand explanation. For example, it might generate text such as "A quadratic equation is of the form ax^2 + bx + c = 0, and the formula for solving it is...", along with gentle audio and corresponding video data, and send them to the user's terminal. The user's terminal receives this and displays or plays it for the user.

[0652] This system allows users to effectively and quickly resolve their questions and learn while receiving emotionally tailored support.

[0653] The following describes the processing flow.

[0654] Step 1:

[0655] Users install the educational application on their smartphones, tablets, or computers. After installation, users launch the app, enter their email address and password to create an account, and log in.

[0656] Step 2:

[0657] The device sends login information to the server. The server authenticates the received information and returns the authentication result to the device. If the login is successful, the device displays the home screen.

[0658] Step 3:

[0659] Upon initial login, the device requests the latest lecture data from the server. The server retrieves the lecture data from the database and sends it to the device. The device saves the received lecture data to local storage and displays a "synchronization complete" notification to the user.

[0660] Step 4:

[0661] The user operates the app to select a subject or topic they have questions about during their studies. After selecting, they enter their specific question in the text box and press the submit button.

[0662] Step 5:

[0663] The terminal sends the user-entered question data to the server. This question data includes the entered text, sentiment data, and metadata (such as a timestamp and user ID). Sentiment data is generated by an emotion engine that analyzes the user's facial expressions and voice.

[0664] Step 6:

[0665] The server passes the received question data to a natural language processing (NLP) model. The NLP model analyzes the question content and extracts keywords and the intent behind the question.

[0666] Step 7:

[0667] The server receives emotional data analyzed by the emotion engine. This allows it to understand the user's current emotional state (e.g., confused, frustrated, excited).

[0668] Step 8:

[0669] The server searches the database for relevant lecture content based on the analysis results of the NLP model and sentiment data. This search includes teaching materials, whiteboard notes, and past questions and answers related to the question.

[0670] Step 9:

[0671] The server generates the optimal response pattern based on the user's emotional state. For example, for a confused user, it generates text, audio, and video data that includes more detailed and thorough explanations.

[0672] Step 10:

[0673] The server sends the generated response pattern to the terminal. The terminal analyzes the received response data and displays or plays it in the appropriate format. For example, it might display text such as, "A quadratic equation is of the form ax^2 + bx + c = 0, and the formula for solving it is..." and play audio or video as needed.

[0674] Step 11:

[0675] Users rate the provided answers and provide feedback. Ratings are given in the format of "Good" or "Bad."

[0676] Step 12:

[0677] The device sends user feedback data to the server. The server stores the received feedback data in a database and uses it to improve the accuracy of future responses and enhance the overall system. The feedback data also includes information about emotional states, allowing the system to provide more adaptive learning support.

[0678] Specific example:

[0679] 1. User input and emotion recognition

[0680] When a user asks a question about "how to solve a quadratic equation," they type "how to solve a quadratic equation" into the text box and press the submit button. The emotion engine recognizes the user's state of confusion, and the emotion data is sent to the server along with the question data.

[0681] 2. Question analysis and answer generation

[0682] The server uses an NLP model to extract the keywords "quadratic equation" and "solution method" from the question and searches for related lecture content. It generates answer patterns that include detailed and easy-to-understand explanations for confused users and sends them to the terminal.

[0683] 3. Providing responses and feedback to users

[0684] The terminal displays the received response data, showing the text "The quadratic equation is in the form ax^2 + bx + c = 0, and the quadratic formula is..." along with the corresponding audio or video. The user is satisfied with the response and sends a positive rating and feedback such as "very easy to understand." The server saves this feedback data and uses it to improve the response system in the future.

[0685] This system allows users to receive effective learning support tailored to their emotions.

[0686] (Example 2)

[0687] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0688] Traditional educational support systems provided uniform answers without considering the individual emotional states of users, resulting in limited learning effectiveness. Furthermore, they lacked mechanisms to incorporate feedback based on user emotions. This led to problems in quickly and appropriately resolving users' questions and concerns.

[0689] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for transmitting question data and emotion data input from the user terminal to the data server; means for analyzing the question data using a natural language processing model and searching for relevant lecture content; means for analyzing the emotion data and understanding the user's emotional state; means for generating an optimal answer pattern based on the retrieved lecture content and emotional state; means for transmitting the generated answer pattern to the user terminal in the form of text, audio, and video; means for displaying or playing back the received answer pattern on the user terminal; and means for analyzing the user's facial expressions and voice and acquiring emotion data. This makes it possible to provide optimal learning support tailored to the user's individual emotional state.

[0690] An "educational program" is a software application specifically designed for learning support and educational purposes.

[0691] A "user terminal" refers to a digital device, such as a smartphone, tablet, or personal computer, used to install and operate educational programs.

[0692] A "data server" is a server system that stores and manages question data, sentiment data, and lecture content data, and communicates with user terminals.

[0693] "Question data" refers to text data containing questions and doubts that users have while learning.

[0694] "Emotional data" refers to data that indicates the emotional state (e.g., confusion, anxiety, excitement) of a user, analyzed from their facial expressions and voice.

[0695] A "natural language processing model" is a machine learning model used to analyze the meaning of text data and extract keywords and syntax.

[0696] An "answer pattern" is the optimal answer format (text, audio, or video) for a user's question, generated based on question data and sentiment data.

[0697] "Means for analyzing user facial expressions and voice" refers to a combination of software and hardware that uses cameras and microphones to analyze the user's facial expressions and voice tone, and to acquire emotional data.

[0698] "Feedback data" refers to data that includes satisfaction levels, additional opinions, and evaluations entered by users regarding the provided responses.

[0699] "Lecture content data" refers to digital content such as teaching materials, lecture notes, and video lectures necessary for learning within an educational program.

[0700] This invention provides more effective and personalized learning support by combining an emotion engine with a learning support system that uses educational programs. This system consists of user terminals, a data server, and a communication network connecting them.

[0701] First, users install the educational program on their smartphones, tablets, or PCs. The program includes various learning materials and functions to support their studies. After installation, users launch the program and enter the necessary information to create an account (name, email address, password, etc.). Once account creation and login are complete, the user's device connects to the data server and synchronizes with the latest lecture content data. Specifically, it communicates with the server using REST API and WebSocket technologies to save new lecture data and updates locally.

[0702] When a user begins learning, the camera and microphone automatically activate to analyze the user's facial expressions and voice tone in real time. OpenCV and TensorFlow are used for facial recognition, and MLKit and speech recognition APIs are used for voice analysis. This allows the user's emotional state (e.g., confusion, frustration, excitement) to be captured. When the user selects a specific subject or topic, enters their questions into a text box, and presses the submit button, the device sends the text data and the real-time emotional data to the data server.

[0703] The data server analyzes the received question data and sentiment data using natural language processing (NLP) models. NLP models such as Transformers and BERT are used to extract the meaning of the questions and important keywords. Furthermore, sentiment data is also analyzed to understand the user's emotional state. Based on the analysis results, the data server searches the database for relevant lecture content. This search includes textbook data, past Q&A, and the user's learning history. Based on the search results and the emotional state, the server generates the optimal response pattern. Responses are provided in text, audio, and video formats.

[0704] The generated answer patterns are sent to the user's device, which receives and displays or plays them. Specifically, HTML, CSS, and JavaScript are used to provide an appropriate UI / UX, presenting the answers in a user-friendly format. For example, if a user asks about "how to solve a quadratic equation" and is confused, an answer adjusted to be detailed and easy to understand will be provided. For instance, an explanation such as "A quadratic equation is in the form ax^2 + bx + c = 0, and the quadratic formula is..." or a corresponding video will be played.

[0705] Users provide satisfaction ratings and additional feedback on the provided answers. This feedback data is then sent back to the data server via the device. The received feedback data is stored in a database and used to improve the accuracy of future answers and enhance the overall system. Because the feedback data also includes information about emotional states, the system can increasingly provide adaptive learning support to users.

[0706] Examples of prompt statements

[0707] For example, the following prompt statements are possible.

[0708] User: I don't know how to solve quadratic equations.

[0709] Emotional state: Confusion

[0710] Based on this prompt, the data server generates an appropriate response pattern and provides optimized support to resolve the user's confusion.

[0711] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0712] Step 1: Initial Setup

[0713] Users install the educational program on their smartphones, tablets, or PCs. After installation, users launch the program and enter the necessary information to create an account (name, email address, password, etc.). An account is generated based on the entered data and saved to the data server. Once logged in, the user's device connects to the data server and synchronizes the latest lecture content data. Specifically, data communication is performed using REST APIs or WebSockets, and the lecture data is saved locally.

[0714] Input: User information (name, email address, password), login information

[0715] Output: Account data generation, lecture content data synchronization

[0716] Step 2: Start facial expression and voice analysis

[0717] When a user begins learning, the device's camera and microphone automatically activate. The device uses these devices to analyze the user's facial expressions and voice tone in real time using an emotion engine. OpenCV and TensorFlow are used for facial recognition, and MLKit and speech recognition APIs are used for voice analysis. At this stage, the user's facial expression and voice tone data are collected and output as emotion data in real time.

[0718] Input: User's facial expression data, voice data

[0719] Output: Emotional data (e.g., confused, dissatisfied, excited)

[0720] Step 3: Enter and submit your question.

[0721] The user selects a specific subject or topic and enters their question into a text box. When the user presses the submit button, the device sends the text data and real-time sentiment data to a data server. The submitted data includes the question, sentiment data, a timestamp, and the user ID. The submitted data is received and stored on the server.

[0722] Input: User's question (text data), sentiment data

[0723] Output: Sending question data and sentiment data to the data server.

[0724] Step 4: Analyzing Question Data

[0725] The data server analyzes the received question data and sentiment data using natural language processing (NLP) models. NLP models such as Transformers and BERT are used to extract the meaning of the question and important keywords. In addition, sentiment data is analyzed to identify the user's emotional state. The analysis results are output as a series of keywords and sentiment state data.

[0726] Input: Question data, sentiment data

[0727] Output: Analysis results (keywords, emotional state)

[0728] Step 5: Search for related lecture content

[0729] The data server searches the database for relevant lecture content based on the analysis results. The database contains textbook data, past Q&A, and user learning history. As a result of the search, the relevant lecture content data is identified and retrieved.

[0730] Input: Analysis results (keywords, emotional state)

[0731] Output: Related lecture content data

[0732] Step 6: Generating response patterns

[0733] The data server generates the optimal response pattern based on the retrieved lecture content data. The response is generated in text, audio, and video formats, depending on the user's emotional state (e.g., detailed and gentle tone if confused). At this stage, the final response pattern is determined and stored on the data server.

[0734] Input: Related lecture content data, emotional state

[0735] Output: Response patterns (text, audio, video)

[0736] Step 7: Provide answer patterns

[0737] The user terminal receives response patterns generated from the data server. Based on the received response patterns, the terminal displays or plays content for the user in an appropriate format. Specifically, it uses HTML, CSS, and JavaScript to provide a user-friendly UI / UX. For example, it plays videos explaining how to solve quadratic equations or providing example problems.

[0738] Input: Response patterns (text, audio, video)

[0739] Output: Learning support for the user (display, playback)

[0740] Step 8: Collecting and submitting feedback

[0741] Users provide satisfaction ratings and additional feedback on the provided answers. This feedback data is sent to a data server via the device. The input feedback data includes evaluations of the quality of the answers and responses to emotions.

[0742] Input: User feedback (satisfaction level, additional comments)

[0743] Output: Sending feedback data to the data server

[0744] Step 9: Feedback Analysis and System Improvement

[0745] The data server stores the received feedback data in a database. The feedback data is analyzed and used to improve future response accuracy and the overall system. Machine learning algorithms are used to analyze the feedback data and improve the system's accuracy and performance.

[0746] Input: Feedback data

[0747] Output: Analysis results, system improvement data

[0748] (Application Example 2)

[0749] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0750] Conventional learning support systems unilaterally provided information without considering the user's emotional state, making it difficult to provide adaptive support tailored to each learner's level of understanding and emotions. Furthermore, in physical stores, customers often felt an emotional distance when asking questions about products, resulting in a lack of interactive customer support.

[0751] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for transmitting question data input from a user terminal to the server, means for analyzing the question data using a natural language processing model and searching for relevant lecture content, means for generating an optimal answer pattern based on the searched lecture content, means for analyzing the user's emotions in real time using an emotion engine with a smart device and adding the analyzed emotion data to the question data, and means for adjusting the generated answer pattern according to the user's emotional state. This makes it possible to provide personalized learning support according to the user's emotional state. Furthermore, even in physical stores, it becomes possible to provide product descriptions and support according to the customer's emotions, thereby improving customer satisfaction.

[0752] An "educational application" is software used by users for learning, providing lecture data and learning materials.

[0753] A "user terminal" refers to a device, such as a smartphone, tablet, or personal computer, on which a user installs and uses educational applications.

[0754] A "server" is a computer system that receives data sent from a user terminal and performs analysis and processing on it.

[0755] "Question data" refers to data that includes the questions that users have and enter during their learning process.

[0756] A "natural language processing model" refers to algorithms and techniques for analyzing natural language and understanding its meaning.

[0757] "Relevant lecture content" refers to appropriate teaching materials and lecture content in response to the question entered by the user.

[0758] An "answer pattern" refers to the format or method used to provide the most appropriate answer to a question.

[0759] An "emotion engine" refers to software and hardware that recognizes and analyzes emotions from a user's facial expressions and voice.

[0760] "Emotional data" refers to data that indicates the user's emotional state, as analyzed by the emotion engine.

[0761] A "smart device" refers to a device that has internet connectivity and is used by a user, such as smart glasses or a smartphone.

[0762] "Feedback data" refers to data that includes evaluations and opinions from users regarding the responses they provide.

[0763] "Synchronization" refers to the process of ensuring that the latest data is matched between the user's terminal and the server.

[0764] "Real-time" means that processing and responses to user actions occur immediately.

[0765] The above are definitions of the important terms included in the scope of the claim.

[0766] This invention applies to a learning support system using educational applications and a customer support system using smart devices in physical stores. User terminals include smartphones, tablets, and personal computers, and the system is built by installing educational applications on these devices. In physical stores, smart devices such as smart glasses are used.

[0767] System Configuration

[0768] This system consists of the following components:

[0769] 1. User terminal: A device equipped with a camera and microphone, and with educational applications installed.

[0770] 2. Server: A computer system that receives data sent from user terminals, performs analysis, and generates responses.

[0771] 3. Emotion Engine: Software that analyzes the user's facial expressions and voice to generate emotion data.

[0772] 4. Natural Language Processing Model (NLP Model): An analytical tool that analyzes user questions and searches for relevant lecture content or product information.

[0773] 5. Database: A storage system that stores lecture content and product information.

[0774] 6. Smart devices: Devices that display information to customers in physical stores, such as smart glasses.

[0775] Flow of operations

[0776] 1. Initial setup of the user terminal:

[0777] After the user installs the application and creates an account upon their first login, they synchronize with the server to retrieve the latest lecture data.

[0778] 2. Emotion recognition and data collection:

[0779] When the application is launched, the camera and microphone are automatically activated, and the emotion engine analyzes the user's facial expressions and voice to generate emotion data. When the user enters a question, the question data and emotion data are sent to the server.

[0780] 3. Question analysis and answer generation:

[0781] The server analyzes the received question data using a natural language processing model and searches the database for relevant lecture content and product information. Based on the search results, it generates the optimal response pattern. This response pattern is adjusted according to the user's emotional state.

[0782] 4. Display the answer:

[0783] The generated response patterns are sent to the user's device in text, audio, or video format. The user's device displays or plays the received responses in the appropriate format.

[0784] For example, if the question is "How do I use this product?" and the user is confused, a detailed and easy-to-understand explanation will be provided.

[0785] 5. Gathering feedback:

[0786] User feedback is sent from the device to the server and stored as feedback data. This data is used to improve the accuracy of the system.

[0787] Specific usage examples

[0788] In the educational application, when a user asks "How to solve a quadratic equation," the emotion engine detects confusion and provides a detailed and easy-to-understand explanation.

[0789] In smart shopping at physical stores, when a customer asks "How do I use this product?" through smart glasses, the store provides appropriate product information and instructions in real time.

[0790] Examples of prompts to input into a generative AI model

[0791] If you ask the "Smart Shopping Assistant" in-store, "How do I use this product?", please generate a detailed explanation of how to use it based on emotion recognition.

[0792] If the customer is confused, please explain gently and in detail.

[0793] If the customer is excited, get straight to the point.

[0794] This system allows users to receive excellent support tailored to their emotions, which is expected to improve learning efficiency and customer satisfaction.

[0795] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0796] Step 1:

[0797] Initial settings:

[0798] Users first install the educational application on their device, such as a smartphone, tablet, or PC. After installation is complete, they launch the application, enter the necessary information to create an account, and then create an account and log in.

[0799] Input: User account information (name, email address, etc.)

[0800] Output: A user account is created on the server and synchronized with the user's terminal.

[0801] Specific operation: When a user enters the required information on the account creation screen and presses the data submission button, that data is sent to the server and registered as a new account.

[0802] Step 2:

[0803] Data synchronization from the server:

[0804] Upon initial login, the user's device connects to the server, and the latest lecture and course material data is synchronized from the server.

[0805] Input: Latest lecture content data stored on the server

[0806] Output: The latest lecture content data is downloaded to the user's terminal.

[0807] Specific operation: After the user presses the login button, a request is sent from the terminal to the server, the server sends the latest lecture data to the terminal, and the data is updated on the terminal.

[0808] Step 3:

[0809] Recognition of emotions:

[0810] When a user begins learning, the camera and microphone on the user's device are automatically activated, and the emotion engine analyzes the user's facial expressions and voice to generate emotion data.

[0811] Input: User's facial expressions and voice data

[0812] Output: Analyzed emotion data (e.g., confusion, excitement, etc.)

[0813] Specific operation: The camera and microphone capture the user's face, and that data is analyzed in real time by an emotion engine to identify the user's emotions.

[0814] Step 4:

[0815] Entering question data:

[0816] When a user has a question while studying, they select a specific subject or topic, enter their question into a text box, and press the submit button.

[0817] Input: Question entered by the user

[0818] Output: Questionnaire data (text format) and corresponding sentiment data

[0819] Specific operation: When a user enters a question in a text box and presses the submit button, the question content and sentiment data are sent to the server.

[0820] Step 5:

[0821] Question analysis on the server:

[0822] The server analyzes the received question data using a natural language processing model and searches the database for relevant lecture content.

[0823] Input: Questionnaire data and sentiment data

[0824] Output: Related lecture content data

[0825] Specific operation: The server analyzes the question data, searches the database for relevant textbooks and lecture content, and retrieves the appropriate information.

[0826] Step 6:

[0827] Generating response patterns:

[0828] The server generates the optimal response pattern based on the search results and adjusts the format of the response according to the user's emotional state.

[0829] Input: Relevant lecture content data, user sentiment data

[0830] Output: Adjusted response patterns (text, audio, video, etc.)

[0831] Specific operation: On the server side, the program decides whether to make the response detailed or concise based on the user's emotions, and then outputs it as text or audio data.

[0832] Step 7:

[0833] Providing responses to user terminals:

[0834] The generated response patterns are sent to the user's device in text, audio, or video format and displayed or played back in the appropriate format.

[0835] Input: Adjusted response patterns (text, audio, video, etc.)

[0836] Output: The answer that the user sees or hears on their device.

[0837] Specific operation: The user terminal analyzes the received response data, displays it in a user-friendly format, and also plays the audio.

[0838] Step 8:

[0839] Gathering feedback:

[0840] Users enter their satisfaction level with the provided answers and any additional feedback, then submit it to the server.

[0841] Input: User feedback data

[0842] Output: Feedback data stored on the server

[0843] Specific operation: When a user enters their opinions and ratings on the feedback screen and presses the submit button, that data is saved to the server.

[0844] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0845] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0846] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0847] [Third Embodiment]

[0848] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0849] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0850] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0851] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0852] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0853] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0854] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0855] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0856] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0857] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0858] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0859] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0860] The system of this invention aims to provide effective learning support using educational applications. This system consists of user terminals, a server, and a communication network connecting them.

[0861] Initial setup

[0862] Settings on the user terminal

[0863] First, users install the educational application on their smartphone, tablet, or computer. After installation, users launch the application and enter the necessary information to create an account. Once account creation and login are complete, the user's device connects to the server and synchronizes with the latest lecture data from the server.

[0864] Enter and submit your question

[0865] If a user has a question while learning, they can select a specific subject or topic through the educational application and enter their question in the text box. Once they have finished entering their question, they press the submit button.

[0866] User terminal operation

[0867] The user terminal sends the entered question data to the server. This question data includes the text entered by the user and other metadata (such as a timestamp and user ID).

[0868] Question analysis and answer generation

[0869] The server receives question data sent from the user's terminal. First, it analyzes the question using a natural language processing (NLP) model. This analysis extracts keywords and intent from the question. For example, in the question "How to solve a quadratic equation," "quadratic equation" and "how to solve" are extracted as important keywords.

[0870] Server operation

[0871] The server searches the database for relevant lecture content based on the analysis results. This search yields potential candidates such as relevant teaching materials, whiteboard notes, and past questions and answers. It then generates the optimal response pattern (text, audio, video) and sends it to the user's terminal.

[0872] Providing answers to users

[0873] The user terminal analyzes the response data received from the server and displays it in an appropriate format. For example, an explanation of how to solve a quadratic equation might be displayed as text, and corresponding audio or video can be played as needed. The user can then review the provided answers and continue their learning.

[0874] Gathering feedback and improving the system

[0875] Users can provide satisfaction ratings and additional feedback on the answers. This feedback data is sent from the user's device to the server, which stores it in a database. This feedback data is used to improve the accuracy of future automated responses and to improve the overall system.

[0876] Specific example

[0877] For example, suppose a user asks a question about quadratic equations in mathematics. The user enters "how to solve a quadratic equation" into a text box and presses the submit button. The user's terminal sends this question data to the server. The server uses an NLP model to extract the important keywords "quadratic equation" and "how to solve," and searches the database for relevant lecture content. From the search results, it generates the optimal answer pattern, for example, explanatory text such as "A quadratic equation is of the form ax^2 + bx + c = 0, and the quadratic formula is...", along with corresponding audio and video data, and sends it to the user's terminal. The user's terminal receives this and displays or plays it for the user.

[0878] This system allows users to efficiently resolve questions and advance their learning anytime, anywhere.

[0879] The following describes the processing flow.

[0880] Step 1:

[0881] Users install the educational application on their smartphone, tablet, or PC and launch the app. They then create an account by entering their email address and password and log in.

[0882] Step 2:

[0883] The device sends login information to the server. The server authenticates the received information and returns the authentication result to the device. If the login is successful, the device displays the home screen.

[0884] Step 3:

[0885] Upon initial login, the device requests the latest lecture data from the server. The server retrieves the lecture data from the database and sends it to the device. The device saves the received lecture data to local storage and displays a "synchronization complete" notification to the user.

[0886] Step 4:

[0887] The user operates the app to select the subject or topic for which they have a question during their studies. After selecting, they enter the specific question in the text box and press the submit button.

[0888] Step 5:

[0889] The terminal sends the user-entered question data to the server. This question data includes the entered text and metadata (e.g., timestamp, user ID).

[0890] Step 6:

[0891] The server passes the received question data to a natural language processing (NLP) model. The NLP model analyzes the question content and extracts important keywords and the intent of the question.

[0892] Step 7:

[0893] Based on the analysis results, the server searches the database for relevant lecture content. The database includes lecture content, whiteboard notes, past questions and answers, etc.

[0894] Step 8:

[0895] The server generates the optimal response pattern based on the searched lecture content. This response pattern is generated in text, audio, and video formats.

[0896] Step 9:

[0897] The server sends the generated response pattern to the terminal. The terminal analyzes the received response data and configures the display settings in the appropriate format.

[0898] Step 10:

[0899] The device displays the answer in text format on the user's screen. For example, it might display, "The quadratic equation is of the form ax^2 + bx + c = 0, and the quadratic formula is..." If necessary, it displays audio or video playback buttons, and playback begins when the user presses them.

[0900] Step 11:

[0901] Users rate the provided answers and provide feedback. Ratings are given in the format of "Good" or "Bad."

[0902] Step 12:

[0903] The device sends user feedback data to the server. The server stores the received feedback data in a database and uses it to improve the accuracy of future responses.

[0904] (Example 1)

[0905] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0906] In today's educational environment, students often lack the means to immediately resolve their questions. Furthermore, the limited availability of teaching materials and lecture content from educational institutions makes it difficult to provide support tailored to the individual needs of learners. Moreover, there are insufficient means of collecting feedback to continuously improve the quality and accuracy of the answers provided. This invention aims to solve these problems and provide a system that enables users to learn effectively.

[0907] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0908] In this invention, the server includes means for transmitting inquiry data entered from a terminal to an information processing device, means for analyzing the inquiry data using a generation AI model and searching for relevant educational content, and means for generating an optimal answer format based on the searched educational content. This makes it possible for users to efficiently resolve their questions anytime, anywhere, improving their satisfaction and learning effectiveness.

[0909] An "educational program" is software intended for use by users to support educational activities.

[0910] A "device" refers to a hardware device used by a user to install and utilize educational programs, such as a smartphone, tablet, or personal computer.

[0911] An "information processing device" is a computer system that analyzes and processes data transmitted from a terminal and provides the necessary data.

[0912] "Inquiry data" refers to the content of questions entered and submitted by users through educational programs, along with related data (such as timestamps and user IDs).

[0913] A "generative AI model" is an artificial intelligence model that uses natural language processing and other machine learning techniques to analyze input text data and extract intent and keywords.

[0914] "Educational content" includes teaching materials, lecture notes, and past questions and answers related to specific subjects or topics.

[0915] "Answer format" refers to the method of explanation provided to the user, and includes formats such as text, audio, and video.

[0916] The system of this invention is intended to support learning using educational programs and consists of a terminal, an information processing device, and a communication network connecting them. The detailed configuration and processing flow of the system are described below.

[0917] Initial setup

[0918] Settings on the user terminal

[0919] Users install the educational program on their smartphones, tablets, PCs, or other devices. After installation, users launch the application and enter the necessary information to create an account. Once account creation and login are complete, the device connects to the information processing unit and synchronizes the latest lecture data from the server. This allows users to access the latest educational content at any time.

[0920] Enter and submit your question

[0921] If a user has a question while learning, they select a specific subject or topic through the educational program and enter their question into the text box. After completing the input, they press the submit button. The terminal then sends the entered inquiry data (including the question text, timestamp, and user ID) to the information processing device.

[0922] Question analysis and answer generation

[0923] The information processing device receives inquiry data sent from the terminal. First, it analyzes the inquiry data using a generative AI model to extract keywords and intent of the question. For example, in the case of the question "How to solve a quadratic equation," "quadratic equation" and "how to solve" are analyzed as important keywords. Next, the information processing device searches a database based on the extracted keywords to find relevant educational content. This database includes teaching materials, lecture notes, and past questions and answers. It then generates the most suitable answer format (text, audio, video) and sends it to the terminal.

[0924] Providing answers to users

[0925] The terminal analyzes the response data received from the information processing device and displays it in an appropriate format. For example, an explanation of how to solve a quadratic equation might be displayed as text, and corresponding audio or video can be played as needed. This allows the user to review the provided answers and continue their learning.

[0926] Gathering feedback and improving the system

[0927] Users can input their satisfaction level and provide additional feedback on the provided answers. The terminal sends this feedback data to the information processing device. The information processing device stores the feedback data in a database and uses it to improve the accuracy of future answers and the overall system. This feedback is crucial for the continuous improvement of the system and the enhancement of the user experience.

[0928] Specific example

[0929] The following are specific examples of how the system can be used.

[0930] For example, if a user asks a question about quadratic equations in mathematics, the user enters "how to solve a quadratic equation" into the text box and presses the submit button. The terminal sends this inquiry data to the information processing device. The information processing device uses a generative AI model to extract the important keywords "quadratic equation" and "how to solve," and searches its database for relevant educational content. From the search results, it generates the optimal answer format, such as "A quadratic equation is in the form ax^2 + bx + c = 0, and the quadratic formula is...", along with corresponding audio and video data, and sends it to the terminal. The terminal receives this and displays or plays it for the user.

[0931] This system allows users to efficiently resolve questions and advance their learning anytime, anywhere.

[0932] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0933] Step 1:

[0934] Initial setup on the user terminal

[0935] Users install the educational program on their smartphones, tablets, or computers. After installation, users launch the application and enter necessary information such as their name, email address, and password to create an account. The entered information is sent from the device to the server, which stores this information in a database. Once authentication is complete, the device synchronizes the latest lecture data from the server and displays it on the device.

[0936] Input: User's personal information (name, email address, password)

[0937] Output: Account creation confirmation and synchronization of the latest lecture data

[0938] Step 2:

[0939] Enter and submit your question

[0940] If a user has a question while learning, they select a specific subject or topic through the educational program and enter their question in a text box. Once this is complete, the user presses the submit button. The device then sends this inquiry data (including the question text, timestamp, and user ID) to the server.

[0941] Input: User's question, timestamp, user ID

[0942] Output: Confirmation of sending query data to the server

[0943] Step 3:

[0944] Question analysis

[0945] The server receives query data sent from the terminal. The server uses a generative AI model to analyze the query data and extract keywords and intent from the question. For example, in the question "How to solve a quadratic equation," "quadratic equation" and "how to solve" are extracted as important keywords.

[0946] Input: Inquiry data (question content, timestamp, user ID)

[0947] Output: Analysis results (keywords and intent)

[0948] Step 4:

[0949] Search for related educational content

[0950] Based on the analysis results, the server searches the database for relevant educational content. This database includes teaching materials, lecture notes, and past questions and answers. The server finds the most suitable educational content that matches the search criteria.

[0951] Input: Analysis results (keywords and intent)

[0952] Output: Related educational content (teaching materials, lecture notes, past questions and answers)

[0953] Step 5:

[0954] Generating an answer format

[0955] The server generates the most suitable response format based on the searched educational content. The response format includes text, audio, and video, and is provided in the format that is easiest for the user to understand.

[0956] Input: Related educational content (textbooks, lecture notes, past questions and answers)

[0957] Output: Optimal response format (text, audio, video)

[0958] Step 6:

[0959] Submit your response

[0960] The server sends the generated response format to the terminal. The transmitted data includes the response text and corresponding audio or video data.

[0961] Input: Optimal response format (text, audio, video)

[0962] Output: Confirmation of sending response data to the terminal

[0963] Step 7:

[0964] Providing answers to users

[0965] The device analyzes the response data received from the server and displays or plays it back to the user in an appropriate format. For example, text may be displayed on the screen, and audio or video may be played. The user can then proceed with their learning based on this information.

[0966] Input: Response data (text, audio, video)

[0967] Output: Display and play back the response to the user.

[0968] Step 8:

[0969] Gathering feedback

[0970] Users can input their satisfaction level and provide additional feedback on the provided answers. The device sends this feedback to the server, which stores it in a database. This helps improve the accuracy of future answers.

[0971] Input: User feedback (satisfaction level, additional comments)

[0972] Output: Confirmation and storage of feedback data sent to the server.

[0973] (Application Example 1)

[0974] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0975] With the increasing prevalence of autonomous vehicles, there is a growing need for ways to ensure passengers have a meaningful experience during long journeys. However, conventional in-car entertainment systems are primarily limited to entertainment and information provision, lacking effective learning tools for education. Therefore, providing interactive educational support systems that passengers can use within autonomous vehicles is a key challenge.

[0976] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0977] In this invention, the server includes means for transmitting question data entered from a user terminal to the server, means for analyzing the question data using a natural language processing model and searching for relevant lecture content, and means for generating an optimal answer pattern based on the searched lecture content. This enables passengers to learn effectively in an autonomous vehicle and spend long journeys meaningfully.

[0978] An "educational application" is software designed for users to learn specific knowledge or skills.

[0979] A "user terminal" is a computer device used by a user to run and operate educational applications.

[0980] A "server" is a central computer system responsible for data processing and storage for educational applications, and for communicating with user terminals.

[0981] A "natural language processing model" is an artificial intelligence technology used to analyze question data and text data entered by users.

[0982] An "infotainment system" is an in-car system that provides information and entertainment to passengers in autonomous vehicles.

[0983] "Question data" refers to information about questions and problems that users have entered into educational applications.

[0984] An "answer pattern" is the format of the explanations and descriptions that an educational application provides in response to a user's question.

[0985] "Feedback data" refers to information in which users have entered evaluations and opinions regarding the provided answers.

[0986] "Synchronization" is the process performed to make data consistent between the user's terminal and the server.

[0987] System Overview

[0988] The system of the present invention is for the effective use of educational applications in autonomous vehicles. It consists of a user terminal, a server, a communication network, and an infotainment system within the autonomous vehicle.

[0989] Program Operation Overview

[0990] The user installs the educational application on the in-car infotainment system and begins the learning process. The system operates in the following steps:

[0991] 1. The user operates the infotainment system and launches an educational application.

[0992] 2. Enter the information needed to create a user account on the infotainment system, create the account, and log in. This will start synchronization with the server.

[0993] 3. If a question arises during learning, the user selects a specific subject or topic within the application and enters the question in the text box.

[0994] 4. The infotainment system sends the question data to the server. This question data includes metadata such as text, timestamp, and user ID.

[0995] 5. The server receives the question data and performs analysis using a natural language processing (NLP) model. This analysis extracts important keywords and intents.

[0996] 6. The server searches the database for relevant lecture content and generates the optimal response pattern (text, audio, video).

[0997] 7. The generated response patterns are sent back to the infotainment system and presented to the user.

[0998] 8. Users review the provided answers and continue learning. Simultaneously, they can provide feedback on the answers. This feedback data is sent to the server and used to improve the accuracy of future answers.

[0999] Hardware and software usage

[1000] Hardware: Infotainment systems for autonomous vehicles, smartphones, tablets

[1001] software:

[1002] Natural Language Processing (NLP) models (e.g., BERT model)

[1003] Communication protocol: Internet Protocol (HTTP, HTTPS)

[1004] Specific example

[1005] For example, if a passenger enters a request to learn "the basics of quantum mechanics" while in the car, the in-car infotainment system automatically sends the question to a server. The server uses an NLP model to extract the keywords "quantum mechanics" and "basics." Based on the keywords, the server searches for the most suitable learning materials (such as introductory videos and text explanations) and sends them to the infotainment system. The infotainment system then displays or plays the received materials for the passenger.

[1006] Examples of prompts to input into a generative AI model

[1007] "The user typed 'Teach me the basics of quantum mechanics.' Please search for and provide appropriate teaching materials."

[1008] This system allows passengers to spend their time productively while efficiently pursuing their studies.

[1009] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1010] Step 1:

[1011] The user launches the educational application through the infotainment system in the autonomous vehicle and enters information to create an account. This information includes username, email address, and password. Once the user enters the information and presses the submit button, the infotainment system sends it to the server. The server receives the input data and creates the user account. This initiates communication between the server and the infotainment system, and the latest lecture content data is synchronized upon the first login.

[1012] Input: Username, email address, password, etc.

[1013] Output: User account created, synchronization with server started

[1014] Step 2:

[1015] When a user has a question, they operate an educational application on the infotainment system, select a specific subject or topic, and enter their question in a text box. For example, they might enter, "Please explain the basics of quantum mechanics." This question data, along with a timestamp and user ID, is sent to the server.

[1016] Input: Question (e.g., "Please explain the basics of quantum mechanics"), timestamp, user ID

[1017] Output: Sending query data to the server

[1018] Step 3:

[1019] The server receives the submitted question data and analyzes the question content using a natural language processing (NLP) model. During the analysis, important keywords and the user's intent are extracted. For example, keywords such as "quantum mechanics" and "fundamentals" may be extracted. Based on these analysis results, the server searches the database for relevant lecture content.

[1020] Input: Question data (text, timestamp, user ID)

[1021] Output: Keyword extraction, search for related lecture content

[1022] Step 4:

[1023] The server generates the most suitable response pattern from the search results. For example, it might select a text explanation or video on "Fundamentals of Quantum Mechanics" for beginners. These responses are generated in various formats such as text, audio, and video, and sent to the infotainment system.

[1024] Input: Analysis results, related lecture content

[1025] Output: Generation of optimal response patterns (text, audio, video)

[1026] Step 5:

[1027] The infotainment system receives the response patterns and presents them to the user. For example, a text explanation may be displayed on the screen, and videos or audio may be played as needed. The user can review the provided answers and continue learning.

[1028] Input: Response patterns (text, audio, video)

[1029] Output: Providing answers to the user (display, play)

[1030] Step 6:

[1031] Users can provide feedback on the provided answers. This feedback data is then sent back to the server via the infotainment system. The server stores this feedback data and uses it to improve the accuracy of future answers.

[1032] Input: User Feedback

[1033] Output: Feedback data is stored on the server and used for future improvements.

[1034] This will enable a system that allows passengers to efficiently learn while inside an autonomous vehicle.

[1035] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1036] This invention provides more effective and personalized learning support by combining an emotion engine with a learning support system that uses educational applications. The system consists of user terminals, a server, and a communication network connecting them.

[1037] Initial setup

[1038] Settings on the user terminal

[1039] First, users install the educational application on their smartphone, tablet, or PC. After installation, users launch the application and enter the necessary information to create an account. Once account creation and login are complete, the user's device connects to the server and synchronizes with the latest lecture data from the server.

[1040] Recognition of emotions and input of questions

[1041] Facial expression and voice analysis

[1042] The application is equipped with a camera and microphone, and includes an emotion engine that uses these to recognize emotions from the user's facial expressions and voice. When the user interacts with the application, the camera and microphone automatically activate and analyze the user's facial expressions and voice tone.

[1043] User question input

[1044] When a user has a question during their studies, they select a specific subject or topic, enter the question into a text box, and press the submit button. At this point, the emotion engine analyzes the user's emotions in real time and adds that data to the question data.

[1045] User terminal operation

[1046] The terminal sends the entered question data and sentiment data to the server. This question data includes text, sentiment data, and metadata (such as a timestamp and user ID).

[1047] Question analysis and answer generation

[1048] Natural language processing and sentiment analysis

[1049] The server receives question data sent from the user's terminal and analyzes the question content using a natural language processing (NLP) model. Furthermore, it also analyzes sentiment data to understand the user's emotional state (e.g., confusion, dissatisfaction, excitement).

[1050] Search related lecture content

[1051] The server searches the database for relevant lecture content based on the analysis results of the NLP model and sentiment data. This search considers not only teaching materials and past answers related to the question, but also answers that are appropriate for the user's emotional state.

[1052] Generating response patterns

[1053] The server generates the optimal response pattern based on the searched lecture content. This response pattern is adjusted according to the user's emotional state and is provided in text, audio, and video formats.

[1054] Providing answers to users

[1055] User terminal operation

[1056] The device analyzes the response data received from the server and sets the display format to an appropriate one. For example, response patterns that include explanations about "how to solve quadratic equations" are displayed in a format that is easy to understand and tailored to the user's level of confusion. Audio and video playback are also provided as needed.

[1057] User feedback

[1058] Users enter their satisfaction level and additional feedback regarding the provided answers. This feedback data includes evaluations of the quality of the answers and responses regarding their feelings.

[1059] Gathering feedback and improving the system

[1060] Server operation

[1061] The device sends user feedback data to the server. The server stores the received feedback data in a database and uses it to improve the accuracy of future responses and enhance the overall system. The feedback data also includes information about emotional states, which allows the system to provide more adaptive learning support.

[1062] Specific example

[1063] For example, suppose a user asks a question about "how to solve a quadratic equation" in mathematics. The user types "how to solve a quadratic equation" into a text box and presses the submit button. The user's terminal sends this question data and emotion data indicating the user's state of confusion to the server. The server uses an NLP model to extract the keywords "quadratic equation" and "how to solve" and searches its database for relevant lecture content. Furthermore, it generates an answer pattern that is tailored to the user's state of confusion, with a detailed and easy-to-understand explanation. For example, it might generate text such as "A quadratic equation is of the form ax^2 + bx + c = 0, and the formula for solving it is...", along with gentle audio and corresponding video data, and send them to the user's terminal. The user's terminal receives this and displays or plays it for the user.

[1064] This system allows users to effectively and quickly resolve their questions and learn while receiving emotionally tailored support.

[1065] The following describes the processing flow.

[1066] Step 1:

[1067] Users install the educational application on their smartphones, tablets, or computers. After installation, users launch the app, enter their email address and password to create an account, and log in.

[1068] Step 2:

[1069] The device sends login information to the server. The server authenticates the received information and returns the authentication result to the device. If the login is successful, the device displays the home screen.

[1070] Step 3:

[1071] Upon initial login, the device requests the latest lecture data from the server. The server retrieves the lecture data from the database and sends it to the device. The device saves the received lecture data to local storage and displays a "synchronization complete" notification to the user.

[1072] Step 4:

[1073] The user operates the app to select a subject or topic they have questions about during their studies. After selecting, they enter their specific question in the text box and press the submit button.

[1074] Step 5:

[1075] The terminal sends the user-entered question data to the server. This question data includes the entered text, sentiment data, and metadata (such as a timestamp and user ID). Sentiment data is generated by an emotion engine that analyzes the user's facial expressions and voice.

[1076] Step 6:

[1077] The server passes the received question data to a natural language processing (NLP) model. The NLP model analyzes the question content and extracts keywords and the intent behind the question.

[1078] Step 7:

[1079] The server receives emotional data analyzed by the emotion engine. This allows it to understand the user's current emotional state (e.g., confused, frustrated, excited).

[1080] Step 8:

[1081] The server searches the database for relevant lecture content based on the analysis results of the NLP model and sentiment data. This search includes teaching materials, whiteboard notes, and past questions and answers related to the question.

[1082] Step 9:

[1083] The server generates the optimal response pattern based on the user's emotional state. For example, for a confused user, it generates text, audio, and video data that includes more detailed and thorough explanations.

[1084] Step 10:

[1085] The server sends the generated response pattern to the terminal. The terminal analyzes the received response data and displays or plays it in the appropriate format. For example, it might display text such as, "A quadratic equation is of the form ax^2 + bx + c = 0, and the formula for solving it is..." and play audio or video as needed.

[1086] Step 11:

[1087] Users rate the provided answers and provide feedback. Ratings are given in the format of "Good" or "Bad."

[1088] Step 12:

[1089] The device sends user feedback data to the server. The server stores the received feedback data in a database and uses it to improve the accuracy of future responses and enhance the overall system. The feedback data also includes information about emotional states, allowing the system to provide more adaptive learning support.

[1090] Specific example:

[1091] 1. User input and emotion recognition

[1092] When a user asks a question about "how to solve a quadratic equation," they type "how to solve a quadratic equation" into the text box and press the submit button. The emotion engine recognizes the user's state of confusion, and the emotion data is sent to the server along with the question data.

[1093] 2. Question analysis and answer generation

[1094] The server uses an NLP model to extract the keywords "quadratic equation" and "solution method" from the question and searches for related lecture content. It generates answer patterns that include detailed and easy-to-understand explanations for confused users and sends them to the terminal.

[1095] 3. Providing responses and feedback to users

[1096] The terminal displays the received response data, showing the text "The quadratic equation is in the form ax^2 + bx + c = 0, and the quadratic formula is..." along with the corresponding audio or video. The user is satisfied with the response and sends a positive rating and feedback such as "very easy to understand." The server saves this feedback data and uses it to improve the response system in the future.

[1097] This system allows users to receive effective learning support tailored to their emotions.

[1098] (Example 2)

[1099] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1100] Traditional educational support systems provided uniform answers without considering the individual emotional states of users, resulting in limited learning effectiveness. Furthermore, they lacked mechanisms to incorporate feedback based on user emotions. This led to problems in quickly and appropriately resolving users' questions and concerns.

[1101] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for transmitting question data and emotion data input from the user terminal to the data server; means for analyzing the question data using a natural language processing model and searching for relevant lecture content; means for analyzing the emotion data and understanding the user's emotional state; means for generating an optimal answer pattern based on the retrieved lecture content and emotional state; means for transmitting the generated answer pattern to the user terminal in the form of text, audio, and video; means for displaying or playing back the received answer pattern on the user terminal; and means for analyzing the user's facial expressions and voice and acquiring emotion data. This makes it possible to provide optimal learning support tailored to the user's individual emotional state.

[1102] An "educational program" is a software application specifically designed for learning support and educational purposes.

[1103] A "user terminal" refers to a digital device, such as a smartphone, tablet, or personal computer, used to install and operate educational programs.

[1104] A "data server" is a server system that stores and manages question data, sentiment data, and lecture content data, and communicates with user terminals.

[1105] "Question data" refers to text data containing questions and doubts that users have while learning.

[1106] "Emotional data" refers to data that indicates the emotional state (e.g., confusion, anxiety, excitement) of a user, analyzed from their facial expressions and voice.

[1107] A "natural language processing model" is a machine learning model used to analyze the meaning of text data and extract keywords and syntax.

[1108] An "answer pattern" is the optimal answer format (text, audio, or video) for a user's question, generated based on question data and sentiment data.

[1109] "Means for analyzing user facial expressions and voice" refers to a combination of software and hardware that uses cameras and microphones to analyze the user's facial expressions and voice tone, and to acquire emotional data.

[1110] "Feedback data" refers to data that includes satisfaction levels, additional opinions, and evaluations entered by users regarding the provided responses.

[1111] "Lecture content data" refers to digital content such as teaching materials, lecture notes, and video lectures necessary for learning within an educational program.

[1112] This invention provides more effective and personalized learning support by combining an emotion engine with a learning support system that uses educational programs. This system consists of user terminals, a data server, and a communication network connecting them.

[1113] First, users install the educational program on their smartphones, tablets, or PCs. The program includes various learning materials and functions to support their studies. After installation, users launch the program and enter the necessary information to create an account (name, email address, password, etc.). Once account creation and login are complete, the user's device connects to the data server and synchronizes with the latest lecture content data. Specifically, it communicates with the server using REST API and WebSocket technologies to save new lecture data and updates locally.

[1114] When a user begins learning, the camera and microphone automatically activate to analyze the user's facial expressions and voice tone in real time. OpenCV and TensorFlow are used for facial recognition, and MLKit and speech recognition APIs are used for voice analysis. This allows the user's emotional state (e.g., confusion, frustration, excitement) to be captured. When the user selects a specific subject or topic, enters their questions into a text box, and presses the submit button, the device sends the text data and the real-time emotional data to the data server.

[1115] The data server analyzes the received question data and sentiment data using natural language processing (NLP) models. NLP models such as Transformers and BERT are used to extract the meaning of the questions and important keywords. Furthermore, sentiment data is also analyzed to understand the user's emotional state. Based on the analysis results, the data server searches the database for relevant lecture content. This search includes textbook data, past Q&A, and the user's learning history. Based on the search results and the emotional state, the server generates the optimal response pattern. Responses are provided in text, audio, and video formats.

[1116] The generated answer patterns are sent to the user's device, which receives and displays or plays them. Specifically, HTML, CSS, and JavaScript are used to provide an appropriate UI / UX, presenting the answers in a user-friendly format. For example, if a user asks about "how to solve a quadratic equation" and is confused, an answer adjusted to be detailed and easy to understand will be provided. For instance, an explanation such as "A quadratic equation is in the form ax^2 + bx + c = 0, and the quadratic formula is..." or a corresponding video will be played.

[1117] Users provide satisfaction ratings and additional feedback on the provided answers. This feedback data is then sent back to the data server via the device. The received feedback data is stored in a database and used to improve the accuracy of future answers and enhance the overall system. Because the feedback data also includes information about emotional states, the system can increasingly provide adaptive learning support to users.

[1118] Examples of prompt statements

[1119] For example, the following prompt statements are possible.

[1120] User: I don't know how to solve quadratic equations.

[1121] Emotional state: Confusion

[1122] Based on this prompt, the data server generates an appropriate response pattern and provides optimized support to resolve the user's confusion.

[1123] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1124] Step 1: Initial Setup

[1125] Users install the educational program on their smartphones, tablets, or PCs. After installation, users launch the program and enter the necessary information to create an account (name, email address, password, etc.). An account is generated based on the entered data and saved to the data server. Once logged in, the user's device connects to the data server and synchronizes the latest lecture content data. Specifically, data communication is performed using REST APIs or WebSockets, and the lecture data is saved locally.

[1126] Input: User information (name, email address, password), login information

[1127] Output: Account data generation, lecture content data synchronization

[1128] Step 2: Start facial expression and voice analysis

[1129] When a user begins learning, the device's camera and microphone automatically activate. The device uses these devices to analyze the user's facial expressions and voice tone in real time using an emotion engine. OpenCV and TensorFlow are used for facial recognition, and MLKit and speech recognition APIs are used for voice analysis. At this stage, the user's facial expression and voice tone data are collected and output as emotion data in real time.

[1130] Input: User's facial expression data, voice data

[1131] Output: Emotional data (e.g., confused, dissatisfied, excited)

[1132] Step 3: Enter and submit your question.

[1133] The user selects a specific subject or topic and enters their question into a text box. When the user presses the submit button, the device sends the text data and real-time sentiment data to a data server. The submitted data includes the question, sentiment data, a timestamp, and the user ID. The submitted data is received and stored on the server.

[1134] Input: User's question (text data), sentiment data

[1135] Output: Sending question data and sentiment data to the data server.

[1136] Step 4: Analyzing Question Data

[1137] The data server analyzes the received question data and sentiment data using natural language processing (NLP) models. NLP models such as Transformers and BERT are used to extract the meaning of the question and important keywords. In addition, sentiment data is analyzed to identify the user's emotional state. The analysis results are output as a series of keywords and sentiment state data.

[1138] Input: Question data, sentiment data

[1139] Output: Analysis results (keywords, emotional state)

[1140] Step 5: Search for related lecture content

[1141] The data server searches the database for relevant lecture content based on the analysis results. The database contains textbook data, past Q&A, and user learning history. As a result of the search, the relevant lecture content data is identified and retrieved.

[1142] Input: Analysis results (keywords, emotional state)

[1143] Output: Related lecture content data

[1144] Step 6: Generating response patterns

[1145] The data server generates the optimal response pattern based on the retrieved lecture content data. The response is generated in text, audio, and video formats, depending on the user's emotional state (e.g., detailed and gentle tone if confused). At this stage, the final response pattern is determined and stored on the data server.

[1146] Input: Related lecture content data, emotional state

[1147] Output: Response patterns (text, audio, video)

[1148] Step 7: Provide answer patterns

[1149] The user terminal receives response patterns generated from the data server. Based on the received response patterns, the terminal displays or plays content for the user in an appropriate format. Specifically, it uses HTML, CSS, and JavaScript to provide a user-friendly UI / UX. For example, it plays videos explaining how to solve quadratic equations or providing example problems.

[1150] Input: Response patterns (text, audio, video)

[1151] Output: Learning support for the user (display, playback)

[1152] Step 8: Collecting and submitting feedback

[1153] Users provide satisfaction ratings and additional feedback on the provided answers. This feedback data is sent to a data server via the device. The input feedback data includes evaluations of the quality of the answers and responses to emotions.

[1154] Input: User feedback (satisfaction level, additional comments)

[1155] Output: Sending feedback data to the data server

[1156] Step 9: Feedback Analysis and System Improvement

[1157] The data server stores the received feedback data in a database. The feedback data is analyzed and used to improve future response accuracy and the overall system. Machine learning algorithms are used to analyze the feedback data and improve the system's accuracy and performance.

[1158] Input: Feedback data

[1159] Output: Analysis results, system improvement data

[1160] (Application Example 2)

[1161] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1162] Conventional learning support systems unilaterally provided information without considering the user's emotional state, making it difficult to provide adaptive support tailored to each learner's level of understanding and emotions. Furthermore, in physical stores, customers often felt an emotional distance when asking questions about products, resulting in a lack of interactive customer support.

[1163] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for transmitting question data input from a user terminal to the server, means for analyzing the question data using a natural language processing model and searching for relevant lecture content, means for generating an optimal answer pattern based on the searched lecture content, means for analyzing the user's emotions in real time using an emotion engine with a smart device and adding the analyzed emotion data to the question data, and means for adjusting the generated answer pattern according to the user's emotional state. This makes it possible to provide personalized learning support according to the user's emotional state. Furthermore, even in physical stores, it becomes possible to provide product descriptions and support according to the customer's emotions, thereby improving customer satisfaction.

[1164] An "educational application" is software used by users for learning, providing lecture data and learning materials.

[1165] A "user terminal" refers to a device, such as a smartphone, tablet, or personal computer, on which a user installs and uses educational applications.

[1166] A "server" is a computer system that receives data sent from a user terminal and performs analysis and processing on it.

[1167] "Question data" refers to data that includes the questions that users have and enter during their learning process.

[1168] A "natural language processing model" refers to algorithms and techniques for analyzing natural language and understanding its meaning.

[1169] "Relevant lecture content" refers to appropriate teaching materials and lecture content in response to the question entered by the user.

[1170] An "answer pattern" refers to the format or method used to provide the most appropriate answer to a question.

[1171] An "emotion engine" refers to software and hardware that recognizes and analyzes emotions from a user's facial expressions and voice.

[1172] "Emotional data" refers to data that indicates the user's emotional state, as analyzed by the emotion engine.

[1173] A "smart device" refers to a device that has internet connectivity and is used by a user, such as smart glasses or a smartphone.

[1174] "Feedback data" refers to data that includes evaluations and opinions from users regarding the responses they provide.

[1175] "Synchronization" refers to the process of ensuring that the latest data is matched between the user's terminal and the server.

[1176] "Real-time" means that processing and responses to user actions occur immediately.

[1177] The above are definitions of the important terms included in the scope of the claim.

[1178] This invention applies to a learning support system using educational applications and a customer support system using smart devices in physical stores. User terminals include smartphones, tablets, and personal computers, and the system is built by installing educational applications on these devices. In physical stores, smart devices such as smart glasses are used.

[1179] System Configuration

[1180] This system consists of the following components:

[1181] 1. User terminal: A device equipped with a camera and microphone, and with educational applications installed.

[1182] 2. Server: A computer system that receives data sent from user terminals, performs analysis, and generates responses.

[1183] 3. Emotion Engine: Software that analyzes the user's facial expressions and voice to generate emotion data.

[1184] 4. Natural Language Processing Model (NLP Model): An analytical tool that analyzes user questions and searches for relevant lecture content or product information.

[1185] 5. Database: A storage system that stores lecture content and product information.

[1186] 6. Smart devices: Devices that display information to customers in physical stores, such as smart glasses.

[1187] Flow of operations

[1188] 1. Initial setup of the user terminal:

[1189] After the user installs the application and creates an account upon their first login, they synchronize with the server to retrieve the latest lecture data.

[1190] 2. Emotion recognition and data collection:

[1191] When the application is launched, the camera and microphone are automatically activated, and the emotion engine analyzes the user's facial expressions and voice to generate emotion data. When the user enters a question, the question data and emotion data are sent to the server.

[1192] 3. Question analysis and answer generation:

[1193] The server analyzes the received question data using a natural language processing model and searches the database for relevant lecture content and product information. Based on the search results, it generates the optimal response pattern. This response pattern is adjusted according to the user's emotional state.

[1194] 4. Display the answer:

[1195] The generated response patterns are sent to the user's device in text, audio, or video format. The user's device displays or plays the received responses in the appropriate format.

[1196] For example, if the question is "How do I use this product?" and the user is confused, a detailed and easy-to-understand explanation will be provided.

[1197] 5. Gathering feedback:

[1198] User feedback is sent from the device to the server and stored as feedback data. This data is used to improve the accuracy of the system.

[1199] Specific usage examples

[1200] In the educational application, when a user asks "How to solve a quadratic equation," the emotion engine detects confusion and provides a detailed and easy-to-understand explanation.

[1201] In smart shopping at physical stores, when a customer asks "How do I use this product?" through smart glasses, the store provides appropriate product information and instructions in real time.

[1202] Examples of prompts to input into a generative AI model

[1203] If you ask the "Smart Shopping Assistant" in-store, "How do I use this product?", please generate a detailed explanation of how to use it based on emotion recognition.

[1204] If the customer is confused, please explain gently and in detail.

[1205] If the customer is excited, get straight to the point.

[1206] This system allows users to receive excellent support tailored to their emotions, which is expected to improve learning efficiency and customer satisfaction.

[1207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1208] Step 1:

[1209] Initial settings:

[1210] Users first install the educational application on their device, such as a smartphone, tablet, or PC. After installation is complete, they launch the application, enter the necessary information to create an account, and then create an account and log in.

[1211] Input: User account information (name, email address, etc.)

[1212] Output: A user account is created on the server and synchronized with the user's terminal.

[1213] Specific operation: When a user enters the required information on the account creation screen and presses the data submission button, that data is sent to the server and registered as a new account.

[1214] Step 2:

[1215] Data synchronization from the server:

[1216] Upon initial login, the user's device connects to the server, and the latest lecture and course material data is synchronized from the server.

[1217] Input: Latest lecture content data stored on the server

[1218] Output: The latest lecture content data is downloaded to the user's terminal.

[1219] Specific operation: After the user presses the login button, a request is sent from the terminal to the server, the server sends the latest lecture data to the terminal, and the data is updated on the terminal.

[1220] Step 3:

[1221] Recognition of emotions:

[1222] When a user begins learning, the camera and microphone on the user's device are automatically activated, and the emotion engine analyzes the user's facial expressions and voice to generate emotion data.

[1223] Input: User's facial expressions and voice data

[1224] Output: Analyzed emotion data (e.g., confusion, excitement, etc.)

[1225] Specific operation: The camera and microphone capture the user's face, and that data is analyzed in real time by an emotion engine to identify the user's emotions.

[1226] Step 4:

[1227] Entering question data:

[1228] When a user has a question while studying, they select a specific subject or topic, enter their question into a text box, and press the submit button.

[1229] Input: Question entered by the user

[1230] Output: Questionnaire data (text format) and corresponding sentiment data

[1231] Specific operation: When a user enters a question in a text box and presses the submit button, the question content and sentiment data are sent to the server.

[1232] Step 5:

[1233] Question analysis on the server:

[1234] The server analyzes the received question data using a natural language processing model and searches the database for relevant lecture content.

[1235] Input: Questionnaire data and sentiment data

[1236] Output: Related lecture content data

[1237] Specific operation: The server analyzes the question data, searches the database for relevant textbooks and lecture content, and retrieves the appropriate information.

[1238] Step 6:

[1239] Generating response patterns:

[1240] The server generates the optimal response pattern based on the search results and adjusts the format of the response according to the user's emotional state.

[1241] Input: Relevant lecture content data, user sentiment data

[1242] Output: Adjusted response patterns (text, audio, video, etc.)

[1243] Specific operation: On the server side, the program decides whether to make the response detailed or concise based on the user's emotions, and then outputs it as text or audio data.

[1244] Step 7:

[1245] Providing responses to user terminals:

[1246] The generated response patterns are sent to the user's device in text, audio, or video format and displayed or played back in the appropriate format.

[1247] Input: Adjusted response patterns (text, audio, video, etc.)

[1248] Output: The answer that the user sees or hears on their device.

[1249] Specific operation: The user terminal analyzes the received response data, displays it in a user-friendly format, and also plays the audio.

[1250] Step 8:

[1251] Gathering feedback:

[1252] Users enter their satisfaction level with the provided answers and any additional feedback, then submit it to the server.

[1253] Input: User feedback data

[1254] Output: Feedback data stored on the server

[1255] Specific operation: When a user enters their opinions and ratings on the feedback screen and presses the submit button, that data is saved to the server.

[1256] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1257] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1258] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1259] [Fourth Embodiment]

[1260] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1261] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1262] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1263] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1264] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1265] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1266] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1267] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1268] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1269] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1270] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1271] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1272] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1273] The system of this invention aims to provide effective learning support using educational applications. This system consists of user terminals, a server, and a communication network connecting them.

[1274] Initial setup

[1275] Settings on the user terminal

[1276] First, users install the educational application on their smartphone, tablet, or computer. After installation, users launch the application and enter the necessary information to create an account. Once account creation and login are complete, the user's device connects to the server and synchronizes with the latest lecture data from the server.

[1277] Enter and submit your question

[1278] If a user has a question while learning, they can select a specific subject or topic through the educational application and enter their question in the text box. Once they have finished entering their question, they press the submit button.

[1279] User terminal operation

[1280] The user terminal sends the entered question data to the server. This question data includes the text entered by the user and other metadata (such as a timestamp and user ID).

[1281] Question analysis and answer generation

[1282] The server receives question data sent from the user's terminal. First, it analyzes the question using a natural language processing (NLP) model. This analysis extracts keywords and intent from the question. For example, in the question "How to solve a quadratic equation," "quadratic equation" and "how to solve" are extracted as important keywords.

[1283] Server operation

[1284] The server searches the database for relevant lecture content based on the analysis results. This search yields potential candidates such as relevant teaching materials, whiteboard notes, and past questions and answers. It then generates the optimal response pattern (text, audio, video) and sends it to the user's terminal.

[1285] Providing answers to users

[1286] The user terminal analyzes the response data received from the server and displays it in an appropriate format. For example, an explanation of how to solve a quadratic equation might be displayed as text, and corresponding audio or video can be played as needed. The user can then review the provided answers and continue their learning.

[1287] Gathering feedback and improving the system

[1288] Users can provide satisfaction ratings and additional feedback on the answers. This feedback data is sent from the user's device to the server, which stores it in a database. This feedback data is used to improve the accuracy of future automated responses and to improve the overall system.

[1289] Specific example

[1290] For example, suppose a user asks a question about quadratic equations in mathematics. The user enters "how to solve a quadratic equation" into a text box and presses the submit button. The user's terminal sends this question data to the server. The server uses an NLP model to extract the important keywords "quadratic equation" and "how to solve," and searches the database for relevant lecture content. From the search results, it generates the optimal answer pattern, for example, explanatory text such as "A quadratic equation is of the form ax^2 + bx + c = 0, and the quadratic formula is...", along with corresponding audio and video data, and sends it to the user's terminal. The user's terminal receives this and displays or plays it for the user.

[1291] This system allows users to efficiently resolve questions and advance their learning anytime, anywhere.

[1292] The following describes the processing flow.

[1293] Step 1:

[1294] Users install the educational application on their smartphone, tablet, or PC and launch the app. They then create an account by entering their email address and password and log in.

[1295] Step 2:

[1296] The device sends login information to the server. The server authenticates the received information and returns the authentication result to the device. If the login is successful, the device displays the home screen.

[1297] Step 3:

[1298] Upon initial login, the device requests the latest lecture data from the server. The server retrieves the lecture data from the database and sends it to the device. The device saves the received lecture data to local storage and displays a "synchronization complete" notification to the user.

[1299] Step 4:

[1300] The user operates the app to select the subject or topic for which they have a question during their studies. After selecting, they enter the specific question in the text box and press the submit button.

[1301] Step 5:

[1302] The terminal sends the user-entered question data to the server. This question data includes the entered text and metadata (e.g., timestamp, user ID).

[1303] Step 6:

[1304] The server passes the received question data to a natural language processing (NLP) model. The NLP model analyzes the question content and extracts important keywords and the intent of the question.

[1305] Step 7:

[1306] Based on the analysis results, the server searches the database for relevant lecture content. The database includes lecture content, whiteboard notes, past questions and answers, etc.

[1307] Step 8:

[1308] The server generates the optimal response pattern based on the searched lecture content. This response pattern is generated in text, audio, and video formats.

[1309] Step 9:

[1310] The server sends the generated response pattern to the terminal. The terminal analyzes the received response data and configures the display settings in the appropriate format.

[1311] Step 10:

[1312] The device displays the answer in text format on the user's screen. For example, it might display, "The quadratic equation is of the form ax^2 + bx + c = 0, and the quadratic formula is..." If necessary, it displays audio or video playback buttons, and playback begins when the user presses them.

[1313] Step 11:

[1314] Users rate the provided answers and provide feedback. Ratings are given in the format of "Good" or "Bad."

[1315] Step 12:

[1316] The device sends user feedback data to the server. The server stores the received feedback data in a database and uses it to improve the accuracy of future responses.

[1317] (Example 1)

[1318] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1319] In today's educational environment, students often lack the means to immediately resolve their questions. Furthermore, the limited availability of teaching materials and lecture content from educational institutions makes it difficult to provide support tailored to the individual needs of learners. Moreover, there are insufficient means of collecting feedback to continuously improve the quality and accuracy of the answers provided. This invention aims to solve these problems and provide a system that enables users to learn effectively.

[1320] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1321] In this invention, the server includes means for transmitting inquiry data entered from a terminal to an information processing device, means for analyzing the inquiry data using a generation AI model and searching for relevant educational content, and means for generating an optimal answer format based on the searched educational content. This makes it possible for users to efficiently resolve their questions anytime, anywhere, improving their satisfaction and learning effectiveness.

[1322] An "educational program" is software intended for use by users to support educational activities.

[1323] A "device" refers to a hardware device used by a user to install and utilize educational programs, such as a smartphone, tablet, or personal computer.

[1324] An "information processing device" is a computer system that analyzes and processes data transmitted from a terminal and provides the necessary data.

[1325] "Inquiry data" refers to the content of questions entered and submitted by users through educational programs, along with related data (such as timestamps and user IDs).

[1326] A "generative AI model" is an artificial intelligence model that uses natural language processing and other machine learning techniques to analyze input text data and extract intent and keywords.

[1327] "Educational content" includes teaching materials, lecture notes, and past questions and answers related to specific subjects or topics.

[1328] "Answer format" refers to the method of explanation provided to the user, and includes formats such as text, audio, and video.

[1329] The system of this invention is intended to support learning using educational programs and consists of a terminal, an information processing device, and a communication network connecting them. The detailed configuration and processing flow of the system are described below.

[1330] Initial setup

[1331] Settings on the user terminal

[1332] Users install the educational program on their smartphones, tablets, PCs, or other devices. After installation, users launch the application and enter the necessary information to create an account. Once account creation and login are complete, the device connects to the information processing unit and synchronizes the latest lecture data from the server. This allows users to access the latest educational content at any time.

[1333] Enter and submit your question

[1334] If a user has a question while learning, they select a specific subject or topic through the educational program and enter their question into the text box. After completing the input, they press the submit button. The terminal then sends the entered inquiry data (including the question text, timestamp, and user ID) to the information processing device.

[1335] Question analysis and answer generation

[1336] The information processing device receives inquiry data sent from the terminal. First, it analyzes the inquiry data using a generative AI model to extract keywords and intent of the question. For example, in the case of the question "How to solve a quadratic equation," "quadratic equation" and "how to solve" are analyzed as important keywords. Next, the information processing device searches a database based on the extracted keywords to find relevant educational content. This database includes teaching materials, lecture notes, and past questions and answers. It then generates the most suitable answer format (text, audio, video) and sends it to the terminal.

[1337] Providing answers to users

[1338] The terminal analyzes the response data received from the information processing device and displays it in an appropriate format. For example, an explanation of how to solve a quadratic equation might be displayed as text, and corresponding audio or video can be played as needed. This allows the user to review the provided answers and continue their learning.

[1339] Gathering feedback and improving the system

[1340] Users can input their satisfaction level and provide additional feedback on the provided answers. The terminal sends this feedback data to the information processing device. The information processing device stores the feedback data in a database and uses it to improve the accuracy of future answers and the overall system. This feedback is crucial for the continuous improvement of the system and the enhancement of the user experience.

[1341] Specific example

[1342] The following are specific examples of how the system can be used.

[1343] For example, if a user asks a question about quadratic equations in mathematics, the user enters "how to solve a quadratic equation" into the text box and presses the submit button. The terminal sends this inquiry data to the information processing device. The information processing device uses a generative AI model to extract the important keywords "quadratic equation" and "how to solve," and searches its database for relevant educational content. From the search results, it generates the optimal answer format, such as "A quadratic equation is in the form ax^2 + bx + c = 0, and the quadratic formula is...", along with corresponding audio and video data, and sends it to the terminal. The terminal receives this and displays or plays it for the user.

[1344] This system allows users to efficiently resolve questions and advance their learning anytime, anywhere.

[1345] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1346] Step 1:

[1347] Initial setup on the user terminal

[1348] Users install the educational program on their smartphones, tablets, or computers. After installation, users launch the application and enter necessary information such as their name, email address, and password to create an account. The entered information is sent from the device to the server, which stores this information in a database. Once authentication is complete, the device synchronizes the latest lecture data from the server and displays it on the device.

[1349] Input: User's personal information (name, email address, password)

[1350] Output: Account creation confirmation and synchronization of the latest lecture data

[1351] Step 2:

[1352] Enter and submit your question

[1353] If a user has a question while learning, they select a specific subject or topic through the educational program and enter their question in a text box. Once this is complete, the user presses the submit button. The device then sends this inquiry data (including the question text, timestamp, and user ID) to the server.

[1354] Input: User's question, timestamp, user ID

[1355] Output: Confirmation of sending query data to the server

[1356] Step 3:

[1357] Question analysis

[1358] The server receives query data sent from the terminal. The server uses a generative AI model to analyze the query data and extract keywords and intent from the question. For example, in the question "How to solve a quadratic equation," "quadratic equation" and "how to solve" are extracted as important keywords.

[1359] Input: Inquiry data (question content, timestamp, user ID)

[1360] Output: Analysis results (keywords and intent)

[1361] Step 4:

[1362] Search for related educational content

[1363] Based on the analysis results, the server searches the database for relevant educational content. This database includes teaching materials, lecture notes, and past questions and answers. The server finds the most suitable educational content that matches the search criteria.

[1364] Input: Analysis results (keywords and intent)

[1365] Output: Related educational content (teaching materials, lecture notes, past questions and answers)

[1366] Step 5:

[1367] Generating an answer format

[1368] The server generates the most suitable response format based on the searched educational content. The response format includes text, audio, and video, and is provided in the format that is easiest for the user to understand.

[1369] Input: Related educational content (textbooks, lecture notes, past questions and answers)

[1370] Output: Optimal response format (text, audio, video)

[1371] Step 6:

[1372] Submit your response

[1373] The server sends the generated response format to the terminal. The transmitted data includes the response text and corresponding audio or video data.

[1374] Input: Optimal response format (text, audio, video)

[1375] Output: Confirmation of sending response data to the terminal

[1376] Step 7:

[1377] Providing answers to users

[1378] The device analyzes the response data received from the server and displays or plays it back to the user in an appropriate format. For example, text may be displayed on the screen, and audio or video may be played. The user can then proceed with their learning based on this information.

[1379] Input: Response data (text, audio, video)

[1380] Output: Display and play back the response to the user.

[1381] Step 8:

[1382] Gathering feedback

[1383] Users can input their satisfaction level and provide additional feedback on the provided answers. The device sends this feedback to the server, which stores it in a database. This helps improve the accuracy of future answers.

[1384] Input: User feedback (satisfaction level, additional comments)

[1385] Output: Confirmation and storage of feedback data sent to the server.

[1386] (Application Example 1)

[1387] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1388] With the increasing prevalence of autonomous vehicles, there is a growing need for ways to ensure passengers have a meaningful experience during long journeys. However, conventional in-car entertainment systems are primarily limited to entertainment and information provision, lacking effective learning tools for education. Therefore, providing interactive educational support systems that passengers can use within autonomous vehicles is a key challenge.

[1389] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1390] In this invention, the server includes means for transmitting question data entered from a user terminal to the server, means for analyzing the question data using a natural language processing model and searching for relevant lecture content, and means for generating an optimal answer pattern based on the searched lecture content. This enables passengers to learn effectively in an autonomous vehicle and spend long journeys meaningfully.

[1391] An "educational application" is software designed for users to learn specific knowledge or skills.

[1392] A "user terminal" is a computer device used by a user to run and operate educational applications.

[1393] A "server" is a central computer system responsible for data processing and storage for educational applications, and for communicating with user terminals.

[1394] A "natural language processing model" is an artificial intelligence technology used to analyze question data and text data entered by users.

[1395] An "infotainment system" is an in-car system that provides information and entertainment to passengers in autonomous vehicles.

[1396] "Question data" refers to information about questions and problems that users have entered into educational applications.

[1397] An "answer pattern" is the format of the explanations and descriptions that an educational application provides in response to a user's question.

[1398] "Feedback data" refers to information in which users have entered evaluations and opinions regarding the provided answers.

[1399] "Synchronization" is the process performed to make data consistent between the user's terminal and the server.

[1400] System Overview

[1401] The system of the present invention is for the effective use of educational applications in autonomous vehicles. It consists of a user terminal, a server, a communication network, and an infotainment system within the autonomous vehicle.

[1402] Program Operation Overview

[1403] The user installs the educational application on the in-car infotainment system and begins the learning process. The system operates in the following steps:

[1404] 1. The user operates the infotainment system and launches an educational application.

[1405] 2. Enter the information needed to create a user account on the infotainment system, create the account, and log in. This will start synchronization with the server.

[1406] 3. If a question arises during learning, the user selects a specific subject or topic within the application and enters the question in the text box.

[1407] 4. The infotainment system sends the question data to the server. This question data includes metadata such as text, timestamp, and user ID.

[1408] 5. The server receives the question data and performs analysis using a natural language processing (NLP) model. This analysis extracts important keywords and intents.

[1409] 6. The server searches the database for relevant lecture content and generates the optimal response pattern (text, audio, video).

[1410] 7. The generated response patterns are sent back to the infotainment system and presented to the user.

[1411] 8. Users review the provided answers and continue learning. Simultaneously, they can provide feedback on the answers. This feedback data is sent to the server and used to improve the accuracy of future answers.

[1412] Hardware and software usage

[1413] Hardware: Infotainment systems for autonomous vehicles, smartphones, tablets

[1414] software:

[1415] Natural Language Processing (NLP) models (e.g., BERT model)

[1416] Communication protocol: Internet Protocol (HTTP, HTTPS)

[1417] Specific example

[1418] For example, if a passenger enters a request to learn "the basics of quantum mechanics" while in the car, the in-car infotainment system automatically sends the question to a server. The server uses an NLP model to extract the keywords "quantum mechanics" and "basics." Based on the keywords, the server searches for the most suitable learning materials (such as introductory videos and text explanations) and sends them to the infotainment system. The infotainment system then displays or plays the received materials for the passenger.

[1419] Examples of prompts to input into a generative AI model

[1420] "The user typed 'Teach me the basics of quantum mechanics.' Please search for and provide appropriate teaching materials."

[1421] This system allows passengers to spend their time productively while efficiently pursuing their studies.

[1422] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1423] Step 1:

[1424] The user launches the educational application through the infotainment system in the autonomous vehicle and enters information to create an account. This information includes username, email address, and password. Once the user enters the information and presses the submit button, the infotainment system sends it to the server. The server receives the input data and creates the user account. This initiates communication between the server and the infotainment system, and the latest lecture content data is synchronized upon the first login.

[1425] Input: Username, email address, password, etc.

[1426] Output: User account created, synchronization with server started

[1427] Step 2:

[1428] When a user has a question, they operate an educational application on the infotainment system, select a specific subject or topic, and enter their question in a text box. For example, they might enter, "Please explain the basics of quantum mechanics." This question data, along with a timestamp and user ID, is sent to the server.

[1429] Input: Question (e.g., "Please explain the basics of quantum mechanics"), timestamp, user ID

[1430] Output: Sending query data to the server

[1431] Step 3:

[1432] The server receives the submitted question data and analyzes the question content using a natural language processing (NLP) model. During the analysis, important keywords and the user's intent are extracted. For example, keywords such as "quantum mechanics" and "fundamentals" may be extracted. Based on these analysis results, the server searches the database for relevant lecture content.

[1433] Input: Question data (text, timestamp, user ID)

[1434] Output: Keyword extraction, search for related lecture content

[1435] Step 4:

[1436] The server generates the most suitable response pattern from the search results. For example, it might select a text explanation or video on "Fundamentals of Quantum Mechanics" for beginners. These responses are generated in various formats such as text, audio, and video, and sent to the infotainment system.

[1437] Input: Analysis results, related lecture content

[1438] Output: Generation of optimal response patterns (text, audio, video)

[1439] Step 5:

[1440] The infotainment system receives the response patterns and presents them to the user. For example, a text explanation may be displayed on the screen, and videos or audio may be played as needed. The user can review the provided answers and continue learning.

[1441] Input: Response patterns (text, audio, video)

[1442] Output: Providing answers to the user (display, play)

[1443] Step 6:

[1444] Users can provide feedback on the provided answers. This feedback data is then sent back to the server via the infotainment system. The server stores this feedback data and uses it to improve the accuracy of future answers.

[1445] Input: User Feedback

[1446] Output: Feedback data is stored on the server and used for future improvements.

[1447] This will enable a system that allows passengers to efficiently learn while inside an autonomous vehicle.

[1448] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1449] This invention provides more effective and personalized learning support by combining an emotion engine with a learning support system that uses educational applications. The system consists of user terminals, a server, and a communication network connecting them.

[1450] Initial setup

[1451] Settings on the user terminal

[1452] First, users install the educational application on their smartphone, tablet, or PC. After installation, users launch the application and enter the necessary information to create an account. Once account creation and login are complete, the user's device connects to the server and synchronizes with the latest lecture data from the server.

[1453] Recognition of emotions and input of questions

[1454] Facial expression and voice analysis

[1455] The application is equipped with a camera and microphone, and includes an emotion engine that uses these to recognize emotions from the user's facial expressions and voice. When the user interacts with the application, the camera and microphone automatically activate and analyze the user's facial expressions and voice tone.

[1456] User question input

[1457] When a user has a question during their studies, they select a specific subject or topic, enter the question into a text box, and press the submit button. At this point, the emotion engine analyzes the user's emotions in real time and adds that data to the question data.

[1458] User terminal operation

[1459] The terminal sends the entered question data and sentiment data to the server. This question data includes text, sentiment data, and metadata (such as a timestamp and user ID).

[1460] Question analysis and answer generation

[1461] Natural language processing and sentiment analysis

[1462] The server receives question data sent from the user's terminal and analyzes the question content using a natural language processing (NLP) model. Furthermore, it also analyzes sentiment data to understand the user's emotional state (e.g., confusion, dissatisfaction, excitement).

[1463] Search related lecture content

[1464] The server searches the database for relevant lecture content based on the analysis results of the NLP model and sentiment data. This search considers not only teaching materials and past answers related to the question, but also answers that are appropriate for the user's emotional state.

[1465] Generating response patterns

[1466] The server generates the optimal response pattern based on the searched lecture content. This response pattern is adjusted according to the user's emotional state and is provided in text, audio, and video formats.

[1467] Providing answers to users

[1468] User terminal operation

[1469] The device analyzes the response data received from the server and sets the display format to an appropriate one. For example, response patterns that include explanations about "how to solve quadratic equations" are displayed in a format that is easy to understand and tailored to the user's level of confusion. Audio and video playback are also provided as needed.

[1470] User feedback

[1471] Users enter their satisfaction level and additional feedback regarding the provided answers. This feedback data includes evaluations of the quality of the answers and responses regarding their feelings.

[1472] Gathering feedback and improving the system

[1473] Server operation

[1474] The device sends user feedback data to the server. The server stores the received feedback data in a database and uses it to improve the accuracy of future responses and enhance the overall system. The feedback data also includes information about emotional states, which allows the system to provide more adaptive learning support.

[1475] Specific example

[1476] For example, suppose a user asks a question about "how to solve a quadratic equation" in mathematics. The user types "how to solve a quadratic equation" into a text box and presses the submit button. The user's terminal sends this question data and emotion data indicating the user's state of confusion to the server. The server uses an NLP model to extract the keywords "quadratic equation" and "how to solve" and searches its database for relevant lecture content. Furthermore, it generates an answer pattern that is tailored to the user's state of confusion, with a detailed and easy-to-understand explanation. For example, it might generate text such as "A quadratic equation is of the form ax^2 + bx + c = 0, and the formula for solving it is...", along with gentle audio and corresponding video data, and send them to the user's terminal. The user's terminal receives this and displays or plays it for the user.

[1477] This system allows users to effectively and quickly resolve their questions and learn while receiving emotionally tailored support.

[1478] The following describes the processing flow.

[1479] Step 1:

[1480] Users install the educational application on their smartphones, tablets, or computers. After installation, users launch the app, enter their email address and password to create an account, and log in.

[1481] Step 2:

[1482] The device sends login information to the server. The server authenticates the received information and returns the authentication result to the device. If the login is successful, the device displays the home screen.

[1483] Step 3:

[1484] Upon initial login, the device requests the latest lecture data from the server. The server retrieves the lecture data from the database and sends it to the device. The device saves the received lecture data to local storage and displays a "synchronization complete" notification to the user.

[1485] Step 4:

[1486] The user operates the app to select a subject or topic they have questions about during their studies. After selecting, they enter their specific question in the text box and press the submit button.

[1487] Step 5:

[1488] The terminal sends the user-entered question data to the server. This question data includes the entered text, sentiment data, and metadata (such as a timestamp and user ID). Sentiment data is generated by an emotion engine that analyzes the user's facial expressions and voice.

[1489] Step 6:

[1490] The server passes the received question data to a natural language processing (NLP) model. The NLP model analyzes the question content and extracts keywords and the intent behind the question.

[1491] Step 7:

[1492] The server receives emotional data analyzed by the emotion engine. This allows it to understand the user's current emotional state (e.g., confused, frustrated, excited).

[1493] Step 8:

[1494] The server searches the database for relevant lecture content based on the analysis results of the NLP model and sentiment data. This search includes teaching materials, whiteboard notes, and past questions and answers related to the question.

[1495] Step 9:

[1496] The server generates the optimal response pattern based on the user's emotional state. For example, for a confused user, it generates text, audio, and video data that includes more detailed and thorough explanations.

[1497] Step 10:

[1498] The server sends the generated response pattern to the terminal. The terminal analyzes the received response data and displays or plays it in the appropriate format. For example, it might display text such as, "A quadratic equation is of the form ax^2 + bx + c = 0, and the formula for solving it is..." and play audio or video as needed.

[1499] Step 11:

[1500] Users rate the provided answers and provide feedback. Ratings are given in the format of "Good" or "Bad."

[1501] Step 12:

[1502] The device sends user feedback data to the server. The server stores the received feedback data in a database and uses it to improve the accuracy of future responses and enhance the overall system. The feedback data also includes information about emotional states, allowing the system to provide more adaptive learning support.

[1503] Specific example:

[1504] 1. User input and emotion recognition

[1505] When a user asks a question about "how to solve a quadratic equation," they type "how to solve a quadratic equation" into the text box and press the submit button. The emotion engine recognizes the user's state of confusion, and the emotion data is sent to the server along with the question data.

[1506] 2. Question analysis and answer generation

[1507] The server uses an NLP model to extract the keywords "quadratic equation" and "solution method" from the question and searches for related lecture content. It generates answer patterns that include detailed and easy-to-understand explanations for confused users and sends them to the terminal.

[1508] 3. Providing responses and feedback to users

[1509] The terminal displays the received response data, showing the text "The quadratic equation is in the form ax^2 + bx + c = 0, and the quadratic formula is..." along with the corresponding audio or video. The user is satisfied with the response and sends a positive rating and feedback such as "very easy to understand." The server saves this feedback data and uses it to improve the response system in the future.

[1510] This system allows users to receive effective learning support tailored to their emotions.

[1511] (Example 2)

[1512] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1513] Traditional educational support systems provided uniform answers without considering the individual emotional states of users, resulting in limited learning effectiveness. Furthermore, they lacked mechanisms to incorporate feedback based on user emotions. This led to problems in quickly and appropriately resolving users' questions and concerns.

[1514] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for transmitting question data and emotion data input from the user terminal to the data server; means for analyzing the question data using a natural language processing model and searching for relevant lecture content; means for analyzing the emotion data and understanding the user's emotional state; means for generating an optimal answer pattern based on the retrieved lecture content and emotional state; means for transmitting the generated answer pattern to the user terminal in the form of text, audio, and video; means for displaying or playing back the received answer pattern on the user terminal; and means for analyzing the user's facial expressions and voice and acquiring emotion data. This makes it possible to provide optimal learning support tailored to the user's individual emotional state.

[1515] An "educational program" is a software application specifically designed for learning support and educational purposes.

[1516] A "user terminal" refers to a digital device, such as a smartphone, tablet, or personal computer, used to install and operate educational programs.

[1517] A "data server" is a server system that stores and manages question data, sentiment data, and lecture content data, and communicates with user terminals.

[1518] "Question data" refers to text data containing questions and doubts that users have while learning.

[1519] "Emotional data" refers to data that indicates the emotional state (e.g., confusion, anxiety, excitement) of a user, analyzed from their facial expressions and voice.

[1520] A "natural language processing model" is a machine learning model used to analyze the meaning of text data and extract keywords and syntax.

[1521] An "answer pattern" is the optimal answer format (text, audio, or video) for a user's question, generated based on question data and sentiment data.

[1522] "Means for analyzing user facial expressions and voice" refers to a combination of software and hardware that uses cameras and microphones to analyze the user's facial expressions and voice tone, and to acquire emotional data.

[1523] "Feedback data" refers to data that includes satisfaction levels, additional opinions, and evaluations entered by users regarding the provided responses.

[1524] "Lecture content data" refers to digital content such as teaching materials, lecture notes, and video lectures necessary for learning within an educational program.

[1525] This invention provides more effective and personalized learning support by combining an emotion engine with a learning support system that uses educational programs. This system consists of user terminals, a data server, and a communication network connecting them.

[1526] First, users install the educational program on their smartphones, tablets, or PCs. The program includes various learning materials and functions to support their studies. After installation, users launch the program and enter the necessary information to create an account (name, email address, password, etc.). Once account creation and login are complete, the user's device connects to the data server and synchronizes with the latest lecture content data. Specifically, it communicates with the server using REST API and WebSocket technologies to save new lecture data and updates locally.

[1527] When a user begins learning, the camera and microphone automatically activate to analyze the user's facial expressions and voice tone in real time. OpenCV and TensorFlow are used for facial recognition, and MLKit and speech recognition APIs are used for voice analysis. This allows the user's emotional state (e.g., confusion, frustration, excitement) to be captured. When the user selects a specific subject or topic, enters their questions into a text box, and presses the submit button, the device sends the text data and the real-time emotional data to the data server.

[1528] The data server analyzes the received question data and sentiment data using natural language processing (NLP) models. NLP models such as Transformers and BERT are used to extract the meaning of the questions and important keywords. Furthermore, sentiment data is also analyzed to understand the user's emotional state. Based on the analysis results, the data server searches the database for relevant lecture content. This search includes textbook data, past Q&A, and the user's learning history. Based on the search results and the emotional state, the server generates the optimal response pattern. Responses are provided in text, audio, and video formats.

[1529] The generated answer patterns are sent to the user's device, which receives and displays or plays them. Specifically, HTML, CSS, and JavaScript are used to provide an appropriate UI / UX, presenting the answers in a user-friendly format. For example, if a user asks about "how to solve a quadratic equation" and is confused, an answer adjusted to be detailed and easy to understand will be provided. For instance, an explanation such as "A quadratic equation is in the form ax^2 + bx + c = 0, and the quadratic formula is..." or a corresponding video will be played.

[1530] Users provide satisfaction ratings and additional feedback on the provided answers. This feedback data is then sent back to the data server via the device. The received feedback data is stored in a database and used to improve the accuracy of future answers and enhance the overall system. Because the feedback data also includes information about emotional states, the system can increasingly provide adaptive learning support to users.

[1531] Examples of prompt statements

[1532] For example, the following prompt statements are possible.

[1533] User: I don't know how to solve quadratic equations.

[1534] Emotional state: Confusion

[1535] Based on this prompt, the data server generates an appropriate response pattern and provides optimized support to resolve the user's confusion.

[1536] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1537] Step 1: Initial Setup

[1538] Users install the educational program on their smartphones, tablets, or PCs. After installation, users launch the program and enter the necessary information to create an account (name, email address, password, etc.). An account is generated based on the entered data and saved to the data server. Once logged in, the user's device connects to the data server and synchronizes the latest lecture content data. Specifically, data communication is performed using REST APIs or WebSockets, and the lecture data is saved locally.

[1539] Input: User information (name, email address, password), login information

[1540] Output: Account data generation, lecture content data synchronization

[1541] Step 2: Start facial expression and voice analysis

[1542] When a user begins learning, the device's camera and microphone automatically activate. The device uses these devices to analyze the user's facial expressions and voice tone in real time using an emotion engine. OpenCV and TensorFlow are used for facial recognition, and MLKit and speech recognition APIs are used for voice analysis. At this stage, the user's facial expression and voice tone data are collected and output as emotion data in real time.

[1543] Input: User's facial expression data, voice data

[1544] Output: Emotional data (e.g., confused, dissatisfied, excited)

[1545] Step 3: Enter and submit your question.

[1546] The user selects a specific subject or topic and enters their question into a text box. When the user presses the submit button, the device sends the text data and real-time sentiment data to a data server. The submitted data includes the question, sentiment data, a timestamp, and the user ID. The submitted data is received and stored on the server.

[1547] Input: User's question (text data), sentiment data

[1548] Output: Sending question data and sentiment data to the data server.

[1549] Step 4: Analyzing Question Data

[1550] The data server analyzes the received question data and sentiment data using natural language processing (NLP) models. NLP models such as Transformers and BERT are used to extract the meaning of the question and important keywords. In addition, sentiment data is analyzed to identify the user's emotional state. The analysis results are output as a series of keywords and sentiment state data.

[1551] Input: Question data, sentiment data

[1552] Output: Analysis results (keywords, emotional state)

[1553] Step 5: Search for related lecture content

[1554] The data server searches the database for relevant lecture content based on the analysis results. The database contains textbook data, past Q&A, and user learning history. As a result of the search, the relevant lecture content data is identified and retrieved.

[1555] Input: Analysis results (keywords, emotional state)

[1556] Output: Related lecture content data

[1557] Step 6: Generating response patterns

[1558] The data server generates the optimal response pattern based on the retrieved lecture content data. The response is generated in text, audio, and video formats, depending on the user's emotional state (e.g., detailed and gentle tone if confused). At this stage, the final response pattern is determined and stored on the data server.

[1559] Input: Related lecture content data, emotional state

[1560] Output: Response patterns (text, audio, video)

[1561] Step 7: Provide answer patterns

[1562] The user terminal receives response patterns generated from the data server. Based on the received response patterns, the terminal displays or plays content for the user in an appropriate format. Specifically, it uses HTML, CSS, and JavaScript to provide a user-friendly UI / UX. For example, it plays videos explaining how to solve quadratic equations or providing example problems.

[1563] Input: Response patterns (text, audio, video)

[1564] Output: Learning support for the user (display, playback)

[1565] Step 8: Collecting and submitting feedback

[1566] Users provide satisfaction ratings and additional feedback on the provided answers. This feedback data is sent to a data server via the device. The input feedback data includes evaluations of the quality of the answers and responses to emotions.

[1567] Input: User feedback (satisfaction level, additional comments)

[1568] Output: Sending feedback data to the data server

[1569] Step 9: Feedback Analysis and System Improvement

[1570] The data server stores the received feedback data in a database. The feedback data is analyzed and used to improve future response accuracy and the overall system. Machine learning algorithms are used to analyze the feedback data and improve the system's accuracy and performance.

[1571] Input: Feedback data

[1572] Output: Analysis results, system improvement data

[1573] (Application Example 2)

[1574] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1575] Conventional learning support systems unilaterally provided information without considering the user's emotional state, making it difficult to provide adaptive support tailored to each learner's level of understanding and emotions. Furthermore, in physical stores, customers often felt an emotional distance when asking questions about products, resulting in a lack of interactive customer support.

[1576] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for transmitting question data input from a user terminal to the server, means for analyzing the question data using a natural language processing model and searching for relevant lecture content, means for generating an optimal answer pattern based on the searched lecture content, means for analyzing the user's emotions in real time using an emotion engine with a smart device and adding the analyzed emotion data to the question data, and means for adjusting the generated answer pattern according to the user's emotional state. This makes it possible to provide personalized learning support according to the user's emotional state. Furthermore, even in physical stores, it becomes possible to provide product descriptions and support according to the customer's emotions, thereby improving customer satisfaction.

[1577] An "educational application" is software used by users for learning, providing lecture data and learning materials.

[1578] A "user terminal" refers to a device, such as a smartphone, tablet, or personal computer, on which a user installs and uses educational applications.

[1579] A "server" is a computer system that receives data sent from a user terminal and performs analysis and processing on it.

[1580] "Question data" refers to data that includes the questions that users have and enter during their learning process.

[1581] A "natural language processing model" refers to algorithms and techniques for analyzing natural language and understanding its meaning.

[1582] "Relevant lecture content" refers to appropriate teaching materials and lecture content in response to the question entered by the user.

[1583] An "answer pattern" refers to the format or method used to provide the most appropriate answer to a question.

[1584] An "emotion engine" refers to software and hardware that recognizes and analyzes emotions from a user's facial expressions and voice.

[1585] "Emotional data" refers to data that indicates the user's emotional state, as analyzed by the emotion engine.

[1586] A "smart device" refers to a device that has internet connectivity and is used by a user, such as smart glasses or a smartphone.

[1587] "Feedback data" refers to data that includes evaluations and opinions from users regarding the responses they provide.

[1588] "Synchronization" refers to the process of ensuring that the latest data is matched between the user's terminal and the server.

[1589] "Real-time" means that processing and responses to user actions occur immediately.

[1590] The above are definitions of the important terms included in the scope of the claim.

[1591] This invention applies to a learning support system using educational applications and a customer support system using smart devices in physical stores. User terminals include smartphones, tablets, and personal computers, and the system is built by installing educational applications on these devices. In physical stores, smart devices such as smart glasses are used.

[1592] System Configuration

[1593] This system consists of the following components:

[1594] 1. User terminal: A device equipped with a camera and microphone, and with educational applications installed.

[1595] 2. Server: A computer system that receives data sent from user terminals, performs analysis, and generates responses.

[1596] 3. Emotion Engine: Software that analyzes the user's facial expressions and voice to generate emotion data.

[1597] 4. Natural Language Processing Model (NLP Model): An analytical tool that analyzes user questions and searches for relevant lecture content or product information.

[1598] 5. Database: A storage system that stores lecture content and product information.

[1599] 6. Smart devices: Devices that display information to customers in physical stores, such as smart glasses.

[1600] Flow of operations

[1601] 1. Initial setup of the user terminal:

[1602] After the user installs the application and creates an account upon their first login, they synchronize with the server to retrieve the latest lecture data.

[1603] 2. Emotion recognition and data collection:

[1604] When the application is launched, the camera and microphone are automatically activated, and the emotion engine analyzes the user's facial expressions and voice to generate emotion data. When the user enters a question, the question data and emotion data are sent to the server.

[1605] 3. Question analysis and answer generation:

[1606] The server analyzes the received question data using a natural language processing model and searches the database for relevant lecture content and product information. Based on the search results, it generates the optimal response pattern. This response pattern is adjusted according to the user's emotional state.

[1607] 4. Display the answer:

[1608] The generated response patterns are sent to the user's device in text, audio, or video format. The user's device displays or plays the received responses in the appropriate format.

[1609] For example, if the question is "How do I use this product?" and the user is confused, a detailed and easy-to-understand explanation will be provided.

[1610] 5. Gathering feedback:

[1611] User feedback is sent from the device to the server and stored as feedback data. This data is used to improve the accuracy of the system.

[1612] Specific usage examples

[1613] In the educational application, when a user asks "How to solve a quadratic equation," the emotion engine detects confusion and provides a detailed and easy-to-understand explanation.

[1614] In smart shopping at physical stores, when a customer asks "How do I use this product?" through smart glasses, the store provides appropriate product information and instructions in real time.

[1615] Examples of prompts to input into a generative AI model

[1616] If you ask the "Smart Shopping Assistant" in-store, "How do I use this product?", please generate a detailed explanation of how to use it based on emotion recognition.

[1617] If the customer is confused, please explain gently and in detail.

[1618] If the customer is excited, get straight to the point.

[1619] This system allows users to receive excellent support tailored to their emotions, which is expected to improve learning efficiency and customer satisfaction.

[1620] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1621] Step 1:

[1622] Initial settings:

[1623] Users first install the educational application on their device, such as a smartphone, tablet, or PC. After installation is complete, they launch the application, enter the necessary information to create an account, and then create an account and log in.

[1624] Input: User account information (name, email address, etc.)

[1625] Output: A user account is created on the server and synchronized with the user's terminal.

[1626] Specific operation: When a user enters the required information on the account creation screen and presses the data submission button, that data is sent to the server and registered as a new account.

[1627] Step 2:

[1628] Data synchronization from the server:

[1629] Upon initial login, the user's device connects to the server, and the latest lecture and course material data is synchronized from the server.

[1630] Input: Latest lecture content data stored on the server

[1631] Output: The latest lecture content data is downloaded to the user's terminal.

[1632] Specific operation: After the user presses the login button, a request is sent from the terminal to the server, the server sends the latest lecture data to the terminal, and the data is updated on the terminal.

[1633] Step 3:

[1634] Recognition of emotions:

[1635] When a user begins learning, the camera and microphone on the user's device are automatically activated, and the emotion engine analyzes the user's facial expressions and voice to generate emotion data.

[1636] Input: User's facial expressions and voice data

[1637] Output: Analyzed emotion data (e.g., confusion, excitement, etc.)

[1638] Specific operation: The camera and microphone capture the user's face, and that data is analyzed in real time by an emotion engine to identify the user's emotions.

[1639] Step 4:

[1640] Entering question data:

[1641] When a user has a question while studying, they select a specific subject or topic, enter their question into a text box, and press the submit button.

[1642] Input: Question entered by the user

[1643] Output: Questionnaire data (text format) and corresponding sentiment data

[1644] Specific operation: When a user enters a question in a text box and presses the submit button, the question content and sentiment data are sent to the server.

[1645] Step 5:

[1646] Question analysis on the server:

[1647] The server analyzes the received question data using a natural language processing model and searches the database for relevant lecture content.

[1648] Input: Questionnaire data and sentiment data

[1649] Output: Related lecture content data

[1650] Specific operation: The server analyzes the question data, searches the database for relevant textbooks and lecture content, and retrieves the appropriate information.

[1651] Step 6:

[1652] Generating response patterns:

[1653] The server generates the optimal response pattern based on the search results and adjusts the format of the response according to the user's emotional state.

[1654] Input: Relevant lecture content data, user sentiment data

[1655] Output: Adjusted response patterns (text, audio, video, etc.)

[1656] Specific operation: On the server side, the program decides whether to make the response detailed or concise based on the user's emotions, and then outputs it as text or audio data.

[1657] Step 7:

[1658] Providing responses to user terminals:

[1659] The generated response patterns are sent to the user's device in text, audio, or video format and displayed or played back in the appropriate format.

[1660] Input: Adjusted response patterns (text, audio, video, etc.)

[1661] Output: The answer that the user sees or hears on their device.

[1662] Specific operation: The user terminal analyzes the received response data, displays it in a user-friendly format, and also plays the audio.

[1663] Step 8:

[1664] Gathering feedback:

[1665] Users enter their satisfaction level with the provided answers and any additional feedback, then submit it to the server.

[1666] Input: User feedback data

[1667] Output: Feedback data stored on the server

[1668] Specific operation: When a user enters their opinions and ratings on the feedback screen and presses the submit button, that data is saved to the server.

[1669] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1670] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1671] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1672] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1673] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1674] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1675] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1676] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1677] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1678] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1679] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1680] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1681] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1682] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1683] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1684] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1685] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1686] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1687] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1688] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1689] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1690] The following is further disclosed regarding the embodiments described above.

[1691] (Claim 1)

[1692] A system including a user terminal with an educational application installed and a server,

[1693] A means of sending question data entered from the user terminal to the server,

[1694] A method for analyzing question data using a natural language processing model and searching for relevant lecture content,

[1695] A means of generating the optimal answer pattern based on the searched lecture content,

[1696] A means for sending the generated response patterns to the user's terminal in text, audio, and video formats,

[1697] A system that includes means for displaying and playing back response patterns received on a user's terminal.

[1698] (Claim 2)

[1699] The system according to claim 1, comprising means for storing user-submitted feedback data on a server and using it to improve the accuracy of future responses.

[1700] (Claim 3)

[1701] The system according to claim 1, further comprising means for synchronizing the latest lecture content data from the server when the user terminal first logs in.

[1702] "Example 1"

[1703] (Claim 1)

[1704] A system including a terminal with an educational program installed and an information processing device,

[1705] A means for transmitting inquiry data entered from a terminal to an information processing device,

[1706] A means of analyzing inquiry data using a generated AI model and searching for relevant educational content,

[1707] A means of generating the optimal response format based on the searched educational content,

[1708] A means of sending the generated response format to the terminal in text, audio, and video formats,

[1709] A system that includes means for displaying and playing back the response format received on a terminal.

[1710] (Claim 2)

[1711] The system according to claim 1, comprising means for storing evaluation data submitted by a user in an information processing device and using it to improve the accuracy of future responses.

[1712] (Claim 3)

[1713] The system according to claim 1, further comprising means for synchronizing the latest educational content data from an information processing device when the terminal first accesses it.

[1714] "Application Example 1"

[1715] (Claim 1)

[1716] A system including a user terminal with an educational application installed and a server,

[1717] A means of sending question data entered from the user terminal to the server,

[1718] A method for analyzing question data using a natural language processing model and searching for relevant lecture content,

[1719] A means of generating the optimal answer pattern based on the searched lecture content,

[1720] A means for sending the generated response patterns to the user's terminal in text, audio, and video formats,

[1721] A means of displaying and playing back response patterns received on the user's terminal,

[1722] A means of displaying an educational application on the infotainment system so that passengers can learn while inside an autonomous vehicle,

[1723] A means of receiving question data from passengers and sending it to a server,

[1724] A system that includes means for appropriately displaying or playing back the provided lecture content to passengers.

[1725] (Claim 2)

[1726] The system according to claim 1, comprising means for storing user-submitted feedback data on a server and using it to improve the accuracy of future responses.

[1727] (Claim 3)

[1728] The system according to claim 1, further comprising means for synchronizing the latest lecture content data from the server when the user terminal first logs in.

[1729] "Example 2 of combining an emotion engine"

[1730] (Claim 1)

[1731] A system including a user terminal with an educational program installed and a data server,

[1732] A means for transmitting question data and sentiment data entered from a user terminal to a data server,

[1733] A method for analyzing question data using a natural language processing model and searching for relevant lecture content,

[1734] A means of analyzing emotional data to understand the user's emotional state,

[1735] A means for generating the optimal response pattern based on the retrieved lecture content and emotional state,

[1736] A means for sending the generated response patterns to the user's terminal in text, audio, and video formats,

[1737] A means of displaying and playing back response patterns received on the user's terminal,

[1738] A system that includes means for analyzing the user's facial expressions and voice to acquire emotional data.

[1739] (Claim 2)

[1740] The system according to claim 1, comprising means for storing user-submitted feedback data in a data server and using it to improve the accuracy of future responses.

[1741] (Claim 3)

[1742] The system according to claim 1, further comprising means for synchronizing the latest lecture content data from a data server when the user terminal first logs in.

[1743] "Application example 2 when combining with an emotional engine"

[1744] Additional claims

[1745] (Claim 1)

[1746] A system including a user terminal with an educational application installed and a server,

[1747] A means of sending question data entered from the user terminal to the server,

[1748] A method for analyzing question data using a natural language processing model and searching for relevant lecture content,

[1749] A means of generating the optimal answer pattern based on the searched lecture content,

[1750] A means for sending the generated response patterns to the user's terminal in text, audio, and video formats,

[1751] A means of displaying and playing back response patterns received on the user's terminal,

[1752] A means of using a smart device to analyze a user's emotions in real time with an emotion engine and adding the analyzed emotion data to the question data,

[1753] A system that includes means for adjusting the generated response patterns according to the user's emotional state.

[1754] (Claim 2)

[1755] The system according to claim 1, comprising means for storing user-submitted feedback data on a server and using it to improve the accuracy of future responses.

[1756] (Claim 3)

[1757] The system according to claim 1, further comprising means for synchronizing the latest lecture content data from the server when the user terminal first logs in. [Explanation of Symbols]

[1758] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A system including a user terminal with an educational application installed and a server, A means of sending question data entered from the user terminal to the server, A method for analyzing question data using a natural language processing model and searching for relevant lecture content, A means of generating the optimal answer pattern based on the searched lecture content, A means for sending the generated response patterns to the user's terminal in text, audio, and video formats, A system that includes means for displaying and playing back response patterns received on a user's terminal.

2. The system according to claim 1, further comprising means for storing user-submitted feedback data on a server and using it to improve the accuracy of future responses.

3. The system according to claim 1, further comprising means for synchronizing the latest lecture content data from the server when the user terminal first logs in.

Images