system

Abstract

We provide the system. [Solution] A means of providing educational programs via information terminals, A means of recording and analyzing the learning progress of users, A means of providing optimal feedback and next-priority learning content based on the analysis results, Means of presenting educational information using multiple languages, A means of pre-storing information necessary to enable operation even in environments with unstable internet connections, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Children in regions and families with insufficient educational resources tend not to receive sufficient education, which has a negative impact on the development of society as a whole and the cultivation of talents. In particular, diverse language environments and unstable Internet connections are further widening the educational gap. Existing systems have difficulty providing feedback optimized for individual learners and the next learning steps, and there is a problem that the quality of education cannot be maintained uniformly.

Means for Solving the Problems

[0005] This invention is a system that provides educational programs via an information terminal and has the function of recording and analyzing the user's learning progress. Based on the analysis results, it presents optimal feedback and the next learning content. Furthermore, it supports multilingual environments by presenting educational information in multiple languages ​​and pre-stores necessary information on the terminal so that it can operate even if the internet connection is unstable. This system aims to improve the quality of education and correct educational disparities by enabling smooth interaction with the user using natural language processing and presenting recommendations based on interests and abilities.

[0006] An "information terminal" is an electronic device used by users to access learning programs, and is a device that has the function of sending, receiving, and recording data.

[0007] An "educational program" is content designed for the purpose of educating learners, and includes teaching materials, exercises, and interactive teaching elements.

[0008] "Learning progress" refers to information that indicates the progress and level of understanding a learner has made as they work through an educational program.

[0009] "Feedback" refers to information that provides evaluations and suggestions for improvement regarding a learner's responses and actions, with the aim of improving learning.

[0010] "Multiple languages" refers to a set of different languages, indicating that the system can be used in diverse language environments.

[0011] "Pre-storing" refers to saving necessary data to the device in advance so that it can be used even when offline.

[0012] "Natural language processing" is a technology that enables computers to understand and process natural human language, and is used to support interaction with users.

[0013] "Recommended content" refers to information indicating the next learning steps and content that are considered optimal based on the learner's interests and abilities. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention is a system that records and analyzes learners' progress in real time by utilizing educational programs provided via information terminals. This system records user activity, generates optimal feedback based on the data, and automatically suggests the next learning objectives. Furthermore, it supports multiple languages, making it effective in multilingual environments. It also features an offline mode, allowing for uninterrupted learning even with minimal internet connectivity.

[0036] First, the user accesses the application using their device and logs into their individual user account. At this point, the server provides a highly relevant learning program based on the user's profile information. For example, if there is a primary school student user, the server generates a curriculum and materials appropriate to the user's grade level and sends them to the device.

[0037] The user then selects a specific educational program and begins learning. As learning progresses, the device records the user's responses and actions in real time. This data is sent to a server, which uses generative AI to analyze the user's learning progress and understanding. For example, if the user continues to answer multiple multiple-choice questions correctly, the server evaluates their understanding and suggests more difficult questions or new learning topics.

[0038] The feedback generated based on the analysis results is presented to the user through the device. This feedback includes specific areas for improvement and serves as a guide for the user to take the next step. For example, if a user answers a question incorrectly, the device provides hints and additional materials to help them understand the mistake and support their learning.

[0039] Furthermore, using natural language processing technology, the device interacts with the user, providing interactions that facilitate the learning process. By inputting questions, the system can provide appropriate answers and relevant information.

[0040] As described above, the present invention is a system that aims to provide individualized learning experiences and ensure high-quality educational opportunities in diverse environments.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The user activates their information terminal and logs into the educational application. The server receives the user's authentication information and authenticates the user by referring to the database. If authentication is successful, the server sends the user's learning history and profile information to the terminal.

[0044] Step 2:

[0045] The terminal displays a selection menu of learning programs based on user information received from the server. The user selects a program based on their learning objectives and interests.

[0046] Step 3:

[0047] When a user selects a learning program, the device downloads the necessary learning materials and exercises. The server then sends important information to the device to enable offline use.

[0048] Step 4:

[0049] The user starts the learning program, and the device records the user's responses and actions in real time. This information is sent to the server as learning progress.

[0050] Step 5:

[0051] The server analyzes the received progress data and uses AI technology to evaluate the user's understanding. The next step is to prepare to generate appropriate learning content and feedback.

[0052] Step 6:

[0053] The server-generated feedback is sent to the terminal and presented to the user. This feedback may include analysis of incorrect answers, suggestions for improvement, and additional learning materials.

[0054] Step 7:

[0055] The user reviews the feedback, views explanations as needed, or selects a new learning program to start again. The device then monitors the user's activity again and prepares for the next learning step.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] Traditional education systems lacked sufficient feedback tailored to individual user progress and understanding, suffered from a shortage of multilingual learning materials, and made it difficult to continue learning in environments with unstable internet connections. Furthermore, interaction with users was not sufficiently smooth, highlighting the need for a system that provides a personalized learning experience.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for providing educational programs via an information processing device, means for recording the user's learning progress and analyzing it with a generative AI model, and means for optimizing the content to be learned next using prompt sentences in the generative AI model. This enables personalized feedback and suggestions for the next learning content for each user, realizing a stable multilingual education and a learning environment that can be continued even offline.

[0061] An "information processing device" is an electronic device used for inputting, processing, storing, and outputting data, and includes a terminal that accepts user input.

[0062] An "educational program" is a software application that includes teaching materials and assignments designed for learners to acquire specific knowledge or skills.

[0063] "User learning progress" refers to information that indicates the degree of learning and the progress of acquisition achieved by users through the educational program.

[0064] A "generative AI model" is an algorithm designed to analyze data using artificial intelligence technology and make predictions or suggestions for a specific task.

[0065] "Feedback" refers to information provided as guidance, evaluation, or improvement based on the user's actions and results.

[0066] A "prompt" is a text-based instruction or question that is input to a generative AI model to guide it to produce a specific output.

[0067] "Multilingual education" refers to a learning environment that includes educational content provided in multiple languages, accommodating learners who speak different languages.

[0068] An "offline learning environment" is an environment that allows learning activities to continue even when internet connectivity is unstable or unavailable.

[0069] This invention relates to an information processing system for providing personalized educational experiences. It realizes technology that effectively advances learning through interaction between servers, terminals, and users.

[0070] When a user connects via a terminal, the server functions as an information processing device, selecting the optimal educational program based on the user's profile information. This process utilizes a generative AI model, which analyzes the user's progress data to accurately suggest what they should learn next. The server also pre-stores necessary information to support offline learning, even in cases of unstable network connections.

[0071] The terminal is the primary hardware component that receives user input and runs educational programs. It starts the program selected by the user, records learning progress in real time, and sends data to the server. If the user answers incorrectly, the terminal provides hints and additional materials related to the question to support learning.

[0072] Users operate their devices and learn educational content provided through the interface. Multilingual support enables smooth learning even in different language environments. Furthermore, users can input questions into their devices using natural language processing technology to receive appropriate answers from the server.

[0073] As a concrete example, when an elementary school student uses a math learning program, the prompt they input to the generative AI model might be something like, "If the user answers four questions correctly in a row, what topic should be suggested next?" This ensures that educational content is provided that is tailored to the user's learning ability.

[0074] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0075] Step 1:

[0076] The user accesses the application using their device and enters their account information. The device sends this information to the server. The server authenticates the user and retrieves their profile information. This profile information includes their grade level and past learning data, which forms the basis for a customized educational program.

[0077] Step 2:

[0078] The server uses a generated AI model based on the user's profile information to select highly relevant educational programs. The server outputs these selection results to the terminal. In this process, the most suitable content is suggested according to the user's skill level and learning goals.

[0079] Step 3:

[0080] The user starts the educational program from their device. The device records the user's learning activity and sends the results to the server in real time. Specifically, the user's answers and learning speed are input, and data such as the user's level of understanding is generated based on this.

[0081] Step 4:

[0082] The server analyzes the training data received from the terminal using a generated AI model. The analysis results are generated as feedback and output to the terminal. This output specifically shows the user's progress and areas for improvement, and is designed to help them efficiently progress to the next learning step.

[0083] Step 5:

[0084] The device presents the user with feedback from the server. If the user answers a particular question incorrectly, the device provides additional hints and explanations to support learning and understanding. This allows the user to identify their weaknesses and learn more effectively.

[0085] (Application Example 1)

[0086] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0087] With advancements in educational technology, providing effective individualized instruction at home has become increasingly difficult. In particular, differences in home environment, language barriers, and unstable internet connections hinder the continuity of learning. Furthermore, real-time assessment of learners' progress and providing appropriate feedback are crucial challenges. Therefore, a new system is needed to meet these requirements while providing an individualized educational experience.

[0088] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0089] In this invention, the server includes means for providing educational programs via an information processing device, means for evaluating learning progress in real time and providing feedback through audio and visual means using intelligent devices that support learning in a home environment, and means for presenting educational information in multiple languages. This makes high-quality individualized learning possible even at home.

[0090] An "information processing device" is a computer or electronic device used for receiving, processing, and transmitting data, and is used as a means of providing educational programs.

[0091] An "educational program" refers to a curriculum and materials designed to help learners acquire specific skills and knowledge, and is content that enables optimal learning.

[0092] "Home environment" refers to the place and conditions in which learners receive education within their homes, and is a setting intended to realize educational support in diverse environments.

[0093] An "intelligent device" is a device equipped with audio and visual interfaces designed to provide educational support within the home, and is capable of real-time learning assessment.

[0094] "Learning support" refers to support activities aimed at enabling learners to efficiently acquire skills and knowledge, and includes providing specific feedback and learning materials.

[0095] "Real-time assessment" refers to a process of immediately observing, evaluating, and providing appropriate feedback on learners' activities.

[0096] "Feedback" is information provided based on a learner's activities and responses, which guides the learner in taking the next step.

[0097] "Multilingual" refers to a system that presents educational information in different languages ​​and can accommodate learners with diverse linguistic backgrounds.

[0098] This embodiment of the invention is a system that uses an information processing device to provide educational programs and evaluate learners' learning progress in real time. The server utilizes a database to manage user profiles in order to provide individualized educational content to each learner. This makes it possible to recommend optimal learning materials based on the learner's interests and level of understanding.

[0099] The device connects to an intelligent device and records the learner's real-time activity data. This data is sent to a server and analyzed in detail using a generative AI model. Based on the analysis results, the learner is provided with appropriate feedback in both audio and visual formats. The intelligent device has a multi-language interface and utilizes natural language processing technology to enable effective communication with the learner.

[0100] In a home environment, even with an unstable internet connection, intelligent devices can continue to support learning using pre-stored information. This system provides a more effective learning experience by offering recommendations based on learners' interests and skill levels when selecting educational programs.

[0101] A concrete example is when an intelligent device provides real-time feedback to a learner solving a math problem. It instantly determines whether the learner's answer is correct or incorrect, providing the next step if correct, or hints for further consideration if incorrect. This feedback is delivered verbally, which helps facilitate the learner's understanding.

[0102] An example of a prompt might be, "How would you encourage a child to move on to the next step when they make a mistake on a math problem?"

[0103] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0104] Step 1:

[0105] When a user logs in via a terminal, the server uses an information processing device to retrieve the user's profile from the database. The input is the user's login information, and the output is profile data including the user's interests and level of understanding. Based on this profile data, the server selects highly relevant learning content. Specifically, the server analyzes the user's learning history and past responses to prepare to suggest the most suitable learning materials for the user.

[0106] Step 2:

[0107] The terminal receives educational content transmitted from the server and presents it to the user in conjunction with intelligent devices. The input is content data from the server, and the output is learning materials presented to the user visually and audibly. Specifically, the terminal displays learning questions on its screen and provides navigation and question text audibly through an audio output device.

[0108] Step 3:

[0109] When a user works on a learning problem, the device records the user's answers in real time and sends them to the server as input. The output is the answer data, which is analyzed through communication with the server. Specifically, when a user solves a multiple-choice question, their selection is immediately uploaded to the server via the device.

[0110] Step 4:

[0111] The server uses a generated AI model to analyze the received user answer data and evaluate learning progress and comprehension. The input is the answer data, and the output is the comprehension score and recommended learning materials as evaluation results. Specifically, the server determines whether the answer is correct or incorrect, analyzes the cause of the error if it is incorrect, and determines what should be learned next.

[0112] Step 5:

[0113] The server generates optimal feedback based on the analysis results and provides it to the user via the terminal. The input is the analysis data, and the output is the audio and visual feedback presented to the user. Specifically, the feedback provides educational guidance to the user, including advice on the next learning step and hints about what was done incorrectly.

[0114] Step 6:

[0115] The user receives feedback from their device and starts the next learning session based on newly provided learning content. The input is feedback information, and the output is the selection of content for the new learning activity. In concrete terms, the user refers to new problems and materials and proceeds with learning independently.

[0116] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0117] This invention combines an emotion engine with a system that provides educational programs via an information terminal and records and analyzes the user's learning progress. The emotion engine recognizes the user's emotional responses and utilizes that information in the educational process. The aim of this system is to provide users with an appropriate learning experience and enhance the effectiveness of education.

[0118] First, the user logs into the application on their device and selects an educational program based on their profile. The server performs initial setup to understand the user's basic emotional information and activates the emotion engine. The emotion engine analyzes the user's facial expressions and voice through input devices such as the camera and microphone, and recognizes emotions in real time.

[0119] As the user progresses through the learning process, the device records not only data related to normal learning progress but also emotional data generated by the emotion engine. The server integrates this data to gain a comprehensive understanding of the user's learning progress. For example, if the user is feeling bored with the learning material, the emotion engine recognizes this and sends data to the server.

[0120] The server generates feedback based on sentiment analysis. For example, it instructs the device to present learning content in a more engaging way for users who are feeling bored. It can also provide additional explanations or hints if it detects that the user is confused.

[0121] Furthermore, the analysis results from the emotion engine are also used as data to form long-term learning patterns. In this way, the server aims to provide users with a continuous, personalized, and optimal learning experience, thereby improving the quality of education.

[0122] In this way, by adding an emotion recognition function using an emotion engine, the present invention generates flexible learning plans that are tailored to the user's emotional responses, thereby realizing efficient educational support.

[0123] The following describes the processing flow.

[0124] Step 1:

[0125] The user accesses the educational application using an information terminal and logs in. The server verifies the user's authentication information and completes the login process.

[0126] Step 2:

[0127] The server retrieves the user's profile information and sends a selection of optimal educational programs to the device based on their learning history. The device then displays these options, allowing the user to choose a program that suits their interests and needs.

[0128] Step 3:

[0129] Once the user selects a program, the device activates the emotion engine and confirms access permissions for the camera and microphone. This prepares the device to use the user's facial expressions and voice for real-time emotion analysis.

[0130] Step 4:

[0131] As the user learns, the emotion engine performs facial recognition and voice analysis to analyze the user's emotions. For example, if the user's facial expression is smiling, the engine will determine that the user is enjoying themselves.

[0132] Step 5:

[0133] The device simultaneously sends emotion engine data and user learning progress to the server. Based on this data, the server comprehensively analyzes the user's current learning and emotional state.

[0134] Step 6:

[0135] Based on the analysis results, the server generates emotionally responsive feedback. For example, if a user displays a confused expression, the server generates a helpful explanation and sends it to the device.

[0136] Step 7:

[0137] The device receives feedback from the server and presents the user with appropriate information. The user then uses this information to further their learning or make improvements.

[0138] Step 8:

[0139] Once a learning session ends, the server saves the data and prepares to analyze it for use in future learning plans. The emotional data generated by the emotion engine is accumulated as data for improvement in long-term learning.

[0140] (Example 2)

[0141] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0142] Traditional education systems provide feedback based on users' learning progress and performance, but they have faced the challenge of providing personalized learning experiences that take into account users' emotional states. Therefore, there is a need to immediately capture the user's feelings of interest, boredom, and lack of understanding during learning, and to flexibly adjust the content of the learning program based on that feedback.

[0143] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0144] In this invention, the server includes means for recognizing the user's emotional state and applying it to the educational process, means for providing an educational program via an information processing device and recording learning progress, and means for integrating and analyzing the recorded learning progress and emotional data. This makes it possible to optimize the learning program according to the user's emotional state and provide an individualized educational experience.

[0145] A "user" is an individual who uses the educational system to engage in learning activities.

[0146] "Emotional analysis means" refers to a technical device or method for detecting and interpreting an emotional state from a user's facial expressions or voice.

[0147] An "information processing device" is a computer or similar electronic device that collects, stores, processes, and communicates data.

[0148] An "educational program" refers to teaching materials and their structure that aim to improve the knowledge and skills of users.

[0149] "Learning progress" is an indicator that shows the degree or stage of learning that a user has achieved through an educational program.

[0150] "Feedback" refers to information and suggestions provided to users to communicate their learning results and level of understanding, and to encourage further improvement in their learning.

[0151] "Analytical tools" refer to the techniques and methods used to evaluate collected data and derive certain conclusions or views.

[0152] A "learning pattern" is a regularity or tendency that can be found based on a user's past learning behavior and results.

[0153] This invention is a system that provides users with personalized educational experiences by combining an information processing device and an emotion analysis means. The system consists of a server and terminals, where the server is responsible for data integration and analysis, and the terminals primarily provide educational programs through user interaction.

[0154] Hardware and software

[0155] The device has a built-in camera and microphone, and these hardware components are used to capture the user's facial expressions and voice data as a means of sentiment analysis. The video and voice data are sent to the sentiment analysis engine in real time and analyzed by OpenCV and speech recognition libraries. The server uses machine learning algorithms with programming languages ​​such as Python and R to integrate the collected sentiment data and learning progress data, and generates optimal feedback for the next step.

[0156] Specifically, the emotion analysis engine identifies the user's emotional state and sends this information to the server. The server then uses machine learning models to generate a personalized learning program based on the accumulated data. For example, if the server detects that the user is feeling bored, it will recommend new learning content that will pique the user's interest.

[0157] Specific example

[0158] For example, if a user is participating in a program to "learn about historical events," and the emotion analysis engine detects from the user's facial expressions that they are feeling bored, the server will sense this. The server will then analyze the user's past interests and present content such as "documentary about famous historical figures" on the device. This will rekindle the user's interest and allow them to continue learning effectively.

[0159] Example of a prompt

[0160] "If a user's facial expression shows confusion while they are solving a math problem, what additional explanation should be provided?"

[0161] In this way, the system enables users to receive a flexible and effective learning experience through dynamic feedback mediated by emotion analysis.

[0162] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0163] Step 1:

[0164] The user logs into their device and enters their profile information. This information is sent from the device to the server, where it is compared against the user's past learning history and registration information. Based on this data, the server generates personalized educational program suggestions and sends them to the device. The user is then provided with input for selection, and a list of program suggestions is displayed as output.

[0165] Step 2:

[0166] The user selects their desired educational program from the displayed program options. The selected program information is sent from the terminal to the server. The server receives the selection information, organizes the detailed data of the corresponding program, and sends it to the terminal. In response to the user's input to begin learning, the learning content is displayed as output.

[0167] Step 3:

[0168] The device activates its camera and microphone to perform emotion analysis as the user begins learning. The emotion analysis engine captures the user's facial expressions and voice, and analyzes the acquired data in real time. The captured data is acquired as input, and the user's emotional state is recognized as output.

[0169] Step 4:

[0170] The emotion data recognized by the emotion analysis engine is sent to the server via the terminal. The server integrates the emotion data and learning progress data and performs analysis. The input consists of emotion data and learning data, and based on this, the server evaluates the user's current level of understanding and emotional state, and generates appropriate feedback as output.

[0171] Step 5:

[0172] The server sends the generated feedback to the terminal and presents it to the user. Specifically, if it determines that the user is bored, it instructs the server to display engaging videos or additional content. The input for presenting the feedback is the evaluation result described above, and the output is the feedback displayed on the user's interface.

[0173] Step 6:

[0174] Long-term data collected through sentiment analysis is stored on a server and used to shape the user's learning patterns. The server analyzes this historical data and uses it as training data for a generative AI model that proposes future learning programs. The input is the stored learning history, and the output is a prediction of future learning patterns.

[0175] The data and processing obtained at each step are used to provide the most effective learning experience for users, ultimately aiming to improve the quality of education.

[0176] (Application Example 2)

[0177] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0178] Traditional educational support systems can provide feedback based on a user's learning progress, but they have struggled to provide adaptive support based on the user's emotional responses. As a result, the impact of the user's emotional aspects on learning effectiveness is often ignored, and it has been difficult to provide a learning experience optimized for each individual user. Furthermore, there is the challenge of limited operation in environments with unstable internet connections.

[0179] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0180] In this invention, the server includes means for providing educational programs via an information processing device, means for recording and analyzing the user's learning progress, means for providing the optimal response and next-priority educational content based on the analysis results, means for presenting educational information in multiple languages, means for pre-storing information to enable operation even under unstable network connection conditions, means for recognizing the user's emotional response using an emotion recognition device, and means for dynamically adjusting the educational program in accordance with the recognized emotional response. This makes it possible to provide an individualized learning experience that corresponds to the user's emotional state and improve the quality of education.

[0181] An "information processing device" is a device that has the function of providing and processing educational programs with users.

[0182] A "user" refers to a person who uses an educational program and whose learning progress is recorded.

[0183] "Learning progress" refers to information that indicates the state or stage of learning achieved by a user through an educational program.

[0184] "Analysis" is the process of analyzing user learning progress data and emotional response data to improve educational content.

[0185] "Feedback" refers to information and instructions provided to improve the user's learning experience.

[0186] "Educational information" refers to teaching materials and knowledge used to support users' learning.

[0187] A "network connection" is a connection that allows information processing devices to communicate data over the internet.

[0188] An "emotion recognition device" is a device that analyzes a user's facial expressions and voice to identify their emotions.

[0189] "Emotional response" refers to the changes or states of emotion that a user exhibits while learning.

[0190] An "educational program" refers to the learning content and educational methods provided through information processing devices.

[0191] This invention is an educational support system for providing users with an individualized learning experience. The system mainly consists of an information processing device, an emotion recognition device, and a server.

[0192] The information processing device primarily functions as a user interface and provides educational programs. It also captures the user's facial expressions and voice using input devices such as cameras and microphones, and transmits this information to an emotion recognition device.

[0193] The emotion recognition device analyzes the user's emotional responses and generates emotion data. This data is transmitted to the server in real time. Specific software used includes the Facial Expression Recognition API and voice emotion analysis libraries. The server integrates and analyzes the learning progress data and emotion data transmitted from the information processing device.

[0194] Based on this data, the server generates optimal feedback and selects the next priority educational content. The server also uses natural language processing techniques to facilitate user interaction and dynamically adjust the educational program. For example, if the server determines that a user is experiencing stress while working on a problem, it provides helpful hints.

[0195] As a concrete example, imagine a child practicing the piano and attempting a difficult piece. If the child shows signs of distress, the server will generate advice such as, "Try that part again slowly." An example of a prompt for this generating AI model would be, "Analyze how the user's voice tone has changed and evaluate their stress level."

[0196] The system is also designed to continue providing learning support by pre-storing necessary information even in environments with unstable network connections. This makes it possible to deliver a consistent educational experience in any environment.

[0197] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0198] Step 1:

[0199] Users log in to the educational program using an information processing device. The user's login information (username and password) is required as input. The output displays the user's individual profile on the terminal and transitions to the educational program selection screen. User information is sent to the server upon login, and the program starts.

[0200] Step 2:

[0201] The device uses its camera and microphone to capture the user's facial expressions and voice. The input consists of the user's facial expression data and voice data. The output is that this data is transmitted in real time to an emotion recognition device, where it is converted into emotion data. Specifically, the device acquires data every 10 seconds and immediately uploads it to the emotion recognition device.

[0202] Step 3:

[0203] The server integrates emotion data obtained from the emotion recognition device with learning progress data sent from the terminal. It receives emotion data and learning progress data as input. The output is the result of analyzing this data to evaluate the user's learning state. The data processing performed here involves comparing it with past learning history stored in a database.

[0204] Step 4:

[0205] The server generates optimal feedback based on the analysis results and sends it to the terminal. The input for feedback generation is the result of a comprehensive analysis of emotions and learning. The output includes helpful feedback for the user and guidance for the next steps. Specifically, dynamic guidance content is created using a generative AI model. For example, a generated prompt might say, "The user seems confused. Let's try the exercise again slowly."

[0206] Step 5:

[0207] The terminal displays feedback received from the server to the user and suggests new learning content. The input is feedback data from the server, and the output is new learning content and instructions displayed on the user's screen. The terminal updates the user's learning screen and provides navigation to the next learning stage.

[0208] Step 6:

[0209] The user progresses to the suggested new learning stage and continues learning. The input is the new learning task selected by the user, and the output is a record of its progress, which then returns to step 2. This mechanism allows the user to continuously progress in their learning.

[0210] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0211] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0212] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0213] [Second Embodiment]

[0214] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0215] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0216] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0217] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0218] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0219] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0220] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0221] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0222] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0223] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0224] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0225] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0226] This invention is a system that records and analyzes learners' progress in real time by utilizing educational programs provided via information terminals. This system records user activity, generates optimal feedback based on the data, and automatically suggests the next learning objectives. Furthermore, it supports multiple languages, making it effective in multilingual environments. It also features an offline mode, allowing for uninterrupted learning even with minimal internet connectivity.

[0227] First, the user accesses the application using their device and logs into their individual user account. At this point, the server provides a highly relevant learning program based on the user's profile information. For example, if there is a primary school student user, the server generates a curriculum and materials appropriate to the user's grade level and sends them to the device.

[0228] The user then selects a specific educational program and begins learning. As learning progresses, the device records the user's responses and actions in real time. This data is sent to a server, which uses generative AI to analyze the user's learning progress and understanding. For example, if the user continues to answer multiple multiple-choice questions correctly, the server evaluates their understanding and suggests more difficult questions or new learning topics.

[0229] The feedback generated based on the analysis results is presented to the user through the device. This feedback includes specific areas for improvement and serves as a guide for the user to take the next step. For example, if a user answers a question incorrectly, the device provides hints and additional materials to help them understand the mistake and support their learning.

[0230] Furthermore, using natural language processing technology, the device interacts with the user, providing interactions that facilitate the learning process. By inputting questions, the system can provide appropriate answers and relevant information.

[0231] As described above, the present invention is a system that aims to provide individualized learning experiences and ensure high-quality educational opportunities in diverse environments.

[0232] The following describes the processing flow.

[0233] Step 1:

[0234] The user activates their information terminal and logs into the educational application. The server receives the user's authentication information and authenticates the user by referring to the database. If authentication is successful, the server sends the user's learning history and profile information to the terminal.

[0235] Step 2:

[0236] The terminal displays a selection menu of learning programs based on user information received from the server. The user selects a program based on their learning objectives and interests.

[0237] Step 3:

[0238] When a user selects a learning program, the device downloads the necessary learning materials and exercises. The server then sends important information to the device to enable offline use.

[0239] Step 4:

[0240] The user starts the learning program, and the device records the user's responses and actions in real time. This information is sent to the server as learning progress.

[0241] Step 5:

[0242] The server analyzes the received progress data and uses AI technology to evaluate the user's understanding. The next step is to prepare to generate appropriate learning content and feedback.

[0243] Step 6:

[0244] The server-generated feedback is sent to the terminal and presented to the user. This feedback may include analysis of incorrect answers, suggestions for improvement, and additional learning materials.

[0245] Step 7:

[0246] The user reviews the feedback, views explanations as needed, or selects a new learning program to start again. The device then monitors the user's activity again and prepares for the next learning step.

[0247] (Example 1)

[0248] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0249] Traditional education systems lacked sufficient feedback tailored to individual user progress and understanding, suffered from a shortage of multilingual learning materials, and made it difficult to continue learning in environments with unstable internet connections. Furthermore, interaction with users was not sufficiently smooth, highlighting the need for a system that provides a personalized learning experience.

[0250] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0251] In this invention, the server includes means for providing educational programs via an information processing device, means for recording the user's learning progress and analyzing it with a generative AI model, and means for optimizing the content to be learned next using prompt sentences in the generative AI model. This enables personalized feedback and suggestions for the next learning content for each user, realizing a stable multilingual education and a learning environment that can be continued even offline.

[0252] An "information processing device" is an electronic device used for inputting, processing, storing, and outputting data, and includes a terminal that accepts user input.

[0253] An "educational program" is a software application that includes teaching materials and assignments designed for learners to acquire specific knowledge or skills.

[0254] "User learning progress" refers to information that indicates the degree of learning and the progress of acquisition achieved by users through the educational program.

[0255] A "generative AI model" is an algorithm designed to analyze data using artificial intelligence technology and make predictions or suggestions for a specific task.

[0256] "Feedback" refers to information provided as guidance, evaluation, or improvement based on the user's actions and results.

[0257] A "prompt" is a text-based instruction or question that is input to a generative AI model to guide it to produce a specific output.

[0258] "Multilingual education" refers to a learning environment that includes educational content provided in multiple languages, accommodating learners who speak different languages.

[0259] An "offline learning environment" is an environment that allows learning activities to continue even when internet connectivity is unstable or unavailable.

[0260] This invention relates to an information processing system for providing personalized educational experiences. It realizes technology that effectively advances learning through interaction between servers, terminals, and users.

[0261] When a user connects via a terminal, the server functions as an information processing device, selecting the optimal educational program based on the user's profile information. This process utilizes a generative AI model, which analyzes the user's progress data to accurately suggest what they should learn next. The server also pre-stores necessary information to support offline learning, even in cases of unstable network connections.

[0262] The terminal is the primary hardware component that receives user input and runs educational programs. It starts the program selected by the user, records learning progress in real time, and sends data to the server. If the user answers incorrectly, the terminal provides hints and additional materials related to the question to support learning.

[0263] Users operate their devices and learn educational content provided through the interface. Multilingual support enables smooth learning even in different language environments. Furthermore, users can input questions into their devices using natural language processing technology to receive appropriate answers from the server.

[0264] As a concrete example, when an elementary school student uses a math learning program, the prompt they input to the generative AI model might be something like, "If the user answers four questions correctly in a row, what topic should be suggested next?" This ensures that educational content is provided that is tailored to the user's learning ability.

[0265] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0266] Step 1:

[0267] The user accesses the application using their device and enters their account information. The device sends this information to the server. The server authenticates the user and retrieves their profile information. This profile information includes their grade level and past learning data, which forms the basis for a customized educational program.

[0268] Step 2:

[0269] The server uses a generated AI model based on the user's profile information to select highly relevant educational programs. The server outputs these selection results to the terminal. In this process, the most suitable content is suggested according to the user's skill level and learning goals.

[0270] Step 3:

[0271] The user starts the educational program from their device. The device records the user's learning activity and sends the results to the server in real time. Specifically, the user's answers and learning speed are input, and data such as the user's level of understanding is generated based on this.

[0272] Step 4:

[0273] The server analyzes the training data received from the terminal using a generated AI model. The analysis results are generated as feedback and output to the terminal. This output specifically shows the user's progress and areas for improvement, and is designed to help them efficiently progress to the next learning step.

[0274] Step 5:

[0275] The device presents the user with feedback from the server. If the user answers a particular question incorrectly, the device provides additional hints and explanations to support learning and understanding. This allows the user to identify their weaknesses and learn more effectively.

[0276] (Application Example 1)

[0277] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0278] With advancements in educational technology, providing effective individualized instruction at home has become increasingly difficult. In particular, differences in home environment, language barriers, and unstable internet connections hinder the continuity of learning. Furthermore, real-time assessment of learners' progress and providing appropriate feedback are crucial challenges. Therefore, a new system is needed to meet these requirements while providing an individualized educational experience.

[0279] The specific processing by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is realized by the following means.

[0280] In this invention, the server includes means for providing an educational program via an information processing device, means for using an intelligent device that provides learning support in a home environment to evaluate the learning progress in real time and provide feedback through voice and vision, and means for presenting educational information in multiple languages. Thereby, high-quality individual learning is possible even within a home.

[0281] The "information processing device" is a computer or electronic device for receiving, processing, and transmitting data, and is a device used as means for providing an educational program.

[0282] The "educational program" refers to a curriculum or teaching material designed for a learner to acquire specific skills or knowledge, and is content that enables optimal learning.

[0283] The "home environment" represents the place and conditions where a learner receives education within a home, and is a setting aimed at realizing educational support in various environments.

[0284] The "intelligent device" is a device designed to provide educational support within a home, equipped with voice and visual interfaces, and is a device capable of performing real-time learning evaluation.

[0285] "Learning support" refers to support activities for enabling a learner to efficiently acquire skills and knowledge, including providing specific feedback and teaching materials.

[0286] "Evaluation in real time" refers to a process of immediately observing a learner's activities, evaluating them, and providing appropriate feedback.

[0287] "Feedback" is information provided based on a learner's activities and responses, which guides the learner in taking the next step.

[0288] "Multilingual" refers to a system that presents educational information in different languages ​​and can accommodate learners with diverse linguistic backgrounds.

[0289] This embodiment of the invention is a system that uses an information processing device to provide educational programs and evaluate learners' learning progress in real time. The server utilizes a database to manage user profiles in order to provide individualized educational content to each learner. This makes it possible to recommend optimal learning materials based on the learner's interests and level of understanding.

[0290] The device connects to an intelligent device and records the learner's real-time activity data. This data is sent to a server and analyzed in detail using a generative AI model. Based on the analysis results, the learner is provided with appropriate feedback in both audio and visual formats. The intelligent device has a multi-language interface and utilizes natural language processing technology to enable effective communication with the learner.

[0291] In a home environment, even with an unstable internet connection, intelligent devices can continue to support learning using pre-stored information. This system provides a more effective learning experience by offering recommendations based on learners' interests and skill levels when selecting educational programs.

[0292] A concrete example is when an intelligent device provides real-time feedback to a learner solving a math problem. It instantly determines whether the learner's answer is correct or incorrect, providing the next step if correct, or hints for further consideration if incorrect. This feedback is delivered verbally, which helps facilitate the learner's understanding.

[0293] An example of a prompt might be, "How would you encourage a child to move on to the next step when they make a mistake on a math problem?"

[0294] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0295] Step 1:

[0296] When a user logs in via a terminal, the server uses an information processing device to retrieve the user's profile from the database. The input is the user's login information, and the output is profile data including the user's interests and level of understanding. Based on this profile data, the server selects highly relevant learning content. Specifically, the server analyzes the user's learning history and past responses to prepare to suggest the most suitable learning materials for the user.

[0297] Step 2:

[0298] The terminal receives educational content transmitted from the server and presents it to the user in conjunction with intelligent devices. The input is content data from the server, and the output is learning materials presented to the user visually and audibly. Specifically, the terminal displays learning questions on its screen and provides navigation and question text audibly through an audio output device.

[0299] Step 3:

[0300] When a user works on a learning problem, the device records the user's answers in real time and sends them to the server as input. The output is the answer data, which is analyzed through communication with the server. Specifically, when a user solves a multiple-choice question, their selection is immediately uploaded to the server via the device.

[0301] Step 4:

[0302] The server analyzes using an AI model generated based on the received user answer data, and evaluates the learning progress and understanding level. The input is the answer data, and the output is the understanding score as an evaluation result and the recommended teaching materials. As a specific operation, the server determines whether the answer is correct or incorrect, analyzes the cause if it is incorrect, and then determines what should be learned next.

[0303] Step 5:

[0304] The server generates optimal feedback based on the analysis results and provides it to the user via the terminal. The input is the data of the analysis results, and the output is the audio and visual feedback presented to the user. As a specific operation, the feedback includes advice for the next learning step and hints about the incorrect content, providing educational guidance to the user.

[0305] Step 6:

[0306] The user receives the feedback from the terminal and starts the next learning session based on the newly provided learning content. The input is the feedback information, and the output is the selection of content in the new learning activity. As a specific operation, the user refers to new questions or materials and proceeds with learning independently.

[0307] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion recognition model 59 and perform specific processing using the user's emotion.

[0308] The present invention combines an emotion engine with a system that provides an educational program via an information terminal and records and analyzes the learning progress of a user. The emotion engine recognizes the user's emotional reaction and utilizes that information in the educational process. This system aims to provide an appropriate learning experience for the user and enhance the effectiveness of education.

[0309] First, the user logs into the application on their device and selects an educational program based on their profile. The server performs initial setup to understand the user's basic emotional information and activates the emotion engine. The emotion engine analyzes the user's facial expressions and voice through input devices such as the camera and microphone, and recognizes emotions in real time.

[0310] As the user progresses through the learning process, the device records not only data related to normal learning progress but also emotional data generated by the emotion engine. The server integrates this data to gain a comprehensive understanding of the user's learning progress. For example, if the user is feeling bored with the learning material, the emotion engine recognizes this and sends data to the server.

[0311] The server generates feedback based on sentiment analysis. For example, it instructs the device to present learning content in a more engaging way for users who are feeling bored. It can also provide additional explanations or hints if it detects that the user is confused.

[0312] Furthermore, the analysis results from the emotion engine are also used as data to form long-term learning patterns. In this way, the server aims to provide users with a continuous, personalized, and optimal learning experience, thereby improving the quality of education.

[0313] In this way, by adding an emotion recognition function using an emotion engine, the present invention generates flexible learning plans that are tailored to the user's emotional responses, thereby realizing efficient educational support.

[0314] The following describes the processing flow.

[0315] Step 1:

[0316] The user accesses the educational application using an information terminal and logs in. The server verifies the user's authentication information and completes the login process.

[0317] Step 2:

[0318] The server retrieves the user's profile information and sends a selection of optimal educational programs to the device based on their learning history. The device then displays these options, allowing the user to choose a program that suits their interests and needs.

[0319] Step 3:

[0320] Once the user selects a program, the device activates the emotion engine and confirms access permissions for the camera and microphone. This prepares the device to use the user's facial expressions and voice for real-time emotion analysis.

[0321] Step 4:

[0322] As the user learns, the emotion engine performs facial recognition and voice analysis to analyze the user's emotions. For example, if the user's facial expression is smiling, the engine will determine that the user is enjoying themselves.

[0323] Step 5:

[0324] The device simultaneously sends emotion engine data and user learning progress to the server. Based on this data, the server comprehensively analyzes the user's current learning and emotional state.

[0325] Step 6:

[0326] Based on the analysis results, the server generates emotionally responsive feedback. For example, if a user displays a confused expression, the server generates a helpful explanation and sends it to the device.

[0327] Step 7:

[0328] The device receives feedback from the server and presents the user with appropriate information. The user then uses this information to further their learning or make improvements.

[0329] Step 8:

[0330] Once a learning session ends, the server saves the data and prepares to analyze it for use in future learning plans. The emotional data generated by the emotion engine is accumulated as data for improvement in long-term learning.

[0331] (Example 2)

[0332] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0333] Traditional education systems provide feedback based on users' learning progress and performance, but they have faced the challenge of providing personalized learning experiences that take into account users' emotional states. Therefore, there is a need to immediately capture the user's feelings of interest, boredom, and lack of understanding during learning, and to flexibly adjust the content of the learning program based on that feedback.

[0334] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0335] In this invention, the server includes means for recognizing the user's emotional state and applying it to the educational process, means for providing an educational program via an information processing device and recording learning progress, and means for integrating and analyzing the recorded learning progress and emotional data. This makes it possible to optimize the learning program according to the user's emotional state and provide an individualized educational experience.

[0336] A "user" is an individual who uses the educational system to engage in learning activities.

[0337] "Emotional analysis means" refers to a technical device or method for detecting and interpreting an emotional state from a user's facial expressions or voice.

[0338] An "information processing device" is a computer or similar electronic device that collects, stores, processes, and communicates data.

[0339] An "educational program" refers to teaching materials and their structure that aim to improve the knowledge and skills of users.

[0340] "Learning progress" is an indicator that shows the degree or stage of learning that a user has achieved through an educational program.

[0341] "Feedback" refers to information and suggestions provided to users to communicate their learning results and level of understanding, and to encourage further improvement in their learning.

[0342] "Analytical tools" refer to the techniques and methods used to evaluate collected data and derive certain conclusions or views.

[0343] A "learning pattern" is a regularity or tendency that can be found based on a user's past learning behavior and results.

[0344] This invention is a system that provides users with personalized educational experiences by combining an information processing device and an emotion analysis means. The system consists of a server and terminals, where the server is responsible for data integration and analysis, and the terminals primarily provide educational programs through user interaction.

[0345] Hardware and software

[0346] The device has a built-in camera and microphone, and these hardware components are used to capture the user's facial expressions and voice data as a means of sentiment analysis. The video and voice data are sent to the sentiment analysis engine in real time and analyzed by OpenCV and speech recognition libraries. The server uses machine learning algorithms with programming languages ​​such as Python and R to integrate the collected sentiment data and learning progress data, and generates optimal feedback for the next step.

[0347] Specifically, the emotion analysis engine identifies the user's emotional state and sends this information to the server. The server then uses machine learning models to generate a personalized learning program based on the accumulated data. For example, if the server detects that the user is feeling bored, it will recommend new learning content that will pique the user's interest.

[0348] Specific example

[0349] For example, if a user is participating in a program to "learn about historical events," and the emotion analysis engine detects from the user's facial expressions that they are feeling bored, the server will sense this. The server will then analyze the user's past interests and present content such as "documentary about famous historical figures" on the device. This will rekindle the user's interest and allow them to continue learning effectively.

[0350] Example of a prompt

[0351] "If a user's facial expression shows confusion while they are solving a math problem, what additional explanation should be provided?"

[0352] In this way, the system enables users to receive a flexible and effective learning experience through dynamic feedback mediated by emotion analysis.

[0353] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0354] Step 1:

[0355] The user logs into their device and enters their profile information. This information is sent from the device to the server, where it is compared against the user's past learning history and registration information. Based on this data, the server generates personalized educational program suggestions and sends them to the device. The user is then provided with input for selection, and a list of program suggestions is displayed as output.

[0356] Step 2:

[0357] The user selects their desired educational program from the displayed program options. The selected program information is sent from the terminal to the server. The server receives the selection information, organizes the detailed data of the corresponding program, and sends it to the terminal. In response to the user's input to begin learning, the learning content is displayed as output.

[0358] Step 3:

[0359] The device activates its camera and microphone to perform emotion analysis as the user begins learning. The emotion analysis engine captures the user's facial expressions and voice, and analyzes the acquired data in real time. The captured data is acquired as input, and the user's emotional state is recognized as output.

[0360] Step 4:

[0361] The emotion data recognized by the emotion analysis engine is sent to the server via the terminal. The server integrates the emotion data and learning progress data and performs analysis. The input consists of emotion data and learning data, and based on this, the server evaluates the user's current level of understanding and emotional state, and generates appropriate feedback as output.

[0362] Step 5:

[0363] The server sends the generated feedback to the terminal and presents it to the user. Specifically, if it determines that the user is bored, it instructs the server to display engaging videos or additional content. The input for presenting the feedback is the evaluation result described above, and the output is the feedback displayed on the user's interface.

[0364] Step 6:

[0365] Long-term data collected through sentiment analysis is stored on a server and used to shape the user's learning patterns. The server analyzes this historical data and uses it as training data for a generative AI model that proposes future learning programs. The input is the stored learning history, and the output is a prediction of future learning patterns.

[0366] The data and processing obtained at each step are used to provide the most effective learning experience for users, ultimately aiming to improve the quality of education.

[0367] (Application Example 2)

[0368] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0369] Traditional educational support systems can provide feedback based on a user's learning progress, but they have struggled to provide adaptive support based on the user's emotional responses. As a result, the impact of the user's emotional aspects on learning effectiveness is often ignored, and it has been difficult to provide a learning experience optimized for each individual user. Furthermore, there is the challenge of limited operation in environments with unstable internet connections.

[0370] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0371] In this invention, the server includes means for providing educational programs via an information processing device, means for recording and analyzing the user's learning progress, means for providing the optimal response and next-priority educational content based on the analysis results, means for presenting educational information in multiple languages, means for pre-storing information to enable operation even under unstable network connection conditions, means for recognizing the user's emotional response using an emotion recognition device, and means for dynamically adjusting the educational program in accordance with the recognized emotional response. This makes it possible to provide an individualized learning experience that corresponds to the user's emotional state and improve the quality of education.

[0372] An "information processing device" is a device that has the function of providing and processing educational programs with users.

[0373] A "user" refers to a person who uses an educational program and whose learning progress is recorded.

[0374] "Learning progress" refers to information that indicates the state or stage of learning achieved by a user through an educational program.

[0375] "Analysis" is the process of analyzing user learning progress data and emotional response data to improve educational content.

[0376] "Feedback" refers to information and instructions provided to improve the user's learning experience.

[0377] "Educational information" refers to teaching materials and knowledge used to support users' learning.

[0378] A "network connection" is a connection that allows information processing devices to communicate data over the internet.

[0379] An "emotion recognition device" is a device that analyzes a user's facial expressions and voice to identify their emotions.

[0380] "Emotional response" refers to the changes or states of emotion that a user exhibits while learning.

[0381] An "educational program" refers to the learning content and educational methods provided through information processing devices.

[0382] This invention is an educational support system for providing users with an individualized learning experience. The system mainly consists of an information processing device, an emotion recognition device, and a server.

[0383] The information processing device primarily functions as a user interface and provides educational programs. It also captures the user's facial expressions and voice using input devices such as cameras and microphones, and transmits this information to an emotion recognition device.

[0384] The emotion recognition device analyzes the user's emotional responses and generates emotion data. This data is transmitted to the server in real time. Specific software used includes the Facial Expression Recognition API and voice emotion analysis libraries. The server integrates and analyzes the learning progress data and emotion data transmitted from the information processing device.

[0385] Based on this data, the server generates optimal feedback and selects the next priority educational content. The server also uses natural language processing techniques to facilitate user interaction and dynamically adjust the educational program. For example, if the server determines that a user is experiencing stress while working on a problem, it provides helpful hints.

[0386] As a concrete example, imagine a child practicing the piano and attempting a difficult piece. If the child shows signs of distress, the server will generate advice such as, "Try that part again slowly." An example of a prompt for this generating AI model would be, "Analyze how the user's voice tone has changed and evaluate their stress level."

[0387] The system is also designed to continue providing learning support by pre-storing necessary information even in environments with unstable network connections. This makes it possible to deliver a consistent educational experience in any environment.

[0388] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0389] Step 1:

[0390] Users log in to the educational program using an information processing device. The user's login information (username and password) is required as input. The output displays the user's individual profile on the terminal and transitions to the educational program selection screen. User information is sent to the server upon login, and the program starts.

[0391] Step 2:

[0392] The device uses its camera and microphone to capture the user's facial expressions and voice. The input consists of the user's facial expression data and voice data. The output is that this data is transmitted in real time to an emotion recognition device, where it is converted into emotion data. Specifically, the device acquires data every 10 seconds and immediately uploads it to the emotion recognition device.

[0393] Step 3:

[0394] The server integrates emotion data obtained from the emotion recognition device with learning progress data sent from the terminal. It receives emotion data and learning progress data as input. The output is the result of analyzing this data to evaluate the user's learning state. The data processing performed here involves comparing it with past learning history stored in a database.

[0395] Step 4:

[0396] The server generates optimal feedback based on the analysis results and sends it to the terminal. The input for feedback generation is the result of a comprehensive analysis of emotions and learning. The output includes helpful feedback for the user and guidance for the next steps. Specifically, dynamic guidance content is created using a generative AI model. For example, a generated prompt might say, "The user seems confused. Let's try the exercise again slowly."

[0397] Step 5:

[0398] The terminal displays feedback received from the server to the user and suggests new learning content. The input is feedback data from the server, and the output is new learning content and instructions displayed on the user's screen. The terminal updates the user's learning screen and provides navigation to the next learning stage.

[0399] Step 6:

[0400] The user progresses to the suggested new learning stage and continues learning. The input is the new learning task selected by the user, and the output is a record of its progress, which then returns to step 2. This mechanism allows the user to continuously progress in their learning.

[0401] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0402] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0403] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0404] [Third Embodiment]

[0405] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0406] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0407] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0408] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0409] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0410] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0411] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0412] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0413] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0414] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0415] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0416] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0417] This invention is a system that records and analyzes learners' progress in real time by utilizing educational programs provided via information terminals. This system records user activity, generates optimal feedback based on the data, and automatically suggests the next learning objectives. Furthermore, it supports multiple languages, making it effective in multilingual environments. It also features an offline mode, allowing for uninterrupted learning even with minimal internet connectivity.

[0418] First, the user accesses the application using their device and logs into their individual user account. At this point, the server provides a highly relevant learning program based on the user's profile information. For example, if there is a primary school student user, the server generates a curriculum and materials appropriate to the user's grade level and sends them to the device.

[0419] The user then selects a specific educational program and begins learning. As learning progresses, the device records the user's responses and actions in real time. This data is sent to a server, which uses generative AI to analyze the user's learning progress and understanding. For example, if the user continues to answer multiple multiple-choice questions correctly, the server evaluates their understanding and suggests more difficult questions or new learning topics.

[0420] The feedback generated based on the analysis results is presented to the user through the device. This feedback includes specific areas for improvement and serves as a guide for the user to take the next step. For example, if a user answers a question incorrectly, the device provides hints and additional materials to help them understand the mistake and support their learning.

[0421] Furthermore, using natural language processing technology, the device interacts with the user, providing interactions that facilitate the learning process. By inputting questions, the system can provide appropriate answers and relevant information.

[0422] As described above, the present invention is a system that aims to provide individualized learning experiences and ensure high-quality educational opportunities in diverse environments.

[0423] The following describes the processing flow.

[0424] Step 1:

[0425] The user activates their information terminal and logs into the educational application. The server receives the user's authentication information and authenticates the user by referring to the database. If authentication is successful, the server sends the user's learning history and profile information to the terminal.

[0426] Step 2:

[0427] The terminal displays a selection menu of learning programs based on user information received from the server. The user selects a program based on their learning objectives and interests.

[0428] Step 3:

[0429] When a user selects a learning program, the device downloads the necessary learning materials and exercises. The server then sends important information to the device to enable offline use.

[0430] Step 4:

[0431] The user starts the learning program, and the device records the user's responses and actions in real time. This information is sent to the server as learning progress.

[0432] Step 5:

[0433] The server analyzes the received progress data and uses AI technology to evaluate the user's understanding. The next step is to prepare to generate appropriate learning content and feedback.

[0434] Step 6:

[0435] The server-generated feedback is sent to the terminal and presented to the user. This feedback may include analysis of incorrect answers, suggestions for improvement, and additional learning materials.

[0436] Step 7:

[0437] The user reviews the feedback, views explanations as needed, or selects a new learning program to start again. The device then monitors the user's activity again and prepares for the next learning step.

[0438] (Example 1)

[0439] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0440] Traditional education systems lacked sufficient feedback tailored to individual user progress and understanding, suffered from a shortage of multilingual learning materials, and made it difficult to continue learning in environments with unstable internet connections. Furthermore, interaction with users was not sufficiently smooth, highlighting the need for a system that provides a personalized learning experience.

[0441] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0442] In this invention, the server includes means for providing educational programs via an information processing device, means for recording the user's learning progress and analyzing it with a generative AI model, and means for optimizing the content to be learned next using prompt sentences in the generative AI model. This enables personalized feedback and suggestions for the next learning content for each user, realizing a stable multilingual education and a learning environment that can be continued even offline.

[0443] An "information processing device" is an electronic device used for inputting, processing, storing, and outputting data, and includes a terminal that accepts user input.

[0444] An "educational program" is a software application that includes teaching materials and assignments designed for learners to acquire specific knowledge or skills.

[0445] "User learning progress" refers to information that indicates the degree of learning and the progress of acquisition achieved by users through the educational program.

[0446] A "generative AI model" is an algorithm designed to analyze data using artificial intelligence technology and make predictions or suggestions for a specific task.

[0447] "Feedback" refers to information provided as guidance, evaluation, or improvement based on the user's actions and results.

[0448] A "prompt" is a text-based instruction or question that is input to a generative AI model to guide it to produce a specific output.

[0449] "Multilingual education" refers to a learning environment that includes educational content provided in multiple languages, accommodating learners who speak different languages.

[0450] An "offline learning environment" is an environment that allows learning activities to continue even when internet connectivity is unstable or unavailable.

[0451] This invention relates to an information processing system for providing personalized educational experiences. It realizes technology that effectively advances learning through interaction between servers, terminals, and users.

[0452] When a user connects via a terminal, the server functions as an information processing device, selecting the optimal educational program based on the user's profile information. This process utilizes a generative AI model, which analyzes the user's progress data to accurately suggest what they should learn next. The server also pre-stores necessary information to support offline learning, even in cases of unstable network connections.

[0453] The terminal is the primary hardware component that receives user input and runs educational programs. It starts the program selected by the user, records learning progress in real time, and sends data to the server. If the user answers incorrectly, the terminal provides hints and additional materials related to the question to support learning.

[0454] Users operate their devices and learn educational content provided through the interface. Multilingual support enables smooth learning even in different language environments. Furthermore, users can input questions into their devices using natural language processing technology to receive appropriate answers from the server.

[0455] As a concrete example, when an elementary school student uses a math learning program, the prompt they input to the generative AI model might be something like, "If the user answers four questions correctly in a row, what topic should be suggested next?" This ensures that educational content is provided that is tailored to the user's learning ability.

[0456] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0457] Step 1:

[0458] The user accesses the application using their device and enters their account information. The device sends this information to the server. The server authenticates the user and retrieves their profile information. This profile information includes their grade level and past learning data, which forms the basis for a customized educational program.

[0459] Step 2:

[0460] The server uses a generated AI model based on the user's profile information to select highly relevant educational programs. The server outputs these selection results to the terminal. In this process, the most suitable content is suggested according to the user's skill level and learning goals.

[0461] Step 3:

[0462] The user starts the educational program from their device. The device records the user's learning activity and sends the results to the server in real time. Specifically, the user's answers and learning speed are input, and data such as the user's level of understanding is generated based on this.

[0463] Step 4:

[0464] The server analyzes the training data received from the terminal using a generated AI model. The analysis results are generated as feedback and output to the terminal. This output specifically shows the user's progress and areas for improvement, and is designed to help them efficiently progress to the next learning step.

[0465] Step 5:

[0466] The device presents the user with feedback from the server. If the user answers a particular question incorrectly, the device provides additional hints and explanations to support learning and understanding. This allows the user to identify their weaknesses and learn more effectively.

[0467] (Application Example 1)

[0468] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0469] With advancements in educational technology, providing effective individualized instruction at home has become increasingly difficult. In particular, differences in home environment, language barriers, and unstable internet connections hinder the continuity of learning. Furthermore, real-time assessment of learners' progress and providing appropriate feedback are crucial challenges. Therefore, a new system is needed to meet these requirements while providing an individualized educational experience.

[0470] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0471] In this invention, the server includes means for providing educational programs via an information processing device, means for evaluating learning progress in real time and providing feedback through audio and visual means using intelligent devices that support learning in a home environment, and means for presenting educational information in multiple languages. This makes high-quality individualized learning possible even at home.

[0472] An "information processing device" is a computer or electronic device used for receiving, processing, and transmitting data, and is used as a means of providing educational programs.

[0473] An "educational program" refers to a curriculum and materials designed to help learners acquire specific skills and knowledge, and is content that enables optimal learning.

[0474] "Home environment" refers to the place and conditions in which learners receive education within their homes, and is a setting intended to realize educational support in diverse environments.

[0475] An "intelligent device" is a device equipped with audio and visual interfaces designed to provide educational support within the home, and is capable of real-time learning assessment.

[0476] "Learning support" refers to support activities aimed at enabling learners to efficiently acquire skills and knowledge, and includes providing specific feedback and learning materials.

[0477] "Real-time assessment" refers to a process of immediately observing, evaluating, and providing appropriate feedback on learners' activities.

[0478] "Feedback" is information provided based on a learner's activities and responses, which guides the learner in taking the next step.

[0479] "Multilingual" refers to a system that presents educational information in different languages ​​and can accommodate learners with diverse linguistic backgrounds.

[0480] This embodiment of the invention is a system that uses an information processing device to provide educational programs and evaluate learners' learning progress in real time. The server utilizes a database to manage user profiles in order to provide individualized educational content to each learner. This makes it possible to recommend optimal learning materials based on the learner's interests and level of understanding.

[0481] The device connects to an intelligent device and records the learner's real-time activity data. This data is sent to a server and analyzed in detail using a generative AI model. Based on the analysis results, the learner is provided with appropriate feedback in both audio and visual formats. The intelligent device has a multi-language interface and utilizes natural language processing technology to enable effective communication with the learner.

[0482] In a home environment, even with an unstable internet connection, intelligent devices can continue to support learning using pre-stored information. This system provides a more effective learning experience by offering recommendations based on learners' interests and skill levels when selecting educational programs.

[0483] A concrete example is when an intelligent device provides real-time feedback to a learner solving a math problem. It instantly determines whether the learner's answer is correct or incorrect, providing the next step if correct, or hints for further consideration if incorrect. This feedback is delivered verbally, which helps facilitate the learner's understanding.

[0484] An example of a prompt might be, "How would you encourage a child to move on to the next step when they make a mistake on a math problem?"

[0485] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0486] Step 1:

[0487] When a user logs in via a terminal, the server uses an information processing device to retrieve the user's profile from the database. The input is the user's login information, and the output is profile data including the user's interests and level of understanding. Based on this profile data, the server selects highly relevant learning content. Specifically, the server analyzes the user's learning history and past responses to prepare to suggest the most suitable learning materials for the user.

[0488] Step 2:

[0489] The terminal receives educational content transmitted from the server and presents it to the user in conjunction with intelligent devices. The input is content data from the server, and the output is learning materials presented to the user visually and audibly. Specifically, the terminal displays learning questions on its screen and provides navigation and question text audibly through an audio output device.

[0490] Step 3:

[0491] When a user works on a learning problem, the device records the user's answers in real time and sends them to the server as input. The output is the answer data, which is analyzed through communication with the server. Specifically, when a user solves a multiple-choice question, their selection is immediately uploaded to the server via the device.

[0492] Step 4:

[0493] The server uses a generated AI model to analyze the received user answer data and evaluate learning progress and comprehension. The input is the answer data, and the output is the comprehension score and recommended learning materials as evaluation results. Specifically, the server determines whether the answer is correct or incorrect, analyzes the cause of the error if it is incorrect, and determines what should be learned next.

[0494] Step 5:

[0495] The server generates optimal feedback based on the analysis results and provides it to the user via the terminal. The input is the analysis data, and the output is the audio and visual feedback presented to the user. Specifically, the feedback provides educational guidance to the user, including advice on the next learning step and hints about what was done incorrectly.

[0496] Step 6:

[0497] The user receives feedback from their device and starts the next learning session based on newly provided learning content. The input is feedback information, and the output is the selection of content for the new learning activity. In concrete terms, the user refers to new problems and materials and proceeds with learning independently.

[0498] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0499] This invention combines an emotion engine with a system that provides educational programs via an information terminal and records and analyzes the user's learning progress. The emotion engine recognizes the user's emotional responses and utilizes that information in the educational process. The aim of this system is to provide users with an appropriate learning experience and enhance the effectiveness of education.

[0500] First, the user logs into the application on their device and selects an educational program based on their profile. The server performs initial setup to understand the user's basic emotional information and activates the emotion engine. The emotion engine analyzes the user's facial expressions and voice through input devices such as the camera and microphone, and recognizes emotions in real time.

[0501] As the user progresses through the learning process, the device records not only data related to normal learning progress but also emotional data generated by the emotion engine. The server integrates this data to gain a comprehensive understanding of the user's learning progress. For example, if the user is feeling bored with the learning material, the emotion engine recognizes this and sends data to the server.

[0502] The server generates feedback based on sentiment analysis. For example, it instructs the device to present learning content in a more engaging way for users who are feeling bored. It can also provide additional explanations or hints if it detects that the user is confused.

[0503] Furthermore, the analysis results from the emotion engine are also used as data to form long-term learning patterns. In this way, the server aims to provide users with a continuous, personalized, and optimal learning experience, thereby improving the quality of education.

[0504] In this way, by adding an emotion recognition function using an emotion engine, the present invention generates flexible learning plans that are tailored to the user's emotional responses, thereby realizing efficient educational support.

[0505] The following describes the processing flow.

[0506] Step 1:

[0507] The user accesses the educational application using an information terminal and logs in. The server verifies the user's authentication information and completes the login process.

[0508] Step 2:

[0509] The server retrieves the user's profile information and sends a selection of optimal educational programs to the device based on their learning history. The device then displays these options, allowing the user to choose a program that suits their interests and needs.

[0510] Step 3:

[0511] Once the user selects a program, the device activates the emotion engine and confirms access permissions for the camera and microphone. This prepares the device to use the user's facial expressions and voice for real-time emotion analysis.

[0512] Step 4:

[0513] As the user learns, the emotion engine performs facial recognition and voice analysis to analyze the user's emotions. For example, if the user's facial expression is smiling, the engine will determine that the user is enjoying themselves.

[0514] Step 5:

[0515] The device simultaneously sends emotion engine data and user learning progress to the server. Based on this data, the server comprehensively analyzes the user's current learning and emotional state.

[0516] Step 6:

[0517] Based on the analysis results, the server generates emotionally responsive feedback. For example, if a user displays a confused expression, the server generates a helpful explanation and sends it to the device.

[0518] Step 7:

[0519] The device receives feedback from the server and presents the user with appropriate information. The user then uses this information to further their learning or make improvements.

[0520] Step 8:

[0521] Once a learning session ends, the server saves the data and prepares to analyze it for use in future learning plans. The emotional data generated by the emotion engine is accumulated as data for improvement in long-term learning.

[0522] (Example 2)

[0523] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0524] Traditional education systems provide feedback based on users' learning progress and performance, but they have faced the challenge of providing personalized learning experiences that take into account users' emotional states. Therefore, there is a need to immediately capture the user's feelings of interest, boredom, and lack of understanding during learning, and to flexibly adjust the content of the learning program based on that feedback.

[0525] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0526] In this invention, the server includes means for recognizing the user's emotional state and applying it to the educational process, means for providing an educational program via an information processing device and recording learning progress, and means for integrating and analyzing the recorded learning progress and emotional data. This makes it possible to optimize the learning program according to the user's emotional state and provide an individualized educational experience.

[0527] A "user" is an individual who uses the educational system to engage in learning activities.

[0528] "Emotional analysis means" refers to a technical device or method for detecting and interpreting an emotional state from a user's facial expressions or voice.

[0529] An "information processing device" is a computer or similar electronic device that collects, stores, processes, and communicates data.

[0530] An "educational program" refers to teaching materials and their structure that aim to improve the knowledge and skills of users.

[0531] "Learning progress" is an indicator that shows the degree or stage of learning that a user has achieved through an educational program.

[0532] "Feedback" refers to information and suggestions provided to users to communicate their learning results and level of understanding, and to encourage further improvement in their learning.

[0533] "Analytical tools" refer to the techniques and methods used to evaluate collected data and derive certain conclusions or views.

[0534] A "learning pattern" is a regularity or tendency that can be found based on a user's past learning behavior and results.

[0535] This invention is a system that provides users with personalized educational experiences by combining an information processing device and an emotion analysis means. The system consists of a server and terminals, where the server is responsible for data integration and analysis, and the terminals primarily provide educational programs through user interaction.

[0536] Hardware and software

[0537] The device has a built-in camera and microphone, and these hardware components are used to capture the user's facial expressions and voice data as a means of sentiment analysis. The video and voice data are sent to the sentiment analysis engine in real time and analyzed by OpenCV and speech recognition libraries. The server uses machine learning algorithms with programming languages ​​such as Python and R to integrate the collected sentiment data and learning progress data, and generates optimal feedback for the next step.

[0538] Specifically, the emotion analysis engine identifies the user's emotional state and sends this information to the server. The server then uses machine learning models to generate a personalized learning program based on the accumulated data. For example, if the server detects that the user is feeling bored, it will recommend new learning content that will pique the user's interest.

[0539] Specific example

[0540] For example, if a user is participating in a program to "learn about historical events," and the emotion analysis engine detects from the user's facial expressions that they are feeling bored, the server will sense this. The server will then analyze the user's past interests and present content such as "documentary about famous historical figures" on the device. This will rekindle the user's interest and allow them to continue learning effectively.

[0541] Example of a prompt

[0542] "If a user's facial expression shows confusion while they are solving a math problem, what additional explanation should be provided?"

[0543] In this way, the system enables users to receive a flexible and effective learning experience through dynamic feedback mediated by emotion analysis.

[0544] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0545] Step 1:

[0546] The user logs into their device and enters their profile information. This information is sent from the device to the server, where it is compared against the user's past learning history and registration information. Based on this data, the server generates personalized educational program suggestions and sends them to the device. The user is then provided with input for selection, and a list of program suggestions is displayed as output.

[0547] Step 2:

[0548] The user selects their desired educational program from the displayed program options. The selected program information is sent from the terminal to the server. The server receives the selection information, organizes the detailed data of the corresponding program, and sends it to the terminal. In response to the user's input to begin learning, the learning content is displayed as output.

[0549] Step 3:

[0550] The device activates its camera and microphone to perform emotion analysis as the user begins learning. The emotion analysis engine captures the user's facial expressions and voice, and analyzes the acquired data in real time. The captured data is acquired as input, and the user's emotional state is recognized as output.

[0551] Step 4:

[0552] The emotion data recognized by the emotion analysis engine is sent to the server via the terminal. The server integrates the emotion data and learning progress data and performs analysis. The input consists of emotion data and learning data, and based on this, the server evaluates the user's current level of understanding and emotional state, and generates appropriate feedback as output.

[0553] Step 5:

[0554] The server sends the generated feedback to the terminal and presents it to the user. Specifically, if it determines that the user is bored, it instructs the server to display engaging videos or additional content. The input for presenting the feedback is the evaluation result described above, and the output is the feedback displayed on the user's interface.

[0555] Step 6:

[0556] Long-term data collected through sentiment analysis is stored on a server and used to shape the user's learning patterns. The server analyzes this historical data and uses it as training data for a generative AI model that proposes future learning programs. The input is the stored learning history, and the output is a prediction of future learning patterns.

[0557] The data and processing obtained at each step are used to provide the most effective learning experience for users, ultimately aiming to improve the quality of education.

[0558] (Application Example 2)

[0559] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0560] Traditional educational support systems can provide feedback based on a user's learning progress, but they have struggled to provide adaptive support based on the user's emotional responses. As a result, the impact of the user's emotional aspects on learning effectiveness is often ignored, and it has been difficult to provide a learning experience optimized for each individual user. Furthermore, there is the challenge of limited operation in environments with unstable internet connections.

[0561] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0562] In this invention, the server includes means for providing educational programs via an information processing device, means for recording and analyzing the user's learning progress, means for providing the optimal response and next-priority educational content based on the analysis results, means for presenting educational information in multiple languages, means for pre-storing information to enable operation even under unstable network connection conditions, means for recognizing the user's emotional response using an emotion recognition device, and means for dynamically adjusting the educational program in accordance with the recognized emotional response. This makes it possible to provide an individualized learning experience that corresponds to the user's emotional state and improve the quality of education.

[0563] An "information processing device" is a device that has the function of providing and processing educational programs with users.

[0564] A "user" refers to a person who uses an educational program and whose learning progress is recorded.

[0565] "Learning progress" refers to information that indicates the state or stage of learning achieved by a user through an educational program.

[0566] "Analysis" is the process of analyzing user learning progress data and emotional response data to improve educational content.

[0567] "Feedback" refers to information and instructions provided to improve the user's learning experience.

[0568] "Educational information" refers to teaching materials and knowledge used to support users' learning.

[0569] A "network connection" is a connection that allows information processing devices to communicate data over the internet.

[0570] An "emotion recognition device" is a device that analyzes a user's facial expressions and voice to identify their emotions.

[0571] "Emotional response" refers to the changes or states of emotion that a user exhibits while learning.

[0572] An "educational program" refers to the learning content and educational methods provided through information processing devices.

[0573] This invention is an educational support system for providing users with an individualized learning experience. The system mainly consists of an information processing device, an emotion recognition device, and a server.

[0574] The information processing device primarily functions as a user interface and provides educational programs. It also captures the user's facial expressions and voice using input devices such as cameras and microphones, and transmits this information to an emotion recognition device.

[0575] The emotion recognition device analyzes the user's emotional responses and generates emotion data. This data is transmitted to the server in real time. Specific software used includes the Facial Expression Recognition API and voice emotion analysis libraries. The server integrates and analyzes the learning progress data and emotion data transmitted from the information processing device.

[0576] Based on this data, the server generates optimal feedback and selects the next priority educational content. The server also uses natural language processing techniques to facilitate user interaction and dynamically adjust the educational program. For example, if the server determines that a user is experiencing stress while working on a problem, it provides helpful hints.

[0577] As a concrete example, imagine a child practicing the piano and attempting a difficult piece. If the child shows signs of distress, the server will generate advice such as, "Try that part again slowly." An example of a prompt for this generating AI model would be, "Analyze how the user's voice tone has changed and evaluate their stress level."

[0578] The system is also designed to continue providing learning support by pre-storing necessary information even in environments with unstable network connections. This makes it possible to deliver a consistent educational experience in any environment.

[0579] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0580] Step 1:

[0581] Users log in to the educational program using an information processing device. The user's login information (username and password) is required as input. The output displays the user's individual profile on the terminal and transitions to the educational program selection screen. User information is sent to the server upon login, and the program starts.

[0582] Step 2:

[0583] The device uses its camera and microphone to capture the user's facial expressions and voice. The input consists of the user's facial expression data and voice data. The output is that this data is transmitted in real time to an emotion recognition device, where it is converted into emotion data. Specifically, the device acquires data every 10 seconds and immediately uploads it to the emotion recognition device.

[0584] Step 3:

[0585] The server integrates emotion data obtained from the emotion recognition device with learning progress data sent from the terminal. It receives emotion data and learning progress data as input. The output is the result of analyzing this data to evaluate the user's learning state. The data processing performed here involves comparing it with past learning history stored in a database.

[0586] Step 4:

[0587] The server generates optimal feedback based on the analysis results and sends it to the terminal. The input for feedback generation is the result of a comprehensive analysis of emotions and learning. The output includes helpful feedback for the user and guidance for the next steps. Specifically, dynamic guidance content is created using a generative AI model. For example, a generated prompt might say, "The user seems confused. Let's try the exercise again slowly."

[0588] Step 5:

[0589] The terminal displays feedback received from the server to the user and suggests new learning content. The input is feedback data from the server, and the output is new learning content and instructions displayed on the user's screen. The terminal updates the user's learning screen and provides navigation to the next learning stage.

[0590] Step 6:

[0591] The user progresses to the suggested new learning stage and continues learning. The input is the new learning task selected by the user, and the output is a record of its progress, which then returns to step 2. This mechanism allows the user to continuously progress in their learning.

[0592] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0593] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0594] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0595] [Fourth Embodiment]

[0596] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0597] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0598] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0599] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0600] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0601] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0602] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0603] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0604] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0605] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0606] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0607] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0608] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0609] This invention is a system that records and analyzes learners' progress in real time by utilizing educational programs provided via information terminals. This system records user activity, generates optimal feedback based on the data, and automatically suggests the next learning objectives. Furthermore, it supports multiple languages, making it effective in multilingual environments. It also features an offline mode, allowing for uninterrupted learning even with minimal internet connectivity.

[0610] First, the user accesses the application using their device and logs into their individual user account. At this point, the server provides a highly relevant learning program based on the user's profile information. For example, if there is a primary school student user, the server generates a curriculum and materials appropriate to the user's grade level and sends them to the device.

[0611] The user then selects a specific educational program and begins learning. As learning progresses, the device records the user's responses and actions in real time. This data is sent to a server, which uses generative AI to analyze the user's learning progress and understanding. For example, if the user continues to answer multiple multiple-choice questions correctly, the server evaluates their understanding and suggests more difficult questions or new learning topics.

[0612] The feedback generated based on the analysis results is presented to the user through the device. This feedback includes specific areas for improvement and serves as a guide for the user to take the next step. For example, if a user answers a question incorrectly, the device provides hints and additional materials to help them understand the mistake and support their learning.

[0613] Furthermore, using natural language processing technology, the device interacts with the user, providing interactions that facilitate the learning process. By inputting questions, the system can provide appropriate answers and relevant information.

[0614] As described above, the present invention is a system that aims to provide individualized learning experiences and ensure high-quality educational opportunities in diverse environments.

[0615] The following describes the processing flow.

[0616] Step 1:

[0617] The user activates their information terminal and logs into the educational application. The server receives the user's authentication information and authenticates the user by referring to the database. If authentication is successful, the server sends the user's learning history and profile information to the terminal.

[0618] Step 2:

[0619] The terminal displays a selection menu of learning programs based on user information received from the server. The user selects a program based on their learning objectives and interests.

[0620] Step 3:

[0621] When a user selects a learning program, the device downloads the necessary learning materials and exercises. The server then sends important information to the device to enable offline use.

[0622] Step 4:

[0623] The user starts the learning program, and the device records the user's responses and actions in real time. This information is sent to the server as learning progress.

[0624] Step 5:

[0625] The server analyzes the received progress data and uses AI technology to evaluate the user's understanding. The next step is to prepare to generate appropriate learning content and feedback.

[0626] Step 6:

[0627] The server-generated feedback is sent to the terminal and presented to the user. This feedback may include analysis of incorrect answers, suggestions for improvement, and additional learning materials.

[0628] Step 7:

[0629] The user reviews the feedback, views explanations as needed, or selects a new learning program to start again. The device then monitors the user's activity again and prepares for the next learning step.

[0630] (Example 1)

[0631] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0632] Traditional education systems lacked sufficient feedback tailored to individual user progress and understanding, suffered from a shortage of multilingual learning materials, and made it difficult to continue learning in environments with unstable internet connections. Furthermore, interaction with users was not sufficiently smooth, highlighting the need for a system that provides a personalized learning experience.

[0633] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0634] In this invention, the server includes means for providing educational programs via an information processing device, means for recording the user's learning progress and analyzing it with a generative AI model, and means for optimizing the content to be learned next using prompt sentences in the generative AI model. This enables personalized feedback and suggestions for the next learning content for each user, realizing a stable multilingual education and a learning environment that can be continued even offline.

[0635] An "information processing device" is an electronic device used for inputting, processing, storing, and outputting data, and includes a terminal that accepts user input.

[0636] An "educational program" is a software application that includes teaching materials and assignments designed for learners to acquire specific knowledge or skills.

[0637] "User learning progress" refers to information that indicates the degree of learning and the progress of acquisition achieved by users through the educational program.

[0638] A "generative AI model" is an algorithm designed to analyze data using artificial intelligence technology and make predictions or suggestions for a specific task.

[0639] "Feedback" refers to information provided as guidance, evaluation, or improvement based on the user's actions and results.

[0640] A "prompt" is a text-based instruction or question that is input to a generative AI model to guide it to produce a specific output.

[0641] "Multilingual education" refers to a learning environment that includes educational content provided in multiple languages, accommodating learners who speak different languages.

[0642] An "offline learning environment" is an environment that allows learning activities to continue even when internet connectivity is unstable or unavailable.

[0643] This invention relates to an information processing system for providing personalized educational experiences. It realizes technology that effectively advances learning through interaction between servers, terminals, and users.

[0644] When a user connects via a terminal, the server functions as an information processing device, selecting the optimal educational program based on the user's profile information. This process utilizes a generative AI model, which analyzes the user's progress data to accurately suggest what they should learn next. The server also pre-stores necessary information to support offline learning, even in cases of unstable network connections.

[0645] The terminal is the primary hardware component that receives user input and runs educational programs. It starts the program selected by the user, records learning progress in real time, and sends data to the server. If the user answers incorrectly, the terminal provides hints and additional materials related to the question to support learning.

[0646] Users operate their devices and learn educational content provided through the interface. Multilingual support enables smooth learning even in different language environments. Furthermore, users can input questions into their devices using natural language processing technology to receive appropriate answers from the server.

[0647] As a concrete example, when an elementary school student uses a math learning program, the prompt they input to the generative AI model might be something like, "If the user answers four questions correctly in a row, what topic should be suggested next?" This ensures that educational content is provided that is tailored to the user's learning ability.

[0648] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0649] Step 1:

[0650] The user accesses the application using their device and enters their account information. The device sends this information to the server. The server authenticates the user and retrieves their profile information. This profile information includes their grade level and past learning data, which forms the basis for a customized educational program.

[0651] Step 2:

[0652] The server uses a generated AI model based on the user's profile information to select highly relevant educational programs. The server outputs these selection results to the terminal. In this process, the most suitable content is suggested according to the user's skill level and learning goals.

[0653] Step 3:

[0654] The user starts the educational program from their device. The device records the user's learning activity and sends the results to the server in real time. Specifically, the user's answers and learning speed are input, and data such as the user's level of understanding is generated based on this.

[0655] Step 4:

[0656] The server analyzes the training data received from the terminal using a generated AI model. The analysis results are generated as feedback and output to the terminal. This output specifically shows the user's progress and areas for improvement, and is designed to help them efficiently progress to the next learning step.

[0657] Step 5:

[0658] The device presents the user with feedback from the server. If the user answers a particular question incorrectly, the device provides additional hints and explanations to support learning and understanding. This allows the user to identify their weaknesses and learn more effectively.

[0659] (Application Example 1)

[0660] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0661] With advancements in educational technology, providing effective individualized instruction at home has become increasingly difficult. In particular, differences in home environment, language barriers, and unstable internet connections hinder the continuity of learning. Furthermore, real-time assessment of learners' progress and providing appropriate feedback are crucial challenges. Therefore, a new system is needed to meet these requirements while providing an individualized educational experience.

[0662] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0663] In this invention, the server includes means for providing educational programs via an information processing device, means for evaluating learning progress in real time and providing feedback through audio and visual means using intelligent devices that support learning in a home environment, and means for presenting educational information in multiple languages. This makes high-quality individualized learning possible even at home.

[0664] An "information processing device" is a computer or electronic device used for receiving, processing, and transmitting data, and is used as a means of providing educational programs.

[0665] An "educational program" refers to a curriculum and materials designed to help learners acquire specific skills and knowledge, and is content that enables optimal learning.

[0666] "Home environment" refers to the place and conditions in which learners receive education within their homes, and is a setting intended to realize educational support in diverse environments.

[0667] An "intelligent device" is a device equipped with audio and visual interfaces designed to provide educational support within the home, and is capable of real-time learning assessment.

[0668] "Learning support" refers to support activities aimed at enabling learners to efficiently acquire skills and knowledge, and includes providing specific feedback and learning materials.

[0669] "Real-time assessment" refers to a process of immediately observing, evaluating, and providing appropriate feedback on learners' activities.

[0670] "Feedback" is information provided based on a learner's activities and responses, which guides the learner in taking the next step.

[0671] "Multilingual" refers to a system that presents educational information in different languages ​​and can accommodate learners with diverse linguistic backgrounds.

[0672] This embodiment of the invention is a system that uses an information processing device to provide educational programs and evaluate learners' learning progress in real time. The server utilizes a database to manage user profiles in order to provide individualized educational content to each learner. This makes it possible to recommend optimal learning materials based on the learner's interests and level of understanding.

[0673] The device connects to an intelligent device and records the learner's real-time activity data. This data is sent to a server and analyzed in detail using a generative AI model. Based on the analysis results, the learner is provided with appropriate feedback in both audio and visual formats. The intelligent device has a multi-language interface and utilizes natural language processing technology to enable effective communication with the learner.

[0674] In a home environment, even with an unstable internet connection, intelligent devices can continue to support learning using pre-stored information. This system provides a more effective learning experience by offering recommendations based on learners' interests and skill levels when selecting educational programs.

[0675] A concrete example is when an intelligent device provides real-time feedback to a learner solving a math problem. It instantly determines whether the learner's answer is correct or incorrect, providing the next step if correct, or hints for further consideration if incorrect. This feedback is delivered verbally, which helps facilitate the learner's understanding.

[0676] An example of a prompt might be, "How would you encourage a child to move on to the next step when they make a mistake on a math problem?"

[0677] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0678] Step 1:

[0679] When a user logs in via a terminal, the server uses an information processing device to retrieve the user's profile from the database. The input is the user's login information, and the output is profile data including the user's interests and level of understanding. Based on this profile data, the server selects highly relevant learning content. Specifically, the server analyzes the user's learning history and past responses to prepare to suggest the most suitable learning materials for the user.

[0680] Step 2:

[0681] The terminal receives educational content transmitted from the server and presents it to the user in conjunction with intelligent devices. The input is content data from the server, and the output is learning materials presented to the user visually and audibly. Specifically, the terminal displays learning questions on its screen and provides navigation and question text audibly through an audio output device.

[0682] Step 3:

[0683] When a user works on a learning problem, the device records the user's answers in real time and sends them to the server as input. The output is the answer data, which is analyzed through communication with the server. Specifically, when a user solves a multiple-choice question, their selection is immediately uploaded to the server via the device.

[0684] Step 4:

[0685] The server uses a generated AI model to analyze the received user answer data and evaluate learning progress and comprehension. The input is the answer data, and the output is the comprehension score and recommended learning materials as evaluation results. Specifically, the server determines whether the answer is correct or incorrect, analyzes the cause of the error if it is incorrect, and determines what should be learned next.

[0686] Step 5:

[0687] The server generates optimal feedback based on the analysis results and provides it to the user via the terminal. The input is the analysis data, and the output is the audio and visual feedback presented to the user. Specifically, the feedback provides educational guidance to the user, including advice on the next learning step and hints about what was done incorrectly.

[0688] Step 6:

[0689] The user receives feedback from their device and starts the next learning session based on newly provided learning content. The input is feedback information, and the output is the selection of content for the new learning activity. In concrete terms, the user refers to new problems and materials and proceeds with learning independently.

[0690] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0691] This invention combines an emotion engine with a system that provides educational programs via an information terminal and records and analyzes the user's learning progress. The emotion engine recognizes the user's emotional responses and utilizes that information in the educational process. The aim of this system is to provide users with an appropriate learning experience and enhance the effectiveness of education.

[0692] First, the user logs into the application on their device and selects an educational program based on their profile. The server performs initial setup to understand the user's basic emotional information and activates the emotion engine. The emotion engine analyzes the user's facial expressions and voice through input devices such as the camera and microphone, and recognizes emotions in real time.

[0693] As the user progresses through the learning process, the device records not only data related to normal learning progress but also emotional data generated by the emotion engine. The server integrates this data to gain a comprehensive understanding of the user's learning progress. For example, if the user is feeling bored with the learning material, the emotion engine recognizes this and sends data to the server.

[0694] The server generates feedback based on sentiment analysis. For example, it instructs the device to present learning content in a more engaging way for users who are feeling bored. It can also provide additional explanations or hints if it detects that the user is confused.

[0695] Furthermore, the analysis results from the emotion engine are also used as data to form long-term learning patterns. In this way, the server aims to provide users with a continuous, personalized, and optimal learning experience, thereby improving the quality of education.

[0696] In this way, by adding an emotion recognition function using an emotion engine, the present invention generates flexible learning plans that are tailored to the user's emotional responses, thereby realizing efficient educational support.

[0697] The following describes the processing flow.

[0698] Step 1:

[0699] The user accesses the educational application using an information terminal and logs in. The server verifies the user's authentication information and completes the login process.

[0700] Step 2:

[0701] The server retrieves the user's profile information and sends a selection of optimal educational programs to the device based on their learning history. The device then displays these options, allowing the user to choose a program that suits their interests and needs.

[0702] Step 3:

[0703] Once the user selects a program, the device activates the emotion engine and confirms access permissions for the camera and microphone. This prepares the device to use the user's facial expressions and voice for real-time emotion analysis.

[0704] Step 4:

[0705] As the user learns, the emotion engine performs facial recognition and voice analysis to analyze the user's emotions. For example, if the user's facial expression is smiling, the engine will determine that the user is enjoying themselves.

[0706] Step 5:

[0707] The device simultaneously sends emotion engine data and user learning progress to the server. Based on this data, the server comprehensively analyzes the user's current learning and emotional state.

[0708] Step 6:

[0709] Based on the analysis results, the server generates emotionally responsive feedback. For example, if a user displays a confused expression, the server generates a helpful explanation and sends it to the device.

[0710] Step 7:

[0711] The device receives feedback from the server and presents the user with appropriate information. The user then uses this information to further their learning or make improvements.

[0712] Step 8:

[0713] Once a learning session ends, the server saves the data and prepares to analyze it for use in future learning plans. The emotional data generated by the emotion engine is accumulated as data for improvement in long-term learning.

[0714] (Example 2)

[0715] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0716] Traditional education systems provide feedback based on users' learning progress and performance, but they have faced the challenge of providing personalized learning experiences that take into account users' emotional states. Therefore, there is a need to immediately capture the user's feelings of interest, boredom, and lack of understanding during learning, and to flexibly adjust the content of the learning program based on that feedback.

[0717] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0718] In this invention, the server includes means for recognizing the user's emotional state and applying it to the educational process, means for providing an educational program via an information processing device and recording learning progress, and means for integrating and analyzing the recorded learning progress and emotional data. This makes it possible to optimize the learning program according to the user's emotional state and provide an individualized educational experience.

[0719] A "user" is an individual who uses the educational system to engage in learning activities.

[0720] "Emotional analysis means" refers to a technical device or method for detecting and interpreting an emotional state from a user's facial expressions or voice.

[0721] An "information processing device" is a computer or similar electronic device that collects, stores, processes, and communicates data.

[0722] An "educational program" refers to teaching materials and their structure that aim to improve the knowledge and skills of users.

[0723] "Learning progress" is an indicator that shows the degree or stage of learning that a user has achieved through an educational program.

[0724] "Feedback" refers to information and suggestions provided to users to communicate their learning results and level of understanding, and to encourage further improvement in their learning.

[0725] "Analytical tools" refer to the techniques and methods used to evaluate collected data and derive certain conclusions or views.

[0726] A "learning pattern" is a regularity or tendency that can be found based on a user's past learning behavior and results.

[0727] This invention is a system that provides users with personalized educational experiences by combining an information processing device and an emotion analysis means. The system consists of a server and terminals, where the server is responsible for data integration and analysis, and the terminals primarily provide educational programs through user interaction.

[0728] Hardware and software

[0729] The device has a built-in camera and microphone, and these hardware components are used to capture the user's facial expressions and voice data as a means of sentiment analysis. The video and voice data are sent to the sentiment analysis engine in real time and analyzed by OpenCV and speech recognition libraries. The server uses machine learning algorithms with programming languages ​​such as Python and R to integrate the collected sentiment data and learning progress data, and generates optimal feedback for the next step.

[0730] Specifically, the emotion analysis engine identifies the user's emotional state and sends this information to the server. The server then uses machine learning models to generate a personalized learning program based on the accumulated data. For example, if the server detects that the user is feeling bored, it will recommend new learning content that will pique the user's interest.

[0731] Specific example

[0732] For example, if a user is participating in a program to "learn about historical events," and the emotion analysis engine detects from the user's facial expressions that they are feeling bored, the server will sense this. The server will then analyze the user's past interests and present content such as "documentary about famous historical figures" on the device. This will rekindle the user's interest and allow them to continue learning effectively.

[0733] Example of a prompt

[0734] "If a user's facial expression shows confusion while they are solving a math problem, what additional explanation should be provided?"

[0735] In this way, the system enables users to receive a flexible and effective learning experience through dynamic feedback mediated by emotion analysis.

[0736] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0737] Step 1:

[0738] The user logs into their device and enters their profile information. This information is sent from the device to the server, where it is compared against the user's past learning history and registration information. Based on this data, the server generates personalized educational program suggestions and sends them to the device. The user is then provided with input for selection, and a list of program suggestions is displayed as output.

[0739] Step 2:

[0740] The user selects their desired educational program from the displayed program options. The selected program information is sent from the terminal to the server. The server receives the selection information, organizes the detailed data of the corresponding program, and sends it to the terminal. In response to the user's input to begin learning, the learning content is displayed as output.

[0741] Step 3:

[0742] The device activates its camera and microphone to perform emotion analysis as the user begins learning. The emotion analysis engine captures the user's facial expressions and voice, and analyzes the acquired data in real time. The captured data is acquired as input, and the user's emotional state is recognized as output.

[0743] Step 4:

[0744] The emotion data recognized by the emotion analysis engine is sent to the server via the terminal. The server integrates the emotion data and learning progress data and performs analysis. The input consists of emotion data and learning data, and based on this, the server evaluates the user's current level of understanding and emotional state, and generates appropriate feedback as output.

[0745] Step 5:

[0746] The server sends the generated feedback to the terminal and presents it to the user. Specifically, if it determines that the user is bored, it instructs the server to display engaging videos or additional content. The input for presenting the feedback is the evaluation result described above, and the output is the feedback displayed on the user's interface.

[0747] Step 6:

[0748] Long-term data collected through sentiment analysis is stored on a server and used to shape the user's learning patterns. The server analyzes this historical data and uses it as training data for a generative AI model that proposes future learning programs. The input is the stored learning history, and the output is a prediction of future learning patterns.

[0749] The data and processing obtained at each step are used to provide the most effective learning experience for users, ultimately aiming to improve the quality of education.

[0750] (Application Example 2)

[0751] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0752] Traditional educational support systems can provide feedback based on a user's learning progress, but they have struggled to provide adaptive support based on the user's emotional responses. As a result, the impact of the user's emotional aspects on learning effectiveness is often ignored, and it has been difficult to provide a learning experience optimized for each individual user. Furthermore, there is the challenge of limited operation in environments with unstable internet connections.

[0753] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0754] In this invention, the server includes means for providing educational programs via an information processing device, means for recording and analyzing the user's learning progress, means for providing the optimal response and next-priority educational content based on the analysis results, means for presenting educational information in multiple languages, means for pre-storing information to enable operation even under unstable network connection conditions, means for recognizing the user's emotional response using an emotion recognition device, and means for dynamically adjusting the educational program in accordance with the recognized emotional response. This makes it possible to provide an individualized learning experience that corresponds to the user's emotional state and improve the quality of education.

[0755] An "information processing device" is a device that has the function of providing and processing educational programs with users.

[0756] A "user" refers to a person who uses an educational program and whose learning progress is recorded.

[0757] "Learning progress" refers to information that indicates the state or stage of learning achieved by a user through an educational program.

[0758] "Analysis" is the process of analyzing user learning progress data and emotional response data to improve educational content.

[0759] "Feedback" refers to information and instructions provided to improve the user's learning experience.

[0760] "Educational information" refers to teaching materials and knowledge used to support users' learning.

[0761] A "network connection" is a connection that allows information processing devices to communicate data over the internet.

[0762] An "emotion recognition device" is a device that analyzes a user's facial expressions and voice to identify their emotions.

[0763] "Emotional response" refers to the changes or states of emotion that a user exhibits while learning.

[0764] An "educational program" refers to the learning content and educational methods provided through information processing devices.

[0765] This invention is an educational support system for providing users with an individualized learning experience. The system mainly consists of an information processing device, an emotion recognition device, and a server.

[0766] The information processing device primarily functions as a user interface and provides educational programs. It also captures the user's facial expressions and voice using input devices such as cameras and microphones, and transmits this information to an emotion recognition device.

[0767] The emotion recognition device analyzes the user's emotional responses and generates emotion data. This data is transmitted to the server in real time. Specific software used includes the Facial Expression Recognition API and voice emotion analysis libraries. The server integrates and analyzes the learning progress data and emotion data transmitted from the information processing device.

[0768] Based on this data, the server generates optimal feedback and selects the next priority educational content. The server also uses natural language processing techniques to facilitate user interaction and dynamically adjust the educational program. For example, if the server determines that a user is experiencing stress while working on a problem, it provides helpful hints.

[0769] As a concrete example, imagine a child practicing the piano and attempting a difficult piece. If the child shows signs of distress, the server will generate advice such as, "Try that part again slowly." An example of a prompt for this generating AI model would be, "Analyze how the user's voice tone has changed and evaluate their stress level."

[0770] The system is also designed to continue providing learning support by pre-storing necessary information even in environments with unstable network connections. This makes it possible to deliver a consistent educational experience in any environment.

[0771] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0772] Step 1:

[0773] Users log in to the educational program using an information processing device. The user's login information (username and password) is required as input. The output displays the user's individual profile on the terminal and transitions to the educational program selection screen. User information is sent to the server upon login, and the program starts.

[0774] Step 2:

[0775] The device uses its camera and microphone to capture the user's facial expressions and voice. The input consists of the user's facial expression data and voice data. The output is that this data is transmitted in real time to an emotion recognition device, where it is converted into emotion data. Specifically, the device acquires data every 10 seconds and immediately uploads it to the emotion recognition device.

[0776] Step 3:

[0777] The server integrates emotion data obtained from the emotion recognition device with learning progress data sent from the terminal. It receives emotion data and learning progress data as input. The output is the result of analyzing this data to evaluate the user's learning state. The data processing performed here involves comparing it with past learning history stored in a database.

[0778] Step 4:

[0779] The server generates optimal feedback based on the analysis results and sends it to the terminal. The input for feedback generation is the result of a comprehensive analysis of emotions and learning. The output includes helpful feedback for the user and guidance for the next steps. Specifically, dynamic guidance content is created using a generative AI model. For example, a generated prompt might say, "The user seems confused. Let's try the exercise again slowly."

[0780] Step 5:

[0781] The terminal displays feedback received from the server to the user and suggests new learning content. The input is feedback data from the server, and the output is new learning content and instructions displayed on the user's screen. The terminal updates the user's learning screen and provides navigation to the next learning stage.

[0782] Step 6:

[0783] The user progresses to the suggested new learning stage and continues learning. The input is the new learning task selected by the user, and the output is a record of its progress, which then returns to step 2. This mechanism allows the user to continuously progress in their learning.

[0784] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0785] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0786] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0787] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0788] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0789] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0790] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0791] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0792] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0793] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0794] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0795] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0796] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0797] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0798] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0799] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0800] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0801] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0802] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0803] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0804] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0805] The following is further disclosed regarding the embodiments described above.

[0806] (Claim 1)

[0807] A means of providing educational programs via information terminals,

[0808] A means of recording and analyzing the learning progress of users,

[0809] A means of providing optimal feedback and next-priority learning content based on the analysis results,

[0810] Means of presenting educational information using multiple languages,

[0811] A means of pre-storing information necessary to enable operation even in environments with unstable internet connections,

[0812] A system that includes this.

[0813] (Claim 2)

[0814] The system according to claim 1, which uses natural language processing technology to enable smooth dialogue with the user.

[0815] (Claim 3)

[0816] The system according to claim 1, which presents predicted recommendations based on the user's interests and abilities when the user selects an educational program.

[0817] "Example 1"

[0818] (Claim 1)

[0819] A means of providing educational programs via an information processing device,

[0820] A means of recording the user's learning progress and analyzing it with a generative AI model,

[0821] A means of providing optimal feedback and next-priority learning content based on the analysis results,

[0822] Means of presenting educational information using multiple languages,

[0823] A means of pre-storing information necessary to enable operation even in environments with unstable network connections,

[0824] A method for optimizing the content to be learned next in a generative AI model using prompt sentences,

[0825] A system that includes this.

[0826] (Claim 2)

[0827] The system according to claim 1, which enables smooth dialogue with users using natural language technology.

[0828] (Claim 3)

[0829] The system according to claim 1, which presents predicted recommendations based on the user's preferences and abilities when the user selects an educational program.

[0830] "Application Example 1"

[0831] (Claim 1)

[0832] A means of providing educational programs via an information processing device,

[0833] A means of recording and analyzing learners' learning progress,

[0834] A means of providing optimal feedback and next-priority learning content based on the analysis results,

[0835] Means of presenting educational information in multiple languages,

[0836] A means of pre-storing information necessary to enable operation even in environments with unstable communication connections,

[0837] A means of providing learning support in a home environment using intelligent devices to evaluate learning progress in real time and provide feedback through audio and visual means,

[0838] A system that includes this.

[0839] (Claim 2)

[0840] The system according to claim 1, which facilitates smooth dialogue with learners using natural language processing technology.

[0841] (Claim 3)

[0842] The system according to claim 1, which presents predicted recommendations based on the learner's interests and proficiency when the learner selects an educational program.

[0843] "Example 2 of combining an emotion engine"

[0844] (Claim 1)

[0845] A means of emotional analysis for recognizing the emotional state of users and applying it to the educational process,

[0846] A means of providing educational programs via an information processing device and recording learning progress,

[0847] A means of integrating and analyzing recorded learning progress and emotional data,

[0848] A means of generating optimal feedback based on analysis and adjusting educational content,

[0849] A means of recording users' learning attitudes and emotional changes over the long term and forming learning patterns,

[0850] A system that includes this.

[0851] (Claim 2)

[0852] The system according to claim 1, which uses natural language processing technology to facilitate smooth interaction with the user.

[0853] (Claim 3)

[0854] The system according to claim 1, which recommends educational programs that match the user's interests and abilities based on the user's emotional recognition.

[0855] "Application example 2 when combining with an emotional engine"

[0856] (Claim 1)

[0857] A means of providing educational programs via an information processing device,

[0858] A means of recording and analyzing the user's learning progress,

[0859] A means of providing the optimal response and the next priority educational content based on the analysis results,

[0860] A means of presenting educational information using multiple languages,

[0861] A means of pre-storing information to enable operation even under conditions of unstable network connectivity,

[0862] A means of recognizing a user's emotional response using an emotion recognition device,

[0863] Means for dynamically adjusting educational programs in response to recognized emotional responses,

[0864] A system that includes this.

[0865] (Claim 2)

[0866] The system according to claim 1, which uses natural language processing technology to enable smooth interaction with the user.

[0867] (Claim 3)

[0868] The system according to claim 1, which presents predicted suggestions based on the user's interests and knowledge when the user selects an educational program. [Explanation of symbols]

[0869] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of providing educational programs via information terminals, A means of recording and analyzing the learning progress of users, A means of providing optimal feedback and next-priority learning content based on the analysis results, Means of presenting educational information using multiple languages, A means of pre-storing information necessary to enable operation even in environments with unstable internet connections, A system that includes this.

2. The system according to claim 1, which uses natural language processing technology to enable smooth dialogue with the user.

3. The system according to claim 1, which presents predicted recommendations based on the user's interests and abilities when the user selects an educational program.

Images