System

Abstract

A system is provided.SOLUTION: The system includes a means for recording the learning progress of the student, a means for analyzing the recorded learning progress to specify the understanding degree of the student, a means for specifying a part that the student does not understand based on the specified understanding degree and generating an explanation about it, a means for providing the generated explanation to the student's terminal, a means for automatically generating a practice problem according to the understanding degree, a means for generating an individual learning plan based on the target, the academic ability, the understanding degree, and schedule information of the student, and a means for providing the generated individual learning plan to the student's terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] A major problem faced by junior high school students, especially those enrolled in junior and senior high schools, is that they continue their studies without sufficient understanding. Even if students feel they "don't understand" or "can't understand" specific points, they are unable to ask their teachers, resulting in a decline in the quality of their learning. The objective of this invention is to provide personalized learning support according to each student's learning progress and level of understanding, thereby improving learning effectiveness. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes the following means: a means for recording a student's learning progress in detail using a tablet device, a camera, and an IoT sensor; a means for analyzing the recorded learning progress to determine the student's level of understanding; a means for identifying areas where the student does not understand based on the determined level of understanding and generating explanations for those areas; a means for providing the generated explanations to the student's device; a means for automatically generating exercises according to the student's level of understanding; and a means for generating an individualized learning plan based on the student's goals, academic ability, level of understanding, and schedule information. The system also includes a means for providing the generated individualized learning plan to the student's device, receiving the student's questions, and generating answers using a large-scale language model with reference to information search results. This makes it possible to provide an environment in which students can study efficiently and effectively.

[0006] "Study progress" is an indicator that shows how far a student has progressed in their studies in a particular subject or unit.

[0007] "Level of understanding" is an indicator of how well a student understands a particular subject or unit.

[0008] "Explanation" refers to teaching materials or information that explains specific content in a way that is easy for students to understand.

[0009] "Exercises" are questions that students answer to deepen a particular understanding.

[0010] An "individual learning plan" is a learning schedule and learning content plan that is individually created based on the student's goals, academic ability, level of understanding, and schedule.

[0011] A "tablet device" is a portable computing device used by students for learning.

[0012] A "camera" is a device that captures images and videos and is used to record students' expressions and behavior.

[0013] "IoT sensors" are sensor devices connected to the internet that are used to collect physical information and acquire data about the learning environment.

[0014] A "large-scale language model" is an AI model trained on massive amounts of text data, and has the ability to understand and generate natural language.

[0015] "Information search results" refers to information retrieved from a search engine or database in response to a particular question or query. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The system of the present invention is designed to provide personalized learning support according to the learning progress and level of understanding of each student. Specific embodiments for carrying out the present invention will be described below.

[0038] Acquiring learning records

[0039] Subject: Terminal

[0040] First, when a student logs in to a tablet device, the device starts the student's study session. It records the learning materials (digital texts, video materials, etc.) used by the student and their learning progress (study time, answer status, etc.). It also uses the device's built-in camera and IoT sensors to collect data such as facial expressions and posture. This data is used to comprehensively understand the student's learning situation.

[0041] Analyzing learning data and identifying understanding

[0042] Subject: Server

[0043] When the learning data sent from the device is received by the server, the server analyzes the data and identifies the student's level of understanding. Specifically, it uses machine learning algorithms to analyze the student's answer patterns and learning progress and identify which parts of the learning content they have not understood.

[0044] AI-powered commentary

[0045] Subject: Server

[0046] The server then uses AI technology to generate explanations for the identified areas of incomprehension. The explanations are generated in a format that is easy for students to understand (e.g., text, diagrams, videos, etc.). These explanations are then provided to students' devices so that they can access them at any time.

[0047] Automatic generation of exercises

[0048] Subject: Server

[0049] Based on the students' level of understanding, the server automatically generates appropriate exercises using a generative AI model, allowing them to tackle problems that are tailored to their individual level of understanding. The generated exercises are provided to the students' devices, and they can further deepen their understanding by answering them.

[0050] Generate personalized learning plans

[0051] Subject: Server

[0052] The server then generates an individual learning plan based on the student's goals, academic ability, level of understanding, and schedule information. This learning plan is customized by interacting with the student, incorporating their wishes and goals. The generated learning plan is provided to the student's device, allowing the student to proceed with their studies accordingly.

[0053] RAG (Retriever-Augmented Generation) module

[0054] Subject: Server

[0055] Finally, if a student has a question during their study, the server receives the question and generates the best answer using a large-scale language model based on the information search results. This answer is then provided to the student's device, allowing the student to immediately resolve their doubts.

[0056] Specific examples

[0057] 1. User logs in

[0058] The device verifies the student's credentials and begins the learning session.

[0059] 2. Obtaining training data

[0060] The device collects learning progress information and facial expression data and sends them to the server.

[0061] 3. Providing analysis and commentary

[0062] The server analyzes the data to identify areas where students lack understanding, and then uses AI to generate explanations that are provided to the device.

[0063] 4. Exercises and Study Plans

[0064] The server automatically generates appropriate exercises and creates a learning plan based on the student's level of understanding, which is then provided to the terminal.

[0065] 5. Answering questions

[0066] When a student submits a question, the server searches for information, generates the most appropriate answer, and provides it to the device.

[0067] This allows students to study effectively at their own pace and at their own level of understanding.

[0068] The processing flow will be explained below.

[0069] Step 1:

[0070] The device checks the student's credentials

[0071] The device receives the student's login information (user ID and password) and authenticates it with the database. If authentication is successful, the device accesses the student's individual account and begins the learning session.

[0072] Step 2:

[0073] Your device records your study session

[0074] The device records the learning materials selected by the student (e.g., mathematics textbook, history video, etc.) and also records input data from the touch panel and keyboard (answer time, options, answer results, etc.).

[0075] Step 3:

[0076] The device collects data using cameras and IoT sensors

[0077] The device's built-in camera captures students' facial expressions, and IoT sensors monitor their posture and movements while they study, collecting data in real time.

[0078] Step 4:

[0079] The device sends the learning data to the server.

[0080] When the learning session ends or a certain period of time has passed, the device sends all collected data (teaching materials, answer data, facial expression data, etc.) to the server.

[0081] Step 5:

[0082] The server receives and analyzes the learning data.

[0083] The server receives the learning data sent from the device and analyzes it using machine learning algorithms, which identifies areas where the student does not understand and their learning trends.

[0084] Step 6:

[0085] The server generates an explanation for any missing information

[0086] The server uses AI technology to create detailed explanations of the identified areas of lack of understanding, which can be in the form of text, illustrations, videos, etc.

[0087] Step 7:

[0088] The server sends the commentary to the device.

[0089] The server generates explanations that are sent to students' devices so that they can access them at any time. The explanations are saved in individual student accounts.

[0090] Step 8:

[0091] The server automatically generates exercises

[0092] The server uses the generative AI model to automatically generate practice questions that address the identified gaps in understanding, adjusting the difficulty and format of the questions according to the student's level of understanding.

[0093] Step 9:

[0094] The server sends the exercises to the device.

[0095] The server sends the generated exercises to the students' devices so that they can work on them. The answers are also recorded sequentially.

[0096] Step 10:

[0097] The server generates an individualized learning plan.

[0098] The server generates an individualized learning plan based on the student's goals, academic ability, level of understanding, and schedule information. This plan is interactive and incorporates the student's requests.

[0099] Step 11:

[0100] The server sends the lesson plan to the device.

[0101] The server transmits the generated individual learning plan to the student's terminal, allowing the student to progress with their learning according to the daily learning plan.

[0102] Step 12:

[0103] The device will carry out learning according to the learning plan.

[0104] Students progress through their studies according to the study plan displayed on their device and access explanations and practice problems as needed.

[0105] Step 13:

[0106] The server receives the student's questions and generates answers.

[0107] When a student submits a question during their studies, the server receives the question and generates an answer using a large-scale language model based on the information search results.

[0108] Step 14:

[0109] The server generates a response and sends it to the device.

[0110] The server generates answers that are sent to students' devices to help them resolve their questions.

[0111] This will help each student to progress effectively in their studies.

[0112] Example 1

[0113] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0114] In recent years, there has been a demand for personalized learning support tailored to each learner's individual learning progress and level of understanding. However, conventional learning support systems have had problems in that they are insufficient in providing specific explanations and practice problems based on each learner's individual level of understanding, and in responding immediately to students' questions. This has made it difficult for students to study efficiently and effectively. In addition, the accuracy of collecting and analyzing learning data has been low, making it difficult to provide accurate learning support.

[0115] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0116] In this invention, the server includes means for recording the student's learning progress, means for analyzing the recorded learning progress to determine the student's level of understanding, means for determining the student's lack of understanding based on the determined level of understanding and generating explanations for the identified areas, means for providing the generated explanations to the student's terminal, means for automatically generating exercises according to the student's level of understanding, means for generating an individual learning plan based on the student's goals, academic ability, level of understanding, and schedule information, means for providing the generated individual learning plan to the student's terminal, means for receiving the student's questions and generating answers using natural language processing technology with reference to information search results, means for collecting the student's facial expression and posture data at the terminal, means for transmitting the learning data to the server in real time, means for analyzing the data using a machine learning algorithm, and means for generating explanations using artificial intelligence. This makes it possible to provide specific and effective support according to each student's individual learning progress.

[0117] "Learning progress" is an indicator that shows how much content a student has learned through learning activities and how much time they have spent.

[0118] "Recording means" refers to devices or software for storing students' learning progress, study time, answer status, and other related data in a database or cloud storage.

[0119] "Means for analysis" refers to software or systems that apply machine learning algorithms and statistical methods to assess students' learning progress and understanding using collected learning data.

[0120] "Level of understanding" is an indicator of how well a student has understood the learning content, including the accuracy and speed of answers and identification of areas of lack of understanding.

[0121] A "means for generating explanations" is a device or software that uses artificial intelligence or other automatic generation technology to create easy-to-understand explanations for students in the form of text, illustrations, videos, etc. for identified areas of lack of understanding.

[0122] "Practice questions" are questions that students answer to review what they have learned and check their understanding.

[0123] A "means for automatically generating exercises" is a device or software that uses artificial intelligence or generation technology to automatically generate exercises of appropriate difficulty and content based on the student's level of understanding.

[0124] A "study plan" is a plan that includes the learning content and schedule established for a student to achieve their goals.

[0125] A "means for generating an individual learning plan" is a device or software that creates and provides an individually optimized learning plan based on a student's goals, academic ability, level of understanding, schedule information, etc.

[0126] A "question receiving" means is an interface, communication device, or software that allows students to submit questions or concerns online while studying.

[0127] "Information search results" are the results obtained by searching databases and the Internet for information related to a student's question.

[0128] "Natural language processing technology" is a field of artificial intelligence and is a technology for understanding and generating human language.

[0129] "Facial expression and posture data" refers to information about facial expressions and body postures captured to understand students' learning status.

[0130] "Means for transmitting in real time" refers to communication equipment and communication protocols for transmitting learning data from the collection source to the server without delay.

[0131] A "machine learning algorithm" is a mathematical model and computational method for analyzing collected data and extracting patterns and features.

[0132] "Artificial intelligence" refers to systems and software that mimic human intelligence and perform learning and reasoning, and is used to generate explanations and questions.

[0133] The system of the present invention provides personalized learning support according to the student's learning progress and level of understanding. Specific embodiments for carrying out the present invention will be described below.

[0134] Hardware and Software Configuration

[0135] This system uses tablet devices, cameras, and IoT sensors to collect learning data, which is then analyzed on a server. Specifically, it uses the following hardware and software:

[0136] Hardware:

[0137] Tablet: A device used by students for learning.

[0138] Camera: Used to capture students' facial expressions.

[0139] IoT sensors: Used to collect students' posture data.

[0140] software:

[0141] Authentication system: Authentication system using OAuth or JWT.

[0142] Data collection software: Software for collecting learning progress, facial expressions, and posture data.

[0143] Data analysis software: TensorFlow and Scikit-learn are used to analyze data using machine learning algorithms.

[0144] Generative AI model: Generates explanations and practice problems using models such as OpenAI's GPT-4.

[0145] Natural language processing systems: Answering student questions using large-scale language models.

[0146] Processing flow and specific examples

[0147] Students log in and start a study session

[0148] When a student logs in, the device checks the authentication information and starts the learning session. For example, authentication is performed by entering a user ID and password. At this time, the device ensures security by using JWT authentication.

[0149] Collecting and sending learning data

[0150] During a learning session, the device records data such as the student's viewing status, answer status, and study time. It also records the student's facial expressions and posture data acquired through built-in cameras and IoT sensors. The acquired data is sent to a server in real time.

[0151] Analysis of training data

[0152] The server analyzes the data received from the devices using machine learning algorithms. Specifically, it uses the collected data to evaluate students' answer patterns and learning progress, and identifies their level of understanding. This analysis is performed using TensorFlow and Scikit-learn.

[0153] Generate explanations for areas of incomprehension

[0154] The server then uses a generative AI model to generate explanations for identified gaps in understanding. For example, OpenAI's GPT-4 is used to generate explanations in the form of text, diagrams, or videos. These explanations are then provided to students' devices for access.

[0155] Automatic generation of exercises

[0156] Based on the student's level of understanding, the server uses a generative AI model to generate appropriate exercises. The generated exercises are then provided to the student's device, and the student deepens their understanding by answering them.

[0157] Prompt Sentence Examples

[0158] Here are some examples of specific prompt sentences:

[0159] To generate explanations for areas of incomprehension:

[0160] "My students have a limited understanding of differential calculus in Chapter 5. I would like you to write an easy-to-understand explanation."

[0161] For generating practice questions:

[0162] "Generate basic differential calculus problems based on students' understanding."

[0163] Generate personalized learning plans

[0164] The server generates an individual learning plan based on the student's goals, schedule, and level of understanding, and provides it to the student's device. This plan incorporates the student's goals and optimizes daily learning.

[0165] Answering questions during study

[0166] When a student submits a question during their study, the RAG module on the server receives the question and uses natural language processing technology to generate the most appropriate answer, which is then provided to the device.

[0167] This detailed processing flow makes it possible to provide specific and effective support according to each student's learning progress.

[0168] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0169] Step 1: Students log in and start a study session

[0170] Subject: Terminal

[0171] When a student logs in to the device, they are prompted to enter their user ID and password. The entered information is sent to the authentication server, which performs authentication using JWT authentication. If authentication is successful, the device starts a learning session and displays the dashboard screen. The input is the user ID and password, and the output is the authentication result and the start of the learning session.

[0172] Specific behavior:

[0173] Input: User ID, Password

[0174] Data processing: JWT authentication

[0175] Output: Authentication results, dashboard screen

[0176] Step 2: Collect and send training data

[0177] Subject: Terminal

[0178] During a learning session, the device records data such as the student's viewing status, answer status, and study time. It also uses a built-in camera and IoT sensors to collect data on the student's facial expression and posture. This data is sent to a server in real time. The input is learning progress information, facial expression data, and posture data, and the output is data sent to the server.

[0179] Specific behavior:

[0180] Input: learning progress information, facial expression data, posture data

[0181] Data processing: Data recording, sensor information collection

[0182] Output: Send data to the server

[0183] Step 3: Analyze the training data

[0184] Subject: Server

[0185] The server receives the data sent from the device and analyzes it using machine learning algorithms. The analysis evaluates the student's answer patterns and learning progress, and identifies their level of understanding. TensorFlow and Scikit-learn are used for this analysis. The input is the learning data sent from the device, and the output is the level of understanding information obtained as a result of the analysis.

[0186] Specific behavior:

[0187] Input: Training data from the device

[0188] Data processing: Analysis using machine learning algorithms

[0189] Output: Comprehension information

[0190] Step 4: Generate explanations for areas of incomprehension

[0191] Subject: Server

[0192] The server uses a generative AI model to generate explanations for the identified areas of incomprehension. For example, by inputting a prompt into the GPT-4 model, it generates explanatory text, illustrations, and videos. The input is information about the area of ​​incomprehension and the prompt, and the output is the generated explanation.

[0193] Specific behavior:

[0194] Input: Information on areas of incomprehension, prompt text

[0195] Data processing: Generative AI model for generating explanations

[0196] Output: explanatory text, illustrations, videos

[0197] Step 5: Automatic generation of exercises

[0198] Subject: Server

[0199] The server generates appropriate exercises using a generative AI model based on the student's level of comprehension. The generated exercises are sent to the student's device. The input is comprehension information and a prompt, and the output is the generated exercise.

[0200] Specific behavior:

[0201] Input: Comprehension information, prompt

[0202] Data processing: Generative AI model for generating exercises

[0203] Output: Exercises

[0204] Step 6: Generate an individualized learning plan

[0205] Subject: Server

[0206] The server generates an individual learning plan based on the student's goals, academic ability, level of understanding, and schedule information. The generated learning plan is provided to the student's device. The input is the student's goals, academic ability, level of understanding, and schedule information, and the output is the individual learning plan.

[0207] Specific behavior:

[0208] Input: Goals, academic ability, level of understanding, schedule information

[0209] Data processing: Planning using learning plan generation algorithms

[0210] Output: Individualized Learning Plan

[0211] Step 7: Answering questions while studying

[0212] Subject: Server

[0213] The server receives questions sent by students during their studies, and generates optimal answers using a large-scale language model based on information search results. These answers are then provided to the students' devices. The input is the student's question, and the output is the generated answer.

[0214] Specific behavior:

[0215] Input: Student question

[0216] Data Processing: Natural Language Processing and Information Retrieval

[0217] Output: Best answer

[0218] (Application example 1)

[0219] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0220] Conventional factory worker education and training systems lacked personalized support tailored to each worker's learning progress and skill level, making it difficult to efficiently improve skills. They also lacked a means to respond to workers' questions in real time and provide specific feedback. This created the challenge of not maximizing the effectiveness of worker training.

[0221] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0222] In this invention, the server includes means for recording the worker's learning progress, means for analyzing the recorded learning progress to identify the worker's level of understanding, means for identifying areas where the worker does not understand based on the identified level of understanding and generating explanations for those areas, means for providing the generated explanations to the worker's terminal, means for automatically generating exercises according to the worker's level of understanding, means for generating an individual learning plan based on the worker's goals, skills, level of understanding, and schedule information, means for providing the generated individual learning plan to the worker's terminal, means for collecting and analyzing the worker's operation records and posture data, and means for receiving the worker's questions and generating answers using a generative AI model with reference to the information acquisition results. This makes it possible to provide personalized training plans based on each worker's level of understanding and respond to questions in real time.

[0223] "Workers" refers to the technicians and workers who operate machinery and equipment in factories and manufacturing sites. They are the recipients of training according to their individual skill levels and learning progress.

[0224] "Learning progress" refers to the progress a worker makes along a training or education program, specifically the degree to which knowledge and skills have been acquired.

[0225] "Level of understanding" refers to the standard for assessing how well a worker understands the learning content and operating procedures. It is an important indicator for providing feedback to improve identified areas of lack of understanding.

[0226] "Explanation" refers to content that provides additional information or explanations for areas where workers are struggling, often in the form of text, video, or AR guides.

[0227] "Devices" refer to electronic devices used by workers for operation and learning, including tablet devices and head-mounted displays.

[0228] "Exercises" refer to assignments and tasks provided to workers to allow them to try out what they have learned. They are automatically generated based on each worker's level of understanding.

[0229] "Individualized Learning Plan" refers to a customized education and training schedule based on a worker's goals, skills, level of understanding, and schedule information.

[0230] "Operation records" refer to the history of specific operations and tasks performed by workers. Recording these records will be useful for analysis.

[0231] "Posture data" refers to data on the body movements and posture of workers when operating a machine. It is acquired using sensors and cameras.

[0232] A "generative AI model" refers to an artificial intelligence algorithm or system that automatically generates information based on large datasets, which can be used to generate answers to human questions.

[0233] "Information acquisition results" refers to information collected from the internet or other databases in response to worker questions. This is part of the data used by the generative AI model.

[0234] This invention is a system for personalizing education and training for factory workers. This system records and analyzes the worker's learning progress and provides feedback, practice questions, and individual learning plans based on the worker's level of understanding.

[0235] Hardware and software used

[0236] The hardware used includes a tablet computer, a head-mounted display (HMD), a camera, and IoT sensors. This hardware is used to acquire the worker's operation records and posture data. The software used includes a learning data collection app, a data analysis server, an AI explanation system, an exercise problem generation system, and a RAG (Retriever-Augmented Generation) module.

[0237] Acquiring and analyzing training data

[0238] The device records the worker's learning progress. Specifically, it collects the worker's operations and work history, as well as posture and movement data acquired by cameras and IoT sensors. This data is sent to a server in real time.

[0239] The server analyzes the received data and identifies the worker's level of understanding. The analysis uses a machine learning algorithm to identify the areas where each worker lacks understanding. The server then generates an explanation corresponding to the area of ​​lack of understanding and provides it to the worker's device.

[0240] Providing AI explanations and practice questions

[0241] The server uses AI technology to generate easy-to-understand explanations for the identified areas of insufficient understanding. These explanations are created in the form of text, video, AR guides, etc. and are provided to the worker's device. Exercises are also automatically generated based on the worker's level of understanding. The exercises are set at an appropriate level of difficulty with the aim of improving the worker's skills.

[0242] Generate and provide individualized learning plans

[0243] The server generates an individual learning plan based on the worker's goals, skills, level of understanding, and schedule information. This allows the worker to learn at their own pace, enabling effective training. The generated learning plan is provided to the terminal, and the worker follows it to carry out training.

[0244] Real-time question response (RAG module)

[0245] If a worker has a question during learning, the server receives the question and generates an answer using the generative AI model based on the information acquisition results. This answer is provided to the worker's device in real time, instantly resolving the worker's question.

[0246] Examples of concrete examples and prompts

[0247] As an example, consider a worker wearing a head-mounted display and undergoing training to operate a robot. The worker's operation records and posture data are acquired by the terminal and sent to the server. After analyzing the data, the server generates video explanations for any operating procedures that the worker does not fully understand and provides them to the terminal. In addition, it automatically generates exercises related to the operation and presents them to the worker as assignments. If the worker asks a question during this process, the server uses the RAG module to instantly generate an answer and provides it to the terminal.

[0248] An example of a prompt sentence might be:

[0249] "We would like to develop a training system for robot operation in factories. Please come up with an application that provides individual feedback, practice assignments, and training plans based on the worker's skill level and progress."

[0250] As described above, the system of the present invention provides personalized education and training to workers, and supports efficient skill improvement.

[0251] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0252] Step 1:

[0253] The user logs in to the device. The device receives the user's authentication information as input and performs authentication to start a learning session. If authentication is successful, the device obtains the user's past learning data and current learning goals.

[0254] Step 2:

[0255] The device records the user's operations and learning progress in real time. Specifically, the device collects operation records and posture data using cameras and sensors installed on the tablet device or head-mounted display. This data is input and sent to a server.

[0256] Step 3:

[0257] The server analyzes the received data. It receives operation records and posture data as input, and uses a machine learning algorithm to evaluate the user's level of understanding. As a result of the analysis, it identifies and outputs the areas where the user does not understand or where they lack skills.

[0258] Step 4:

[0259] The server generates explanations for the identified areas of incomprehension, using AI technology to create explanations in the form of text, video, or AR guides. Using this as input, the server generates explanation data in the appropriate format and outputs it to be provided to the device.

[0260] Step 5:

[0261] The server automatically generates exercises based on the user's level of understanding and skills. It uses the analysis results as input and creates exercises using a generative AI model. This outputs appropriate exercise data aimed at improving the user's skills.

[0262] Step 6:

[0263] The server generates a personalized learning plan based on the user's goals, skills, level of understanding, and schedule information. It uses the user's profile data and analysis results as input to create an optimal learning schedule. This plan is output for delivery to the device.

[0264] Step 7:

[0265] When a user asks a question during learning, the device sends the question to the server. The device receives the question data as input, uses the RAG module to refer to the information acquisition results, and generates an answer using the generative AI model. The generated answer is sent to the device and provided to the user.

[0266] These steps explain how each task is performed and what input data is used to generate output data, enabling personalized training and real-time question answering.

[0267] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0268] The system of the present invention records and analyzes students' learning progress and level of understanding in detail to provide personalized learning support. This system is also combined with an emotion engine that recognizes the user's emotions. Specific embodiments for carrying out the present invention will be described below.

[0269] Acquiring training data and emotion recognition

[0270] Subject: Terminal

[0271] When a student logs in to a tablet device, the device begins a learning session. While recording data used during the learning process (such as learning material content and answer status), the device also collects the student's facial expression data using the built-in camera. This collected facial expression data is sent in real time to an emotion engine that recognizes the student's emotional state (concentration, confusion, anxiety, etc.).

[0272] Sending training data and emotion data

[0273] Subject: Terminal

[0274] When the learning session ends or after a certain period of time has passed, the device sends the collected learning data and emotion data to the server. This data includes the learning content, answer data, facial expression data, and emotion recognition results.

[0275] Analyzing data and identifying understanding

[0276] Subject: Server

[0277] The server receives the learning data and emotional data sent from the device and first analyzes the student's learning progress and level of understanding using a machine learning algorithm. From the analysis results, it identifies areas where understanding is lacking, and at the same time, taking into account the results of the emotional engine, it also understands the emotional state the student was in while studying.

[0278] Generate personalized commentary

[0279] Subject: Server

[0280] For identified gaps in understanding, the server uses AI technology to generate detailed explanations. The explanations take into account the student's emotional state and are created in context-appropriate formats such as text, diagrams, and videos. Furthermore, these explanations are provided to students' devices so they can review them repeatedly.

[0281] Automatic generation of exercises

[0282] Subject: Server

[0283] The server uses a generative AI model to automatically generate practice questions that address areas of incomplete understanding, allowing students to tackle questions of the appropriate difficulty and format. The practice questions are also provided to the device, and the answers are collected again.

[0284] Generate personalized learning plans

[0285] Subject: Server

[0286] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. This learning plan is optimized by incorporating the student's requests through an interactive format. The generated learning plan is provided to the student's device and serves as a guide for daily learning.

[0287] Question handling and emotional care

[0288] Subject: Server

[0289] When a student submits a question during learning, the server receives the question, performs an information search, and uses the results to generate the optimal answer using a large-scale language model. The answer is then provided to the device, instantly resolving the student's question. The system also uses an emotion engine to provide appropriate feedback and emotional support according to the student's emotional state during learning.

[0290] Specific examples

[0291] 1. User logs in

[0292] The device verifies the student's credentials and begins the learning session.

[0293] 2. Collecting training data and emotion data

[0294] The device collects learning progress information and facial expression data, which are then analyzed in real time using an emotion engine.

[0295] 3. Sending and Receiving Data

[0296] The terminal sends the collected data to the server, which receives it and analyzes it.

[0297] 4. Providing analysis and commentary

[0298] The server analyzes the data to identify areas of lack of understanding, and then uses AI to generate explanations that are provided to the device.

[0299] 5. Exercises and Study Plans

[0300] The server automatically generates appropriate practice questions, creates an individual learning plan, and provides it to the terminal.

[0301] 6. Answering questions and providing emotional support

[0302] The server receives the question, generates an answer using a large-scale language model based on the information search results, and provides it to the device. It also provides appropriate feedback depending on the user's emotional state.

[0303] In this way, this system provides more personalized learning support that also takes into account the student's emotional state.

[0304] The processing flow will be explained below.

[0305] Step 1:

[0306] The device checks the student's credentials

[0307] The device receives the student's login information (user ID and password) and authenticates it with the database. If authentication is successful, the device accesses the student's individual account and begins the learning session.

[0308] Step 2:

[0309] Your device records your study session

[0310] The device records the learning materials selected by the student (e.g., mathematics textbook, history video, etc.) and also records input data from the touch panel and keyboard (answer time, options, answer results, etc.).

[0311] Step 3:

[0312] The device collects data using cameras and IoT sensors

[0313] The device's built-in camera captures students' facial expressions, and IoT sensors monitor their posture and movements while they study, collecting data in real time.

[0314] Step 4:

[0315] The device sends facial expression data to the emotion engine to recognize emotions.

[0316] The device sends the collected facial expression data to the emotion engine, which then analyzes it to recognize the student's emotional state (e.g., concentration, confusion, anxiety, etc.) in real time.

[0317] Step 5:

[0318] The device sends the learning data and emotion data to the server.

[0319] When the learning session ends or after a certain period of time has passed, the device sends all collected data (teaching materials, answer data, facial expression data, emotional data, etc.) to the server.

[0320] Step 6:

[0321] The server receives and analyzes the learning data and emotion data.

[0322] The server receives the learning data and emotion data sent from the device, then analyzes the data using machine learning algorithms and an emotion engine to identify the student's learning progress and level of understanding.

[0323] Step 7:

[0324] The server generates an explanation for any missing information

[0325] The server uses AI technology to generate detailed explanations for the identified areas of lack of understanding. These explanations are provided to students in the form of text, illustrations, videos, etc.

[0326] Step 8:

[0327] Sends server-generated commentary to the device

[0328] The server generates explanations that are sent to students' devices so that they can access them at any time. The explanations are saved in individual student accounts.

[0329] Step 9:

[0330] The server automatically generates exercises

[0331] The server uses the generative AI model to automatically generate practice questions that address areas of incomprehension, adjusting the difficulty and format of the questions according to the student's level of understanding and emotional state.

[0332] Step 10:

[0333] The server sends the generated exercises to the device.

[0334] The server sends the generated exercises to the students' devices so that they can work on them. The answers are collected again and sent to the server.

[0335] Step 11:

[0336] The server generates an individualized learning plan.

[0337] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. This plan is interactive and incorporates the student's requests.

[0338] Step 12:

[0339] The server sends the generated lesson plan to the device.

[0340] The server transmits the generated individual learning plan to the student's terminal, allowing the student to proceed with their studies in accordance with it.

[0341] Step 13:

[0342] The device will carry out learning according to the learning plan.

[0343] Students progress through their studies according to the study plan displayed on their device and access explanations and practice problems as needed.

[0344] Step 14:

[0345] The server receives the student's questions and generates answers.

[0346] When students submit questions during their studies, the server receives the questions and generates the best answer using a large-scale language model based on the information search results.

[0347] Step 15:

[0348] The server generates a response and sends it to the device.

[0349] The server generates answers and sends them to students' devices to help them resolve their questions. It also uses an emotion engine to provide appropriate feedback and emotional support according to the student's emotional state during the course of their studies.

[0350] Example 2

[0351] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0352] Existing learning support systems are not only unable to record and analyze students' learning progress and comprehension, but also unable to provide personalized learning support that takes into account students' emotional state. Furthermore, they lack the ability to generate quick and appropriate answers to students' questions. This makes it difficult to provide learning plans tailored to individual students' needs.

[0353] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for recording the student's learning progress, means for analyzing the recorded learning progress to identify the student's level of understanding, means for analyzing the recorded facial expression data to identify the student's emotional state, means for identifying areas where the student does not understand based on the identified level of understanding and emotional state and generating explanations for those areas, means for automatically generating exercises according to the student's level of understanding, means for generating an individual learning plan based on the student's goals, academic ability, level of understanding, emotional state, and schedule information, means for providing the generated individual learning plan to the student's terminal, and means for receiving the student's question and generating an answer using a large-scale language model with reference to information search results. This enables individual learning support based on the student's learning progress, level of understanding, and emotional state.

[0354] "Learning progress" is data that shows how far a student has progressed with a particular assignment or material.

[0355] "Understanding" is data that indicates how well a student understands a particular material or concept.

[0356] "Emotional state" is data that indicates a student's psychological state (concentration, confusion, anxiety, etc.) while studying.

[0357] A "tablet device" is a portable computer device used by students when studying.

[0358] "Means of recording" refers to a system for collecting and storing data such as students' learning progress and emotional state.

[0359] "Means for analysis" are algorithms or devices that analyze collected data and assess learning progress, comprehension, and emotional state.

[0360] "Generative means" refers to algorithms or devices that create new explanations, exercises, study plans, etc. based on data.

[0361] The "means of provision" refers to a mechanism for sending and displaying the generated explanations, exercises, and study plans on students' devices.

[0362] "Exercises" are problems that students should solve to overcome specific areas of understanding.

[0363] A "learning plan" is a plan that designs the optimal learning process based on a student's goals and schedule.

[0364] "Means for receiving questions" refers to the system for accepting questions from students and interpreting the content of those questions.

[0365] The "means for generating answers" are large-scale language models and information retrieval systems that provide optimal answers to students' questions.

[0366] "Facial expression data" refers to data showing the facial expressions of students captured using the built-in camera.

[0367] An "emotion engine" is software or hardware that analyzes facial expression data to identify a student's emotional state in real time.

[0368] A "large-scale language model" is an advanced AI model for natural language processing that is trained using large amounts of text data.

[0369] The system of the present invention records and analyzes students' learning progress and level of understanding in detail to provide personalized learning support. The system is also combined with an emotion engine that recognizes the user's emotions. Specific embodiments for carrying out the present invention will be described below.

[0370] System Overview

[0371] The system consists of a tablet device, a server, a built-in camera, sensors, an emotion engine, and a generative AI model. The system's purpose is to provide individualized learning support based on a student's learning progress, level of understanding, and emotional state.

[0372] Student Login

[0373] Subject: User

[0374] The user logs in by entering their user ID and password on the tablet device. The device sends this authentication information to the authentication server and receives the authentication result.

[0375] Collection of training data and emotion data

[0376] Subject: Terminal

[0377] During a learning session, the device records the learning materials used by the student, their answers, and facial expression data captured by a built-in camera in real time. The facial expression data is then sent to an emotion engine to recognize their emotional state (concentration, confusion, anxiety, etc.).

[0378] Sending data

[0379] Subject: Terminal

[0380] When the learning session ends or after a certain period of time has passed, the device sends the collected learning data and emotion data to the server.

[0381] Analyzing data and identifying understanding

[0382] Subject: Server

[0383] The server receives the learning data and emotional data sent from the device and uses machine learning algorithms to analyze the student's learning progress and level of understanding.

[0384] Generate personalized commentary

[0385] Subject: Server

[0386] The server uses AI technology to generate detailed explanations for areas of learning deficiencies, taking into account the student's emotional state, and is created in the form of text, diagrams, videos, etc.

[0387] Automatic generation of exercises

[0388] Subject: Server

[0389] The server uses a generative AI model to automatically generate exercises that address areas of insufficient understanding, and these exercises are provided to the device.

[0390] Generate personalized learning plans

[0391] Subject: Server

[0392] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. The generated learning plan is provided to the device and serves as a daily learning guide.

[0393] Question handling and emotional care

[0394] Subject: Server

[0395] When students submit questions during learning, the server receives the questions and generates optimal answers using information retrieval and large-scale language models. It also provides appropriate feedback and emotional care through an emotion engine.

[0396] Specific examples

[0397] 1. User logs in

[0398] The device verifies the student's credentials and begins the learning session.

[0399] 2. Collecting training data and emotion data

[0400] The device collects learning progress information and facial expression data, which are then analyzed in real time using an emotion engine.

[0401] 3. Data submission and analysis

[0402] The terminal sends the collected data to the server, which receives it and analyzes it.

[0403] 4. Generating explanations based on analysis results

[0404] The server analyzes the data to identify areas of lack of understanding, and then uses AI to generate explanations that are provided to the device.

[0405] 5. Generating exercises and study plans

[0406] The server automatically generates appropriate practice questions, creates an individual learning plan, and provides it to the terminal.

[0407] 6. Answering questions and providing emotional support

[0408] The server receives the question, generates an answer using a large-scale language model based on the information search results, and provides it to the device. It also provides appropriate feedback depending on the user's emotional state.

[0409] Examples of prompts for generative AI models

[0410] 1. Prompt for student questions:

[0411] "A student is asking about XXX, with a specific point YYY. Based on this information, please suggest how to provide an answer that will satisfy the student."

[0412] 2. Prompt for generating explanation:

[0413] "Student demonstrates a lack of understanding about XXX. Student's emotional state is ZZZ. Please take this information into consideration and create a detailed explanation including text, illustrations, and video."

[0414] This system realizes more personalized learning support that also takes into account the student's emotional state, thereby improving students' learning efficiency and providing effective support tailored to their individual learning needs.

[0415] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0416] Step 1:

[0417] Input: The user enters their user ID and password on the tablet device.

[0418] Operation: The terminal sends the entered user ID and password to the authentication server.

[0419] Data calculation: The authentication server checks the received user ID and password against the information in the database.

[0420] Output: An authentication result is generated and sent to the device, which then starts a training session.

[0421] Step 2:

[0422] Input: Authenticated student start learning operation.

[0423] How it works: The device displays the learning content and collects facial expression data from the student using a built-in camera. At the same time, it also records how the learning material is used and how the student answers the questions.

[0424] Data calculation: The collected facial expression data is sent to the emotion engine in real time to analyze the emotional state.

[0425] Output: The analysis results of the emotion engine and learning data are generated and recorded.

[0426] Step 3:

[0427] Input: End of study session or passage of a certain period of time.

[0428] How it works: The device packages the learning and emotion data collected during the session.

[0429] Data calculation: Formats the training data and emotion data for transmission to the server.

[0430] Output: The packaged data is sent to the server.

[0431] Step 4:

[0432] Input: Training data and emotion data received by the server.

[0433] How it works: The server stores the training data and emotion data in a database.

[0434] Data calculation: Using machine learning algorithms to analyze learning progress and comprehension, and an emotion engine to analyze emotional states.

[0435] Output: Analysis results are generated regarding learning progress, comprehension, gaps, and emotional state.

[0436] Step 5:

[0437] Input: Identified learning gaps and emotional state.

[0438] How it works: Based on the identified gaps in understanding, the server passes prompts to the generative AI model to generate explanations.

[0439] Data calculation: A generative AI model generates an explanation based on the prompt.

[0440] Output: Explanatory content (text, illustrations, video) is generated and sent to the device.

[0441] Step 6:

[0442] Input: Learning gaps and understanding gaps identified by the server.

[0443] How it works: The server uses a generative AI model to create prompts that generate exercises.

[0444] Data computation: Generative AI models generate exercises of appropriate difficulty and format.

[0445] Output: Exercises are generated and sent to the terminal.

[0446] Step 7:

[0447] Input: Learning data, comprehension, emotional state, goals, and schedule information collected by the server.

[0448] How it works: The server uses the collected data to generate a personalized lesson plan.

[0449] Data computation: Using learning algorithms to optimize personalized learning plans.

[0450] Output: The generated lesson plan is sent to the terminal and provided to the student.

[0451] Step 8:

[0452] Input: Questions asked by students.

[0453] Operation: The device sends the query to the server.

[0454] Data Computation: The server analyzes the question and generates an answer using information retrieval and large-scale language models.

[0455] Output: The generated answer is sent to the device and provided to the student, and emotional feedback is also provided using the emotion engine.

[0456] (Application example 2)

[0457] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0458] Conventional learning support systems only record and analyze students' learning progress and comprehension, but are unable to incorporate real-time emotional analysis. As a result, it is difficult to immediately grasp the frustration or confusion students feel during learning and provide appropriate support. Furthermore, content recommendations based on the viewer's emotional state are not performed, preventing the maximization of learning effectiveness. The present invention aims to solve these problems and provide more personalized learning support.

[0459] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording the student's learning progress, means for analyzing the recorded learning progress to identify the student's level of understanding, means for identifying areas where the student does not understand based on the identified level of understanding and generating explanations related to these, means for providing the generated explanations to the student's terminal, means for automatically generating exercises according to the student's level of understanding, means for generating individual study plans based on the student's goals, academic ability, level of understanding, and schedule information, means for providing the generated individual study plans to the student's terminal, means for collecting viewer facial expression data in real time and analyzing their emotional state, and means for recommending learning content based on their emotional state. This incorporates real-time emotion analysis, enabling personalized learning support and content recommendations according to the student's emotional state.

[0460] "Learning progress" refers to the content and progress a student is currently learning.

[0461] "Level of understanding" is an indicator that shows how well a student understands the learning content.

[0462] "Analysis" is the act of carefully analyzing collected data to find meaning and patterns.

[0463] "Explanation" is content that provides supplementary explanations and answers to identified areas of lack of understanding.

[0464] "Devices" refers to electronic devices such as tablets, smartphones, and computers used by students.

[0465] "Exercises" are problems designed to deepen students' understanding.

[0466] A "study plan" is a detailed plan that outlines the schedule and content for students to study efficiently.

[0467] "Facial expression data" refers to information obtained from a viewer's facial expressions and is used to analyze their emotional state.

[0468] "Emotional state" refers to the emotions the viewer feels while learning (concentration, confusion, anxiety, etc.).

[0469] "Content recommendation" is the act of suggesting suitable learning content to maximize the viewer's learning effect.

[0470] A "server" refers to a computer system used to analyze and store data and perform various processing.

[0471] The system of the present invention records and analyzes students' learning progress and understanding in detail to provide personalized learning support. Furthermore, it combines a function to recognize users' emotions to optimize the learning experience. Specific embodiments for implementing the present invention are described below.

[0472] System Configuration

[0473] The system includes the following major components:

[0474] A means of recording student progress

[0475] A means of identifying understanding

[0476] A means of generating explanations for areas of incomprehension

[0477] Devices that provide commentary

[0478] A method for automatically generating exercises according to the level of understanding

[0479] A means of generating personalized learning plans

[0480] A means of collecting facial expression data in real time and analyzing emotional states

[0481] A means of recommending learning content based on emotional state

[0482] A means of receiving student questions and generating answers using large-scale language models

[0483] Hardware and Software

[0484] Hardware: Viewing devices such as tablets, smartphones, smart glasses, or head-mounted displays are used. These devices have built-in cameras that are used to collect facial expression data from viewers.

[0485] Software: Python, OpenCV, and Keras are used to analyze facial expressions in real time, and the data is sent to the server via a REST API.

[0486] Processing Flow

[0487] 1. Data Collection:

[0488] Using the device's built-in camera, facial expression data is collected in real time from the moment the student logs in while studying. In addition, learning content and answer data are also recorded. The facial expression data is analyzed using facial expression recognition technology (e.g., using OpenCV and Keras) to identify emotional states (e.g., concentration, confusion, anxiety).

[0489] 2. Data transmission:

[0490] Once a certain learning session is completed, the collected data is sent to a server, including the learning content, answer data, facial expression data, and emotion recognition results.

[0491] 3. Data analysis and learning support:

[0492] The server uses machine learning algorithms to analyze students' learning progress and comprehension, identifying areas where they lack understanding. It also takes into account their emotional state and uses AI technology to generate appropriate explanations. These explanations are provided in the form of text, illustrations, videos, and other formats, and can be reviewed by students.

[0493] 4. Exercises and Study Plan:

[0494] The server uses a generative AI model to automatically generate practice questions that address areas of incomplete understanding. The generated individualized learning plan is optimized by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data.

[0495] 5. Questions and feedback:

[0496] When students submit questions during their studies, the server receives the questions and generates optimal answers using information search and large-scale language models. It also provides appropriate feedback and emotional care through an emotion engine.

[0497] Specific examples

[0498] For example, imagine a user is watching an online math course. The camera will recognize any confused faces while they are watching and send that data to a server, which can then recommend more understandable videos or supplementary materials in real time.

[0499] Prompt Sentence Examples

[0500] "Generate an algorithm that recommends optimal learning content based on the user's facial expression data and viewing history while they are viewing. Facial expression data is transmitted in real time, and dynamically adjust learning content based on the user's emotional state (e.g., concentration, confusion, anxiety)."

[0501] In this way, the system of the present invention incorporates real-time emotion analysis and realizes personalized learning support according to the student's emotional state.

[0502] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0503] Step 1:

[0504] Data collection

[0505] The device's built-in camera collects students' facial expression data in real time. This happens automatically as soon as the learning session begins. Learning content and answer data are also recorded at the same time. The video data captured by the camera is first identified as a facial region, and then facial expression recognition technology (using OpenCV and Keras) is used to identify emotional states such as "concentration," "confusion," and "anxiety."

[0506] Input: Camera footage, learning content, answer data

[0507] Output: Facial expression data (emotional state)

[0508] Step 2:

[0509] Data transmission

[0510] The collected facial expression data and learning data are sent to the server after a certain learning session has ended or a specified time has passed. The data sent includes the learning content, answer data, facial expression data, and emotion recognition results. This data is sent to the server via a REST API.

[0511] Input: facial expression data, learning content, answer data

[0512] Output: Data sent to the server

[0513] Step 3:

[0514] Data analysis

[0515] The server analyzes the received facial expression data and learning data. It uses machine learning algorithms to identify the student's learning progress and level of understanding, and determines where they are lacking in understanding. At the same time, it takes into account their emotional state. For example, if a student is "confused," it performs additional analysis to find out why.

[0516] Input: Transmitted data (facial expression data, learning content, answer data)

[0517] Output: Learning progress, comprehension, and emotional state identification results

[0518] Step 4:

[0519] Explanation Generation

[0520] The server uses AI technology to generate detailed explanations for identified gaps in understanding. These explanations take into account the student's emotional state and are provided in the most appropriate format, such as text, illustrations, or video. The generated explanations are then retransmitted to the device.

[0521] Input: Areas of incomprehension, emotional state

[0522] Output: explanatory data

[0523] Step 5:

[0524] Exercise problem generation

[0525] The server uses a generative AI model to automatically generate exercises that address areas where students lack understanding, allowing students to tackle problems of the appropriate difficulty and format. The generated exercises are then provided back to the device.

[0526] Input: Area of ​​incomprehension

[0527] Output: Exercise data

[0528] Step 6:

[0529] Learning plan generation

[0530] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. This learning plan is optimized by incorporating the student's requests through an interactive format. The generated learning plan is provided to the student's device and serves as a guide for the student's daily learning.

[0531] Input: Student goals, academic achievement, comprehension, schedule information, emotional data

[0532] Output: Learning plan data

[0533] Step 7:

[0534] Questions and feedback

[0535] When a student submits a question during their studies, the server receives the question and generates the best answer using information retrieval and a large-scale language model. The answer is then provided to the device. The emotion engine also provides appropriate feedback and emotional support according to the student's emotional state.

[0536] Input: Student question

[0537] Output: Answers to questions, emotional care feedback

[0538] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0539] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0540] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0541] [Second embodiment]

[0542] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0543] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0544] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0545] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0546] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0547] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0548] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0549] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0550] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0551] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0552] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0553] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0554] The system of the present invention is designed to provide personalized learning support according to the learning progress and level of understanding of each student. Specific embodiments for carrying out the present invention will be described below.

[0555] Acquiring learning records

[0556] Subject: Terminal

[0557] First, when a student logs in to a tablet device, the device starts the student's study session. It records the learning materials (digital texts, video materials, etc.) used by the student and their learning progress (study time, answer status, etc.). It also uses the device's built-in camera and IoT sensors to collect data such as facial expressions and posture. This data is used to comprehensively understand the student's learning situation.

[0558] Analyzing learning data and identifying understanding

[0559] Subject: Server

[0560] When the learning data sent from the device is received by the server, the server analyzes the data and identifies the student's level of understanding. Specifically, it uses machine learning algorithms to analyze the student's answer patterns and learning progress and identify which parts of the learning content they have not understood.

[0561] AI-powered commentary

[0562] Subject: Server

[0563] The server then uses AI technology to generate explanations for the identified areas of incomprehension. The explanations are generated in a format that is easy for students to understand (e.g., text, diagrams, videos, etc.). These explanations are then provided to students' devices so that they can access them at any time.

[0564] Automatic generation of exercises

[0565] Subject: Server

[0566] Based on the students' level of understanding, the server automatically generates appropriate exercises using a generative AI model, allowing them to tackle problems that are tailored to their individual level of understanding. The generated exercises are provided to the students' devices, and they can further deepen their understanding by answering them.

[0567] Generate personalized learning plans

[0568] Subject: Server

[0569] The server then generates an individual learning plan based on the student's goals, academic ability, level of understanding, and schedule information. This learning plan is customized by interacting with the student, incorporating their wishes and goals. The generated learning plan is provided to the student's device, allowing the student to proceed with their studies accordingly.

[0570] RAG (Retriever-Augmented Generation) module

[0571] Subject: Server

[0572] Finally, if a student has a question during their study, the server receives the question and generates the best answer using a large-scale language model based on the information search results. This answer is then provided to the student's device, allowing the student to immediately resolve their doubts.

[0573] Specific examples

[0574] 1. User logs in

[0575] The device verifies the student's credentials and begins the learning session.

[0576] 2. Obtaining training data

[0577] The device collects learning progress information and facial expression data and sends them to the server.

[0578] 3. Providing analysis and commentary

[0579] The server analyzes the data to identify areas where students lack understanding, and then uses AI to generate explanations that are provided to the device.

[0580] 4. Exercises and Study Plans

[0581] The server automatically generates appropriate exercises and creates a learning plan based on the student's level of understanding, which is then provided to the terminal.

[0582] 5. Answering questions

[0583] When a student submits a question, the server searches for information, generates the most appropriate answer, and provides it to the device.

[0584] This allows students to study effectively at their own pace and at their own level of understanding.

[0585] The processing flow will be explained below.

[0586] Step 1:

[0587] The device checks the student's credentials

[0588] The device receives the student's login information (user ID and password) and authenticates it with the database. If authentication is successful, the device accesses the student's individual account and begins the learning session.

[0589] Step 2:

[0590] Your device records your study session

[0591] The device records the learning materials selected by the student (e.g., mathematics textbook, history video, etc.) and also records input data from the touch panel and keyboard (answer time, options, answer results, etc.).

[0592] Step 3:

[0593] The device collects data using cameras and IoT sensors

[0594] The device's built-in camera captures students' facial expressions, and IoT sensors monitor their posture and movements while they study, collecting data in real time.

[0595] Step 4:

[0596] The device sends the learning data to the server.

[0597] When the learning session ends or a certain period of time has passed, the device sends all collected data (teaching materials, answer data, facial expression data, etc.) to the server.

[0598] Step 5:

[0599] The server receives and analyzes the learning data.

[0600] The server receives the learning data sent from the device and analyzes it using machine learning algorithms, which identifies areas where the student does not understand and their learning trends.

[0601] Step 6:

[0602] The server generates an explanation for any missing information

[0603] The server uses AI technology to create detailed explanations of the identified areas of lack of understanding, which can be in the form of text, illustrations, videos, etc.

[0604] Step 7:

[0605] The server sends the commentary to the device.

[0606] The server generates explanations that are sent to students' devices so that they can access them at any time. The explanations are saved in individual student accounts.

[0607] Step 8:

[0608] The server automatically generates exercises

[0609] The server uses the generative AI model to automatically generate practice questions that address the identified gaps in understanding, adjusting the difficulty and format of the questions according to the student's level of understanding.

[0610] Step 9:

[0611] The server sends the exercises to the device.

[0612] The server sends the generated exercises to the students' devices so that they can work on them. The answers are also recorded sequentially.

[0613] Step 10:

[0614] The server generates an individualized learning plan.

[0615] The server generates an individualized learning plan based on the student's goals, academic ability, level of understanding, and schedule information. This plan is interactive and incorporates the student's requests.

[0616] Step 11:

[0617] The server sends the lesson plan to the device.

[0618] The server transmits the generated individual learning plan to the student's terminal, allowing the student to progress with their learning according to the daily learning plan.

[0619] Step 12:

[0620] The device will carry out learning according to the learning plan.

[0621] Students progress through their studies according to the study plan displayed on their device and access explanations and practice problems as needed.

[0622] Step 13:

[0623] The server receives the student's questions and generates answers.

[0624] When a student submits a question during their studies, the server receives the question and generates an answer using a large-scale language model based on the information search results.

[0625] Step 14:

[0626] The server generates a response and sends it to the device.

[0627] The server generates answers that are sent to students' devices to help them resolve their questions.

[0628] This will help each student to progress effectively in their studies.

[0629] Example 1

[0630] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0631] In recent years, there has been a demand for personalized learning support tailored to each learner's individual learning progress and level of understanding. However, conventional learning support systems have had problems in that they are insufficient in providing specific explanations and practice problems based on each learner's individual level of understanding, and in responding immediately to students' questions. This has made it difficult for students to study efficiently and effectively. In addition, the accuracy of collecting and analyzing learning data has been low, making it difficult to provide accurate learning support.

[0632] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0633] In this invention, the server includes means for recording the student's learning progress, means for analyzing the recorded learning progress to determine the student's level of understanding, means for determining the student's lack of understanding based on the determined level of understanding and generating explanations for the identified areas, means for providing the generated explanations to the student's terminal, means for automatically generating exercises according to the student's level of understanding, means for generating an individual learning plan based on the student's goals, academic ability, level of understanding, and schedule information, means for providing the generated individual learning plan to the student's terminal, means for receiving the student's questions and generating answers using natural language processing technology with reference to information search results, means for collecting the student's facial expression and posture data at the terminal, means for transmitting the learning data to the server in real time, means for analyzing the data using a machine learning algorithm, and means for generating explanations using artificial intelligence. This makes it possible to provide specific and effective support according to each student's individual learning progress.

[0634] "Learning progress" is an indicator that shows how much content a student has learned through learning activities and how much time they have spent.

[0635] "Recording means" refers to devices or software for storing students' learning progress, study time, answer status, and other related data in a database or cloud storage.

[0636] "Means for analysis" refers to software or systems that apply machine learning algorithms and statistical methods to assess students' learning progress and understanding using collected learning data.

[0637] "Level of understanding" is an indicator of how well a student has understood the learning content, including the accuracy and speed of answers and identification of areas of lack of understanding.

[0638] A "means for generating explanations" is a device or software that uses artificial intelligence or other automatic generation technology to create easy-to-understand explanations for students in the form of text, illustrations, videos, etc. for identified areas of lack of understanding.

[0639] "Practice questions" are questions that students answer to review what they have learned and check their understanding.

[0640] A "means for automatically generating exercises" is a device or software that uses artificial intelligence or generation technology to automatically generate exercises of appropriate difficulty and content based on the student's level of understanding.

[0641] A "study plan" is a plan that includes the learning content and schedule established for a student to achieve their goals.

[0642] A "means for generating an individual learning plan" is a device or software that creates and provides an individually optimized learning plan based on a student's goals, academic ability, level of understanding, schedule information, etc.

[0643] A "question receiving" means is an interface, communication device, or software that allows students to submit questions or concerns online while studying.

[0644] "Information search results" are the results obtained by searching databases and the Internet for information related to a student's question.

[0645] "Natural language processing technology" is a field of artificial intelligence and is a technology for understanding and generating human language.

[0646] "Facial expression and posture data" refers to information about facial expressions and body postures captured to understand students' learning status.

[0647] "Means for transmitting in real time" refers to communication equipment and communication protocols for transmitting learning data from the collection source to the server without delay.

[0648] A "machine learning algorithm" is a mathematical model and computational method for analyzing collected data and extracting patterns and features.

[0649] "Artificial intelligence" refers to systems and software that mimic human intelligence and perform learning and reasoning, and is used to generate explanations and questions.

[0650] The system of the present invention provides personalized learning support according to the student's learning progress and level of understanding. Specific embodiments for carrying out the present invention will be described below.

[0651] Hardware and Software Configuration

[0652] This system uses tablet devices, cameras, and IoT sensors to collect learning data, which is then analyzed on a server. Specifically, it uses the following hardware and software:

[0653] Hardware:

[0654] Tablet: A device used by students for learning.

[0655] Camera: Used to capture students' facial expressions.

[0656] IoT sensors: Used to collect students' posture data.

[0657] software:

[0658] Authentication system: Authentication system using OAuth or JWT.

[0659] Data collection software: Software for collecting learning progress, facial expressions, and posture data.

[0660] Data analysis software: TensorFlow and Scikit-learn are used to analyze data using machine learning algorithms.

[0661] Generative AI model: Generates explanations and practice problems using models such as OpenAI's GPT-4.

[0662] Natural language processing systems: Answering student questions using large-scale language models.

[0663] Processing flow and specific examples

[0664] Students log in and start a study session

[0665] When a student logs in, the device checks the authentication information and starts the learning session. For example, authentication is performed by entering a user ID and password. At this time, the device ensures security by using JWT authentication.

[0666] Collecting and sending learning data

[0667] During a learning session, the device records data such as the student's viewing status, answer status, and study time. It also records the student's facial expressions and posture data acquired through built-in cameras and IoT sensors. The acquired data is sent to a server in real time.

[0668] Analysis of training data

[0669] The server analyzes the data received from the devices using machine learning algorithms. Specifically, it uses the collected data to evaluate students' answer patterns and learning progress, and identifies their level of understanding. This analysis is performed using TensorFlow and Scikit-learn.

[0670] Generate explanations for areas of incomprehension

[0671] The server then uses a generative AI model to generate explanations for identified gaps in understanding. For example, OpenAI's GPT-4 is used to generate explanations in the form of text, diagrams, or videos. These explanations are then provided to students' devices for access.

[0672] Automatic generation of exercises

[0673] Based on the student's level of understanding, the server uses a generative AI model to generate appropriate exercises. The generated exercises are then provided to the student's device, and the student deepens their understanding by answering them.

[0674] Prompt Sentence Examples

[0675] Here are some examples of specific prompt sentences:

[0676] To generate explanations for areas of incomprehension:

[0677] "My students have a limited understanding of differential calculus in Chapter 5. I would like you to write an easy-to-understand explanation."

[0678] For generating practice questions:

[0679] "Generate basic differential calculus problems based on students' understanding."

[0680] Generate personalized learning plans

[0681] The server generates an individual learning plan based on the student's goals, schedule, and level of understanding, and provides it to the student's device. This plan incorporates the student's goals and optimizes daily learning.

[0682] Answering questions during study

[0683] When a student submits a question during their study, the RAG module on the server receives the question and uses natural language processing technology to generate the most appropriate answer, which is then provided to the device.

[0684] This detailed processing flow makes it possible to provide specific and effective support according to each student's learning progress.

[0685] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0686] Step 1: Students log in and start a study session

[0687] Subject: Terminal

[0688] When a student logs in to the device, they are prompted to enter their user ID and password. The entered information is sent to the authentication server, which performs authentication using JWT authentication. If authentication is successful, the device starts a learning session and displays the dashboard screen. The input is the user ID and password, and the output is the authentication result and the start of the learning session.

[0689] Specific behavior:

[0690] Input: User ID, Password

[0691] Data processing: JWT authentication

[0692] Output: Authentication results, dashboard screen

[0693] Step 2: Collect and send training data

[0694] Subject: Terminal

[0695] During a learning session, the device records data such as the student's viewing status, answer status, and study time. It also uses a built-in camera and IoT sensors to collect data on the student's facial expression and posture. This data is sent to a server in real time. The input is learning progress information, facial expression data, and posture data, and the output is data sent to the server.

[0696] Specific behavior:

[0697] Input: learning progress information, facial expression data, posture data

[0698] Data processing: Data recording, sensor information collection

[0699] Output: Send data to the server

[0700] Step 3: Analyze the training data

[0701] Subject: Server

[0702] The server receives the data sent from the device and analyzes it using machine learning algorithms. The analysis evaluates the student's answer patterns and learning progress, and identifies their level of understanding. TensorFlow and Scikit-learn are used for this analysis. The input is the learning data sent from the device, and the output is the level of understanding information obtained as a result of the analysis.

[0703] Specific behavior:

[0704] Input: Training data from the device

[0705] Data processing: Analysis using machine learning algorithms

[0706] Output: Comprehension information

[0707] Step 4: Generate explanations for areas of incomprehension

[0708] Subject: Server

[0709] The server uses a generative AI model to generate explanations for the identified areas of incomprehension. For example, by inputting a prompt into the GPT-4 model, it generates explanatory text, illustrations, and videos. The input is information about the area of ​​incomprehension and the prompt, and the output is the generated explanation.

[0710] Specific behavior:

[0711] Input: Information on areas of incomprehension, prompt text

[0712] Data processing: Generative AI model for generating explanations

[0713] Output: explanatory text, illustrations, videos

[0714] Step 5: Automatic generation of exercises

[0715] Subject: Server

[0716] The server generates appropriate exercises using a generative AI model based on the student's level of comprehension. The generated exercises are sent to the student's device. The input is comprehension information and a prompt, and the output is the generated exercise.

[0717] Specific behavior:

[0718] Input: Comprehension information, prompt

[0719] Data processing: Generative AI model for generating exercises

[0720] Output: Exercises

[0721] Step 6: Generate an individualized learning plan

[0722] Subject: Server

[0723] The server generates an individual learning plan based on the student's goals, academic ability, level of understanding, and schedule information. The generated learning plan is provided to the student's device. The input is the student's goals, academic ability, level of understanding, and schedule information, and the output is the individual learning plan.

[0724] Specific behavior:

[0725] Input: Goals, academic ability, level of understanding, schedule information

[0726] Data processing: Planning using learning plan generation algorithms

[0727] Output: Individualized Learning Plan

[0728] Step 7: Answering questions while studying

[0729] Subject: Server

[0730] The server receives questions sent by students during their studies, and generates optimal answers using a large-scale language model based on information search results. These answers are then provided to the students' devices. The input is the student's question, and the output is the generated answer.

[0731] Specific behavior:

[0732] Input: Student question

[0733] Data Processing: Natural Language Processing and Information Retrieval

[0734] Output: Best answer

[0735] (Application example 1)

[0736] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0737] Conventional factory worker education and training systems lacked personalized support tailored to each worker's learning progress and skill level, making it difficult to efficiently improve skills. They also lacked a means to respond to workers' questions in real time and provide specific feedback. This created the challenge of not maximizing the effectiveness of worker training.

[0738] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0739] In this invention, the server includes means for recording the worker's learning progress, means for analyzing the recorded learning progress to identify the worker's level of understanding, means for identifying areas where the worker does not understand based on the identified level of understanding and generating explanations for those areas, means for providing the generated explanations to the worker's terminal, means for automatically generating exercises according to the worker's level of understanding, means for generating an individual learning plan based on the worker's goals, skills, level of understanding, and schedule information, means for providing the generated individual learning plan to the worker's terminal, means for collecting and analyzing the worker's operation records and posture data, and means for receiving the worker's questions and generating answers using a generative AI model with reference to the information acquisition results. This makes it possible to provide personalized training plans based on each worker's level of understanding and respond to questions in real time.

[0740] "Workers" refers to the technicians and workers who operate machinery and equipment in factories and manufacturing sites. They are the recipients of training according to their individual skill levels and learning progress.

[0741] "Learning progress" refers to the progress a worker makes along a training or education program, specifically the degree to which knowledge and skills have been acquired.

[0742] "Level of understanding" refers to the standard for assessing how well a worker understands the learning content and operating procedures. It is an important indicator for providing feedback to improve identified areas of lack of understanding.

[0743] "Explanation" refers to content that provides additional information or explanations for areas where workers are struggling, often in the form of text, video, or AR guides.

[0744] "Devices" refer to electronic devices used by workers for operation and learning, including tablet devices and head-mounted displays.

[0745] "Exercises" refer to assignments and tasks provided to workers to allow them to try out what they have learned. They are automatically generated based on each worker's level of understanding.

[0746] "Individualized Learning Plan" refers to a customized education and training schedule based on a worker's goals, skills, level of understanding, and schedule information.

[0747] "Operation records" refer to the history of specific operations and tasks performed by workers. Recording these records will be useful for analysis.

[0748] "Posture data" refers to data on the body movements and posture of workers when operating a machine. It is acquired using sensors and cameras.

[0749] A "generative AI model" refers to an artificial intelligence algorithm or system that automatically generates information based on large datasets, which can be used to generate answers to human questions.

[0750] "Information acquisition results" refers to information collected from the internet or other databases in response to worker questions. This is part of the data used by the generative AI model.

[0751] This invention is a system for personalizing education and training for factory workers. This system records and analyzes the worker's learning progress and provides feedback, practice questions, and individual learning plans based on the worker's level of understanding.

[0752] Hardware and software used

[0753] The hardware used includes a tablet computer, a head-mounted display (HMD), a camera, and IoT sensors. This hardware is used to acquire the worker's operation records and posture data. The software used includes a learning data collection app, a data analysis server, an AI explanation system, an exercise problem generation system, and a RAG (Retriever-Augmented Generation) module.

[0754] Acquiring and analyzing training data

[0755] The device records the worker's learning progress. Specifically, it collects the worker's operations and work history, as well as posture and movement data acquired by cameras and IoT sensors. This data is sent to a server in real time.

[0756] The server analyzes the received data and identifies the worker's level of understanding. The analysis uses a machine learning algorithm to identify the areas where each worker lacks understanding. The server then generates an explanation corresponding to the area of ​​lack of understanding and provides it to the worker's device.

[0757] Providing AI explanations and practice questions

[0758] The server uses AI technology to generate easy-to-understand explanations for the identified areas of insufficient understanding. These explanations are created in the form of text, video, AR guides, etc. and are provided to the worker's device. Exercises are also automatically generated based on the worker's level of understanding. The exercises are set at an appropriate level of difficulty with the aim of improving the worker's skills.

[0759] Generate and provide individualized learning plans

[0760] The server generates an individual learning plan based on the worker's goals, skills, level of understanding, and schedule information. This allows the worker to learn at their own pace, enabling effective training. The generated learning plan is provided to the terminal, and the worker follows it to carry out training.

[0761] Real-time question response (RAG module)

[0762] If a worker has a question during learning, the server receives the question and generates an answer using the generative AI model based on the information acquisition results. This answer is provided to the worker's device in real time, instantly resolving the worker's question.

[0763] Examples of concrete examples and prompts

[0764] As an example, consider a worker wearing a head-mounted display and undergoing training to operate a robot. The worker's operation records and posture data are acquired by the terminal and sent to the server. After analyzing the data, the server generates video explanations for any operating procedures that the worker does not fully understand and provides them to the terminal. In addition, it automatically generates exercises related to the operation and presents them to the worker as assignments. If the worker asks a question during this process, the server uses the RAG module to instantly generate an answer and provides it to the terminal.

[0765] An example of a prompt sentence might be:

[0766] "We would like to develop a training system for robot operation in factories. Please come up with an application that provides individual feedback, practice assignments, and training plans based on the worker's skill level and progress."

[0767] As described above, the system of the present invention provides personalized education and training to workers, and supports efficient skill improvement.

[0768] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0769] Step 1:

[0770] The user logs in to the device. The device receives the user's authentication information as input and performs authentication to start a learning session. If authentication is successful, the device obtains the user's past learning data and current learning goals.

[0771] Step 2:

[0772] The device records the user's operations and learning progress in real time. Specifically, the device collects operation records and posture data using cameras and sensors installed on the tablet device or head-mounted display. This data is input and sent to a server.

[0773] Step 3:

[0774] The server analyzes the received data. It receives operation records and posture data as input, and uses a machine learning algorithm to evaluate the user's level of understanding. As a result of the analysis, it identifies and outputs the areas where the user does not understand or where they lack skills.

[0775] Step 4:

[0776] The server generates explanations for the identified areas of incomprehension, using AI technology to create explanations in the form of text, video, or AR guides. Using this as input, the server generates explanation data in the appropriate format and outputs it to be provided to the device.

[0777] Step 5:

[0778] The server automatically generates exercises based on the user's level of understanding and skills. It uses the analysis results as input and creates exercises using a generative AI model. This outputs appropriate exercise data aimed at improving the user's skills.

[0779] Step 6:

[0780] The server generates a personalized learning plan based on the user's goals, skills, level of understanding, and schedule information. It uses the user's profile data and analysis results as input to create an optimal learning schedule. This plan is output for delivery to the device.

[0781] Step 7:

[0782] When a user asks a question during learning, the device sends the question to the server. The device receives the question data as input, uses the RAG module to refer to the information acquisition results, and generates an answer using the generative AI model. The generated answer is sent to the device and provided to the user.

[0783] These steps explain how each task is performed and what input data is used to generate output data, enabling personalized training and real-time question answering.

[0784] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0785] The system of the present invention records and analyzes students' learning progress and level of understanding in detail to provide personalized learning support. This system is also combined with an emotion engine that recognizes the user's emotions. Specific embodiments for carrying out the present invention will be described below.

[0786] Acquiring training data and emotion recognition

[0787] Subject: Terminal

[0788] When a student logs in to a tablet device, the device begins a learning session. While recording data used during the learning process (such as learning material content and answer status), the device also collects the student's facial expression data using the built-in camera. This collected facial expression data is sent in real time to an emotion engine that recognizes the student's emotional state (concentration, confusion, anxiety, etc.).

[0789] Sending training data and emotion data

[0790] Subject: Terminal

[0791] When the learning session ends or after a certain period of time has passed, the device sends the collected learning data and emotion data to the server. This data includes the learning content, answer data, facial expression data, and emotion recognition results.

[0792] Analyzing data and identifying understanding

[0793] Subject: Server

[0794] The server receives the learning data and emotional data sent from the device and first analyzes the student's learning progress and level of understanding using a machine learning algorithm. From the analysis results, it identifies areas where understanding is lacking, and at the same time, taking into account the results of the emotional engine, it also understands the emotional state the student was in while studying.

[0795] Generate personalized commentary

[0796] Subject: Server

[0797] For identified gaps in understanding, the server uses AI technology to generate detailed explanations. The explanations take into account the student's emotional state and are created in context-appropriate formats such as text, diagrams, and videos. Furthermore, these explanations are provided to students' devices so they can review them repeatedly.

[0798] Automatic generation of exercises

[0799] Subject: Server

[0800] The server uses a generative AI model to automatically generate practice questions that address areas of incomplete understanding, allowing students to tackle questions of the appropriate difficulty and format. The practice questions are also provided to the device, and the answers are collected again.

[0801] Generate personalized learning plans

[0802] Subject: Server

[0803] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. This learning plan is optimized by incorporating the student's requests through an interactive format. The generated learning plan is provided to the student's device and serves as a guide for daily learning.

[0804] Question handling and emotional care

[0805] Subject: Server

[0806] When a student submits a question during learning, the server receives the question, performs an information search, and uses the results to generate the optimal answer using a large-scale language model. The answer is then provided to the device, instantly resolving the student's question. The system also uses an emotion engine to provide appropriate feedback and emotional support according to the student's emotional state during learning.

[0807] Specific examples

[0808] 1. User logs in

[0809] The device verifies the student's credentials and begins the learning session.

[0810] 2. Collecting training data and emotion data

[0811] The device collects learning progress information and facial expression data, which are then analyzed in real time using an emotion engine.

[0812] 3. Sending and Receiving Data

[0813] The terminal sends the collected data to the server, which receives it and analyzes it.

[0814] 4. Providing analysis and commentary

[0815] The server analyzes the data to identify areas of lack of understanding, and then uses AI to generate explanations that are provided to the device.

[0816] 5. Exercises and Study Plans

[0817] The server automatically generates appropriate practice questions, creates an individual learning plan, and provides it to the terminal.

[0818] 6. Answering questions and providing emotional support

[0819] The server receives the question, generates an answer using a large-scale language model based on the information search results, and provides it to the device. It also provides appropriate feedback depending on the user's emotional state.

[0820] In this way, this system provides more personalized learning support that also takes into account the student's emotional state.

[0821] The processing flow will be explained below.

[0822] Step 1:

[0823] The device checks the student's credentials

[0824] The device receives the student's login information (user ID and password) and authenticates it with the database. If authentication is successful, the device accesses the student's individual account and begins the learning session.

[0825] Step 2:

[0826] Your device records your study session

[0827] The device records the learning materials selected by the student (e.g., mathematics textbook, history video, etc.) and also records input data from the touch panel and keyboard (answer time, options, answer results, etc.).

[0828] Step 3:

[0829] The device collects data using cameras and IoT sensors

[0830] The device's built-in camera captures students' facial expressions, and IoT sensors monitor their posture and movements while they study, collecting data in real time.

[0831] Step 4:

[0832] The device sends facial expression data to the emotion engine to recognize emotions.

[0833] The device sends the collected facial expression data to the emotion engine, which then analyzes it to recognize the student's emotional state (e.g., concentration, confusion, anxiety, etc.) in real time.

[0834] Step 5:

[0835] The device sends the learning data and emotion data to the server.

[0836] When the learning session ends or after a certain period of time has passed, the device sends all collected data (teaching materials, answer data, facial expression data, emotional data, etc.) to the server.

[0837] Step 6:

[0838] The server receives and analyzes the learning data and emotion data.

[0839] The server receives the learning data and emotion data sent from the device, then analyzes the data using machine learning algorithms and an emotion engine to identify the student's learning progress and level of understanding.

[0840] Step 7:

[0841] The server generates an explanation for any missing information

[0842] The server uses AI technology to generate detailed explanations for the identified areas of lack of understanding. These explanations are provided to students in the form of text, illustrations, videos, etc.

[0843] Step 8:

[0844] Sends server-generated commentary to the device

[0845] The server generates explanations that are sent to students' devices so that they can access them at any time. The explanations are saved in individual student accounts.

[0846] Step 9:

[0847] The server automatically generates exercises

[0848] The server uses the generative AI model to automatically generate practice questions that address areas of incomprehension, adjusting the difficulty and format of the questions according to the student's level of understanding and emotional state.

[0849] Step 10:

[0850] The server sends the generated exercises to the device.

[0851] The server sends the generated exercises to the students' devices so that they can work on them. The answers are collected again and sent to the server.

[0852] Step 11:

[0853] The server generates an individualized learning plan.

[0854] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. This plan is interactive and incorporates the student's requests.

[0855] Step 12:

[0856] The server sends the generated lesson plan to the device.

[0857] The server transmits the generated individual learning plan to the student's terminal, allowing the student to proceed with their studies in accordance with it.

[0858] Step 13:

[0859] The device will carry out learning according to the learning plan.

[0860] Students progress through their studies according to the study plan displayed on their device and access explanations and practice problems as needed.

[0861] Step 14:

[0862] The server receives the student's questions and generates answers.

[0863] When students submit questions during their studies, the server receives the questions and generates the best answer using a large-scale language model based on the information search results.

[0864] Step 15:

[0865] The server generates a response and sends it to the device.

[0866] The server generates answers and sends them to students' devices to help them resolve their questions. It also uses an emotion engine to provide appropriate feedback and emotional support according to the student's emotional state during the course of their studies.

[0867] Example 2

[0868] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0869] Existing learning support systems are not only unable to record and analyze students' learning progress and comprehension, but also unable to provide personalized learning support that takes into account students' emotional state. Furthermore, they lack the ability to generate quick and appropriate answers to students' questions. This makes it difficult to provide learning plans tailored to individual students' needs.

[0870] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for recording the student's learning progress, means for analyzing the recorded learning progress to identify the student's level of understanding, means for analyzing the recorded facial expression data to identify the student's emotional state, means for identifying areas where the student does not understand based on the identified level of understanding and emotional state and generating explanations for those areas, means for automatically generating exercises according to the student's level of understanding, means for generating an individual learning plan based on the student's goals, academic ability, level of understanding, emotional state, and schedule information, means for providing the generated individual learning plan to the student's terminal, and means for receiving the student's question and generating an answer using a large-scale language model with reference to information search results. This enables individual learning support based on the student's learning progress, level of understanding, and emotional state.

[0871] "Learning progress" is data that shows how far a student has progressed with a particular assignment or material.

[0872] "Understanding" is data that indicates how well a student understands a particular material or concept.

[0873] "Emotional state" is data that indicates a student's psychological state (concentration, confusion, anxiety, etc.) while studying.

[0874] A "tablet device" is a portable computer device used by students when studying.

[0875] "Means of recording" refers to a system for collecting and storing data such as students' learning progress and emotional state.

[0876] "Means for analysis" are algorithms or devices that analyze collected data and assess learning progress, comprehension, and emotional state.

[0877] "Generative means" refers to algorithms or devices that create new explanations, exercises, study plans, etc. based on data.

[0878] The "means of provision" refers to a mechanism for sending and displaying the generated explanations, exercises, and study plans on students' devices.

[0879] "Exercises" are problems that students should solve to overcome specific areas of understanding.

[0880] A "learning plan" is a plan that designs the optimal learning process based on a student's goals and schedule.

[0881] "Means for receiving questions" refers to the system for accepting questions from students and interpreting the content of those questions.

[0882] The "means for generating answers" are large-scale language models and information retrieval systems that provide optimal answers to students' questions.

[0883] "Facial expression data" refers to data showing the facial expressions of students captured using the built-in camera.

[0884] An "emotion engine" is software or hardware that analyzes facial expression data to identify a student's emotional state in real time.

[0885] A "large-scale language model" is an advanced AI model for natural language processing that is trained using large amounts of text data.

[0886] The system of the present invention records and analyzes students' learning progress and level of understanding in detail to provide personalized learning support. The system is also combined with an emotion engine that recognizes the user's emotions. Specific embodiments for carrying out the present invention will be described below.

[0887] System Overview

[0888] The system consists of a tablet device, a server, a built-in camera, sensors, an emotion engine, and a generative AI model. The system's purpose is to provide individualized learning support based on a student's learning progress, level of understanding, and emotional state.

[0889] Student Login

[0890] Subject: User

[0891] The user logs in by entering their user ID and password on the tablet device. The device sends this authentication information to the authentication server and receives the authentication result.

[0892] Collection of training data and emotion data

[0893] Subject: Terminal

[0894] During a learning session, the device records the learning materials used by the student, their answers, and facial expression data captured by a built-in camera in real time. The facial expression data is then sent to an emotion engine to recognize their emotional state (concentration, confusion, anxiety, etc.).

[0895] Sending data

[0896] Subject: Terminal

[0897] When the learning session ends or after a certain period of time has passed, the device sends the collected learning data and emotion data to the server.

[0898] Analyzing data and identifying understanding

[0899] Subject: Server

[0900] The server receives the learning data and emotional data sent from the device and uses machine learning algorithms to analyze the student's learning progress and level of understanding.

[0901] Generate personalized commentary

[0902] Subject: Server

[0903] The server uses AI technology to generate detailed explanations for areas of learning deficiencies, taking into account the student's emotional state, and is created in the form of text, diagrams, videos, etc.

[0904] Automatic generation of exercises

[0905] Subject: Server

[0906] The server uses a generative AI model to automatically generate exercises that address areas of insufficient understanding, and these exercises are provided to the device.

[0907] Generate personalized learning plans

[0908] Subject: Server

[0909] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. The generated learning plan is provided to the device and serves as a daily learning guide.

[0910] Question handling and emotional care

[0911] Subject: Server

[0912] When students submit questions during learning, the server receives the questions and generates optimal answers using information retrieval and large-scale language models. It also provides appropriate feedback and emotional care through an emotion engine.

[0913] Specific examples

[0914] 1. User logs in

[0915] The device verifies the student's credentials and begins the learning session.

[0916] 2. Collecting training data and emotion data

[0917] The device collects learning progress information and facial expression data, which are then analyzed in real time using an emotion engine.

[0918] 3. Data submission and analysis

[0919] The terminal sends the collected data to the server, which receives it and analyzes it.

[0920] 4. Generating explanations based on analysis results

[0921] The server analyzes the data to identify areas of lack of understanding, and then uses AI to generate explanations that are provided to the device.

[0922] 5. Generating exercises and study plans

[0923] The server automatically generates appropriate practice questions, creates an individual learning plan, and provides it to the terminal.

[0924] 6. Answering questions and providing emotional support

[0925] The server receives the question, generates an answer using a large-scale language model based on the information search results, and provides it to the device. It also provides appropriate feedback depending on the user's emotional state.

[0926] Examples of prompts for generative AI models

[0927] 1. Prompt for student questions:

[0928] "A student is asking about XXX, with a specific point YYY. Based on this information, please suggest how to provide an answer that will satisfy the student."

[0929] 2. Prompt for generating explanation:

[0930] "Student demonstrates a lack of understanding about XXX. Student's emotional state is ZZZ. Please take this information into consideration and create a detailed explanation including text, illustrations, and video."

[0931] This system realizes more personalized learning support that also takes into account the student's emotional state, thereby improving students' learning efficiency and providing effective support tailored to their individual learning needs.

[0932] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0933] Step 1:

[0934] Input: The user enters their user ID and password on the tablet device.

[0935] Operation: The terminal sends the entered user ID and password to the authentication server.

[0936] Data calculation: The authentication server checks the received user ID and password against the information in the database.

[0937] Output: An authentication result is generated and sent to the device, which then starts a training session.

[0938] Step 2:

[0939] Input: Authenticated student start learning operation.

[0940] How it works: The device displays the learning content and collects facial expression data from the student using a built-in camera. At the same time, it also records how the learning material is used and how the student answers the questions.

[0941] Data calculation: The collected facial expression data is sent to the emotion engine in real time to analyze the emotional state.

[0942] Output: The analysis results of the emotion engine and learning data are generated and recorded.

[0943] Step 3:

[0944] Input: End of study session or passage of a certain period of time.

[0945] How it works: The device packages the learning and emotion data collected during the session.

[0946] Data calculation: Formats the training data and emotion data for transmission to the server.

[0947] Output: The packaged data is sent to the server.

[0948] Step 4:

[0949] Input: Training data and emotion data received by the server.

[0950] How it works: The server stores the training data and emotion data in a database.

[0951] Data calculation: Using machine learning algorithms to analyze learning progress and comprehension, and an emotion engine to analyze emotional states.

[0952] Output: Analysis results are generated regarding learning progress, comprehension, gaps, and emotional state.

[0953] Step 5:

[0954] Input: Identified learning gaps and emotional state.

[0955] How it works: Based on the identified gaps in understanding, the server passes prompts to the generative AI model to generate explanations.

[0956] Data calculation: A generative AI model generates an explanation based on the prompt.

[0957] Output: Explanatory content (text, illustrations, video) is generated and sent to the device.

[0958] Step 6:

[0959] Input: Learning gaps and understanding gaps identified by the server.

[0960] How it works: The server uses a generative AI model to create prompts that generate exercises.

[0961] Data computation: Generative AI models generate exercises of appropriate difficulty and format.

[0962] Output: Exercises are generated and sent to the terminal.

[0963] Step 7:

[0964] Input: Learning data, comprehension, emotional state, goals, and schedule information collected by the server.

[0965] How it works: The server uses the collected data to generate a personalized lesson plan.

[0966] Data computation: Using learning algorithms to optimize personalized learning plans.

[0967] Output: The generated lesson plan is sent to the terminal and provided to the student.

[0968] Step 8:

[0969] Input: Questions asked by students.

[0970] Operation: The device sends the query to the server.

[0971] Data Computation: The server analyzes the question and generates an answer using information retrieval and large-scale language models.

[0972] Output: The generated answer is sent to the device and provided to the student, and emotional feedback is also provided using the emotion engine.

[0973] (Application example 2)

[0974] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0975] Conventional learning support systems only record and analyze students' learning progress and comprehension, but are unable to incorporate real-time emotional analysis. As a result, it is difficult to immediately grasp the frustration or confusion students feel during learning and provide appropriate support. Furthermore, content recommendations based on the viewer's emotional state are not performed, preventing the maximization of learning effectiveness. The present invention aims to solve these problems and provide more personalized learning support.

[0976] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording the student's learning progress, means for analyzing the recorded learning progress to identify the student's level of understanding, means for identifying areas where the student does not understand based on the identified level of understanding and generating explanations related to these, means for providing the generated explanations to the student's terminal, means for automatically generating exercises according to the student's level of understanding, means for generating individual study plans based on the student's goals, academic ability, level of understanding, and schedule information, means for providing the generated individual study plans to the student's terminal, means for collecting viewer facial expression data in real time and analyzing their emotional state, and means for recommending learning content based on their emotional state. This incorporates real-time emotion analysis, enabling personalized learning support and content recommendations according to the student's emotional state.

[0977] "Learning progress" refers to the content and progress a student is currently learning.

[0978] "Level of understanding" is an indicator that shows how well a student understands the learning content.

[0979] "Analysis" is the act of carefully analyzing collected data to find meaning and patterns.

[0980] "Explanation" is content that provides supplementary explanations and answers to identified areas of lack of understanding.

[0981] "Devices" refers to electronic devices such as tablets, smartphones, and computers used by students.

[0982] "Exercises" are problems designed to deepen students' understanding.

[0983] A "study plan" is a detailed plan that outlines the schedule and content for students to study efficiently.

[0984] "Facial expression data" refers to information obtained from a viewer's facial expressions and is used to analyze their emotional state.

[0985] "Emotional state" refers to the emotions the viewer feels while learning (concentration, confusion, anxiety, etc.).

[0986] "Content recommendation" is the act of suggesting suitable learning content to maximize the viewer's learning effect.

[0987] A "server" refers to a computer system used to analyze and store data and perform various processing.

[0988] The system of the present invention records and analyzes students' learning progress and understanding in detail to provide personalized learning support. Furthermore, it combines a function to recognize users' emotions to optimize the learning experience. Specific embodiments for implementing the present invention are described below.

[0989] System Configuration

[0990] The system includes the following major components:

[0991] A means of recording student progress

[0992] A means of identifying understanding

[0993] A means of generating explanations for areas of incomprehension

[0994] Devices that provide commentary

[0995] A method for automatically generating exercises according to the level of understanding

[0996] A means of generating personalized learning plans

[0997] A means of collecting facial expression data in real time and analyzing emotional states

[0998] A means of recommending learning content based on emotional state

[0999] A means of receiving student questions and generating answers using large-scale language models

[1000] Hardware and Software

[1001] Hardware: Viewing devices such as tablets, smartphones, smart glasses, or head-mounted displays are used. These devices have built-in cameras that are used to collect facial expression data from viewers.

[1002] Software: Python, OpenCV, and Keras are used to analyze facial expressions in real time, and the data is sent to the server via a REST API.

[1003] Processing Flow

[1004] 1. Data Collection:

[1005] Using the device's built-in camera, facial expression data is collected in real time from the moment the student logs in while studying. In addition, learning content and answer data are also recorded. The facial expression data is analyzed using facial expression recognition technology (e.g., using OpenCV and Keras) to identify emotional states (e.g., concentration, confusion, anxiety).

[1006] 2. Data transmission:

[1007] Once a certain learning session is completed, the collected data is sent to a server, including the learning content, answer data, facial expression data, and emotion recognition results.

[1008] 3. Data analysis and learning support:

[1009] The server uses machine learning algorithms to analyze students' learning progress and comprehension, identifying areas where they lack understanding. It also takes into account their emotional state and uses AI technology to generate appropriate explanations. These explanations are provided in the form of text, illustrations, videos, and other formats, and can be reviewed by students.

[1010] 4. Exercises and Study Plan:

[1011] The server uses a generative AI model to automatically generate practice questions that address areas of incomplete understanding. The generated individualized learning plan is optimized by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data.

[1012] 5. Questions and feedback:

[1013] When students submit questions during their studies, the server receives the questions and generates optimal answers using information search and large-scale language models. It also provides appropriate feedback and emotional care through an emotion engine.

[1014] Specific examples

[1015] For example, imagine a user is watching an online math course. The camera will recognize any confused faces while they are watching and send that data to a server, which can then recommend more understandable videos or supplementary materials in real time.

[1016] Prompt Sentence Examples

[1017] "Generate an algorithm that recommends optimal learning content based on the user's facial expression data and viewing history while they are viewing. Facial expression data is transmitted in real time, and dynamically adjust learning content based on the user's emotional state (e.g., concentration, confusion, anxiety)."

[1018] In this way, the system of the present invention incorporates real-time emotion analysis and realizes personalized learning support according to the student's emotional state.

[1019] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1020] Step 1:

[1021] Data collection

[1022] The device's built-in camera collects students' facial expression data in real time. This happens automatically as soon as the learning session begins. Learning content and answer data are also recorded at the same time. The video data captured by the camera is first identified as a facial region, and then facial expression recognition technology (using OpenCV and Keras) is used to identify emotional states such as "concentration," "confusion," and "anxiety."

[1023] Input: Camera footage, learning content, answer data

[1024] Output: Facial expression data (emotional state)

[1025] Step 2:

[1026] Data transmission

[1027] The collected facial expression data and learning data are sent to the server after a certain learning session has ended or a specified time has passed. The data sent includes the learning content, answer data, facial expression data, and emotion recognition results. This data is sent to the server via a REST API.

[1028] Input: facial expression data, learning content, answer data

[1029] Output: Data sent to the server

[1030] Step 3:

[1031] Data analysis

[1032] The server analyzes the received facial expression data and learning data. It uses machine learning algorithms to identify the student's learning progress and level of understanding, and determines where they are lacking in understanding. At the same time, it takes into account their emotional state. For example, if a student is "confused," it performs additional analysis to find out why.

[1033] Input: Transmitted data (facial expression data, learning content, answer data)

[1034] Output: Learning progress, comprehension, and emotional state identification results

[1035] Step 4:

[1036] Explanation Generation

[1037] The server uses AI technology to generate detailed explanations for identified gaps in understanding. These explanations take into account the student's emotional state and are provided in the most appropriate format, such as text, illustrations, or video. The generated explanations are then retransmitted to the device.

[1038] Input: Areas of incomprehension, emotional state

[1039] Output: explanatory data

[1040] Step 5:

[1041] Exercise problem generation

[1042] The server uses a generative AI model to automatically generate exercises that address areas where students lack understanding, allowing students to tackle problems of the appropriate difficulty and format. The generated exercises are then provided back to the device.

[1043] Input: Area of ​​incomprehension

[1044] Output: Exercise data

[1045] Step 6:

[1046] Learning plan generation

[1047] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. This learning plan is optimized by incorporating the student's requests through an interactive format. The generated learning plan is provided to the student's device and serves as a guide for the student's daily learning.

[1048] Input: Student goals, academic achievement, comprehension, schedule information, emotional data

[1049] Output: Learning plan data

[1050] Step 7:

[1051] Questions and feedback

[1052] When a student submits a question during their studies, the server receives the question and generates the best answer using information retrieval and a large-scale language model. The answer is then provided to the device. The emotion engine also provides appropriate feedback and emotional support according to the student's emotional state.

[1053] Input: Student question

[1054] Output: Answers to questions, emotional care feedback

[1055] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1056] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1057] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1058] [Third embodiment]

[1059] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1060] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1061] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1062] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1063] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1064] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1065] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1066] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1067] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1068] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1069] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1070] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1071] The system of the present invention is designed to provide personalized learning support according to the learning progress and level of understanding of each student. Specific embodiments for carrying out the present invention will be described below.

[1072] Acquiring learning records

[1073] Subject: Terminal

[1074] First, when a student logs in to a tablet device, the device starts the student's study session. It records the learning materials (digital texts, video materials, etc.) used by the student and their learning progress (study time, answer status, etc.). It also uses the device's built-in camera and IoT sensors to collect data such as facial expressions and posture. This data is used to comprehensively understand the student's learning situation.

[1075] Analyzing learning data and identifying understanding

[1076] Subject: Server

[1077] When the learning data sent from the device is received by the server, the server analyzes the data and identifies the student's level of understanding. Specifically, it uses machine learning algorithms to analyze the student's answer patterns and learning progress and identify which parts of the learning content they have not understood.

[1078] AI-powered commentary

[1079] Subject: Server

[1080] The server then uses AI technology to generate explanations for the identified areas of incomprehension. The explanations are generated in a format that is easy for students to understand (e.g., text, diagrams, videos, etc.). These explanations are then provided to students' devices so that they can access them at any time.

[1081] Automatic generation of exercises

[1082] Subject: Server

[1083] Based on the students' level of understanding, the server automatically generates appropriate exercises using a generative AI model, allowing them to tackle problems that are tailored to their individual level of understanding. The generated exercises are provided to the students' devices, and they can further deepen their understanding by answering them.

[1084] Generate personalized learning plans

[1085] Subject: Server

[1086] The server then generates an individual learning plan based on the student's goals, academic ability, level of understanding, and schedule information. This learning plan is customized by interacting with the student, incorporating their wishes and goals. The generated learning plan is provided to the student's device, allowing the student to proceed with their studies accordingly.

[1087] RAG (Retriever-Augmented Generation) module

[1088] Subject: Server

[1089] Finally, if a student has a question during their study, the server receives the question and generates the best answer using a large-scale language model based on the information search results. This answer is then provided to the student's device, allowing the student to immediately resolve their doubts.

[1090] Specific examples

[1091] 1. User logs in

[1092] The device verifies the student's credentials and begins the learning session.

[1093] 2. Obtaining training data

[1094] The device collects learning progress information and facial expression data and sends them to the server.

[1095] 3. Providing analysis and commentary

[1096] The server analyzes the data to identify areas where students lack understanding, and then uses AI to generate explanations that are provided to the device.

[1097] 4. Exercises and Study Plans

[1098] The server automatically generates appropriate exercises and creates a learning plan based on the student's level of understanding, which is then provided to the terminal.

[1099] 5. Answering questions

[1100] When a student submits a question, the server searches for information, generates the most appropriate answer, and provides it to the device.

[1101] This allows students to study effectively at their own pace and at their own level of understanding.

[1102] The processing flow will be explained below.

[1103] Step 1:

[1104] The device checks the student's credentials

[1105] The device receives the student's login information (user ID and password) and authenticates it with the database. If authentication is successful, the device accesses the student's individual account and begins the learning session.

[1106] Step 2:

[1107] Your device records your study session

[1108] The device records the learning materials selected by the student (e.g., mathematics textbook, history video, etc.) and also records input data from the touch panel and keyboard (answer time, options, answer results, etc.).

[1109] Step 3:

[1110] The device collects data using cameras and IoT sensors

[1111] The device's built-in camera captures students' facial expressions, and IoT sensors monitor their posture and movements while they study, collecting data in real time.

[1112] Step 4:

[1113] The device sends the learning data to the server.

[1114] When the learning session ends or a certain period of time has passed, the device sends all collected data (teaching materials, answer data, facial expression data, etc.) to the server.

[1115] Step 5:

[1116] The server receives and analyzes the learning data.

[1117] The server receives the learning data sent from the device and analyzes it using machine learning algorithms, which identifies areas where the student does not understand and their learning trends.

[1118] Step 6:

[1119] The server generates an explanation for any missing information

[1120] The server uses AI technology to create detailed explanations of the identified areas of lack of understanding, which can be in the form of text, illustrations, videos, etc.

[1121] Step 7:

[1122] The server sends the commentary to the device.

[1123] The server generates explanations that are sent to students' devices so that they can access them at any time. The explanations are saved in individual student accounts.

[1124] Step 8:

[1125] The server automatically generates exercises

[1126] The server uses the generative AI model to automatically generate practice questions that address the identified gaps in understanding, adjusting the difficulty and format of the questions according to the student's level of understanding.

[1127] Step 9:

[1128] The server sends the exercises to the device.

[1129] The server sends the generated exercises to the students' devices so that they can work on them. The answers are also recorded sequentially.

[1130] Step 10:

[1131] The server generates an individualized learning plan.

[1132] The server generates an individualized learning plan based on the student's goals, academic ability, level of understanding, and schedule information. This plan is interactive and incorporates the student's requests.

[1133] Step 11:

[1134] The server sends the lesson plan to the device.

[1135] The server transmits the generated individual learning plan to the student's terminal, allowing the student to progress with their learning according to the daily learning plan.

[1136] Step 12:

[1137] The device will carry out learning according to the learning plan.

[1138] Students progress through their studies according to the study plan displayed on their device and access explanations and practice problems as needed.

[1139] Step 13:

[1140] The server receives the student's questions and generates answers.

[1141] When a student submits a question during their studies, the server receives the question and generates an answer using a large-scale language model based on the information search results.

[1142] Step 14:

[1143] The server generates a response and sends it to the device.

[1144] The server generates answers that are sent to students' devices to help them resolve their questions.

[1145] This will help each student to progress effectively in their studies.

[1146] Example 1

[1147] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1148] In recent years, there has been a demand for personalized learning support tailored to each learner's individual learning progress and level of understanding. However, conventional learning support systems have had problems in that they are insufficient in providing specific explanations and practice problems based on each learner's individual level of understanding, and in responding immediately to students' questions. This has made it difficult for students to study efficiently and effectively. In addition, the accuracy of collecting and analyzing learning data has been low, making it difficult to provide accurate learning support.

[1149] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1150] In this invention, the server includes means for recording the student's learning progress, means for analyzing the recorded learning progress to determine the student's level of understanding, means for determining the student's lack of understanding based on the determined level of understanding and generating explanations for the identified areas, means for providing the generated explanations to the student's terminal, means for automatically generating exercises according to the student's level of understanding, means for generating an individual learning plan based on the student's goals, academic ability, level of understanding, and schedule information, means for providing the generated individual learning plan to the student's terminal, means for receiving the student's questions and generating answers using natural language processing technology with reference to information search results, means for collecting the student's facial expression and posture data at the terminal, means for transmitting the learning data to the server in real time, means for analyzing the data using a machine learning algorithm, and means for generating explanations using artificial intelligence. This makes it possible to provide specific and effective support according to each student's individual learning progress.

[1151] "Learning progress" is an indicator that shows how much content a student has learned through learning activities and how much time they have spent.

[1152] "Recording means" refers to devices or software for storing students' learning progress, study time, answer status, and other related data in a database or cloud storage.

[1153] "Means for analysis" refers to software or systems that apply machine learning algorithms and statistical methods to assess students' learning progress and understanding using collected learning data.

[1154] "Level of understanding" is an indicator of how well a student has understood the learning content, including the accuracy and speed of answers and identification of areas of lack of understanding.

[1155] A "means for generating explanations" is a device or software that uses artificial intelligence or other automatic generation technology to create easy-to-understand explanations for students in the form of text, illustrations, videos, etc. for identified areas of lack of understanding.

[1156] "Practice questions" are questions that students answer to review what they have learned and check their understanding.

[1157] A "means for automatically generating exercises" is a device or software that uses artificial intelligence or generation technology to automatically generate exercises of appropriate difficulty and content based on the student's level of understanding.

[1158] A "study plan" is a plan that includes the learning content and schedule established for a student to achieve their goals.

[1159] A "means for generating an individual learning plan" is a device or software that creates and provides an individually optimized learning plan based on a student's goals, academic ability, level of understanding, schedule information, etc.

[1160] A "question receiving" means is an interface, communication device, or software that allows students to submit questions or concerns online while studying.

[1161] "Information search results" are the results obtained by searching databases and the Internet for information related to a student's question.

[1162] "Natural language processing technology" is a field of artificial intelligence and is a technology for understanding and generating human language.

[1163] "Facial expression and posture data" refers to information about facial expressions and body postures captured to understand students' learning status.

[1164] "Means for transmitting in real time" refers to communication equipment and communication protocols for transmitting learning data from the collection source to the server without delay.

[1165] A "machine learning algorithm" is a mathematical model and computational method for analyzing collected data and extracting patterns and features.

[1166] "Artificial intelligence" refers to systems and software that mimic human intelligence and perform learning and reasoning, and is used to generate explanations and questions.

[1167] The system of the present invention provides personalized learning support according to the student's learning progress and level of understanding. Specific embodiments for carrying out the present invention will be described below.

[1168] Hardware and Software Configuration

[1169] This system uses tablet devices, cameras, and IoT sensors to collect learning data, which is then analyzed on a server. Specifically, it uses the following hardware and software:

[1170] Hardware:

[1171] Tablet: A device used by students for learning.

[1172] Camera: Used to capture students' facial expressions.

[1173] IoT sensors: Used to collect students' posture data.

[1174] software:

[1175] Authentication system: Authentication system using OAuth or JWT.

[1176] Data collection software: Software for collecting learning progress, facial expressions, and posture data.

[1177] Data analysis software: TensorFlow and Scikit-learn are used to analyze data using machine learning algorithms.

[1178] Generative AI model: Generates explanations and practice problems using models such as OpenAI's GPT-4.

[1179] Natural language processing systems: Answering student questions using large-scale language models.

[1180] Processing flow and specific examples

[1181] Students log in and start a study session

[1182] When a student logs in, the device checks the authentication information and starts the learning session. For example, authentication is performed by entering a user ID and password. At this time, the device ensures security by using JWT authentication.

[1183] Collecting and sending learning data

[1184] During a learning session, the device records data such as the student's viewing status, answer status, and study time. It also records the student's facial expressions and posture data acquired through built-in cameras and IoT sensors. The acquired data is sent to a server in real time.

[1185] Analysis of training data

[1186] The server analyzes the data received from the devices using machine learning algorithms. Specifically, it uses the collected data to evaluate students' answer patterns and learning progress, and identifies their level of understanding. This analysis is performed using TensorFlow and Scikit-learn.

[1187] Generate explanations for areas of incomprehension

[1188] The server then uses a generative AI model to generate explanations for identified gaps in understanding. For example, OpenAI's GPT-4 is used to generate explanations in the form of text, diagrams, or videos. These explanations are then provided to students' devices for access.

[1189] Automatic generation of exercises

[1190] Based on the student's level of understanding, the server uses a generative AI model to generate appropriate exercises. The generated exercises are then provided to the student's device, and the student deepens their understanding by answering them.

[1191] Prompt Sentence Examples

[1192] Here are some examples of specific prompt sentences:

[1193] To generate explanations for areas of incomprehension:

[1194] "My students have a limited understanding of differential calculus in Chapter 5. I would like you to write an easy-to-understand explanation."

[1195] For generating practice questions:

[1196] "Generate basic differential calculus problems based on students' understanding."

[1197] Generate personalized learning plans

[1198] The server generates an individual learning plan based on the student's goals, schedule, and level of understanding, and provides it to the student's device. This plan incorporates the student's goals and optimizes daily learning.

[1199] Answering questions during study

[1200] When a student submits a question during their study, the RAG module on the server receives the question and uses natural language processing technology to generate the most appropriate answer, which is then provided to the device.

[1201] This detailed processing flow makes it possible to provide specific and effective support according to each student's learning progress.

[1202] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1203] Step 1: Students log in and start a study session

[1204] Subject: Terminal

[1205] When a student logs in to the device, they are prompted to enter their user ID and password. The entered information is sent to the authentication server, which performs authentication using JWT authentication. If authentication is successful, the device starts a learning session and displays the dashboard screen. The input is the user ID and password, and the output is the authentication result and the start of the learning session.

[1206] Specific behavior:

[1207] Input: User ID, Password

[1208] Data processing: JWT authentication

[1209] Output: Authentication results, dashboard screen

[1210] Step 2: Collect and send training data

[1211] Subject: Terminal

[1212] During a learning session, the device records data such as the student's viewing status, answer status, and study time. It also uses a built-in camera and IoT sensors to collect data on the student's facial expression and posture. This data is sent to a server in real time. The input is learning progress information, facial expression data, and posture data, and the output is data sent to the server.

[1213] Specific behavior:

[1214] Input: learning progress information, facial expression data, posture data

[1215] Data processing: Data recording, sensor information collection

[1216] Output: Send data to the server

[1217] Step 3: Analyze the training data

[1218] Subject: Server

[1219] The server receives the data sent from the device and analyzes it using machine learning algorithms. The analysis evaluates the student's answer patterns and learning progress, and identifies their level of understanding. TensorFlow and Scikit-learn are used for this analysis. The input is the learning data sent from the device, and the output is the level of understanding information obtained as a result of the analysis.

[1220] Specific behavior:

[1221] Input: Training data from the device

[1222] Data processing: Analysis using machine learning algorithms

[1223] Output: Comprehension information

[1224] Step 4: Generate explanations for areas of incomprehension

[1225] Subject: Server

[1226] The server uses a generative AI model to generate explanations for the identified areas of incomprehension. For example, by inputting a prompt into the GPT-4 model, it generates explanatory text, illustrations, and videos. The input is information about the area of ​​incomprehension and the prompt, and the output is the generated explanation.

[1227] Specific behavior:

[1228] Input: Information on areas of incomprehension, prompt text

[1229] Data processing: Generative AI model for generating explanations

[1230] Output: explanatory text, illustrations, videos

[1231] Step 5: Automatic generation of exercises

[1232] Subject: Server

[1233] The server generates appropriate exercises using a generative AI model based on the student's level of comprehension. The generated exercises are sent to the student's device. The input is comprehension information and a prompt, and the output is the generated exercise.

[1234] Specific behavior:

[1235] Input: Comprehension information, prompt

[1236] Data processing: Generative AI model for generating exercises

[1237] Output: Exercises

[1238] Step 6: Generate an individualized learning plan

[1239] Subject: Server

[1240] The server generates an individual learning plan based on the student's goals, academic ability, level of understanding, and schedule information. The generated learning plan is provided to the student's device. The input is the student's goals, academic ability, level of understanding, and schedule information, and the output is the individual learning plan.

[1241] Specific behavior:

[1242] Input: Goals, academic ability, level of understanding, schedule information

[1243] Data processing: Planning using learning plan generation algorithms

[1244] Output: Individualized Learning Plan

[1245] Step 7: Answering questions while studying

[1246] Subject: Server

[1247] The server receives questions sent by students during their studies, and generates optimal answers using a large-scale language model based on information search results. These answers are then provided to the students' devices. The input is the student's question, and the output is the generated answer.

[1248] Specific behavior:

[1249] Input: Student question

[1250] Data Processing: Natural Language Processing and Information Retrieval

[1251] Output: Best answer

[1252] (Application example 1)

[1253] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1254] Conventional factory worker education and training systems lacked personalized support tailored to each worker's learning progress and skill level, making it difficult to efficiently improve skills. They also lacked a means to respond to workers' questions in real time and provide specific feedback. This created the challenge of not maximizing the effectiveness of worker training.

[1255] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1256] In this invention, the server includes means for recording the worker's learning progress, means for analyzing the recorded learning progress to identify the worker's level of understanding, means for identifying areas where the worker does not understand based on the identified level of understanding and generating explanations for those areas, means for providing the generated explanations to the worker's terminal, means for automatically generating exercises according to the worker's level of understanding, means for generating an individual learning plan based on the worker's goals, skills, level of understanding, and schedule information, means for providing the generated individual learning plan to the worker's terminal, means for collecting and analyzing the worker's operation records and posture data, and means for receiving the worker's questions and generating answers using a generative AI model with reference to the information acquisition results. This makes it possible to provide personalized training plans based on each worker's level of understanding and respond to questions in real time.

[1257] "Workers" refers to the technicians and workers who operate machinery and equipment in factories and manufacturing sites. They are the recipients of training according to their individual skill levels and learning progress.

[1258] "Learning progress" refers to the progress a worker makes along a training or education program, specifically the degree to which knowledge and skills have been acquired.

[1259] "Level of understanding" refers to the standard for assessing how well a worker understands the learning content and operating procedures. It is an important indicator for providing feedback to improve identified areas of lack of understanding.

[1260] "Explanation" refers to content that provides additional information or explanations for areas where workers are struggling, often in the form of text, video, or AR guides.

[1261] "Devices" refer to electronic devices used by workers for operation and learning, including tablet devices and head-mounted displays.

[1262] "Exercises" refer to assignments and tasks provided to workers to allow them to try out what they have learned. They are automatically generated based on each worker's level of understanding.

[1263] "Individualized Learning Plan" refers to a customized education and training schedule based on a worker's goals, skills, level of understanding, and schedule information.

[1264] "Operation records" refer to the history of specific operations and tasks performed by workers. Recording these records will be useful for analysis.

[1265] "Posture data" refers to data on the body movements and posture of workers when operating a machine. It is acquired using sensors and cameras.

[1266] A "generative AI model" refers to an artificial intelligence algorithm or system that automatically generates information based on large datasets, which can be used to generate answers to human questions.

[1267] "Information acquisition results" refers to information collected from the internet or other databases in response to worker questions. This is part of the data used by the generative AI model.

[1268] This invention is a system for personalizing education and training for factory workers. This system records and analyzes the worker's learning progress and provides feedback, practice questions, and individual learning plans based on the worker's level of understanding.

[1269] Hardware and software used

[1270] The hardware used includes a tablet computer, a head-mounted display (HMD), a camera, and IoT sensors. This hardware is used to acquire the worker's operation records and posture data. The software used includes a learning data collection app, a data analysis server, an AI explanation system, an exercise problem generation system, and a RAG (Retriever-Augmented Generation) module.

[1271] Acquiring and analyzing training data

[1272] The device records the worker's learning progress. Specifically, it collects the worker's operations and work history, as well as posture and movement data acquired by cameras and IoT sensors. This data is sent to a server in real time.

[1273] The server analyzes the received data and identifies the worker's level of understanding. The analysis uses a machine learning algorithm to identify the areas where each worker lacks understanding. The server then generates an explanation corresponding to the area of ​​lack of understanding and provides it to the worker's device.

[1274] Providing AI explanations and practice questions

[1275] The server uses AI technology to generate easy-to-understand explanations for the identified areas of insufficient understanding. These explanations are created in the form of text, video, AR guides, etc. and are provided to the worker's device. Exercises are also automatically generated based on the worker's level of understanding. The exercises are set at an appropriate level of difficulty with the aim of improving the worker's skills.

[1276] Generate and provide individualized learning plans

[1277] The server generates an individual learning plan based on the worker's goals, skills, level of understanding, and schedule information. This allows the worker to learn at their own pace, enabling effective training. The generated learning plan is provided to the terminal, and the worker follows it to carry out training.

[1278] Real-time question response (RAG module)

[1279] If a worker has a question during learning, the server receives the question and generates an answer using the generative AI model based on the information acquisition results. This answer is provided to the worker's device in real time, instantly resolving the worker's question.

[1280] Examples of concrete examples and prompts

[1281] As an example, consider a worker wearing a head-mounted display and undergoing training to operate a robot. The worker's operation records and posture data are acquired by the terminal and sent to the server. After analyzing the data, the server generates video explanations for any operating procedures that the worker does not fully understand and provides them to the terminal. In addition, it automatically generates exercises related to the operation and presents them to the worker as assignments. If the worker asks a question during this process, the server uses the RAG module to instantly generate an answer and provides it to the terminal.

[1282] An example of a prompt sentence might be:

[1283] "We would like to develop a training system for robot operation in factories. Please come up with an application that provides individual feedback, practice assignments, and training plans based on the worker's skill level and progress."

[1284] As described above, the system of the present invention provides personalized education and training to workers, and supports efficient skill improvement.

[1285] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1286] Step 1:

[1287] The user logs in to the device. The device receives the user's authentication information as input and performs authentication to start a learning session. If authentication is successful, the device obtains the user's past learning data and current learning goals.

[1288] Step 2:

[1289] The device records the user's operations and learning progress in real time. Specifically, the device collects operation records and posture data using cameras and sensors installed on the tablet device or head-mounted display. This data is input and sent to a server.

[1290] Step 3:

[1291] The server analyzes the received data. It receives operation records and posture data as input, and uses a machine learning algorithm to evaluate the user's level of understanding. As a result of the analysis, it identifies and outputs the areas where the user does not understand or where they lack skills.

[1292] Step 4:

[1293] The server generates explanations for the identified areas of incomprehension, using AI technology to create explanations in the form of text, video, or AR guides. Using this as input, the server generates explanation data in the appropriate format and outputs it to be provided to the device.

[1294] Step 5:

[1295] The server automatically generates exercises based on the user's level of understanding and skills. It uses the analysis results as input and creates exercises using a generative AI model. This outputs appropriate exercise data aimed at improving the user's skills.

[1296] Step 6:

[1297] The server generates a personalized learning plan based on the user's goals, skills, level of understanding, and schedule information. It uses the user's profile data and analysis results as input to create an optimal learning schedule. This plan is output for delivery to the device.

[1298] Step 7:

[1299] When a user asks a question during learning, the device sends the question to the server. The device receives the question data as input, uses the RAG module to refer to the information acquisition results, and generates an answer using the generative AI model. The generated answer is sent to the device and provided to the user.

[1300] These steps explain how each task is performed and what input data is used to generate output data, enabling personalized training and real-time question answering.

[1301] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1302] The system of the present invention records and analyzes students' learning progress and level of understanding in detail to provide personalized learning support. This system is also combined with an emotion engine that recognizes the user's emotions. Specific embodiments for carrying out the present invention will be described below.

[1303] Acquiring training data and emotion recognition

[1304] Subject: Terminal

[1305] When a student logs in to a tablet device, the device begins a learning session. While recording data used during the learning process (such as learning material content and answer status), the device also collects the student's facial expression data using the built-in camera. This collected facial expression data is sent in real time to an emotion engine that recognizes the student's emotional state (concentration, confusion, anxiety, etc.).

[1306] Sending training data and emotion data

[1307] Subject: Terminal

[1308] When the learning session ends or after a certain period of time has passed, the device sends the collected learning data and emotion data to the server. This data includes the learning content, answer data, facial expression data, and emotion recognition results.

[1309] Analyzing data and identifying understanding

[1310] Subject: Server

[1311] The server receives the learning data and emotional data sent from the device and first analyzes the student's learning progress and level of understanding using a machine learning algorithm. From the analysis results, it identifies areas where understanding is lacking, and at the same time, taking into account the results of the emotional engine, it also understands the emotional state the student was in while studying.

[1312] Generate personalized commentary

[1313] Subject: Server

[1314] For identified gaps in understanding, the server uses AI technology to generate detailed explanations. The explanations take into account the student's emotional state and are created in context-appropriate formats such as text, diagrams, and videos. Furthermore, these explanations are provided to students' devices so they can review them repeatedly.

[1315] Automatic generation of exercises

[1316] Subject: Server

[1317] The server uses a generative AI model to automatically generate practice questions that address areas of incomplete understanding, allowing students to tackle questions of the appropriate difficulty and format. The practice questions are also provided to the device, and the answers are collected again.

[1318] Generate personalized learning plans

[1319] Subject: Server

[1320] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. This learning plan is optimized by incorporating the student's requests through an interactive format. The generated learning plan is provided to the student's device and serves as a guide for daily learning.

[1321] Question handling and emotional care

[1322] Subject: Server

[1323] When a student submits a question during learning, the server receives the question, performs an information search, and uses the results to generate the optimal answer using a large-scale language model. The answer is then provided to the device, instantly resolving the student's question. The system also uses an emotion engine to provide appropriate feedback and emotional support according to the student's emotional state during learning.

[1324] Specific examples

[1325] 1. User logs in

[1326] The device verifies the student's credentials and begins the learning session.

[1327] 2. Collecting training data and emotion data

[1328] The device collects learning progress information and facial expression data, which are then analyzed in real time using an emotion engine.

[1329] 3. Sending and Receiving Data

[1330] The terminal sends the collected data to the server, which receives it and analyzes it.

[1331] 4. Providing analysis and commentary

[1332] The server analyzes the data to identify areas of lack of understanding, and then uses AI to generate explanations that are provided to the device.

[1333] 5. Exercises and Study Plans

[1334] The server automatically generates appropriate practice questions, creates an individual learning plan, and provides it to the terminal.

[1335] 6. Answering questions and providing emotional support

[1336] The server receives the question, generates an answer using a large-scale language model based on the information search results, and provides it to the device. It also provides appropriate feedback depending on the user's emotional state.

[1337] In this way, this system provides more personalized learning support that also takes into account the student's emotional state.

[1338] The processing flow will be explained below.

[1339] Step 1:

[1340] The device checks the student's credentials

[1341] The device receives the student's login information (user ID and password) and authenticates it with the database. If authentication is successful, the device accesses the student's individual account and begins the learning session.

[1342] Step 2:

[1343] Your device records your study session

[1344] The device records the learning materials selected by the student (e.g., mathematics textbook, history video, etc.) and also records input data from the touch panel and keyboard (answer time, options, answer results, etc.).

[1345] Step 3:

[1346] The device collects data using cameras and IoT sensors

[1347] The device's built-in camera captures students' facial expressions, and IoT sensors monitor their posture and movements while they study, collecting data in real time.

[1348] Step 4:

[1349] The device sends facial expression data to the emotion engine to recognize emotions.

[1350] The device sends the collected facial expression data to the emotion engine, which then analyzes it to recognize the student's emotional state (e.g., concentration, confusion, anxiety, etc.) in real time.

[1351] Step 5:

[1352] The device sends the learning data and emotion data to the server.

[1353] When the learning session ends or after a certain period of time has passed, the device sends all collected data (teaching materials, answer data, facial expression data, emotional data, etc.) to the server.

[1354] Step 6:

[1355] The server receives and analyzes the learning data and emotion data.

[1356] The server receives the learning data and emotion data sent from the device, then analyzes the data using machine learning algorithms and an emotion engine to identify the student's learning progress and level of understanding.

[1357] Step 7:

[1358] The server generates an explanation for any missing information

[1359] The server uses AI technology to generate detailed explanations for the identified areas of lack of understanding. These explanations are provided to students in the form of text, illustrations, videos, etc.

[1360] Step 8:

[1361] Sends server-generated commentary to the device

[1362] The server generates explanations that are sent to students' devices so that they can access them at any time. The explanations are saved in individual student accounts.

[1363] Step 9:

[1364] The server automatically generates exercises

[1365] The server uses the generative AI model to automatically generate practice questions that address areas of incomprehension, adjusting the difficulty and format of the questions according to the student's level of understanding and emotional state.

[1366] Step 10:

[1367] The server sends the generated exercises to the device.

[1368] The server sends the generated exercises to the students' devices so that they can work on them. The answers are collected again and sent to the server.

[1369] Step 11:

[1370] The server generates an individualized learning plan.

[1371] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. This plan is interactive and incorporates the student's requests.

[1372] Step 12:

[1373] The server sends the generated lesson plan to the device.

[1374] The server transmits the generated individual learning plan to the student's terminal, allowing the student to proceed with their studies in accordance with it.

[1375] Step 13:

[1376] The device will carry out learning according to the learning plan.

[1377] Students progress through their studies according to the study plan displayed on their device and access explanations and practice problems as needed.

[1378] Step 14:

[1379] The server receives the student's questions and generates answers.

[1380] When students submit questions during their studies, the server receives the questions and generates the best answer using a large-scale language model based on the information search results.

[1381] Step 15:

[1382] The server generates a response and sends it to the device.

[1383] The server generates answers and sends them to students' devices to help them resolve their questions. It also uses an emotion engine to provide appropriate feedback and emotional support according to the student's emotional state during the course of their studies.

[1384] Example 2

[1385] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1386] Existing learning support systems are not only unable to record and analyze students' learning progress and comprehension, but also unable to provide personalized learning support that takes into account students' emotional state. Furthermore, they lack the ability to generate quick and appropriate answers to students' questions. This makes it difficult to provide learning plans tailored to individual students' needs.

[1387] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for recording the student's learning progress, means for analyzing the recorded learning progress to identify the student's level of understanding, means for analyzing the recorded facial expression data to identify the student's emotional state, means for identifying areas where the student does not understand based on the identified level of understanding and emotional state and generating explanations for those areas, means for automatically generating exercises according to the student's level of understanding, means for generating an individual learning plan based on the student's goals, academic ability, level of understanding, emotional state, and schedule information, means for providing the generated individual learning plan to the student's terminal, and means for receiving the student's question and generating an answer using a large-scale language model with reference to information search results. This enables individual learning support based on the student's learning progress, level of understanding, and emotional state.

[1388] "Learning progress" is data that shows how far a student has progressed with a particular assignment or material.

[1389] "Understanding" is data that indicates how well a student understands a particular material or concept.

[1390] "Emotional state" is data that indicates a student's psychological state (concentration, confusion, anxiety, etc.) while studying.

[1391] A "tablet device" is a portable computer device used by students when studying.

[1392] "Means of recording" refers to a system for collecting and storing data such as students' learning progress and emotional state.

[1393] "Means for analysis" are algorithms or devices that analyze collected data and assess learning progress, comprehension, and emotional state.

[1394] "Generative means" refers to algorithms or devices that create new explanations, exercises, study plans, etc. based on data.

[1395] The "means of provision" refers to a mechanism for sending and displaying the generated explanations, exercises, and study plans on students' devices.

[1396] "Exercises" are problems that students should solve to overcome specific areas of understanding.

[1397] A "learning plan" is a plan that designs the optimal learning process based on a student's goals and schedule.

[1398] "Means for receiving questions" refers to the system for accepting questions from students and interpreting the content of those questions.

[1399] The "means for generating answers" are large-scale language models and information retrieval systems that provide optimal answers to students' questions.

[1400] "Facial expression data" refers to data showing the facial expressions of students captured using the built-in camera.

[1401] An "emotion engine" is software or hardware that analyzes facial expression data to identify a student's emotional state in real time.

[1402] A "large-scale language model" is an advanced AI model for natural language processing that is trained using large amounts of text data.

[1403] The system of the present invention records and analyzes students' learning progress and level of understanding in detail to provide personalized learning support. The system is also combined with an emotion engine that recognizes the user's emotions. Specific embodiments for carrying out the present invention will be described below.

[1404] System Overview

[1405] The system consists of a tablet device, a server, a built-in camera, sensors, an emotion engine, and a generative AI model. The system's purpose is to provide individualized learning support based on a student's learning progress, level of understanding, and emotional state.

[1406] Student Login

[1407] Subject: User

[1408] The user logs in by entering their user ID and password on the tablet device. The device sends this authentication information to the authentication server and receives the authentication result.

[1409] Collection of training data and emotion data

[1410] Subject: Terminal

[1411] During a learning session, the device records the learning materials used by the student, their answers, and facial expression data captured by a built-in camera in real time. The facial expression data is then sent to an emotion engine to recognize their emotional state (concentration, confusion, anxiety, etc.).

[1412] Sending data

[1413] Subject: Terminal

[1414] When the learning session ends or after a certain period of time has passed, the device sends the collected learning data and emotion data to the server.

[1415] Analyzing data and identifying understanding

[1416] Subject: Server

[1417] The server receives the learning data and emotional data sent from the device and uses machine learning algorithms to analyze the student's learning progress and level of understanding.

[1418] Generate personalized commentary

[1419] Subject: Server

[1420] The server uses AI technology to generate detailed explanations for areas of learning deficiencies, taking into account the student's emotional state, and is created in the form of text, diagrams, videos, etc.

[1421] Automatic generation of exercises

[1422] Subject: Server

[1423] The server uses a generative AI model to automatically generate exercises that address areas of insufficient understanding, and these exercises are provided to the device.

[1424] Generate personalized learning plans

[1425] Subject: Server

[1426] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. The generated learning plan is provided to the device and serves as a daily learning guide.

[1427] Question handling and emotional care

[1428] Subject: Server

[1429] When students submit questions during learning, the server receives the questions and generates optimal answers using information retrieval and large-scale language models. It also provides appropriate feedback and emotional care through an emotion engine.

[1430] Specific examples

[1431] 1. User logs in

[1432] The device verifies the student's credentials and begins the learning session.

[1433] 2. Collecting training data and emotion data

[1434] The device collects learning progress information and facial expression data, which are then analyzed in real time using an emotion engine.

[1435] 3. Data submission and analysis

[1436] The terminal sends the collected data to the server, which receives it and analyzes it.

[1437] 4. Generating explanations based on analysis results

[1438] The server analyzes the data to identify areas of lack of understanding, and then uses AI to generate explanations that are provided to the device.

[1439] 5. Generating exercises and study plans

[1440] The server automatically generates appropriate practice questions, creates an individual learning plan, and provides it to the terminal.

[1441] 6. Answering questions and providing emotional support

[1442] The server receives the question, generates an answer using a large-scale language model based on the information search results, and provides it to the device. It also provides appropriate feedback depending on the user's emotional state.

[1443] Examples of prompts for generative AI models

[1444] 1. Prompt for student questions:

[1445] "A student is asking about XXX, with a specific point YYY. Based on this information, please suggest how to provide an answer that will satisfy the student."

[1446] 2. Prompt for generating explanation:

[1447] "Student demonstrates a lack of understanding about XXX. Student's emotional state is ZZZ. Please take this information into consideration and create a detailed explanation including text, illustrations, and video."

[1448] This system realizes more personalized learning support that also takes into account the student's emotional state, thereby improving students' learning efficiency and providing effective support tailored to their individual learning needs.

[1449] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1450] Step 1:

[1451] Input: The user enters their user ID and password on the tablet device.

[1452] Operation: The terminal sends the entered user ID and password to the authentication server.

[1453] Data calculation: The authentication server checks the received user ID and password against the information in the database.

[1454] Output: An authentication result is generated and sent to the device, which then starts a training session.

[1455] Step 2:

[1456] Input: Authenticated student start learning operation.

[1457] How it works: The device displays the learning content and collects facial expression data from the student using a built-in camera. At the same time, it also records how the learning material is used and how the student answers the questions.

[1458] Data calculation: The collected facial expression data is sent to the emotion engine in real time to analyze the emotional state.

[1459] Output: The analysis results of the emotion engine and learning data are generated and recorded.

[1460] Step 3:

[1461] Input: End of study session or passage of a certain period of time.

[1462] How it works: The device packages the learning and emotion data collected during the session.

[1463] Data calculation: Formats the training data and emotion data for transmission to the server.

[1464] Output: The packaged data is sent to the server.

[1465] Step 4:

[1466] Input: Training data and emotion data received by the server.

[1467] How it works: The server stores the training data and emotion data in a database.

[1468] Data calculation: Using machine learning algorithms to analyze learning progress and comprehension, and an emotion engine to analyze emotional states.

[1469] Output: Analysis results are generated regarding learning progress, comprehension, gaps, and emotional state.

[1470] Step 5:

[1471] Input: Identified learning gaps and emotional state.

[1472] How it works: Based on the identified gaps in understanding, the server passes prompts to the generative AI model to generate explanations.

[1473] Data calculation: A generative AI model generates an explanation based on the prompt.

[1474] Output: Explanatory content (text, illustrations, video) is generated and sent to the device.

[1475] Step 6:

[1476] Input: Learning gaps and understanding gaps identified by the server.

[1477] How it works: The server uses a generative AI model to create prompts that generate exercises.

[1478] Data computation: Generative AI models generate exercises of appropriate difficulty and format.

[1479] Output: Exercises are generated and sent to the terminal.

[1480] Step 7:

[1481] Input: Learning data, comprehension, emotional state, goals, and schedule information collected by the server.

[1482] How it works: The server uses the collected data to generate a personalized lesson plan.

[1483] Data computation: Using learning algorithms to optimize personalized learning plans.

[1484] Output: The generated lesson plan is sent to the terminal and provided to the student.

[1485] Step 8:

[1486] Input: Questions asked by students.

[1487] Operation: The device sends the query to the server.

[1488] Data Computation: The server analyzes the question and generates an answer using information retrieval and large-scale language models.

[1489] Output: The generated answer is sent to the device and provided to the student, and emotional feedback is also provided using the emotion engine.

[1490] (Application example 2)

[1491] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1492] Conventional learning support systems only record and analyze students' learning progress and comprehension, but are unable to incorporate real-time emotional analysis. As a result, it is difficult to immediately grasp the frustration or confusion students feel during learning and provide appropriate support. Furthermore, content recommendations based on the viewer's emotional state are not performed, preventing the maximization of learning effectiveness. The present invention aims to solve these problems and provide more personalized learning support.

[1493] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording the student's learning progress, means for analyzing the recorded learning progress to identify the student's level of understanding, means for identifying areas where the student does not understand based on the identified level of understanding and generating explanations related to these, means for providing the generated explanations to the student's terminal, means for automatically generating exercises according to the student's level of understanding, means for generating individual study plans based on the student's goals, academic ability, level of understanding, and schedule information, means for providing the generated individual study plans to the student's terminal, means for collecting viewer facial expression data in real time and analyzing their emotional state, and means for recommending learning content based on their emotional state. This incorporates real-time emotion analysis, enabling personalized learning support and content recommendations according to the student's emotional state.

[1494] "Learning progress" refers to the content and progress a student is currently learning.

[1495] "Level of understanding" is an indicator that shows how well a student understands the learning content.

[1496] "Analysis" is the act of carefully analyzing collected data to find meaning and patterns.

[1497] "Explanation" is content that provides supplementary explanations and answers to identified areas of lack of understanding.

[1498] "Devices" refers to electronic devices such as tablets, smartphones, and computers used by students.

[1499] "Exercises" are problems designed to deepen students' understanding.

[1500] A "study plan" is a detailed plan that outlines the schedule and content for students to study efficiently.

[1501] "Facial expression data" refers to information obtained from a viewer's facial expressions and is used to analyze their emotional state.

[1502] "Emotional state" refers to the emotions the viewer feels while learning (concentration, confusion, anxiety, etc.).

[1503] "Content recommendation" is the act of suggesting suitable learning content to maximize the viewer's learning effect.

[1504] A "server" refers to a computer system used to analyze and store data and perform various processing.

[1505] The system of the present invention records and analyzes students' learning progress and understanding in detail to provide personalized learning support. Furthermore, it combines a function to recognize users' emotions to optimize the learning experience. Specific embodiments for implementing the present invention are described below.

[1506] System Configuration

[1507] The system includes the following major components:

[1508] A means of recording student progress

[1509] A means of identifying understanding

[1510] A means of generating explanations for areas of incomprehension

[1511] Devices that provide commentary

[1512] A method for automatically generating exercises according to the level of understanding

[1513] A means of generating personalized learning plans

[1514] A means of collecting facial expression data in real time and analyzing emotional states

[1515] A means of recommending learning content based on emotional state

[1516] A means of receiving student questions and generating answers using large-scale language models

[1517] Hardware and Software

[1518] Hardware: Viewing devices such as tablets, smartphones, smart glasses, or head-mounted displays are used. These devices have built-in cameras that are used to collect facial expression data from viewers.

[1519] Software: Python, OpenCV, and Keras are used to analyze facial expressions in real time, and the data is sent to the server via a REST API.

[1520] Processing Flow

[1521] 1. Data Collection:

[1522] Using the device's built-in camera, facial expression data is collected in real time from the moment the student logs in while studying. In addition, learning content and answer data are also recorded. The facial expression data is analyzed using facial expression recognition technology (e.g., using OpenCV and Keras) to identify emotional states (e.g., concentration, confusion, anxiety).

[1523] 2. Data transmission:

[1524] Once a certain learning session is completed, the collected data is sent to a server, including the learning content, answer data, facial expression data, and emotion recognition results.

[1525] 3. Data analysis and learning support:

[1526] The server uses machine learning algorithms to analyze students' learning progress and comprehension, identifying areas where they lack understanding. It also takes into account their emotional state and uses AI technology to generate appropriate explanations. These explanations are provided in the form of text, illustrations, videos, and other formats, and can be reviewed by students.

[1527] 4. Exercises and Study Plan:

[1528] The server uses a generative AI model to automatically generate practice questions that address areas of incomplete understanding. The generated individualized learning plan is optimized by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data.

[1529] 5. Questions and feedback:

[1530] When students submit questions during their studies, the server receives the questions and generates optimal answers using information search and large-scale language models. It also provides appropriate feedback and emotional care through an emotion engine.

[1531] Specific examples

[1532] For example, imagine a user is watching an online math course. The camera will recognize any confused faces while they are watching and send that data to a server, which can then recommend more understandable videos or supplementary materials in real time.

[1533] Prompt Sentence Examples

[1534] "Generate an algorithm that recommends optimal learning content based on the user's facial expression data and viewing history while they are viewing. Facial expression data is transmitted in real time, and dynamically adjust learning content based on the user's emotional state (e.g., concentration, confusion, anxiety)."

[1535] In this way, the system of the present invention incorporates real-time emotion analysis and realizes personalized learning support according to the student's emotional state.

[1536] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1537] Step 1:

[1538] Data collection

[1539] The device's built-in camera collects students' facial expression data in real time. This happens automatically as soon as the learning session begins. Learning content and answer data are also recorded at the same time. The video data captured by the camera is first identified as a facial region, and then facial expression recognition technology (using OpenCV and Keras) is used to identify emotional states such as "concentration," "confusion," and "anxiety."

[1540] Input: Camera footage, learning content, answer data

[1541] Output: Facial expression data (emotional state)

[1542] Step 2:

[1543] Data transmission

[1544] The collected facial expression data and learning data are sent to the server after a certain learning session has ended or a specified time has passed. The data sent includes the learning content, answer data, facial expression data, and emotion recognition results. This data is sent to the server via a REST API.

[1545] Input: facial expression data, learning content, answer data

[1546] Output: Data sent to the server

[1547] Step 3:

[1548] Data analysis

[1549] The server analyzes the received facial expression data and learning data. It uses machine learning algorithms to identify the student's learning progress and level of understanding, and determines where they are lacking in understanding. At the same time, it takes into account their emotional state. For example, if a student is "confused," it performs additional analysis to find out why.

[1550] Input: Transmitted data (facial expression data, learning content, answer data)

[1551] Output: Learning progress, comprehension, and emotional state identification results

[1552] Step 4:

[1553] Explanation Generation

[1554] The server uses AI technology to generate detailed explanations for identified gaps in understanding. These explanations take into account the student's emotional state and are provided in the most appropriate format, such as text, illustrations, or video. The generated explanations are then retransmitted to the device.

[1555] Input: Areas of incomprehension, emotional state

[1556] Output: explanatory data

[1557] Step 5:

[1558] Exercise problem generation

[1559] The server uses a generative AI model to automatically generate exercises that address areas where students lack understanding, allowing students to tackle problems of the appropriate difficulty and format. The generated exercises are then provided back to the device.

[1560] Input: Area of ​​incomprehension

[1561] Output: Exercise data

[1562] Step 6:

[1563] Learning plan generation

[1564] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. This learning plan is optimized by incorporating the student's requests through an interactive format. The generated learning plan is provided to the student's device and serves as a guide for the student's daily learning.

[1565] Input: Student goals, academic achievement, comprehension, schedule information, emotional data

[1566] Output: Learning plan data

[1567] Step 7:

[1568] Questions and feedback

[1569] When a student submits a question during their studies, the server receives the question and generates the best answer using information retrieval and a large-scale language model. The answer is then provided to the device. The emotion engine also provides appropriate feedback and emotional support according to the student's emotional state.

[1570] Input: Student question

[1571] Output: Answers to questions, emotional care feedback

[1572] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1573] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1574] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1575] [Fourth embodiment]

[1576] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1577] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1578] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1579] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1580] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1581] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1582] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1583] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1584] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1585] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1586] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1587] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1588] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1589] The system of the present invention is designed to provide personalized learning support according to the learning progress and level of understanding of each student. Specific embodiments for carrying out the present invention will be described below.

[1590] Acquiring learning records

[1591] Subject: Terminal

[1592] First, when a student logs in to a tablet device, the device starts the student's study session. It records the learning materials (digital texts, video materials, etc.) used by the student and their learning progress (study time, answer status, etc.). It also uses the device's built-in camera and IoT sensors to collect data such as facial expressions and posture. This data is used to comprehensively understand the student's learning situation.

[1593] Analyzing learning data and identifying understanding

[1594] Subject: Server

[1595] When the learning data sent from the device is received by the server, the server analyzes the data and identifies the student's level of understanding. Specifically, it uses machine learning algorithms to analyze the student's answer patterns and learning progress and identify which parts of the learning content they have not understood.

[1596] AI-powered commentary

[1597] Subject: Server

[1598] The server then uses AI technology to generate explanations for the identified areas of incomprehension. The explanations are generated in a format that is easy for students to understand (e.g., text, diagrams, videos, etc.). These explanations are then provided to students' devices so that they can access them at any time.

[1599] Automatic generation of exercises

[1600] Subject: Server

[1601] Based on the students' level of understanding, the server automatically generates appropriate exercises using a generative AI model, allowing them to tackle problems that are tailored to their individual level of understanding. The generated exercises are provided to the students' devices, and they can further deepen their understanding by answering them.

[1602] Generate personalized learning plans

[1603] Subject: Server

[1604] The server then generates an individual learning plan based on the student's goals, academic ability, level of understanding, and schedule information. This learning plan is customized by interacting with the student, incorporating their wishes and goals. The generated learning plan is provided to the student's device, allowing the student to proceed with their studies accordingly.

[1605] RAG (Retriever-Augmented Generation) module

[1606] Subject: Server

[1607] Finally, if a student has a question during their study, the server receives the question and generates the best answer using a large-scale language model based on the information search results. This answer is then provided to the student's device, allowing the student to immediately resolve their doubts.

[1608] Specific examples

[1609] 1. User logs in

[1610] The device verifies the student's credentials and begins the learning session.

[1611] 2. Obtaining training data

[1612] The device collects learning progress information and facial expression data and sends them to the server.

[1613] 3. Providing analysis and commentary

[1614] The server analyzes the data to identify areas where students lack understanding, and then uses AI to generate explanations that are provided to the device.

[1615] 4. Exercises and Study Plans

[1616] The server automatically generates appropriate exercises and creates a learning plan based on the student's level of understanding, which is then provided to the terminal.

[1617] 5. Answering questions

[1618] When a student submits a question, the server searches for information, generates the most appropriate answer, and provides it to the device.

[1619] This allows students to study effectively at their own pace and at their own level of understanding.

[1620] The processing flow will be explained below.

[1621] Step 1:

[1622] The device checks the student's credentials

[1623] The device receives the student's login information (user ID and password) and authenticates it with the database. If authentication is successful, the device accesses the student's individual account and begins the learning session.

[1624] Step 2:

[1625] Your device records your study session

[1626] The device records the learning materials selected by the student (e.g., mathematics textbook, history video, etc.) and also records input data from the touch panel and keyboard (answer time, options, answer results, etc.).

[1627] Step 3:

[1628] The device collects data using cameras and IoT sensors

[1629] The device's built-in camera captures students' facial expressions, and IoT sensors monitor their posture and movements while they study, collecting data in real time.

[1630] Step 4:

[1631] The device sends the learning data to the server.

[1632] When the learning session ends or a certain period of time has passed, the device sends all collected data (teaching materials, answer data, facial expression data, etc.) to the server.

[1633] Step 5:

[1634] The server receives and analyzes the learning data.

[1635] The server receives the learning data sent from the device and analyzes it using machine learning algorithms, which identifies areas where the student does not understand and their learning trends.

[1636] Step 6:

[1637] The server generates an explanation for any missing information

[1638] The server uses AI technology to create detailed explanations of the identified areas of lack of understanding, which can be in the form of text, illustrations, videos, etc.

[1639] Step 7:

[1640] The server sends the commentary to the device.

[1641] The server generates explanations that are sent to students' devices so that they can access them at any time. The explanations are saved in individual student accounts.

[1642] Step 8:

[1643] The server automatically generates exercises

[1644] The server uses the generative AI model to automatically generate practice questions that address the identified gaps in understanding, adjusting the difficulty and format of the questions according to the student's level of understanding.

[1645] Step 9:

[1646] The server sends the exercises to the device.

[1647] The server sends the generated exercises to the students' devices so that they can work on them. The answers are also recorded sequentially.

[1648] Step 10:

[1649] The server generates an individualized learning plan.

[1650] The server generates an individualized learning plan based on the student's goals, academic ability, level of understanding, and schedule information. This plan is interactive and incorporates the student's requests.

[1651] Step 11:

[1652] The server sends the lesson plan to the device.

[1653] The server transmits the generated individual learning plan to the student's terminal, allowing the student to progress with their learning according to the daily learning plan.

[1654] Step 12:

[1655] The device will carry out learning according to the learning plan.

[1656] Students progress through their studies according to the study plan displayed on their device and access explanations and practice problems as needed.

[1657] Step 13:

[1658] The server receives the student's questions and generates answers.

[1659] When a student submits a question during their studies, the server receives the question and generates an answer using a large-scale language model based on the information search results.

[1660] Step 14:

[1661] The server generates a response and sends it to the device.

[1662] The server generates answers that are sent to students' devices to help them resolve their questions.

[1663] This will help each student to progress effectively in their studies.

[1664] Example 1

[1665] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1666] In recent years, there has been a demand for personalized learning support tailored to each learner's individual learning progress and level of understanding. However, conventional learning support systems have had problems in that they are insufficient in providing specific explanations and practice problems based on each learner's individual level of understanding, and in responding immediately to students' questions. This has made it difficult for students to study efficiently and effectively. In addition, the accuracy of collecting and analyzing learning data has been low, making it difficult to provide accurate learning support.

[1667] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1668] In this invention, the server includes means for recording the student's learning progress, means for analyzing the recorded learning progress to determine the student's level of understanding, means for determining the student's lack of understanding based on the determined level of understanding and generating explanations for the identified areas, means for providing the generated explanations to the student's terminal, means for automatically generating exercises according to the student's level of understanding, means for generating an individual learning plan based on the student's goals, academic ability, level of understanding, and schedule information, means for providing the generated individual learning plan to the student's terminal, means for receiving the student's questions and generating answers using natural language processing technology with reference to information search results, means for collecting the student's facial expression and posture data at the terminal, means for transmitting the learning data to the server in real time, means for analyzing the data using a machine learning algorithm, and means for generating explanations using artificial intelligence. This makes it possible to provide specific and effective support according to each student's individual learning progress.

[1669] "Learning progress" is an indicator that shows how much content a student has learned through learning activities and how much time they have spent.

[1670] "Recording means" refers to devices or software for storing students' learning progress, study time, answer status, and other related data in a database or cloud storage.

[1671] "Means for analysis" refers to software or systems that apply machine learning algorithms and statistical methods to assess students' learning progress and understanding using collected learning data.

[1672] "Level of understanding" is an indicator of how well a student has understood the learning content, including the accuracy and speed of answers and identification of areas of lack of understanding.

[1673] A "means for generating explanations" is a device or software that uses artificial intelligence or other automatic generation technology to create easy-to-understand explanations for students in the form of text, illustrations, videos, etc. for identified areas of lack of understanding.

[1674] "Practice questions" are questions that students answer to review what they have learned and check their understanding.

[1675] A "means for automatically generating exercises" is a device or software that uses artificial intelligence or generation technology to automatically generate exercises of appropriate difficulty and content based on the student's level of understanding.

[1676] A "study plan" is a plan that includes the learning content and schedule established for a student to achieve their goals.

[1677] A "means for generating an individual learning plan" is a device or software that creates and provides an individually optimized learning plan based on a student's goals, academic ability, level of understanding, schedule information, etc.

[1678] A "question receiving" means is an interface, communication device, or software that allows students to submit questions or concerns online while studying.

[1679] "Information search results" are the results obtained by searching databases and the Internet for information related to a student's question.

[1680] "Natural language processing technology" is a field of artificial intelligence and is a technology for understanding and generating human language.

[1681] "Facial expression and posture data" refers to information about facial expressions and body postures captured to understand students' learning status.

[1682] "Means for transmitting in real time" refers to communication equipment and communication protocols for transmitting learning data from the collection source to the server without delay.

[1683] A "machine learning algorithm" is a mathematical model and computational method for analyzing collected data and extracting patterns and features.

[1684] "Artificial intelligence" refers to systems and software that mimic human intelligence and perform learning and reasoning, and is used to generate explanations and questions.

[1685] The system of the present invention provides personalized learning support according to the student's learning progress and level of understanding. Specific embodiments for carrying out the present invention will be described below.

[1686] Hardware and Software Configuration

[1687] This system uses tablet devices, cameras, and IoT sensors to collect learning data, which is then analyzed on a server. Specifically, it uses the following hardware and software:

[1688] Hardware:

[1689] Tablet: A device used by students for learning.

[1690] Camera: Used to capture students' facial expressions.

[1691] IoT sensors: Used to collect students' posture data.

[1692] software:

[1693] Authentication system: Authentication system using OAuth or JWT.

[1694] Data collection software: Software for collecting learning progress, facial expressions, and posture data.

[1695] Data analysis software: TensorFlow and Scikit-learn are used to analyze data using machine learning algorithms.

[1696] Generative AI model: Generates explanations and practice problems using models such as OpenAI's GPT-4.

[1697] Natural language processing systems: Answering student questions using large-scale language models.

[1698] Processing flow and specific examples

[1699] Students log in and start a study session

[1700] When a student logs in, the device checks the authentication information and starts the learning session. For example, authentication is performed by entering a user ID and password. At this time, the device ensures security by using JWT authentication.

[1701] Collecting and sending learning data

[1702] During a learning session, the device records data such as the student's viewing status, answer status, and study time. It also records the student's facial expressions and posture data acquired through built-in cameras and IoT sensors. The acquired data is sent to a server in real time.

[1703] Analysis of training data

[1704] The server analyzes the data received from the devices using machine learning algorithms. Specifically, it uses the collected data to evaluate students' answer patterns and learning progress, and identifies their level of understanding. This analysis is performed using TensorFlow and Scikit-learn.

[1705] Generate explanations for areas of incomprehension

[1706] The server then uses a generative AI model to generate explanations for identified gaps in understanding. For example, OpenAI's GPT-4 is used to generate explanations in the form of text, diagrams, or videos. These explanations are then provided to students' devices for access.

[1707] Automatic generation of exercises

[1708] Based on the student's level of understanding, the server uses a generative AI model to generate appropriate exercises. The generated exercises are then provided to the student's device, and the student deepens their understanding by answering them.

[1709] Prompt Sentence Examples

[1710] Here are some examples of specific prompt sentences:

[1711] To generate explanations for areas of incomprehension:

[1712] "My students have a limited understanding of differential calculus in Chapter 5. I would like you to write an easy-to-understand explanation."

[1713] For generating practice questions:

[1714] "Generate basic differential calculus problems based on students' understanding."

[1715] Generate personalized learning plans

[1716] The server generates an individual learning plan based on the student's goals, schedule, and level of understanding, and provides it to the student's device. This plan incorporates the student's goals and optimizes daily learning.

[1717] Answering questions during study

[1718] When a student submits a question during their study, the RAG module on the server receives the question and uses natural language processing technology to generate the most appropriate answer, which is then provided to the device.

[1719] This detailed processing flow makes it possible to provide specific and effective support according to each student's learning progress.

[1720] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1721] Step 1: Students log in and start a study session

[1722] Subject: Terminal

[1723] When a student logs in to the device, they are prompted to enter their user ID and password. The entered information is sent to the authentication server, which performs authentication using JWT authentication. If authentication is successful, the device starts a learning session and displays the dashboard screen. The input is the user ID and password, and the output is the authentication result and the start of the learning session.

[1724] Specific behavior:

[1725] Input: User ID, Password

[1726] Data processing: JWT authentication

[1727] Output: Authentication results, dashboard screen

[1728] Step 2: Collect and send training data

[1729] Subject: Terminal

[1730] During a learning session, the device records data such as the student's viewing status, answer status, and study time. It also uses a built-in camera and IoT sensors to collect data on the student's facial expression and posture. This data is sent to a server in real time. The input is learning progress information, facial expression data, and posture data, and the output is data sent to the server.

[1731] Specific behavior:

[1732] Input: learning progress information, facial expression data, posture data

[1733] Data processing: Data recording, sensor information collection

[1734] Output: Send data to the server

[1735] Step 3: Analyze the training data

[1736] Subject: Server

[1737] The server receives the data sent from the device and analyzes it using machine learning algorithms. The analysis evaluates the student's answer patterns and learning progress, and identifies their level of understanding. TensorFlow and Scikit-learn are used for this analysis. The input is the learning data sent from the device, and the output is the level of understanding information obtained as a result of the analysis.

[1738] Specific behavior:

[1739] Input: Training data from the device

[1740] Data processing: Analysis using machine learning algorithms

[1741] Output: Comprehension information

[1742] Step 4: Generate explanations for areas of incomprehension

[1743] Subject: Server

[1744] The server uses a generative AI model to generate explanations for the identified areas of incomprehension. For example, by inputting a prompt into the GPT-4 model, it generates explanatory text, illustrations, and videos. The input is information about the area of ​​incomprehension and the prompt, and the output is the generated explanation.

[1745] Specific behavior:

[1746] Input: Information on areas of incomprehension, prompt text

[1747] Data processing: Generative AI model for generating explanations

[1748] Output: explanatory text, illustrations, videos

[1749] Step 5: Automatic generation of exercises

[1750] Subject: Server

[1751] The server generates appropriate exercises using a generative AI model based on the student's level of comprehension. The generated exercises are sent to the student's device. The input is comprehension information and a prompt, and the output is the generated exercise.

[1752] Specific behavior:

[1753] Input: Comprehension information, prompt

[1754] Data processing: Generative AI model for generating exercises

[1755] Output: Exercises

[1756] Step 6: Generate an individualized learning plan

[1757] Subject: Server

[1758] The server generates an individual learning plan based on the student's goals, academic ability, level of understanding, and schedule information. The generated learning plan is provided to the student's device. The input is the student's goals, academic ability, level of understanding, and schedule information, and the output is the individual learning plan.

[1759] Specific behavior:

[1760] Input: Goals, academic ability, level of understanding, schedule information

[1761] Data processing: Planning using learning plan generation algorithms

[1762] Output: Individualized Learning Plan

[1763] Step 7: Answering questions while studying

[1764] Subject: Server

[1765] The server receives questions sent by students during their studies, and generates optimal answers using a large-scale language model based on information search results. These answers are then provided to the students' devices. The input is the student's question, and the output is the generated answer.

[1766] Specific behavior:

[1767] Input: Student question

[1768] Data Processing: Natural Language Processing and Information Retrieval

[1769] Output: Best answer

[1770] (Application example 1)

[1771] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1772] Conventional factory worker education and training systems lacked personalized support tailored to each worker's learning progress and skill level, making it difficult to efficiently improve skills. They also lacked a means to respond to workers' questions in real time and provide specific feedback. This created the challenge of not maximizing the effectiveness of worker training.

[1773] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1774] In this invention, the server includes means for recording the worker's learning progress, means for analyzing the recorded learning progress to identify the worker's level of understanding, means for identifying areas where the worker does not understand based on the identified level of understanding and generating explanations for those areas, means for providing the generated explanations to the worker's terminal, means for automatically generating exercises according to the worker's level of understanding, means for generating an individual learning plan based on the worker's goals, skills, level of understanding, and schedule information, means for providing the generated individual learning plan to the worker's terminal, means for collecting and analyzing the worker's operation records and posture data, and means for receiving the worker's questions and generating answers using a generative AI model with reference to the information acquisition results. This makes it possible to provide personalized training plans based on each worker's level of understanding and respond to questions in real time.

[1775] "Workers" refers to the technicians and workers who operate machinery and equipment in factories and manufacturing sites. They are the recipients of training according to their individual skill levels and learning progress.

[1776] "Learning progress" refers to the progress a worker makes along a training or education program, specifically the degree to which knowledge and skills have been acquired.

[1777] "Level of understanding" refers to the standard for assessing how well a worker understands the learning content and operating procedures. It is an important indicator for providing feedback to improve identified areas of lack of understanding.

[1778] "Explanation" refers to content that provides additional information or explanations for areas where workers are struggling, often in the form of text, video, or AR guides.

[1779] "Devices" refer to electronic devices used by workers for operation and learning, including tablet devices and head-mounted displays.

[1780] "Exercises" refer to assignments and tasks provided to workers to allow them to try out what they have learned. They are automatically generated based on each worker's level of understanding.

[1781] "Individualized Learning Plan" refers to a customized education and training schedule based on a worker's goals, skills, level of understanding, and schedule information.

[1782] "Operation records" refer to the history of specific operations and tasks performed by workers. Recording these records will be useful for analysis.

[1783] "Posture data" refers to data on the body movements and posture of workers when operating a machine. It is acquired using sensors and cameras.

[1784] A "generative AI model" refers to an artificial intelligence algorithm or system that automatically generates information based on large datasets, which can be used to generate answers to human questions.

[1785] "Information acquisition results" refers to information collected from the internet or other databases in response to worker questions. This is part of the data used by the generative AI model.

[1786] This invention is a system for personalizing education and training for factory workers. This system records and analyzes the worker's learning progress and provides feedback, practice questions, and individual learning plans based on the worker's level of understanding.

[1787] Hardware and software used

[1788] The hardware used includes a tablet computer, a head-mounted display (HMD), a camera, and IoT sensors. This hardware is used to acquire the worker's operation records and posture data. The software used includes a learning data collection app, a data analysis server, an AI explanation system, an exercise problem generation system, and a RAG (Retriever-Augmented Generation) module.

[1789] Acquiring and analyzing training data

[1790] The device records the worker's learning progress. Specifically, it collects the worker's operations and work history, as well as posture and movement data acquired by cameras and IoT sensors. This data is sent to a server in real time.

[1791] The server analyzes the received data and identifies the worker's level of understanding. The analysis uses a machine learning algorithm to identify the areas where each worker lacks understanding. The server then generates an explanation corresponding to the area of ​​lack of understanding and provides it to the worker's device.

[1792] Providing AI explanations and practice questions

[1793] The server uses AI technology to generate easy-to-understand explanations for the identified areas of insufficient understanding. These explanations are created in the form of text, video, AR guides, etc. and are provided to the worker's device. Exercises are also automatically generated based on the worker's level of understanding. The exercises are set at an appropriate level of difficulty with the aim of improving the worker's skills.

[1794] Generate and provide individualized learning plans

[1795] The server generates an individual learning plan based on the worker's goals, skills, level of understanding, and schedule information. This allows the worker to learn at their own pace, enabling effective training. The generated learning plan is provided to the terminal, and the worker follows it to carry out training.

[1796] Real-time question response (RAG module)

[1797] If a worker has a question during learning, the server receives the question and generates an answer using the generative AI model based on the information acquisition results. This answer is provided to the worker's device in real time, instantly resolving the worker's question.

[1798] Examples of concrete examples and prompts

[1799] As an example, consider a worker wearing a head-mounted display and undergoing training to operate a robot. The worker's operation records and posture data are acquired by the terminal and sent to the server. After analyzing the data, the server generates video explanations for any operating procedures that the worker does not fully understand and provides them to the terminal. In addition, it automatically generates exercises related to the operation and presents them to the worker as assignments. If the worker asks a question during this process, the server uses the RAG module to instantly generate an answer and provides it to the terminal.

[1800] An example of a prompt sentence might be:

[1801] "We would like to develop a training system for robot operation in factories. Please come up with an application that provides individual feedback, practice assignments, and training plans based on the worker's skill level and progress."

[1802] As described above, the system of the present invention provides personalized education and training to workers, and supports efficient skill improvement.

[1803] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1804] Step 1:

[1805] The user logs in to the device. The device receives the user's authentication information as input and performs authentication to start a learning session. If authentication is successful, the device obtains the user's past learning data and current learning goals.

[1806] Step 2:

[1807] The device records the user's operations and learning progress in real time. Specifically, the device collects operation records and posture data using cameras and sensors installed on the tablet device or head-mounted display. This data is input and sent to a server.

[1808] Step 3:

[1809] The server analyzes the received data. It receives operation records and posture data as input, and uses a machine learning algorithm to evaluate the user's level of understanding. As a result of the analysis, it identifies and outputs the areas where the user does not understand or where they lack skills.

[1810] Step 4:

[1811] The server generates explanations for the identified areas of incomprehension, using AI technology to create explanations in the form of text, video, or AR guides. Using this as input, the server generates explanation data in the appropriate format and outputs it to be provided to the device.

[1812] Step 5:

[1813] The server automatically generates exercises based on the user's level of understanding and skills. It uses the analysis results as input and creates exercises using a generative AI model. This outputs appropriate exercise data aimed at improving the user's skills.

[1814] Step 6:

[1815] The server generates a personalized learning plan based on the user's goals, skills, level of understanding, and schedule information. It uses the user's profile data and analysis results as input to create an optimal learning schedule. This plan is output for delivery to the device.

[1816] Step 7:

[1817] When a user asks a question during learning, the device sends the question to the server. The device receives the question data as input, uses the RAG module to refer to the information acquisition results, and generates an answer using the generative AI model. The generated answer is sent to the device and provided to the user.

[1818] These steps explain how each task is performed and what input data is used to generate output data, enabling personalized training and real-time question answering.

[1819] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1820] The system of the present invention records and analyzes students' learning progress and level of understanding in detail to provide personalized learning support. This system is also combined with an emotion engine that recognizes the user's emotions. Specific embodiments for carrying out the present invention will be described below.

[1821] Acquiring training data and emotion recognition

[1822] Subject: Terminal

[1823] When a student logs in to a tablet device, the device begins a learning session. While recording data used during the learning process (such as learning material content and answer status), the device also collects the student's facial expression data using the built-in camera. This collected facial expression data is sent in real time to an emotion engine that recognizes the student's emotional state (concentration, confusion, anxiety, etc.).

[1824] Sending training data and emotion data

[1825] Subject: Terminal

[1826] When the learning session ends or after a certain period of time has passed, the device sends the collected learning data and emotion data to the server. This data includes the learning content, answer data, facial expression data, and emotion recognition results.

[1827] Analyzing data and identifying understanding

[1828] Subject: Server

[1829] The server receives the learning data and emotional data sent from the device and first analyzes the student's learning progress and level of understanding using a machine learning algorithm. From the analysis results, it identifies areas where understanding is lacking, and at the same time, taking into account the results of the emotional engine, it also understands the emotional state the student was in while studying.

[1830] Generate personalized commentary

[1831] Subject: Server

[1832] For identified gaps in understanding, the server uses AI technology to generate detailed explanations. The explanations take into account the student's emotional state and are created in context-appropriate formats such as text, diagrams, and videos. Furthermore, these explanations are provided to students' devices so they can review them repeatedly.

[1833] Automatic generation of exercises

[1834] Subject: Server

[1835] The server uses a generative AI model to automatically generate practice questions that address areas of incomplete understanding, allowing students to tackle questions of the appropriate difficulty and format. The practice questions are also provided to the device, and the answers are collected again.

[1836] Generate personalized learning plans

[1837] Subject: Server

[1838] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. This learning plan is optimized by incorporating the student's requests through an interactive format. The generated learning plan is provided to the student's device and serves as a guide for daily learning.

[1839] Question handling and emotional care

[1840] Subject: Server

[1841] When a student submits a question during learning, the server receives the question, performs an information search, and uses the results to generate the optimal answer using a large-scale language model. The answer is then provided to the device, instantly resolving the student's question. The system also uses an emotion engine to provide appropriate feedback and emotional support according to the student's emotional state during learning.

[1842] Specific examples

[1843] 1. User logs in

[1844] The device verifies the student's credentials and begins the learning session.

[1845] 2. Collecting training data and emotion data

[1846] The device collects learning progress information and facial expression data, which are then analyzed in real time using an emotion engine.

[1847] 3. Sending and Receiving Data

[1848] The terminal sends the collected data to the server, which receives it and analyzes it.

[1849] 4. Providing analysis and commentary

[1850] The server analyzes the data to identify areas of lack of understanding, and then uses AI to generate explanations that are provided to the device.

[1851] 5. Exercises and Study Plans

[1852] The server automatically generates appropriate practice questions, creates an individual learning plan, and provides it to the terminal.

[1853] 6. Answering questions and providing emotional support

[1854] The server receives the question, generates an answer using a large-scale language model based on the information search results, and provides it to the device. It also provides appropriate feedback depending on the user's emotional state.

[1855] In this way, this system provides more personalized learning support that also takes into account the student's emotional state.

[1856] The processing flow will be explained below.

[1857] Step 1:

[1858] The device checks the student's credentials

[1859] The device receives the student's login information (user ID and password) and authenticates it with the database. If authentication is successful, the device accesses the student's individual account and begins the learning session.

[1860] Step 2:

[1861] Your device records your study session

[1862] The device records the learning materials selected by the student (e.g., mathematics textbook, history video, etc.) and also records input data from the touch panel and keyboard (answer time, options, answer results, etc.).

[1863] Step 3:

[1864] The device collects data using cameras and IoT sensors

[1865] The device's built-in camera captures students' facial expressions, and IoT sensors monitor their posture and movements while they study, collecting data in real time.

[1866] Step 4:

[1867] The device sends facial expression data to the emotion engine to recognize emotions.

[1868] The device sends the collected facial expression data to the emotion engine, which then analyzes it to recognize the student's emotional state (e.g., concentration, confusion, anxiety, etc.) in real time.

[1869] Step 5:

[1870] The device sends the learning data and emotion data to the server.

[1871] When the learning session ends or after a certain period of time has passed, the device sends all collected data (teaching materials, answer data, facial expression data, emotional data, etc.) to the server.

[1872] Step 6:

[1873] The server receives and analyzes the learning data and emotion data.

[1874] The server receives the learning data and emotion data sent from the device, then analyzes the data using machine learning algorithms and an emotion engine to identify the student's learning progress and level of understanding.

[1875] Step 7:

[1876] The server generates an explanation for any missing information

[1877] The server uses AI technology to generate detailed explanations for the identified areas of lack of understanding. These explanations are provided to students in the form of text, illustrations, videos, etc.

[1878] Step 8:

[1879] Sends server-generated commentary to the device

[1880] The server generates explanations that are sent to students' devices so that they can access them at any time. The explanations are saved in individual student accounts.

[1881] Step 9:

[1882] The server automatically generates exercises

[1883] The server uses the generative AI model to automatically generate practice questions that address areas of incomprehension, adjusting the difficulty and format of the questions according to the student's level of understanding and emotional state.

[1884] Step 10:

[1885] The server sends the generated exercises to the device.

[1886] The server sends the generated exercises to the students' devices so that they can work on them. The answers are collected again and sent to the server.

[1887] Step 11:

[1888] The server generates an individualized learning plan.

[1889] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. This plan is interactive and incorporates the student's requests.

[1890] Step 12:

[1891] The server sends the generated lesson plan to the device.

[1892] The server transmits the generated individual learning plan to the student's terminal, allowing the student to proceed with their studies in accordance with it.

[1893] Step 13:

[1894] The device will carry out learning according to the learning plan.

[1895] Students progress through their studies according to the study plan displayed on their device and access explanations and practice problems as needed.

[1896] Step 14:

[1897] The server receives the student's questions and generates answers.

[1898] When students submit questions during their studies, the server receives the questions and generates the best answer using a large-scale language model based on the information search results.

[1899] Step 15:

[1900] The server generates a response and sends it to the device.

[1901] The server generates answers and sends them to students' devices to help them resolve their questions. It also uses an emotion engine to provide appropriate feedback and emotional support according to the student's emotional state during the course of their studies.

[1902] Example 2

[1903] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1904] Existing learning support systems are not only unable to record and analyze students' learning progress and comprehension, but also unable to provide personalized learning support that takes into account students' emotional state. Furthermore, they lack the ability to generate quick and appropriate answers to students' questions. This makes it difficult to provide learning plans tailored to individual students' needs.

[1905] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for recording the student's learning progress, means for analyzing the recorded learning progress to identify the student's level of understanding, means for analyzing the recorded facial expression data to identify the student's emotional state, means for identifying areas where the student does not understand based on the identified level of understanding and emotional state and generating explanations for those areas, means for automatically generating exercises according to the student's level of understanding, means for generating an individual learning plan based on the student's goals, academic ability, level of understanding, emotional state, and schedule information, means for providing the generated individual learning plan to the student's terminal, and means for receiving the student's question and generating an answer using a large-scale language model with reference to information search results. This enables individual learning support based on the student's learning progress, level of understanding, and emotional state.

[1906] "Learning progress" is data that shows how far a student has progressed with a particular assignment or material.

[1907] "Understanding" is data that indicates how well a student understands a particular material or concept.

[1908] "Emotional state" is data that indicates a student's psychological state (concentration, confusion, anxiety, etc.) while studying.

[1909] A "tablet device" is a portable computer device used by students when studying.

[1910] "Means of recording" refers to a system for collecting and storing data such as students' learning progress and emotional state.

[1911] "Means for analysis" are algorithms or devices that analyze collected data and assess learning progress, comprehension, and emotional state.

[1912] "Generative means" refers to algorithms or devices that create new explanations, exercises, study plans, etc. based on data.

[1913] The "means of provision" refers to a mechanism for sending and displaying the generated explanations, exercises, and study plans on students' devices.

[1914] "Exercises" are problems that students should solve to overcome specific areas of understanding.

[1915] A "learning plan" is a plan that designs the optimal learning process based on a student's goals and schedule.

[1916] "Means for receiving questions" refers to the system for accepting questions from students and interpreting the content of those questions.

[1917] The "means for generating answers" are large-scale language models and information retrieval systems that provide optimal answers to students' questions.

[1918] "Facial expression data" refers to data showing the facial expressions of students captured using the built-in camera.

[1919] An "emotion engine" is software or hardware that analyzes facial expression data to identify a student's emotional state in real time.

[1920] A "large-scale language model" is an advanced AI model for natural language processing that is trained using large amounts of text data.

[1921] The system of the present invention records and analyzes students' learning progress and level of understanding in detail to provide personalized learning support. The system is also combined with an emotion engine that recognizes the user's emotions. Specific embodiments for carrying out the present invention will be described below.

[1922] System Overview

[1923] The system consists of a tablet device, a server, a built-in camera, sensors, an emotion engine, and a generative AI model. The system's purpose is to provide individualized learning support based on a student's learning progress, level of understanding, and emotional state.

[1924] Student Login

[1925] Subject: User

[1926] The user logs in by entering their user ID and password on the tablet device. The device sends this authentication information to the authentication server and receives the authentication result.

[1927] Collection of training data and emotion data

[1928] Subject: Terminal

[1929] During a learning session, the device records the learning materials used by the student, their answers, and facial expression data captured by a built-in camera in real time. The facial expression data is then sent to an emotion engine to recognize their emotional state (concentration, confusion, anxiety, etc.).

[1930] Sending data

[1931] Subject: Terminal

[1932] When the learning session ends or after a certain period of time has passed, the device sends the collected learning data and emotion data to the server.

[1933] Analyzing data and identifying understanding

[1934] Subject: Server

[1935] The server receives the learning data and emotional data sent from the device and uses machine learning algorithms to analyze the student's learning progress and level of understanding.

[1936] Generate personalized commentary

[1937] Subject: Server

[1938] The server uses AI technology to generate detailed explanations for areas of learning deficiencies, taking into account the student's emotional state, and is created in the form of text, diagrams, videos, etc.

[1939] Automatic generation of exercises

[1940] Subject: Server

[1941] The server uses a generative AI model to automatically generate exercises that address areas of insufficient understanding, and these exercises are provided to the device.

[1942] Generate personalized learning plans

[1943] Subject: Server

[1944] The server generates an individualized learning plan by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data. The generated learning plan is provided to the device and serves as a daily learning guide.

[1945] Question handling and emotional care

[1946] Subject: Server

[1947] When students submit questions during learning, the server receives the questions and generates optimal answers using information retrieval and large-scale language models. It also provides appropriate feedback and emotional care through an emotion engine.

[1948] Specific examples

[1949] 1. User logs in

[1950] The device verifies the student's credentials and begins the learning session.

[1951] 2. Collecting training data and emotion data

[1952] The device collects learning progress information and facial expression data, which are then analyzed in real time using an emotion engine.

[1953] 3. Data submission and analysis

[1954] The terminal sends the collected data to the server, which receives it and analyzes it.

[1955] 4. Generating explanations based on analysis results

[1956] The server analyzes the data to identify areas of lack of understanding, and then uses AI to generate explanations that are provided to the device.

[1957] 5. Generating exercises and study plans

[1958] The server automatically generates appropriate practice questions, creates an individual learning plan, and provides it to the terminal.

[1959] 6. Answering questions and providing emotional support

[1960] The server receives the question, generates an answer using a large-scale language model based on the information search results, and provides it to the device. It also provides appropriate feedback depending on the user's emotional state.

[1961] Examples of prompts for generative AI models

[1962] 1. Prompt for student questions:

[1963] "A student is asking about XXX, with a specific point YYY. Based on this information, please suggest how to provide an answer that will satisfy the student."

[1964] 2. Prompt for generating explanation:

[1965] "Student demonstrates a lack of understanding about XXX. Student's emotional state is ZZZ. Please take this information into consideration and create a detailed explanation including text, illustrations, and video."

[1966] This system realizes more personalized learning support that also takes into account the student's emotional state, thereby improving students' learning efficiency and providing effective support tailored to their individual learning needs.

[1967] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1968] Step 1:

[1969] Input: The user enters their user ID and password on the tablet device.

[1970] Operation: The terminal sends the entered user ID and password to the authentication server.

[1971] Data calculation: The authentication server checks the received user ID and password against the information in the database.

[1972] Output: An authentication result is generated and sent to the device, which then starts a training session.

[1973] Step 2:

[1974] Input: Authenticated student start learning operation.

[1975] How it works: The device displays the learning content and collects facial expression data from the student using a built-in camera. At the same time, it also records how the learning material is used and how the student answers the questions.

[1976] Data calculation: The collected facial expression data is sent to the emotion engine in real time to analyze the emotional state.

[1977] Output: The analysis results of the emotion engine and learning data are generated and recorded.

[1978] Step 3:

[1979] Input: End of study session or passage of a certain period of time.

[1980] How it works: The device packages the learning and emotion data collected during the session.

[1981] Data calculation: Formats the training data and emotion data for transmission to the server.

[1982] Output: The packaged data is sent to the server.

[1983] Step 4:

[1984] Input: Training data and emotion data received by the server.

[1985] How it works: The server stores the training data and emotion data in a database.

[1986] Data calculation: Using machine learning algorithms to analyze learning progress and comprehension, and an emotion engine to analyze emotional states.

[1987] Output: Analysis results are generated regarding learning progress, comprehension, gaps, and emotional state.

[1988] Step 5:

[1989] Input: Identified learning gaps and emotional state.

[1990] How it works: Based on the identified gaps in understanding, the server passes prompts to the generative AI model to generate explanations.

[1991] Data calculation: A generative AI model generates an explanation based on the prompt.

[1992] Output: Explanatory content (text, illustrations, video) is generated and sent to the device.

[1993] Step 6:

[1994] Input: Learning gaps and understanding gaps identified by the server.

[1995] How it works: The server uses a generative AI model to create prompts that generate exercises.

[1996] Data computation: Generative AI models generate exercises of appropriate difficulty and format.

[1997] Output: Exercises are generated and sent to the terminal.

[1998] Step 7:

[1999] Input: Learning data, comprehension, emotional state, goals, and schedule information collected by the server.

[2000] How it works: The server uses the collected data to generate a personalized lesson plan.

[2001] Data computation: Using learning algorithms to optimize personalized learning plans.

[2002] Output: The generated lesson plan is sent to the terminal and provided to the student.

[2003] Step 8:

[2004] Input: Questions asked by students.

[2005] Operation: The device sends the query to the server.

[2006] Data Computation: The server analyzes the question and generates an answer using information retrieval and large-scale language models.

[2007] Output: The generated answer is sent to the device and provided to the student, and emotional feedback is also provided using the emotion engine.

[2008] (Application example 2)

[2009] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2010] Conventional learning support systems only record and analyze students' learning progress and comprehension, but are unable to incorporate real-time emotional analysis. As a result, it is difficult to immediately grasp the frustration or confusion students feel during learning and provide appropriate support. Furthermore, content recommendations based on the viewer's emotional state are not performed, preventing the maximization of learning effectiveness. The present invention aims to solve these problems and provide more personalized learning support.

[2011] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording the student's learning progress, means for analyzing the recorded learning progress to identify the student's level of understanding, means for identifying areas where the student does not understand based on the identified level of understanding and generating explanations related to these, means for providing the generated explanations to the student's terminal, means for automatically generating exercises according to the student's level of understanding, means for generating individual study plans based on the student's goals, academic ability, level of understanding, and schedule information, means for providing the generated individual study plans to the student's terminal, means for collecting viewer facial expression data in real time and analyzing their emotional state, and means for recommending learning content based on their emotional state. This incorporates real-time emotion analysis, enabling personalized learning support and content recommendations according to the student's emotional state.

[2012] "Learning progress" refers to the content and progress a student is currently learning.

[2013] "Level of understanding" is an indicator that shows how well a student understands the learning content.

[2014] "Analysis" is the act of carefully analyzing collected data to find meaning and patterns.

[2015] "Explanation" is content that provides supplementary explanations and answers to identified areas of lack of understanding.

[2016] "Devices" refers to electronic devices such as tablets, smartphones, and computers used by students.

[2017] "Exercises" are problems designed to deepen students' understanding.

[2018] A "study plan" is a detailed plan that outlines the schedule and content for students to study efficiently.

[2019] "Facial expression data" refers to information obtained from a viewer's facial expressions and is used to analyze their emotional state.

[2020] "Emotional state" refers to the emotions the viewer feels while learning (concentration, confusion, anxiety, etc.).

[2021] "Content recommendation" is the act of suggesting suitable learning content to maximize the viewer's learning effect.

[2022] A "server" refers to a computer system used to analyze and store data and perform various processing.

[2023] The system of the present invention records and analyzes students' learning progress and understanding in detail to provide personalized learning support. Furthermore, it combines a function to recognize users' emotions to optimize the learning experience. Specific embodiments for implementing the present invention are described below.

[2024] System Configuration

[2025] The system includes the following major components:

[2026] A means of recording student progress

[2027] A means of identifying understanding

[2028] A means of generating explanations for areas of incomprehension

[2029] Devices that provide commentary

[2030] A method for automatically generating exercises according to the level of understanding

[2031] A means of generating personalized learning plans

[2032] A means of collecting facial expression data in real time and analyzing emotional states

[2033] A means of recommending learning content based on emotional state

[2034] A means of receiving student questions and generating answers using large-scale language models

[2035] Hardware and Software

[2036] Hardware: Viewing devices such as tablets, smartphones, smart glasses, or head-mounted displays are used. These devices have built-in cameras that are used to collect facial expression data from viewers.

[2037] Software: Python, OpenCV, and Keras are used to analyze facial expressions in real time, and the data is sent to the server via a REST API.

[2038] Processing Flow

[2039] 1. Data Collection:

[2040] Using the device's built-in camera, facial expression data is collected in real time from the moment the student logs in while studying. In addition, learning content and answer data are also recorded. The facial expression data is analyzed using facial expression recognition technology (e.g., using OpenCV and Keras) to identify emotional states (e.g., concentration, confusion, anxiety).

[2041] 2. Data transmission:

[2042] Once a certain learning session is completed, the collected data is sent to a server, including the learning content, answer data, facial expression data, and emotion recognition results.

[2043] 3. Data analysis and learning support:

[2044] The server uses machine learning algorithms to analyze students' learning progress and comprehension, identifying areas where they lack understanding. It also takes into account their emotional state and uses AI technology to generate appropriate explanations. These explanations are provided in the form of text, illustrations, videos, and other formats, and can be reviewed by students.

[2045] 4. Exercises and Study Plan:

[2046] The server uses a generative AI model to automatically generate practice questions that address areas of incomplete understanding. The generated individualized learning plan is optimized by comprehensively considering the student's goals, academic ability, level of understanding, schedule information, and emotional data.

[2047] 5. Questions and feedback:

[2048] When students submit questions during their studies, the server receives the ...

Claims

1. a means of recording student learning progress; a means for analyzing the recorded learning progress to identify the student's level of understanding; A means for identifying areas where the student does not understand based on the identified level of understanding and generating commentary thereon; A means for providing the generated commentary to the student's device; A means for automatically generating exercises according to the level of understanding; a means for generating individualized learning plans based on student goals, academic ability, comprehension, and schedule information; A means for providing the generated individual learning plan to the student's device; A system including:

2. The system of claim 1, which records learning progress using a tablet device, a camera, and an IoT sensor.

3. The system of claim 1 , further comprising means for receiving a student question and generating an answer using a large-scale language model with reference to information search results.

Images