System

Abstract

A system is provided.SOLUTION: First camera means for recording lesson contents, second camera means for recording reactions of students, server means for receiving and storing video data from the first camera means and the second camera means, and means for analyzing the video data stored by the server means; An artificial intelligence model unit that evaluates an understanding level of a student, a task generation unit that generates an individual task based on the understanding level evaluated by the artificial intelligence model unit, a transmission unit that transmits the task generated by the task generation unit to a student terminal unit, a display unit that displays the task received by the student terminal unit and causes the student to execute the task, a transmission unit that transmits a result of the task executed by the student terminal unit to a server unit, and a re-evaluation unit that re-evaluates the task based on the result received by the server unit, the system includes a report generation means for generating a comprehension degree report of the whole class, and a transmission means for transmitting the comprehension degree report generated by the report generation means to a teacher terminal means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional group education systems have the problem of making it difficult to provide individualized instruction tailored to each student's level of understanding and learning progress. In particular, grasping students' reactions in real time during class and providing individualized learning support based on that understanding places a heavy burden on teachers. The purpose of this invention is to solve these problems and provide effective education tailored to each student's individual level of understanding. [Means for solving the problem]

[0005] The present invention provides a first camera means for recording lesson content and a second camera means for recording student responses. It also provides a server means for receiving and storing video data from these camera means, and an artificial intelligence model means for analyzing the stored video data and evaluating students' levels of understanding. It also includes assignment generation means for generating individual assignments based on the evaluated levels of understanding, and a transmission means for transmitting the assignments generated by the assignment generation means to student terminal means.

[0006] The student terminal means has a display means for displaying the received assignments and allowing the students to carry out the assignments. It also includes a transmission means for transmitting the results of the assignments carried out by the student terminal means to the server means, and a report generation means for re-evaluating based on the results and generating a comprehension report for the entire class. By adding a transmission means for transmitting the comprehension report to the teacher terminal means, a system is provided that allows teachers to more effectively plan lessons.

[0007] "Camera means" means a device for acquiring video data and transmitting and storing that data.

[0008] The "server means" is a device or system that receives and stores the video data transmitted from the camera means, and further performs processing to analyze the data.

[0009] The "artificial intelligence model means" is a device or system that includes algorithms and programs that analyze data stored in the server means and evaluate the students' level of understanding.

[0010] The "task generation means" is a device or program that generates individual learning tasks for students based on the results of the assessment of the level of understanding by the artificial intelligence model means.

[0011] "Transmission means" refers to a device or program that has the function of transmitting the generated assignments and analysis results to other devices or terminals.

[0012] "Student terminal means" refers to a terminal used by a student, and is a device or system that receives and displays assignments and provides an interface through which students can complete their assignments.

[0013] The "display means" is a device or program having a function for visually presenting the assignment received by the student terminal means to the student.

[0014] The "report generation means" is a device or program that has the function of summarizing the level of understanding of the entire class based on the results of the assignments and presenting this to the teacher in the form of a report.

[0015] The "teacher terminal means" is a terminal used by a teacher, and is a device or system that provides an interface for receiving and displaying a comprehension report. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention relates to an educational support system that can effectively record lesson content and grasp students' understanding in real time. This system is composed of multiple cameras, a server, an artificial intelligence model, an assignment generation unit, a transmission unit, student terminals, a display unit, a report generation unit, and a teacher terminal.

[0038] System program processing and explanation

[0039] Record of lesson content

[0040] At the start of a lesson, the server checks whether the first camera means (Camera 1) that records the lesson content and the second camera means (Camera 2) that records the students' reactions are running normally. When the lesson starts, Camera 1 records the teacher's lesson content in real time and sends the video data to the server. At the same time, Camera 2 records the expressions and actions of all the students in real time and also sends this video data to the server.

[0041] Video data storage and preparation

[0042] The server stores the video data of the lesson content and students' reactions received from the camera means in real time. This stored data will be used for later analysis, so it also checks the video quality and frame rate.

[0043] Data analysis and understanding assessment

[0044] The AI ​​model analyzes the video data stored on the server. A facial recognition algorithm is used to evaluate the student's level of understanding based on their facial expressions and behavior. The evaluation results are saved as a score and listed for each student.

[0045] Creating and submitting individual assignments

[0046] The assignment generation means generates preparation, review, and practice assignments suitable for each student based on the results of the assessment of comprehension by the artificial intelligence model means. The generated assignments are sent to each student's terminal via the transmission means.

[0047] Assignment and feedback of results

[0048] The student's device displays the received assignments and provides an interface for the student to complete them. After the student completes the assignment, the results are sent from the device to the server. The server reassessssments based on the received assignment results and generates a report on the overall class comprehension. This report is used by the teacher to optimize lesson plans.

[0049] Generate reports and provide feedback to teachers

[0050] The report generation tool compiles a summary of the class's overall understanding based on the results of the assignments and creates a detailed report. This report visualizes each student's strengths and weaknesses. The teacher's device receives and displays this report, allowing the teacher to grasp the overall situation in the class and plan appropriate measures for the next lesson.

[0051] Specific examples

[0052] 1. Conducting and recording lessons

[0053] The user (teacher) starts a math lesson. Camera 1 records the teacher's lesson content, and Camera 2 records the students' reactions. For example, when the teacher is explaining the basics of equations, Student A looks confused.

[0054] 2. Data analysis and understanding assessment

[0055] The server stores this video data, and the artificial intelligence model means judges the low level of understanding from the facial expression of Student A. Based on this, an assignment is created for Student A to review basic equation problems.

[0056] 3. Submitting and completing individual assignments

[0057] The assignment is sent to and displayed on Student A's device. Student A answers the assignment and sends the results to the server.

[0058] 4. Feedback on the results and reflecting them in lesson plans

[0059] The server receives Student A's answer and re-evaluates it. A report of the class's understanding is generated and sent to the teacher's terminal. Based on this, the teacher decides to allocate time to the basics of equations in the next lesson.

[0060] In this way, the present invention utilizes AI technology to reduce the burden on teachers while providing learning support that is tailored to each student.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] At the start of a lesson, the server checks whether the first camera means (camera 1) for recording the lesson content and the second camera means (camera 2) for recording the students' reactions are running normally.

[0064] Step 2:

[0065] Camera 1 records the teacher's lesson content in real time and transmits the video data in streaming format to the server.

[0066] Step 3:

[0067] Camera 2 records the facial expressions and behavior of the entire classroom and each student in real time, and transmits the video data in streaming format to a server.

[0068] Step 4:

[0069] The server stores the received video data in real time and monitors the video quality and frame rate.

[0070] Step 5:

[0071] The AI ​​model analyzes the video data of the lesson content and students' reactions stored on the server. Specifically, it uses a facial recognition algorithm to identify students' facial expressions and behaviors and score each student's level of understanding.

[0072] Step 6:

[0073] The server stores the comprehension scores generated by the artificial intelligence model means and lists the learning needs of each student.

[0074] Step 7:

[0075] The assignment generation means generates optimal preparation, review, and practice assignments for each student based on the stored comprehension scores.

[0076] Step 8:

[0077] The transmission means transmits the generated individual assignments to the terminals of the students.

[0078] Step 9:

[0079] The student's device displays the received assignment and provides an interface through which the student can complete the assignment.

[0080] Step 10:

[0081] Students answer the assignments and send the results data from their devices to the server.

[0082] Step 11:

[0083] The server reassessed the students' understanding based on the received assignment results and generated a comprehension report for the entire class.

[0084] Step 12:

[0085] The report generating means transmits the generated comprehension report to the teacher's terminal.

[0086] Step 13:

[0087] The teacher's device will display a comprehension report, allowing the teacher to understand the learning status of the entire class.

[0088] Step 14:

[0089] Based on the comprehension reports, teachers decide what content and focus to cover in the next lesson.

[0090] Through this series of processing steps, the present invention realizes a system that provides learning support suited to each individual student and more effectively supports teachers' lesson plans.

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] Conventional educational support systems have had difficulty accurately assessing each student's level of understanding in real time and providing individualized learning support. They also have limitations in comprehensively recording lesson content and effectively utilizing that data. This makes it difficult for teachers to grasp students' levels of understanding, leading to problems with efficient instruction.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes a first recording means for recording the content of the lesson, a second recording means for recording the responses of students, a data storage means for receiving and storing video data from the first recording means and the second recording means, an analysis means for analyzing the video data stored by the data storage means and evaluating the students' levels of comprehension, an assignment creation means for generating individual assignments based on the levels of comprehension evaluated by the analysis means, a transmission means for transmitting the assignments generated by the assignment creation means to student terminals, a display means for displaying the assignments received by the student terminals and having the students complete them, a transmission means for transmitting the results of the assignments completed by the student terminals to the data storage means, a report creation means for re-evaluating the results received by the data storage means and generating a report on the level of comprehension of the entire class, and a transmission means for transmitting the comprehension report generated by the report creation means to a teacher's terminal. This enables comprehensive recording of the content of the lesson and real-time evaluation and feedback of students' levels of comprehension.

[0096] The "primary recording means for recording the content of the class" is a device that records the lecture content and explanations of the instructor in charge of the class as video and audio.

[0097] The "second recording means for recording student responses" is a device that records students' facial expressions and behavior during class in real time as video.

[0098] The "data storage means" is a system or device for receiving the video data transmitted from the first recording means and the second recording means and storing the data for a long period of time.

[0099] The "analysis means" is a system that includes algorithms and software for analyzing the stored video data and assessing the student's level of understanding.

[0100] The "task creation means" is a system or device for automatically creating individual learning tasks based on the student's level of understanding evaluated by the analysis means.

[0101] The "transmission means" is a system or device for transmitting the generated assignments and comprehension reports to a specified terminal.

[0102] A "student device" is a device such as a computer or tablet used by a student that provides an interface for displaying and completing received assignments.

[0103] The "display means" is software or a system that displays the assignment received on the student terminal and provides an interface for the student to answer the assignment.

[0104] The "report creation means" is a system that compiles the level of understanding of the entire class based on the results of assignments sent from student devices and generates a comprehension report for the teacher.

[0105] A "teacher's terminal" is a device such as a computer or tablet used by a teacher to receive and display comprehension reports.

[0106] The present invention relates to an educational support system that can effectively record lesson content and grasp students' understanding in real time. This system is composed of multiple recording means, data storage means, analysis means, assignment creation means, transmission means, student terminals, display means, report creation means, and teacher terminals.

[0107] Hardware and Software Configuration

[0108] Specific hardware used includes camera devices (first and second recording means), servers (data storage means and analysis means), student tablets or PCs (student terminals), and faculty tablets or PCs (faculty terminals).Software includes video analysis algorithms (face recognition algorithms and facial expression analysis algorithms), AI models, assignment generation algorithms, and database management systems.

[0109] Data collection and storage

[0110] At the start of class, the server checks whether the first recording means (Camera 1), which records the teacher's lecture content, and the second recording means (Camera 2), which records the students' reactions, are operating normally. When class begins, Camera 1 records the teacher's lecture in real time and sends the data to the server. Camera 2 simultaneously records the expressions and movements of all the students and also sends this data to the server. The data storage means saves the received video data to a highly durable storage device.

[0111] Data analysis and understanding assessment

[0112] The saved video data is processed by an analysis means. The analysis means uses a facial recognition algorithm and an expression analysis algorithm to evaluate the student's level of comprehension based on their facial expressions and movements. This evaluation is quantified and recorded as a comprehension score for each student. The analysis results are saved in a database and used for subsequent processing.

[0113] Creating and submitting assignments

[0114] The assignment creation means generates individual preparation, review, and practice assignments based on the comprehension score obtained from the analysis means. The generated assignments are sent to each student's student device via the transmission means. For example, mathematics practice problems created using the generative AI model are distributed to student devices.

[0115] Assignment and feedback of results

[0116] The student terminals display the received assignments and provide an interface for students to answer. When students complete the assignments, their answers are automatically sent to the server. The server reevaluates the received answers and creates a report based on the overall class understanding. This report is generated by the report creation means and sent to the teacher terminal.

[0117] Specific examples

[0118] 1. Lesson Conduct and Recording:

[0119] A user (teacher) starts a math class at 9:00 a.m. Camera 1 records the teacher's lecture, and Camera 2 records the students' reactions. For example, when the teacher is explaining the basics of equations, Student A looks confused.

[0120] 2. Data analysis and comprehension assessment:

[0121] The server stores this video data, and the analysis means determines whether Student A has a low level of understanding based on his / her facial expressions, and records the result as a score. Data is generated to provide Student A with review assignments.

[0122] 3. Submit and complete the assignment:

[0123] The assignment creation means generates a review assignment suitable for Student A and sends it to Student A's student terminal. Student A's terminal displays the review math problems.

[0124] 4. Feedback on results and reflection on next class:

[0125] The server receives Student A's answers and re-evaluates them. A comprehension report is sent to the teacher's terminal, and the teacher uses it to plan the next lesson.

[0126] Prompt Sentence Examples

[0127] Below are some examples of prompts to give instructions to an AI generative model.

[0128] "To determine whether students are understanding during class, consider the following scenario: Camera 1 records a teacher teaching the basics of equations in a math class, while simultaneously recording the students' reactions with Camera 2. If it is determined that Student A has a confused expression, generate appropriate review assignments for Student A based on the results of the facial expression analysis."

[0129] In this way, the present invention utilizes AI technology to reduce the burden on teachers while providing learning support that is tailored to each student.

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Step 1: Start the lesson and check that the equipment is up and running

[0132] At the designated time, the server checks whether the lesson is about to begin. Here, it checks whether the first recording means (camera 1) and the second recording means (camera 2) are operating normally. Specifically, the server sends a PING command to each camera and monitors whether there is a response. If a normal response is confirmed, it determines that the lesson is ready to begin.

[0133] Input: lesson start time, camera status

[0134] Output: Confirmation that the device has started up normally

[0135] Step 2: Record lesson content and student reactions

[0136] When the class begins, the server has Camera 1 record the teacher's lecture in real time and transmits the video data to the server. At the same time, Camera 2 records the reactions of all students and transmits this to the server. The video data being recorded is received in real time and saved by the data saving means.

[0137] Input: Teacher lecture video, student reaction video

[0138] Output: Save real-time video data to the server

[0139] Step 3: Save the video data and check its quality

[0140] The server stores the received video data in a data storage device. At the same time, it checks the quality of the video data. Specifically, it automatically checks whether there are any missing frames and whether the audio data is synchronized. The results of this check are also saved and used to ensure the quality of the data.

[0141] Input: Received video data

[0142] Output: Saved video data, quality check results

[0143] Step 4: Analyze data and assess comprehension

[0144] The saved video data is processed by an analytical device. Specifically, facial recognition algorithms and facial expression analysis algorithms are used to quantify the student's level of comprehension based on their facial expressions and movements. The results of this analysis are recorded as a comprehension score for each student.

[0145] Input: Saved video data

[0146] Output: Comprehension score

[0147] Step 5: Generate assignments and send them to students

[0148] The assignment creation means generates individual assignments based on the comprehension scores. The generated assignments are sent to each student's student device via the transmission means. Based on the generation AI model, more basic questions are provided to students with low scores, and applied questions are provided to students with high scores.

[0149] Input: Comprehension score

[0150] Output: Individual assignment, Submit assignment

[0151] Step 6: Complete and submit the assignment

[0152] The student terminals display the received assignments and provide an interface for students to answer. When students complete the assignments, the answers are automatically sent to the server.

[0153] Input: Received assignment

[0154] Output: Send answer results

[0155] Step 7: Reassess results and provide feedback to teachers

[0156] The server reevaluates the received answers and reassess the level of understanding. Based on the reevaluation results, a report on the level of understanding of the entire class is generated. The report generation means creates this report and sends it to the teacher's terminal.

[0157] Input: Answer result

[0158] Output: Reassessed comprehension, Generate comprehension report

[0159] Step 8: Reflect on the next lesson plan

[0160] The teacher's device displays the received comprehension report and uses it as reference for planning the next lesson. Teachers can identify each student's strengths and weaknesses and optimize their lesson plans.

[0161] Input: Comprehension Report

[0162] Output: Optimized lesson plans

[0163] (Application example 1)

[0164] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0165] Modern education and retail operations require support tailored to diverse individual needs and levels of understanding. In educational environments, it is particularly important to assess each student's level of understanding in real time and provide appropriate learning content based on the results. Meanwhile, in brick-and-mortar stores, analyzing customer behavior and facial expressions and making appropriate product recommendations can be expected to boost purchasing motivation. Current systems face the challenge of being unable to efficiently respond to such diverse demands.

[0166] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0167] In this invention, the server includes a first image acquisition means for recording lesson content, a second image acquisition means for recording student responses, a data storage means for receiving and storing video data from the first image acquisition means and the second image acquisition means, an intelligence assessment means for analyzing the video data stored by the data storage means and evaluating the student's level of comprehension, a learning content generation means for generating individual learning content based on the level of comprehension assessed by the intelligence assessment means, a data transmission means for transmitting the learning content generated by the learning content generation means to a receiving device, a display device for displaying the learning content received by the receiving device and having the student execute it, a data transmission means for transmitting the results of the learning content executed by the receiving device to the data storage means, a report generation means for re-evaluating the results received by the data storage means and generating an overall comprehension report, and a data transmission means for transmitting the comprehension report generated by the report generation means to a reporting device. This enables support to be provided according to individual needs and levels of comprehension in educational and retail environments.

[0168] "Class content" is a term that refers to the content and themes that teachers teach to students in educational activities.

[0169] "Student reactions" is a term that refers to various behaviors that students show during class, such as facial expressions, movements, and actions, that indicate their understanding and interest in what they are learning.

[0170] "Image capture means" is a term that refers to an apparatus or system for capturing video data using a device such as a camera.

[0171] "Data storage means" is a term that refers to a device or system for temporarily or permanently storing acquired video data, analysis results, etc.

[0172] "Intelligence assessment means" is a term used to refer to a device or system that uses artificial intelligence to analyze data obtained and assess an individual's level of understanding and emotions.

[0173] "Learning content generation means" is a term that refers to a device or system for generating learning content optimized for each student based on the evaluation results from the intelligence evaluation means.

[0174] "Data transmission means" is a term that refers to an apparatus or system for transmitting generated learning content, analysis results, reports, etc. to a specific terminal or device.

[0175] "Receiving device" is a term that refers to a device or system for receiving data transmitted from a data transmitting means.

[0176] "Display device" is a term that refers to a device or system that displays data and learning content received by a receiving device so that the user can visually confirm it.

[0177] "Report generator" is a term that refers to a device or system for generating a comprehensive report based on intelligence assessment measures and learning results.

[0178] "Reporting device" is a term that refers to a device or system that ultimately reports generated reports or other information to a user.

[0179] To put this invention into practice, it is necessary to build systems for both education and retail use, and to be able to meet the needs of each environment. Specific examples of such systems are shown below.

[0180] System Overview

[0181] The system mainly includes the following components:

[0182] First image capture method (cameras that record lesson content and customer behavior)

[0183] Secondary image capture means (cameras that record students' reactions and customers' facial expressions)

[0184] Data storage means (server)

[0185] Intelligence evaluation method (analysis system using artificial intelligence model)

[0186] Learning content generation means (a system that generates individual tasks or product proposals)

[0187] Data transmission method (transmission server)

[0188] Receiving device (student device, store display, smartphone)

[0189] Display devices (terminals and displays)

[0190] Report generation means (systems that generate class-wide or customer behavior reports)

[0191] Reporting device (teacher terminal or store management system)

[0192] Hardware and Software

[0193] Hardware:

[0194] Camera (e.g. USB camera)

[0195] Server (data storage and analysis)

[0196] Smartphones, tablets, store displays, robots

[0197] software:

[0198] TensorFlow (a deep learning model for analyzing customer facial expressions)

[0199] OpenCV (acquisition and display of camera images)

[0200] REST API (data sending and receiving)

[0201] Data processing and calculation

[0202] The server processes and calculates the data in the following steps:

[0203] 1. Data Acquisition:

[0204] The first image acquisition means records the contents of the lesson and the behavior of the customer, and transmits the recorded images to the data storage means in real time.

[0205] A second image capturing means records the reactions of the students and the facial expressions of the customers and transmits them to the data storage means as well.

[0206] 2. Analysis and Evaluation:

[0207] The video data stored in the data storage means is analyzed using an intelligent evaluation means (e.g., an artificial intelligence model equipped with a face recognition algorithm) to evaluate the students' understanding and the customers' emotions.

[0208] 3. Data transmission:

[0209] Based on the results of the evaluation by the intelligence evaluation means, the learning content generation means generates appropriate assignments and product suggestions and transmits them to the receiving device.

[0210] 4. Interaction:

[0211] The assignments and product proposals received by the receiving device are displayed on the display device and are carried out by the students or customers.

[0212] 5. Feedback of results:

[0213] The results of the execution on the display device are again transmitted to the data storage means by the data transmission means, and the report generation means generates an understanding level report and a customer behavior report.

[0214] Specific examples

[0215] 1. Educational environment:

[0216] Consider a scenario in which a teacher begins a lesson. A first image acquisition means records the content of the lesson, and a second image acquisition means records the students' responses. An AI model analyzes the students' facial expressions, and if a particular student is struggling to understand, it sends that student an individual assignment. This assignment is displayed on the student's device, and the student answers it. The answers are sent to a server, and the teacher receives a report of the class's understanding on his or her device.

[0217] 2. Store environment:

[0218] Consider a scenario in which a customer enters a store. A first image capture means records the customer's behavior, and a second image capture means analyzes the customer's facial expressions. When the server detects the customer's expressions of interest or confusion, it generates an appropriate product suggestion and sends it to an in-store display or the customer's smartphone. An in-store robot interacts with the customer based on the suggestion (e.g., "Would you like this product?"). Ultimately, store staff optimize their response based on the report generated by the server.

[0219] Prompt Sentence Examples

[0220] For example, if a customer appears unsure, the following prompts can be fed into the generative AI model:

[0221] The customer's expression shows uncertainty. Which of the following products do you predict will most interest the customer?

[0222] This will improve the customer experience and streamline store operations.

[0223] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0224] Step 1:

[0225] Data Acquisition

[0226] Input: Real-time video data acquired by the first image acquisition means (camera 1) and the second image acquisition means (camera 2)

[0227] Output: Video data of lesson content, customer behavior, student reactions, and customer expressions sent to the server

[0228] Specific operation: The device activates Camera 1 and records the entire classroom or store. It then activates Camera 2 and records the facial expressions and behavior of individual students or customers. The captured video data is sent to the server in real time.

[0229] Step 2:

[0230] Data storage

[0231] Input: Real-time video data sent to the server

[0232] Output: Video data stored in a data storage device

[0233] Specific operation: The server stores the received video data in a data storage means (e.g., a database), and also checks the video quality and frame rate to ensure the quality of the stored data.

[0234] Step 3:

[0235] Data analysis and evaluation

[0236] Input: Video data stored in a data storage device

[0237] Output: Comprehension and emotional evaluation scores from intelligence assessment tools

[0238] Specific operation: The server's intelligent evaluation means (for example, an artificial intelligence model using a facial recognition algorithm) analyzes the stored video data. It evaluates the student's understanding and the customer's emotions and quantifies them as a score. This score is then listed for each student or customer.

[0239] Step 4:

[0240] Generate learning content and product suggestions

[0241] Input: Intelligence assessment score

[0242] Output: Individual tasks and product suggestions generated by the learning content generation means

[0243] Specific operation: Based on the score obtained by the intelligence evaluation means, the server uses a generative AI model to generate learning content, which generates individual assignments, review questions, or product suggestions suitable for the customer.

[0244] Step 5:

[0245] Data transmission

[0246] Input: Generated individual issues and product proposals

[0247] Output: Learning content and product suggestions sent to receiving devices (student terminals, store displays, smartphones)

[0248] Specific operation: The server transmits the assignments and product suggestions generated by the learning content generation means to each receiving device using the data transmission means.

[0249] Step 6:

[0250] Interaction

[0251] Input: Issues and product proposals displayed on the receiving device

[0252] Output: Answers and feedback

[0253] Specific operation: The receiving device (e.g., student terminal, smartphone, in-store display) displays the received assignment or product suggestion. The user (student or customer) then operates accordingly, entering an answer in the case of an assignment, or making a selection or purchasing action in the case of a product suggestion.

[0254] Step 7:

[0255] Feedback of results

[0256] Input: Answers and feedback from the receiving device

[0257] Output: Answer results and behavioral data fed back to the server

[0258] Specific operation: The terminal transmits the results of the user's operations (e.g., answers to assignments and product selection information) to the server. The server receives this and records it in the data storage means.

[0259] Step 8:

[0260] Report generation and delivery

[0261] Input: Recorded answer results and behavioral data

[0262] Output: Comprehension reports and customer behavior reports generated by the report generation means, and reports sent to the reporting device

[0263] Specific operation: Based on the received answer results and behavioral data, the server creates a comprehension report and a customer behavior report using the report generation means. The generated report is sent to a reporting device (e.g., a teacher terminal or a store management system) and displayed.

[0264] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0265] The present invention relates to an educational support system that provides effective learning support to each student by recording lesson content and student responses in real time and recognizing the emotional state of the student. This system is composed of multiple camera means, server means, artificial intelligence model means, emotion engine, assignment generation means, transmission means, student terminal means, display means, report generation means, and teacher terminal means.

[0266] System program processing and explanation

[0267] Record of lesson content

[0268] At the start of a lesson, the server checks whether the first camera means (Camera 1) that records the lesson content and the second camera means (Camera 2) that records the students' reactions are running properly. When the lesson starts, Camera 1 records the teacher's lesson content in real time and sends the video data to the server. At the same time, Camera 2 records the expressions and actions of all the students in real time and also sends this video data to the server.

[0269] Video data storage and preparation

[0270] The server stores the video data of the lesson content and students' reactions received from the camera means in real time. This stored data will be used for later analysis, so it also checks the video quality and frame rate.

[0271] Data analysis and understanding assessment

[0272] The AI ​​model analyzes video data of lesson content and student responses stored on a server. A facial recognition algorithm is used to assess students' comprehension based on their facial expressions and behavior. Furthermore, an emotion engine is incorporated to analyze students' emotional states. This emotion engine recognizes emotional states such as enjoyment, interest, confusion, and fatigue in real time, and reflects this data in the assessment of comprehension.

[0273] Creating and submitting individual assignments

[0274] The assignment generation means generates optimal preparation, review, and practice assignments for each student based on the level of understanding and emotional state generated by the artificial intelligence model means and the emotion engine. In particular, it presents assignments that reflect the analysis results of the emotion engine and take learning motivation into consideration. The generated assignments are sent to each student's device via the transmission means.

[0275] Assignment and feedback of results

[0276] The student's device displays the received assignment and provides an interface for the student to complete it. After the student completes the assignment, the results are sent from the device to the server. The server then reassessssments the results and generates a report on the overall class comprehension. This report is used by the teacher to optimize lesson plans.

[0277] Generate reports and provide feedback to teachers

[0278] The report generation tool compiles a summary of the class's overall understanding based on the results of the assignments and creates a detailed report. This report visualizes each student's strengths and weaknesses. The teacher's device receives and displays this report, allowing the teacher to grasp the overall situation in the class and plan appropriate measures for the next lesson.

[0279] Specific examples

[0280] 1. Conducting and recording lessons

[0281] The user (teacher) begins an English lesson. Camera 1 records the teacher's lesson, and Camera 2 records the students' reactions. For example, when the teacher explains the meaning of a new word, the camera captures the expressions of interest or confusion of a particular student.

[0282] 2. Data analysis and understanding assessment

[0283] The server stores these video data, and the artificial intelligence model means and emotion engine analyze the students' facial expressions and emotions, thereby determining whether a particular student has difficulty understanding new words.

[0284] 3. Submitting and completing individual assignments

[0285] Students can receive personalized review tasks on their devices that take into account their level of comprehension and emotional state, such as quiz-style tasks to practice the meaning of new words.

[0286] 4. Feedback on the results and reflecting them in lesson plans

[0287] The server receives the students' answers and re-evaluates them. A class-wide comprehension report is generated and sent to the teacher's device, allowing the teacher to determine which words and expressions need additional explanation in the next lesson.

[0288] In this way, the present invention utilizes AI and emotion recognition technology to provide optimal learning support tailored to each student's level of understanding and emotional state, thereby reducing the burden on teachers while increasing student motivation and creating an effective educational environment.

[0289] The processing flow will be explained below.

[0290] Step 1:

[0291] At the start of a lesson, the server checks whether the first camera means (camera 1) for recording the lesson content and the second camera means (camera 2) for recording the students' reactions are running normally.

[0292] Step 2:

[0293] Camera 1 records the teacher's lesson content in real time and transmits the video data in streaming format to the server.

[0294] Step 3:

[0295] Camera 2 records the facial expressions and behavior of the entire classroom and each student in real time, and transmits the video data in streaming format to a server.

[0296] Step 4:

[0297] The server stores the received video data in real time and monitors the video quality and frame rate.

[0298] Step 5:

[0299] The AI ​​model analyzes the video data of lesson content and student responses stored on the server. Specifically, it uses a facial recognition algorithm to identify students' facial expressions and behaviors and score each student's level of understanding.

[0300] Step 6:

[0301] The emotion engine recognizes students' emotional states (e.g., enjoyment, interest, confusion, fatigue) in real time based on the students' facial expression data acquired by the artificial intelligence model means, and the analysis results are reflected as additional information in the comprehension score.

[0302] Step 7:

[0303] The server stores the comprehension scores and emotion data generated by the artificial intelligence model means and emotion engine, and lists the learning needs of each student.

[0304] Step 8:

[0305] The task generation unit generates optimal preparation, review, and practice tasks for each student based on the stored comprehension scores and emotional data. In particular, tasks that increase the student's learning motivation are generated taking into account the student's emotional state.

[0306] Step 9:

[0307] The transmission means transmits the generated individual assignments to the terminals of the students.

[0308] Step 10:

[0309] The student's device displays the received assignment and provides an interface through which the student can complete the assignment.

[0310] Step 11:

[0311] Students answer the assignments and send the results data from their devices to the server.

[0312] Step 12:

[0313] The server reassessed the students' understanding based on the received assignment results and generated a comprehension report for the entire class.

[0314] Step 13:

[0315] The report generating means transmits the generated comprehension report to the teacher's terminal.

[0316] Step 14:

[0317] The teacher's device will display a comprehension report, allowing the teacher to understand the learning status of the entire class.

[0318] Step 15:

[0319] Based on the generated comprehension reports and student emotion data, the teacher can decide what to cover and what to focus on in the next lesson, for example, by planning to explain in detail a topic that many students found confusing.

[0320] Through this series of processing steps, the present invention combines AI technology and emotion recognition technology to realize a system that provides optimal learning support to each student while reducing the burden on teachers.

[0321] Example 2

[0322] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0323] Conventional educational systems lack efficiency and accuracy in recording lesson content, understanding student responses, and providing individualized instruction, making it difficult to provide individual learning support tailored to each student's level of understanding and emotional state. This also places a heavy burden on teachers, making it difficult to grasp the overall class level of understanding in real time and reflect this in the next lesson plan.

[0324] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a first imaging means for recording the content of the lesson, a second imaging means for recording the reactions of the learners, an information processing means for receiving and storing video data from the first imaging means and the second imaging means, an inference engine means for analyzing the video data stored by the information processing means and evaluating the level of understanding of the learners, a learning material generation means for generating individual learning materials based on the level of understanding evaluated by the inference engine means, a communication means for transmitting the learning materials generated by the learning material generation means to a learner terminal means, a display means for displaying the learning materials received by the learner terminal means and having the learner execute them, a communication means for transmitting the results of the learning materials executed by the learner terminal means to an information processing means, a report generation means for reevaluating based on the results received by the information processing means and generating a comprehension report for the entire group, and a communication means for transmitting the comprehension report generated by the report generation means to an educator terminal means. This will enable the provision of optimal learning support in real time based on each student's level of understanding and emotional state, reducing the burden on teachers while creating an effective educational environment.

[0325] "Capturing means" is a device for acquiring video data.

[0326] An "information processing means" is a device or system that receives, stores, and analyzes the captured video data.

[0327] "Inference engine means" refers to an algorithm or software for analyzing stored video data and assessing the learner's level of comprehension and emotional state.

[0328] The "learning material generating means" is a device or software for generating individual learning materials based on the evaluation results by the inference engine means.

[0329] A "communication means" is an interface for sending and receiving data, and is used to communicate via a network or data link.

[0330] "Student terminal means" refers to a terminal device used by a student, which displays learning materials and accepts input of answers.

[0331] The "display means" refers to a display or monitor for displaying learning materials on the learner terminal means.

[0332] The "report generation means" is a device or software for generating a comprehension report based on the learner's answers.

[0333] The "educator terminal means" is a terminal device used by an educator, which receives and displays the generated comprehension report.

[0334] The educational support system related to this invention provides individualized learning support based on the student's level of understanding and emotional state. This system requires a wide range of hardware and software to comprehensively record lesson content, record student responses, analyze data, generate individualized learning materials, provide feedback on the results, and generate reports.

[0335] Recording lesson content and student responses

[0336] The system is equipped with a first imaging means for recording the content of the lesson and a second imaging means for recording the reactions of students. High-resolution cameras are used for these imaging means. As a specific example, the first imaging means is a fixed camera installed at the front of the classroom, and the second imaging means is a wide-angle camera that can capture the entire student body. When the lesson begins, the server confirms that these cameras are activated and operating normally, and begins acquiring video data in real time.

[0337] Video data storage and preparation

[0338] The server receives the video data sent from the camera in real time and stores it in a database. At the same time, it checks the video quality and frame rate and stores the data in a format suitable for analysis.

[0339] Data analysis and understanding assessment

[0340] The server transmits the stored video data to an inference engine means. This inference engine means incorporates an artificial intelligence model including a facial recognition algorithm. The artificial intelligence model analyzes the students' facial expressions and behavior and quantifies their level of understanding. It also incorporates an emotion recognition engine to determine their emotional state, recognizing their emotional states such as interest, confusion, enjoyment, and fatigue in real time. Based on these results, each student's level of understanding is evaluated.

[0341] Creation and delivery of personalized learning materials

[0342] The learning material generation means generates optimal learning materials based on the data obtained from the inference engine means and the emotion recognition engine. For example, if a student is having difficulty understanding a new English word, a quiz-style assignment for vocabulary practice is generated for that student. The generated learning materials are transmitted to the learner terminal means of each student via the communication means.

[0343] Assignment and feedback of results

[0344] The learner's device displays the received learning materials and provides an interface for the learner to execute them. When the student completes the assignment, the answer is sent from the device to the server. The server re-evaluates the received answer and re-evaluates the overall class and individual comprehension.

[0345] Generate reports and provide feedback to teachers

[0346] The server generates a comprehension report for the entire group based on the answer results. The report generation means creates a detailed report and visualizes each student's strengths and weaknesses. The generated comprehension report is sent to the educator terminal means, where it can be viewed by the teacher. This allows the teacher to grasp the progress of the entire class and provide any additional explanations or individual guidance needed in the next lesson.

[0347] Specific examples

[0348] Prompt Sentence Examples

[0349] "We're starting English class. I'll explain some new words."

[0350] "Take the following quiz-style challenge. Choose the meaning of a new word."

[0351] "Student A seems confused about the meaning of new words."

[0352] "Further explanation will be required in the next class."

[0353] This allows the educational support system to utilize AI and emotion recognition technology to provide optimal learning support in real time according to each student's level of understanding and emotional state, reducing the burden on teachers and creating an effective educational environment.

[0354] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0355] Step 1: Record lesson content and student responses

[0356] 1.1 The server confirms that Camera 1 (for recording lesson content) and Camera 2 (for recording student responses) are running normally.

[0357] (Input): Camera startup status

[0358] (Output): Message confirming successful camera startup

[0359] 1.2 When the lesson begins, Camera 1 records the teacher's lesson content in real time.

[0360] (Input): Lesson start trigger

[0361] (Output): Video data of the teacher's lesson

[0362] 1.3 Camera 2 records students' reactions (facial expressions and behavior) in real time.

[0363] (Input): Lesson start trigger

[0364] (Output): Video data of students' reactions

[0365] 1.4 The video data acquired from Camera 1 and Camera 2 is sent to the server.

[0366] (Input): Video data of teacher's lesson content and students' responses

[0367] (Output): Message that video data has been sent to the server

[0368] Step 2: Saving and preparing your footage

[0369] 2.1 The server receives and stores the video data transmitted from Camera 1 and Camera 2 in real time.

[0370] (Input): Video data of teacher's lesson content and students' responses

[0371] (Output): Message confirming saving to database

[0372] 2.2 The server checks the video quality and frame rate of the stored data.

[0373] (Input): Saved video data

[0374] (Output): Video quality check results

[0375] Step 3: Analyze data and assess comprehension

[0376] 3.1 The server transmits the stored video data to the inference engine means.

[0377] (Input): Saved video data

[0378] (Output): Message that data has been sent to the inference engine

[0379] 3.2 The inference engine means uses a facial recognition algorithm to analyze the student's facial expressions and behavior.

[0380] (Input): Student reaction video data

[0381] (Output): Student facial expression and behavior analysis results

[0382] 3.3 The server sends the analysis results received from the inference engine means to the emotion engine to determine the emotional state of the student.

[0383] (Input): Facial expression and behavior analysis results

[0384] (Output): Student's emotional state assessment result

[0385] Step 4: Generate and send personalized learning materials

[0386] 4.1 The learning material generation means generates optimal learning materials based on the data obtained from the inference engine means and the emotion engine.

[0387] (Input): Comprehension and emotional state data

[0388] (Output): Individual learning materials

[0389] 4.2 The generated learning materials are transmitted to the learner terminal means of each student via communication means.

[0390] (Input): Individual learning materials

[0391] (Output): Message that the teaching materials have been sent to the student's device

[0392] Step 5: Implementing the task and providing feedback

[0393] 5.1 The learner's device displays the transmitted learning materials.

[0394] (Input): Individual learning materials

[0395] (Output): Display of learning materials

[0396] 5.2 Students work on the assignment and send their answers from their devices to the server.

[0397] (Input): Student's answer result

[0398] (Output): Message that the answer result has been sent to the server

[0399] 5.3 The server will re-evaluate the received answer results and re-evaluate the overall class and individual comprehension.

[0400] (Input): Student's answer result

[0401] (Output): Reassessment results (understanding data)

[0402] Step 6: Generate reports and provide feedback to teachers

[0403] 6.1 The server generates a group-wide comprehension report based on the answers.

[0404] (Input): Reassessment results (understanding data)

[0405] (Output): Comprehension report

[0406] 6.2 The report generator will produce detailed reports and visualize each student's strengths and weaknesses.

[0407] (Input): Comprehension Report

[0408] (Output):Detailed report

[0409] 6.3 The generated comprehension report will be sent to the educator terminal means so that the teacher can review it.

[0410] (Input):Detailed Report

[0411] (Output): Report sending completion message to teacher's device

[0412] Through the above processing steps, this system is able to efficiently record and analyze lesson content and student responses in real time, providing optimal learning support to each individual student.

[0413] (Application example 2)

[0414] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0415] Conventional educational support systems have difficulty in properly assessing students' understanding and learning progress, and in particular, they have been unable to provide sufficient learning support that takes into account the emotional state of each student. This can result in a lack of detailed learning support for each student, which can reduce learning effectiveness. Another issue is that teachers are unable to receive effective feedback when grasping the overall understanding level of the class and reflecting this in their next lesson plan.

[0416] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means: In this invention, the server includes a first image capturing device means for recording the content of the lesson, a second image capturing device means for recording the reactions of students, an information processing device means for receiving and storing video information from the first image capturing device and the second image capturing device, a machine learning model device means for analyzing the video information stored by the information processing device and evaluating the level of understanding of the students, an assignment generation device means for generating individual assignments based on the level of understanding evaluated by the machine learning model device, a communication device means for transmitting the assignments generated by the assignment generation device to student terminal devices, a display device means for displaying the assignments received by the student terminal devices and having the students perform them, a communication device means for transmitting the results of the assignments performed by the student terminal devices to the information processing device, a report generation device means for re-evaluating the results received by the information processing device and generating a comprehension report for the entire class, and a communication device means for transmitting the comprehension report generated by the report generation device to a teacher terminal device. This not only makes it possible to create individual tasks based on students' level of understanding and provide learning support that takes into account students' emotional state, but also allows the level of understanding of the entire class to be grasped, providing effective feedback that teachers can reflect in their next lesson plans.

[0417] The "first imaging device for recording the content of the lesson" is a video device for recording the progress of the lesson, and is a camera device for capturing the entire lesson, including what the teacher is saying and what he is writing on the blackboard.

[0418] The "second imaging device for recording students' reactions" is a video device for recording students' facial expressions and behavior, and a camera device for monitoring how students are reacting to lessons.

[0419] An "information processing device" is a computer system for receiving and storing video information that records lesson content and student responses, and is a device that analyzes and stores video data.

[0420] A "machine learning model device" is a data processing device that analyzes video data stored in an information processing device and evaluates students' level of understanding, and is an artificial intelligence device that has algorithms for facial recognition and behavioral analysis.

[0421] The "task generation device" is a device that automatically generates individual tasks based on the level of understanding assessed by the machine learning model device, and is a device for providing learning content adapted to each student.

[0422] The "communication device" is a network device for transmitting and receiving data between the information processing device, student terminal device, and teacher terminal device, and is a device for transmitting assignments and receiving results.

[0423] A "student terminal" is a device that receives and allows a student to complete assignments, and provides an interface for displaying and answering individual assignments.

[0424] The "display device" is a device for showing the assignment received by the student terminal device to the student, and is a monitor or screen for visually displaying the content of the assignment.

[0425] A "report generation device" is a device that compiles the level of understanding of the entire class based on the results of the assignments and creates a detailed report, generating materials that allow teachers to understand the learning situation of their class.

[0426] An "emotion recognition engine" is an algorithm that analyzes students' emotional state from their facial expressions and behavior and evaluates their motivation to learn, and is a technology that recognizes students' emotional changes in real time.

[0427] The present invention relates to an educational support system that analyzes a student's level of understanding and emotional state and provides individual learning support. The detailed configuration and operation of this system will be described below.

[0428] System configuration

[0429] This educational support system consists of the following main components:

[0430] 1. First imaging device

[0431] 2. Secondary imaging device

[0432] 3. Information Processing Device

[0433] 4. Machine Learning Model Device

[0434] 5. Task generation device

[0435] 6. Communications Equipment

[0436] 7. Student terminal devices

[0437] 8. Display device

[0438] 9. Report Generator

[0439] 10. Emotion Recognition Engine

[0440] System Operation

[0441] 1. Recording lesson content and student responses

[0442] When the lesson begins, the first camera records the teacher's lesson content in real time and transmits the video data to the information processing device. At the same time, the second camera records the students' expressions and behaviors and transmits the video data to the information processing device.

[0443] 2. Video data storage and analysis

[0444] The information processing device stores the received video data for later analysis. The quality and frame rate of the video are also checked to ensure optimal conditions are maintained.

[0445] The machine learning model uses the stored video data to assess the student's understanding, using data processing techniques such as facial recognition algorithms.

[0446] 3. Emotional state analysis

[0447] The emotion recognition engine analyzes students' emotional states from their facial expressions and behaviors, allowing real-time assessment of their emotional states, such as enjoyment, interest, confusion, and fatigue.

[0448] 4. Generating individual tasks

[0449] The assignment generator generates optimal preparatory, review, and practice assignments for each student based on data from the machine learning model device and emotion recognition engine.

[0450] In particular, taking into account emotional states provides tasks that are tailored to students' motivation.

[0451] 5. Submit and complete the assignment

[0452] The generated assignment is sent to the student terminal via the communication device, which displays the received assignment and provides an interface for the student to work on it.

[0453] When the student answers the assignment, the results are transmitted to the information processing device via the communication device.

[0454] 6. Review and feedback of results

[0455] The information processing device performs a reassessment based on the received assignment results and generates a report on the level of understanding of the entire class. The report generating device creates this report and sends it to the teacher terminal device to provide feedback to the teacher.

[0456] Specific examples

[0457] For example, consider a situation where students are learning new words during an English class. A first camera records the teacher's lesson content, and a second camera records the students' responses. An information processing device stores and analyzes the video data. A machine learning model device and an emotion recognition engine are used to evaluate whether the students understand the meaning of the words and what their emotional state is. An assignment generation device then generates individual review assignments and sends them to the student terminal device via a communication device. The students answer the assignments on their terminals, and the results are sent back to the server for re-evaluation and feedback.

[0458] Prompt Sentence Examples

[0459] Prompts to assess student understanding:

[0460] Use a facial recognition algorithm to analyze Student A's facial expressions within the most recent lesson data and score his / her level of understanding.

[0461] Prompts for analyzing emotional states:

[0462] Based on Student B's video data, please evaluate his emotional state in real time, such as enjoyment, interest, confusion, and fatigue.

[0463] In this way, the present invention can maximize the effectiveness of education by providing individual learning support that takes into account not only the student's level of understanding of the lesson content but also their emotional state.

[0464] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0465] Step 1:

[0466] Recording of lesson content and student responses

[0467] At the start of a lesson, the server checks whether the first and second image capture devices are running properly, and is then ready to record the lesson content and students' responses.

[0468] Input: Lesson start command, activation status of first and second imaging devices

[0469] Operation: The first camera records the teacher's lesson in real time and sends the video data to a server. At the same time, the second camera records the students' expressions and behavior and sends this video data to the server.

[0470] Output: Video data of teacher's lesson content, video data of students' reactions

[0471] Step 2:

[0472] Video data storage and quality check

[0473] The server temporarily stores the received video data, checking the quality and frame rate of the video data to ensure it is in a state suitable for optimal analysis.

[0474] Input: Video data of lesson content, video data of student reactions

[0475] Operation: Saving video data and checking the frame rate and quality

[0476] Output: High-quality video data stored

[0477] Step 3:

[0478] Analysis of video data and assessment of comprehension

[0479] The server then passes the stored video data to a machine learning model device, which uses a facial recognition algorithm to analyze the student's facial expressions and behavior and generate a score based on their level of understanding.

[0480] Input: High-quality video data

[0481] Action: Analyzing students' facial expressions and behaviors and scoring their understanding using a face recognition algorithm based on a machine learning model device

[0482] Output: Student comprehension score

[0483] Step 4:

[0484] Emotional state analysis

[0485] The server then passes the stored video data to an emotion recognition engine, which analyzes the student's emotional state, assessing emotions such as enjoyment, interest, confusion, and fatigue.

[0486] Input: High-quality video data

[0487] Action: Analyzing students' emotional states using an emotion recognition engine

[0488] Output: Student emotional state data

[0489] Step 5:

[0490] Generate individual assignments

[0491] The task generation device generates optimal preparation, review, and practice tasks for each student based on their comprehension scores and emotional state data.

[0492] Input: Student comprehension scores, student emotional state data

[0493] Action: Individual task generation according to comprehension level and emotional state

[0494] Output: Individual issue data

[0495] Step 6:

[0496] Submitting and viewing assignments

[0497] The created assignment is sent to the student terminal via the communication device, and the student terminal receives the assignment and displays it to the student.

[0498] Input: Individual issue data

[0499] Operation: Sending assignment data and displaying it on student terminal devices

[0500] Output: The displayed issues

[0501] Step 7:

[0502] Implementing the assignment and collecting the results

[0503] Students answer questions using their student devices and send the results to the server, which then receives the results.

[0504] Input: Viewed assignment

[0505] Operation: Student terminal device performs assignments, inputs results, and transmits results via communication device

[0506] Output: Student assignment answer result data

[0507] Step 8:

[0508] Review of results and feedback

[0509] The server analyzes the received answers and evaluates the level of understanding and areas for improvement. The report generator creates a report on the level of understanding of the entire class and sends it to the teacher's terminal device via the communication device.

[0510] Input: Student assignment answer result data

[0511] Actions: Analyze the results data, reassess the level of understanding, generate and send a feedback report

[0512] Output: Whole class comprehension report

[0513] Prompt Sentence Examples

[0514] "Use a facial recognition algorithm to analyze Student A's facial expressions within the most recent lesson data and score their level of understanding."

[0515] "Based on Student B's video data, please rate his emotional state in real time, such as enjoyment, interest, confusion, and fatigue."

[0516] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0517] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0518] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0519] [Second embodiment]

[0520] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0521] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0522] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0523] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0524] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0525] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0526] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0527] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0528] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0529] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0530] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0531] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0532] The present invention relates to an educational support system that can effectively record lesson content and grasp students' understanding in real time. This system is composed of multiple cameras, a server, an artificial intelligence model, an assignment generation unit, a transmission unit, student terminals, a display unit, a report generation unit, and a teacher terminal.

[0533] System program processing and explanation

[0534] Record of lesson content

[0535] At the start of a lesson, the server checks whether the first camera means (Camera 1) that records the lesson content and the second camera means (Camera 2) that records the students' reactions are running normally. When the lesson starts, Camera 1 records the teacher's lesson content in real time and sends the video data to the server. At the same time, Camera 2 records the expressions and actions of all the students in real time and also sends this video data to the server.

[0536] Video data storage and preparation

[0537] The server stores the video data of the lesson content and students' reactions received from the camera means in real time. This stored data will be used for later analysis, so it also checks the video quality and frame rate.

[0538] Data analysis and understanding assessment

[0539] The AI ​​model analyzes the video data stored on the server. A facial recognition algorithm is used to evaluate the student's level of understanding based on their facial expressions and behavior. The evaluation results are saved as a score and listed for each student.

[0540] Creating and submitting individual assignments

[0541] The assignment generation means generates preparation, review, and practice assignments suitable for each student based on the results of the assessment of comprehension by the artificial intelligence model means. The generated assignments are sent to each student's terminal via the transmission means.

[0542] Assignment and feedback of results

[0543] The student's device displays the received assignments and provides an interface for the student to complete them. After the student completes the assignment, the results are sent from the device to the server. The server reassessssments based on the received assignment results and generates a report on the overall class comprehension. This report is used by the teacher to optimize lesson plans.

[0544] Generate reports and provide feedback to teachers

[0545] The report generation tool compiles a summary of the class's overall understanding based on the results of the assignments and creates a detailed report. This report visualizes each student's strengths and weaknesses. The teacher's device receives and displays this report, allowing the teacher to grasp the overall situation in the class and plan appropriate measures for the next lesson.

[0546] Specific examples

[0547] 1. Conducting and recording lessons

[0548] The user (teacher) starts a math lesson. Camera 1 records the teacher's lesson content, and Camera 2 records the students' reactions. For example, when the teacher is explaining the basics of equations, Student A looks confused.

[0549] 2. Data analysis and understanding assessment

[0550] The server stores this video data, and the artificial intelligence model means judges the low level of understanding from the facial expression of Student A. Based on this, an assignment is created for Student A to review basic equation problems.

[0551] 3. Submitting and completing individual assignments

[0552] The assignment is sent to and displayed on Student A's device. Student A answers the assignment and sends the results to the server.

[0553] 4. Feedback on the results and reflecting them in lesson plans

[0554] The server receives Student A's answer and re-evaluates it. A report of the class's understanding is generated and sent to the teacher's terminal. Based on this, the teacher decides to allocate time to the basics of equations in the next lesson.

[0555] In this way, the present invention utilizes AI technology to reduce the burden on teachers while providing learning support that is tailored to each student.

[0556] The processing flow will be explained below.

[0557] Step 1:

[0558] At the start of a lesson, the server checks whether the first camera means (camera 1) for recording the lesson content and the second camera means (camera 2) for recording the students' reactions are running normally.

[0559] Step 2:

[0560] Camera 1 records the teacher's lesson content in real time and transmits the video data in streaming format to the server.

[0561] Step 3:

[0562] Camera 2 records the facial expressions and behavior of the entire classroom and each student in real time, and transmits the video data in streaming format to a server.

[0563] Step 4:

[0564] The server stores the received video data in real time and monitors the video quality and frame rate.

[0565] Step 5:

[0566] The AI ​​model analyzes the video data of the lesson content and students' reactions stored on the server. Specifically, it uses a facial recognition algorithm to identify students' facial expressions and behaviors and score each student's level of understanding.

[0567] Step 6:

[0568] The server stores the comprehension scores generated by the artificial intelligence model means and lists the learning needs of each student.

[0569] Step 7:

[0570] The assignment generation means generates optimal preparation, review, and practice assignments for each student based on the stored comprehension scores.

[0571] Step 8:

[0572] The transmission means transmits the generated individual assignments to the terminals of the students.

[0573] Step 9:

[0574] The student's device displays the received assignment and provides an interface through which the student can complete the assignment.

[0575] Step 10:

[0576] Students answer the assignments and send the results data from their devices to the server.

[0577] Step 11:

[0578] The server reassessed the students' understanding based on the received assignment results and generated a comprehension report for the entire class.

[0579] Step 12:

[0580] The report generating means transmits the generated comprehension report to the teacher's terminal.

[0581] Step 13:

[0582] The teacher's device will display a comprehension report, allowing the teacher to understand the learning status of the entire class.

[0583] Step 14:

[0584] Based on the comprehension reports, teachers decide what content and focus to cover in the next lesson.

[0585] Through this series of processing steps, the present invention realizes a system that provides learning support suited to each individual student and more effectively supports teachers' lesson plans.

[0586] Example 1

[0587] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0588] Conventional educational support systems have had difficulty accurately assessing each student's level of understanding in real time and providing individualized learning support. They also have limitations in comprehensively recording lesson content and effectively utilizing that data. This makes it difficult for teachers to grasp students' levels of understanding, leading to problems with efficient instruction.

[0589] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0590] In this invention, the server includes a first recording means for recording the content of the lesson, a second recording means for recording the responses of students, a data storage means for receiving and storing video data from the first recording means and the second recording means, an analysis means for analyzing the video data stored by the data storage means and evaluating the students' levels of comprehension, an assignment creation means for generating individual assignments based on the levels of comprehension evaluated by the analysis means, a transmission means for transmitting the assignments generated by the assignment creation means to student terminals, a display means for displaying the assignments received by the student terminals and having the students complete them, a transmission means for transmitting the results of the assignments completed by the student terminals to the data storage means, a report creation means for re-evaluating the results received by the data storage means and generating a report on the level of comprehension of the entire class, and a transmission means for transmitting the comprehension report generated by the report creation means to a teacher's terminal. This enables comprehensive recording of the content of the lesson and real-time evaluation and feedback of students' levels of comprehension.

[0591] The "primary recording means for recording the content of the class" is a device that records the lecture content and explanations of the instructor in charge of the class as video and audio.

[0592] The "second recording means for recording student responses" is a device that records students' facial expressions and behavior during class in real time as video.

[0593] The "data storage means" is a system or device for receiving the video data transmitted from the first recording means and the second recording means and storing the data for a long period of time.

[0594] The "analysis means" is a system that includes algorithms and software for analyzing the stored video data and assessing the student's level of understanding.

[0595] The "task creation means" is a system or device for automatically creating individual learning tasks based on the student's level of understanding evaluated by the analysis means.

[0596] The "transmission means" is a system or device for transmitting the generated assignments and comprehension reports to a specified terminal.

[0597] A "student device" is a device such as a computer or tablet used by a student that provides an interface for displaying and completing received assignments.

[0598] The "display means" is software or a system that displays the assignment received on the student terminal and provides an interface for the student to answer the assignment.

[0599] The "report creation means" is a system that compiles the level of understanding of the entire class based on the results of assignments sent from student devices and generates a comprehension report for the teacher.

[0600] A "teacher's terminal" is a device such as a computer or tablet used by a teacher to receive and display comprehension reports.

[0601] The present invention relates to an educational support system that can effectively record lesson content and grasp students' understanding in real time. This system is composed of multiple recording means, data storage means, analysis means, assignment creation means, transmission means, student terminals, display means, report creation means, and teacher terminals.

[0602] Hardware and Software Configuration

[0603] Specific hardware used includes camera devices (first and second recording means), servers (data storage means and analysis means), student tablets or PCs (student terminals), and faculty tablets or PCs (faculty terminals).Software includes video analysis algorithms (face recognition algorithms and facial expression analysis algorithms), AI models, assignment generation algorithms, and database management systems.

[0604] Data collection and storage

[0605] At the start of class, the server checks whether the first recording means (Camera 1), which records the teacher's lecture content, and the second recording means (Camera 2), which records the students' reactions, are operating normally. When class begins, Camera 1 records the teacher's lecture in real time and sends the data to the server. Camera 2 simultaneously records the expressions and movements of all the students and also sends this data to the server. The data storage means saves the received video data to a highly durable storage device.

[0606] Data analysis and understanding assessment

[0607] The saved video data is processed by an analysis means. The analysis means uses a facial recognition algorithm and an expression analysis algorithm to evaluate the student's level of comprehension based on their facial expressions and movements. This evaluation is quantified and recorded as a comprehension score for each student. The analysis results are saved in a database and used for subsequent processing.

[0608] Creating and submitting assignments

[0609] The assignment creation means generates individual preparation, review, and practice assignments based on the comprehension score obtained from the analysis means. The generated assignments are sent to each student's student device via the transmission means. For example, mathematics practice problems created using the generative AI model are distributed to student devices.

[0610] Assignment and feedback of results

[0611] The student terminals display the received assignments and provide an interface for students to answer. When students complete the assignments, their answers are automatically sent to the server. The server reevaluates the received answers and creates a report based on the overall class understanding. This report is generated by the report creation means and sent to the teacher terminal.

[0612] Specific examples

[0613] 1. Lesson Conduct and Recording:

[0614] A user (teacher) starts a math class at 9:00 a.m. Camera 1 records the teacher's lecture, and Camera 2 records the students' reactions. For example, when the teacher is explaining the basics of equations, Student A looks confused.

[0615] 2. Data analysis and comprehension assessment:

[0616] The server stores this video data, and the analysis means determines whether Student A has a low level of understanding based on his / her facial expressions, and records the result as a score. Data is generated to provide Student A with review assignments.

[0617] 3. Submit and complete the assignment:

[0618] The assignment creation means generates a review assignment suitable for Student A and sends it to Student A's student terminal. Student A's terminal displays the review math problems.

[0619] 4. Feedback on results and reflection on next class:

[0620] The server receives Student A's answers and re-evaluates them. A comprehension report is sent to the teacher's terminal, and the teacher uses it to plan the next lesson.

[0621] Prompt Sentence Examples

[0622] Below are some examples of prompts to give instructions to an AI generative model.

[0623] "To determine whether students are understanding during class, consider the following scenario: Camera 1 records a teacher teaching the basics of equations in a math class, while simultaneously recording the students' reactions with Camera 2. If it is determined that Student A has a confused expression, generate appropriate review assignments for Student A based on the results of the facial expression analysis."

[0624] In this way, the present invention utilizes AI technology to reduce the burden on teachers while providing learning support that is tailored to each student.

[0625] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0626] Step 1: Start the lesson and check that the equipment is up and running

[0627] At the designated time, the server checks whether the lesson is about to begin. Here, it checks whether the first recording means (camera 1) and the second recording means (camera 2) are operating normally. Specifically, the server sends a PING command to each camera and monitors whether there is a response. If a normal response is confirmed, it determines that the lesson is ready to begin.

[0628] Input: lesson start time, camera status

[0629] Output: Confirmation that the device has started up normally

[0630] Step 2: Record lesson content and student reactions

[0631] When the class begins, the server has Camera 1 record the teacher's lecture in real time and transmits the video data to the server. At the same time, Camera 2 records the reactions of all students and transmits this to the server. The video data being recorded is received in real time and saved by the data saving means.

[0632] Input: Teacher lecture video, student reaction video

[0633] Output: Save real-time video data to the server

[0634] Step 3: Save the video data and check its quality

[0635] The server stores the received video data in a data storage device. At the same time, it checks the quality of the video data. Specifically, it automatically checks whether there are any missing frames and whether the audio data is synchronized. The results of this check are also saved and used to ensure the quality of the data.

[0636] Input: Received video data

[0637] Output: Saved video data, quality check results

[0638] Step 4: Analyze data and assess comprehension

[0639] The saved video data is processed by an analytical device. Specifically, facial recognition algorithms and facial expression analysis algorithms are used to quantify the student's level of comprehension based on their facial expressions and movements. The results of this analysis are recorded as a comprehension score for each student.

[0640] Input: Saved video data

[0641] Output: Comprehension score

[0642] Step 5: Generate assignments and send them to students

[0643] The assignment creation means generates individual assignments based on the comprehension scores. The generated assignments are sent to each student's student device via the transmission means. Based on the generation AI model, more basic questions are provided to students with low scores, and applied questions are provided to students with high scores.

[0644] Input: Comprehension score

[0645] Output: Individual assignment, Submit assignment

[0646] Step 6: Complete and submit the assignment

[0647] The student terminals display the received assignments and provide an interface for students to answer. When students complete the assignments, the answers are automatically sent to the server.

[0648] Input: Received assignment

[0649] Output: Send answer results

[0650] Step 7: Reassess results and provide feedback to teachers

[0651] The server reevaluates the received answers and reassess the level of understanding. Based on the reevaluation results, a report on the level of understanding of the entire class is generated. The report generation means creates this report and sends it to the teacher's terminal.

[0652] Input: Answer result

[0653] Output: Reassessed comprehension, Generate comprehension report

[0654] Step 8: Reflect on the next lesson plan

[0655] The teacher's device displays the received comprehension report and uses it as reference for planning the next lesson. Teachers can identify each student's strengths and weaknesses and optimize their lesson plans.

[0656] Input: Comprehension Report

[0657] Output: Optimized lesson plans

[0658] (Application example 1)

[0659] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0660] Modern education and retail operations require support tailored to diverse individual needs and levels of understanding. In educational environments, it is particularly important to assess each student's level of understanding in real time and provide appropriate learning content based on the results. Meanwhile, in brick-and-mortar stores, analyzing customer behavior and facial expressions and making appropriate product recommendations can be expected to boost purchasing motivation. Current systems face the challenge of being unable to efficiently respond to such diverse demands.

[0661] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0662] In this invention, the server includes a first image acquisition means for recording lesson content, a second image acquisition means for recording student responses, a data storage means for receiving and storing video data from the first image acquisition means and the second image acquisition means, an intelligence assessment means for analyzing the video data stored by the data storage means and evaluating the student's level of comprehension, a learning content generation means for generating individual learning content based on the level of comprehension assessed by the intelligence assessment means, a data transmission means for transmitting the learning content generated by the learning content generation means to a receiving device, a display device for displaying the learning content received by the receiving device and having the student execute it, a data transmission means for transmitting the results of the learning content executed by the receiving device to the data storage means, a report generation means for re-evaluating the results received by the data storage means and generating an overall comprehension report, and a data transmission means for transmitting the comprehension report generated by the report generation means to a reporting device. This enables support to be provided according to individual needs and levels of comprehension in educational and retail environments.

[0663] "Class content" is a term that refers to the content and themes that teachers teach to students in educational activities.

[0664] "Student reactions" is a term that refers to various behaviors that students show during class, such as facial expressions, movements, and actions, that indicate their understanding and interest in what they are learning.

[0665] "Image capture means" is a term that refers to an apparatus or system for capturing video data using a device such as a camera.

[0666] "Data storage means" is a term that refers to a device or system for temporarily or permanently storing acquired video data, analysis results, etc.

[0667] "Intelligence assessment means" is a term used to refer to a device or system that uses artificial intelligence to analyze data obtained and assess an individual's level of understanding and emotions.

[0668] "Learning content generation means" is a term that refers to a device or system for generating learning content optimized for each student based on the evaluation results from the intelligence evaluation means.

[0669] "Data transmission means" is a term that refers to an apparatus or system for transmitting generated learning content, analysis results, reports, etc. to a specific terminal or device.

[0670] "Receiving device" is a term that refers to a device or system for receiving data transmitted from a data transmitting means.

[0671] "Display device" is a term that refers to a device or system that displays data and learning content received by a receiving device so that the user can visually confirm it.

[0672] "Report generator" is a term that refers to a device or system for generating a comprehensive report based on intelligence assessment measures and learning results.

[0673] "Reporting device" is a term that refers to a device or system that ultimately reports generated reports or other information to a user.

[0674] To put this invention into practice, it is necessary to build systems for both education and retail use, and to be able to meet the needs of each environment. Specific examples of such systems are shown below.

[0675] System Overview

[0676] The system mainly includes the following components:

[0677] First image capture method (cameras that record lesson content and customer behavior)

[0678] Secondary image capture means (cameras that record students' reactions and customers' facial expressions)

[0679] Data storage means (server)

[0680] Intelligence evaluation method (analysis system using artificial intelligence model)

[0681] Learning content generation means (a system that generates individual tasks or product proposals)

[0682] Data transmission method (transmission server)

[0683] Receiving device (student device, store display, smartphone)

[0684] Display devices (terminals and displays)

[0685] Report generation means (systems that generate class-wide or customer behavior reports)

[0686] Reporting device (teacher terminal or store management system)

[0687] Hardware and Software

[0688] Hardware:

[0689] Camera (e.g. USB camera)

[0690] Server (data storage and analysis)

[0691] Smartphones, tablets, store displays, robots

[0692] software:

[0693] TensorFlow (a deep learning model for analyzing customer facial expressions)

[0694] OpenCV (acquisition and display of camera images)

[0695] REST API (data sending and receiving)

[0696] Data processing and calculation

[0697] The server processes and calculates the data in the following steps:

[0698] 1. Data Acquisition:

[0699] The first image acquisition means records the contents of the lesson and the behavior of the customer, and transmits the recorded images to the data storage means in real time.

[0700] A second image capturing means records the reactions of the students and the facial expressions of the customers and transmits them to the data storage means as well.

[0701] 2. Analysis and Evaluation:

[0702] The video data stored in the data storage means is analyzed using an intelligent evaluation means (e.g., an artificial intelligence model equipped with a face recognition algorithm) to evaluate the students' understanding and the customers' emotions.

[0703] 3. Data transmission:

[0704] Based on the results of the evaluation by the intelligence evaluation means, the learning content generation means generates appropriate assignments and product suggestions and transmits them to the receiving device.

[0705] 4. Interaction:

[0706] The assignments and product proposals received by the receiving device are displayed on the display device and are carried out by the students or customers.

[0707] 5. Feedback of results:

[0708] The results of the execution on the display device are again transmitted to the data storage means by the data transmission means, and the report generation means generates an understanding level report and a customer behavior report.

[0709] Specific examples

[0710] 1. Educational environment:

[0711] Consider a scenario in which a teacher begins a lesson. A first image acquisition means records the content of the lesson, and a second image acquisition means records the students' responses. An AI model analyzes the students' facial expressions, and if a particular student is struggling to understand, it sends that student an individual assignment. This assignment is displayed on the student's device, and the student answers it. The answers are sent to a server, and the teacher receives a report of the class's understanding on his or her device.

[0712] 2. Store environment:

[0713] Consider a scenario in which a customer enters a store. A first image capture means records the customer's behavior, and a second image capture means analyzes the customer's facial expressions. When the server detects the customer's expressions of interest or confusion, it generates an appropriate product suggestion and sends it to an in-store display or the customer's smartphone. An in-store robot interacts with the customer based on the suggestion (e.g., "Would you like this product?"). Ultimately, store staff optimize their response based on the report generated by the server.

[0714] Prompt Sentence Examples

[0715] For example, if a customer appears unsure, the following prompts can be fed into the generative AI model:

[0716] The customer's expression shows uncertainty. Which of the following products do you predict will most interest the customer?

[0717] This will improve the customer experience and streamline store operations.

[0718] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0719] Step 1:

[0720] Data Acquisition

[0721] Input: Real-time video data acquired by the first image acquisition means (camera 1) and the second image acquisition means (camera 2)

[0722] Output: Video data of lesson content, customer behavior, student reactions, and customer expressions sent to the server

[0723] Specific operation: The device activates Camera 1 and records the entire classroom or store. It then activates Camera 2 and records the facial expressions and behavior of individual students or customers. The captured video data is sent to the server in real time.

[0724] Step 2:

[0725] Data storage

[0726] Input: Real-time video data sent to the server

[0727] Output: Video data stored in a data storage device

[0728] Specific operation: The server stores the received video data in a data storage means (e.g., a database), and also checks the video quality and frame rate to ensure the quality of the stored data.

[0729] Step 3:

[0730] Data analysis and evaluation

[0731] Input: Video data stored in a data storage device

[0732] Output: Comprehension and emotional evaluation scores from intelligence assessment tools

[0733] Specific operation: The server's intelligent evaluation means (for example, an artificial intelligence model using a facial recognition algorithm) analyzes the stored video data. It evaluates the student's understanding and the customer's emotions and quantifies them as a score. This score is then listed for each student or customer.

[0734] Step 4:

[0735] Generate learning content and product suggestions

[0736] Input: Intelligence assessment score

[0737] Output: Individual tasks and product suggestions generated by the learning content generation means

[0738] Specific operation: Based on the score obtained by the intelligence evaluation means, the server uses a generative AI model to generate learning content, which generates individual assignments, review questions, or product suggestions suitable for the customer.

[0739] Step 5:

[0740] Data transmission

[0741] Input: Generated individual issues and product proposals

[0742] Output: Learning content and product suggestions sent to receiving devices (student terminals, store displays, smartphones)

[0743] Specific operation: The server transmits the assignments and product suggestions generated by the learning content generation means to each receiving device using the data transmission means.

[0744] Step 6:

[0745] Interaction

[0746] Input: Issues and product proposals displayed on the receiving device

[0747] Output: Answers and feedback

[0748] Specific operation: The receiving device (e.g., student terminal, smartphone, in-store display) displays the received assignment or product suggestion. The user (student or customer) then operates accordingly, entering an answer in the case of an assignment, or making a selection or purchasing action in the case of a product suggestion.

[0749] Step 7:

[0750] Feedback of results

[0751] Input: Answers and feedback from the receiving device

[0752] Output: Answer results and behavioral data fed back to the server

[0753] Specific operation: The terminal transmits the results of the user's operations (e.g., answers to assignments and product selection information) to the server. The server receives this and records it in the data storage means.

[0754] Step 8:

[0755] Report generation and delivery

[0756] Input: Recorded answer results and behavioral data

[0757] Output: Comprehension reports and customer behavior reports generated by the report generation means, and reports sent to the reporting device

[0758] Specific operation: Based on the received answer results and behavioral data, the server creates a comprehension report and a customer behavior report using the report generation means. The generated report is sent to a reporting device (e.g., a teacher terminal or a store management system) and displayed.

[0759] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0760] The present invention relates to an educational support system that provides effective learning support to each student by recording lesson content and student responses in real time and recognizing the emotional state of the student. This system is composed of multiple camera means, server means, artificial intelligence model means, emotion engine, assignment generation means, transmission means, student terminal means, display means, report generation means, and teacher terminal means.

[0761] System program processing and explanation

[0762] Record of lesson content

[0763] At the start of a lesson, the server checks whether the first camera means (Camera 1) that records the lesson content and the second camera means (Camera 2) that records the students' reactions are running properly. When the lesson starts, Camera 1 records the teacher's lesson content in real time and sends the video data to the server. At the same time, Camera 2 records the expressions and actions of all the students in real time and also sends this video data to the server.

[0764] Video data storage and preparation

[0765] The server stores the video data of the lesson content and students' reactions received from the camera means in real time. This stored data will be used for later analysis, so it also checks the video quality and frame rate.

[0766] Data analysis and understanding assessment

[0767] The AI ​​model analyzes video data of lesson content and student responses stored on a server. A facial recognition algorithm is used to assess students' comprehension based on their facial expressions and behavior. Furthermore, an emotion engine is incorporated to analyze students' emotional states. This emotion engine recognizes emotional states such as enjoyment, interest, confusion, and fatigue in real time, and reflects this data in the assessment of comprehension.

[0768] Creating and submitting individual assignments

[0769] The assignment generation means generates optimal preparation, review, and practice assignments for each student based on the level of understanding and emotional state generated by the artificial intelligence model means and the emotion engine. In particular, it presents assignments that reflect the analysis results of the emotion engine and take learning motivation into consideration. The generated assignments are sent to each student's device via the transmission means.

[0770] Assignment and feedback of results

[0771] The student's device displays the received assignment and provides an interface for the student to complete it. After the student completes the assignment, the results are sent from the device to the server. The server then reassessssments the results and generates a report on the overall class comprehension. This report is used by the teacher to optimize lesson plans.

[0772] Generate reports and provide feedback to teachers

[0773] The report generation tool compiles a summary of the class's overall understanding based on the results of the assignments and creates a detailed report. This report visualizes each student's strengths and weaknesses. The teacher's device receives and displays this report, allowing the teacher to grasp the overall situation in the class and plan appropriate measures for the next lesson.

[0774] Specific examples

[0775] 1. Conducting and recording lessons

[0776] The user (teacher) begins an English lesson. Camera 1 records the teacher's lesson, and Camera 2 records the students' reactions. For example, when the teacher explains the meaning of a new word, the camera captures the expressions of interest or confusion of a particular student.

[0777] 2. Data analysis and understanding assessment

[0778] The server stores these video data, and the artificial intelligence model means and emotion engine analyze the students' facial expressions and emotions, thereby determining whether a particular student has difficulty understanding new words.

[0779] 3. Submitting and completing individual assignments

[0780] Students can receive personalized review tasks on their devices that take into account their level of comprehension and emotional state, such as quiz-style tasks to practice the meaning of new words.

[0781] 4. Feedback on the results and reflecting them in lesson plans

[0782] The server receives the students' answers and re-evaluates them. A class-wide comprehension report is generated and sent to the teacher's device, allowing the teacher to determine which words and expressions need additional explanation in the next lesson.

[0783] In this way, the present invention utilizes AI and emotion recognition technology to provide optimal learning support tailored to each student's level of understanding and emotional state, thereby reducing the burden on teachers while increasing student motivation and creating an effective educational environment.

[0784] The processing flow will be explained below.

[0785] Step 1:

[0786] At the start of a lesson, the server checks whether the first camera means (camera 1) for recording the lesson content and the second camera means (camera 2) for recording the students' reactions are running normally.

[0787] Step 2:

[0788] Camera 1 records the teacher's lesson content in real time and transmits the video data in streaming format to the server.

[0789] Step 3:

[0790] Camera 2 records the facial expressions and behavior of the entire classroom and each student in real time, and transmits the video data in streaming format to a server.

[0791] Step 4:

[0792] The server stores the received video data in real time and monitors the video quality and frame rate.

[0793] Step 5:

[0794] The AI ​​model analyzes the video data of lesson content and student responses stored on the server. Specifically, it uses a facial recognition algorithm to identify students' facial expressions and behaviors and score each student's level of understanding.

[0795] Step 6:

[0796] The emotion engine recognizes students' emotional states (e.g., enjoyment, interest, confusion, fatigue) in real time based on the students' facial expression data acquired by the artificial intelligence model means, and the analysis results are reflected as additional information in the comprehension score.

[0797] Step 7:

[0798] The server stores the comprehension scores and emotion data generated by the artificial intelligence model means and emotion engine, and lists the learning needs of each student.

[0799] Step 8:

[0800] The task generation unit generates optimal preparation, review, and practice tasks for each student based on the stored comprehension scores and emotional data. In particular, tasks that increase the student's learning motivation are generated taking into account the student's emotional state.

[0801] Step 9:

[0802] The transmission means transmits the generated individual assignments to the terminals of the students.

[0803] Step 10:

[0804] The student's device displays the received assignment and provides an interface through which the student can complete the assignment.

[0805] Step 11:

[0806] Students answer the assignments and send the results data from their devices to the server.

[0807] Step 12:

[0808] The server reassessed the students' understanding based on the received assignment results and generated a comprehension report for the entire class.

[0809] Step 13:

[0810] The report generating means transmits the generated comprehension report to the teacher's terminal.

[0811] Step 14:

[0812] The teacher's device will display a comprehension report, allowing the teacher to understand the learning status of the entire class.

[0813] Step 15:

[0814] Based on the generated comprehension reports and student emotion data, the teacher can decide what to cover and what to focus on in the next lesson, for example, by planning to explain in detail a topic that many students found confusing.

[0815] Through this series of processing steps, the present invention combines AI technology and emotion recognition technology to realize a system that provides optimal learning support to each student while reducing the burden on teachers.

[0816] Example 2

[0817] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0818] Conventional educational systems lack efficiency and accuracy in recording lesson content, understanding student responses, and providing individualized instruction, making it difficult to provide individual learning support tailored to each student's level of understanding and emotional state. This also places a heavy burden on teachers, making it difficult to grasp the overall class level of understanding in real time and reflect this in the next lesson plan.

[0819] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a first imaging means for recording the content of the lesson, a second imaging means for recording the reactions of the learners, an information processing means for receiving and storing video data from the first imaging means and the second imaging means, an inference engine means for analyzing the video data stored by the information processing means and evaluating the level of understanding of the learners, a learning material generation means for generating individual learning materials based on the level of understanding evaluated by the inference engine means, a communication means for transmitting the learning materials generated by the learning material generation means to a learner terminal means, a display means for displaying the learning materials received by the learner terminal means and having the learner execute them, a communication means for transmitting the results of the learning materials executed by the learner terminal means to an information processing means, a report generation means for reevaluating based on the results received by the information processing means and generating a comprehension report for the entire group, and a communication means for transmitting the comprehension report generated by the report generation means to an educator terminal means. This will enable the provision of optimal learning support in real time based on each student's level of understanding and emotional state, reducing the burden on teachers while creating an effective educational environment.

[0820] "Capturing means" is a device for acquiring video data.

[0821] An "information processing means" is a device or system that receives, stores, and analyzes the captured video data.

[0822] "Inference engine means" refers to an algorithm or software for analyzing stored video data and assessing the learner's level of comprehension and emotional state.

[0823] The "learning material generating means" is a device or software for generating individual learning materials based on the evaluation results by the inference engine means.

[0824] A "communication means" is an interface for sending and receiving data, and is used to communicate via a network or data link.

[0825] "Student terminal means" refers to a terminal device used by a student, which displays learning materials and accepts input of answers.

[0826] The "display means" refers to a display or monitor for displaying learning materials on the learner terminal means.

[0827] The "report generation means" is a device or software for generating a comprehension report based on the learner's answers.

[0828] The "educator terminal means" is a terminal device used by an educator, which receives and displays the generated comprehension report.

[0829] The educational support system related to this invention provides individualized learning support based on the student's level of understanding and emotional state. This system requires a wide range of hardware and software to comprehensively record lesson content, record student responses, analyze data, generate individualized learning materials, provide feedback on the results, and generate reports.

[0830] Recording lesson content and student responses

[0831] The system is equipped with a first imaging means for recording the content of the lesson and a second imaging means for recording the reactions of students. High-resolution cameras are used for these imaging means. As a specific example, the first imaging means is a fixed camera installed at the front of the classroom, and the second imaging means is a wide-angle camera that can capture the entire student body. When the lesson begins, the server confirms that these cameras are activated and operating normally, and begins acquiring video data in real time.

[0832] Video data storage and preparation

[0833] The server receives the video data sent from the camera in real time and stores it in a database. At the same time, it checks the video quality and frame rate and stores the data in a format suitable for analysis.

[0834] Data analysis and understanding assessment

[0835] The server transmits the stored video data to an inference engine means. This inference engine means incorporates an artificial intelligence model including a facial recognition algorithm. The artificial intelligence model analyzes the students' facial expressions and behavior and quantifies their level of understanding. It also incorporates an emotion recognition engine to determine their emotional state, recognizing their emotional states such as interest, confusion, enjoyment, and fatigue in real time. Based on these results, each student's level of understanding is evaluated.

[0836] Creation and delivery of personalized learning materials

[0837] The learning material generation means generates optimal learning materials based on the data obtained from the inference engine means and the emotion recognition engine. For example, if a student is having difficulty understanding a new English word, a quiz-style assignment for vocabulary practice is generated for that student. The generated learning materials are transmitted to the learner terminal means of each student via the communication means.

[0838] Assignment and feedback of results

[0839] The learner's device displays the received learning materials and provides an interface for the learner to execute them. When the student completes the assignment, the answer is sent from the device to the server. The server re-evaluates the received answer and re-evaluates the overall class and individual comprehension.

[0840] Generate reports and provide feedback to teachers

[0841] The server generates a comprehension report for the entire group based on the answer results. The report generation means creates a detailed report and visualizes each student's strengths and weaknesses. The generated comprehension report is sent to the educator terminal means, where it can be viewed by the teacher. This allows the teacher to grasp the progress of the entire class and provide any additional explanations or individual guidance needed in the next lesson.

[0842] Specific examples

[0843] Prompt Sentence Examples

[0844] "We're starting English class. I'll explain some new words."

[0845] "Take the following quiz-style challenge. Choose the meaning of a new word."

[0846] "Student A seems confused about the meaning of new words."

[0847] "Further explanation will be required in the next class."

[0848] This allows the educational support system to utilize AI and emotion recognition technology to provide optimal learning support in real time according to each student's level of understanding and emotional state, reducing the burden on teachers and creating an effective educational environment.

[0849] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0850] Step 1: Record lesson content and student responses

[0851] 1.1 The server confirms that Camera 1 (for recording lesson content) and Camera 2 (for recording student responses) are running normally.

[0852] (Input): Camera startup status

[0853] (Output): Message confirming successful camera startup

[0854] 1.2 When the lesson begins, Camera 1 records the teacher's lesson content in real time.

[0855] (Input): Lesson start trigger

[0856] (Output): Video data of the teacher's lesson

[0857] 1.3 Camera 2 records students' reactions (facial expressions and behavior) in real time.

[0858] (Input): Lesson start trigger

[0859] (Output): Video data of students' reactions

[0860] 1.4 The video data acquired from Camera 1 and Camera 2 is sent to the server.

[0861] (Input): Video data of teacher's lesson content and students' responses

[0862] (Output): Message that video data has been sent to the server

[0863] Step 2: Saving and preparing your footage

[0864] 2.1 The server receives and stores the video data transmitted from Camera 1 and Camera 2 in real time.

[0865] (Input): Video data of teacher's lesson content and students' responses

[0866] (Output): Message confirming saving to database

[0867] 2.2 The server checks the video quality and frame rate of the stored data.

[0868] (Input): Saved video data

[0869] (Output): Video quality check results

[0870] Step 3: Analyze data and assess comprehension

[0871] 3.1 The server transmits the stored video data to the inference engine means.

[0872] (Input): Saved video data

[0873] (Output): Message that data has been sent to the inference engine

[0874] 3.2 The inference engine means uses a facial recognition algorithm to analyze the student's facial expressions and behavior.

[0875] (Input): Student reaction video data

[0876] (Output): Student facial expression and behavior analysis results

[0877] 3.3 The server sends the analysis results received from the inference engine means to the emotion engine to determine the emotional state of the student.

[0878] (Input): Facial expression and behavior analysis results

[0879] (Output): Student's emotional state assessment result

[0880] Step 4: Generate and send personalized learning materials

[0881] 4.1 The learning material generation means generates optimal learning materials based on the data obtained from the inference engine means and the emotion engine.

[0882] (Input): Comprehension and emotional state data

[0883] (Output): Individual learning materials

[0884] 4.2 The generated learning materials are transmitted to the learner terminal means of each student via communication means.

[0885] (Input): Individual learning materials

[0886] (Output): Message that the teaching materials have been sent to the student's device

[0887] Step 5: Implementing the task and providing feedback

[0888] 5.1 The learner's device displays the transmitted learning materials.

[0889] (Input): Individual learning materials

[0890] (Output): Display of learning materials

[0891] 5.2 Students work on the assignment and send their answers from their devices to the server.

[0892] (Input): Student's answer result

[0893] (Output): Message that the answer result has been sent to the server

[0894] 5.3 The server will re-evaluate the received answer results and re-evaluate the overall class and individual comprehension.

[0895] (Input): Student's answer result

[0896] (Output): Reassessment results (understanding data)

[0897] Step 6: Generate reports and provide feedback to teachers

[0898] 6.1 The server generates a group-wide comprehension report based on the answers.

[0899] (Input): Reassessment results (understanding data)

[0900] (Output): Comprehension report

[0901] 6.2 The report generator will produce detailed reports and visualize each student's strengths and weaknesses.

[0902] (Input): Comprehension Report

[0903] (Output):Detailed report

[0904] 6.3 The generated comprehension report will be sent to the educator terminal means so that the teacher can review it.

[0905] (Input):Detailed Report

[0906] (Output): Report sending completion message to teacher's device

[0907] Through the above processing steps, this system is able to efficiently record and analyze lesson content and student responses in real time, providing optimal learning support to each individual student.

[0908] (Application example 2)

[0909] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0910] Conventional educational support systems have difficulty in properly assessing students' understanding and learning progress, and in particular, they have been unable to provide sufficient learning support that takes into account the emotional state of each student. This can result in a lack of detailed learning support for each student, which can reduce learning effectiveness. Another issue is that teachers are unable to receive effective feedback when grasping the overall understanding level of the class and reflecting this in their next lesson plan.

[0911] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means: In this invention, the server includes a first image capturing device means for recording the content of the lesson, a second image capturing device means for recording the reactions of students, an information processing device means for receiving and storing video information from the first image capturing device and the second image capturing device, a machine learning model device means for analyzing the video information stored by the information processing device and evaluating the level of understanding of the students, an assignment generation device means for generating individual assignments based on the level of understanding evaluated by the machine learning model device, a communication device means for transmitting the assignments generated by the assignment generation device to student terminal devices, a display device means for displaying the assignments received by the student terminal devices and having the students perform them, a communication device means for transmitting the results of the assignments performed by the student terminal devices to the information processing device, a report generation device means for re-evaluating the results received by the information processing device and generating a comprehension report for the entire class, and a communication device means for transmitting the comprehension report generated by the report generation device to a teacher terminal device. This not only makes it possible to create individual tasks based on students' level of understanding and provide learning support that takes into account students' emotional state, but also allows the level of understanding of the entire class to be grasped, providing effective feedback that teachers can reflect in their next lesson plans.

[0912] The "first imaging device for recording the content of the lesson" is a video device for recording the progress of the lesson, and is a camera device for capturing the entire lesson, including what the teacher is saying and what he is writing on the blackboard.

[0913] The "second imaging device for recording students' reactions" is a video device for recording students' facial expressions and behavior, and a camera device for monitoring how students are reacting to lessons.

[0914] An "information processing device" is a computer system for receiving and storing video information that records lesson content and student responses, and is a device that analyzes and stores video data.

[0915] A "machine learning model device" is a data processing device that analyzes video data stored in an information processing device and evaluates students' level of understanding, and is an artificial intelligence device that has algorithms for facial recognition and behavioral analysis.

[0916] The "task generation device" is a device that automatically generates individual tasks based on the level of understanding assessed by the machine learning model device, and is a device for providing learning content adapted to each student.

[0917] The "communication device" is a network device for transmitting and receiving data between the information processing device, student terminal device, and teacher terminal device, and is a device for transmitting assignments and receiving results.

[0918] A "student terminal" is a device that receives and allows a student to complete assignments, and provides an interface for displaying and answering individual assignments.

[0919] The "display device" is a device for showing the assignment received by the student terminal device to the student, and is a monitor or screen for visually displaying the content of the assignment.

[0920] A "report generation device" is a device that compiles the level of understanding of the entire class based on the results of the assignments and creates a detailed report, generating materials that allow teachers to understand the learning situation of their class.

[0921] An "emotion recognition engine" is an algorithm that analyzes students' emotional state from their facial expressions and behavior and evaluates their motivation to learn, and is a technology that recognizes students' emotional changes in real time.

[0922] The present invention relates to an educational support system that analyzes a student's level of understanding and emotional state and provides individual learning support. The detailed configuration and operation of this system will be described below.

[0923] System configuration

[0924] This educational support system consists of the following main components:

[0925] 1. First imaging device

[0926] 2. Secondary imaging device

[0927] 3. Information Processing Device

[0928] 4. Machine Learning Model Device

[0929] 5. Task generation device

[0930] 6. Communications Equipment

[0931] 7. Student terminal devices

[0932] 8. Display device

[0933] 9. Report Generator

[0934] 10. Emotion Recognition Engine

[0935] System Operation

[0936] 1. Recording lesson content and student responses

[0937] When the lesson begins, the first camera records the teacher's lesson content in real time and transmits the video data to the information processing device. At the same time, the second camera records the students' expressions and behaviors and transmits the video data to the information processing device.

[0938] 2. Video data storage and analysis

[0939] The information processing device stores the received video data for later analysis. The quality and frame rate of the video are also checked to ensure optimal conditions are maintained.

[0940] The machine learning model uses the stored video data to assess the student's understanding, using data processing techniques such as facial recognition algorithms.

[0941] 3. Emotional state analysis

[0942] The emotion recognition engine analyzes students' emotional states from their facial expressions and behaviors, allowing real-time assessment of their emotional states, such as enjoyment, interest, confusion, and fatigue.

[0943] 4. Generating individual tasks

[0944] The assignment generator generates optimal preparatory, review, and practice assignments for each student based on data from the machine learning model device and emotion recognition engine.

[0945] In particular, taking into account emotional states provides tasks that are tailored to students' motivation.

[0946] 5. Submit and complete the assignment

[0947] The generated assignment is sent to the student terminal via the communication device, which displays the received assignment and provides an interface for the student to work on it.

[0948] When the student answers the assignment, the results are transmitted to the information processing device via the communication device.

[0949] 6. Review and feedback of results

[0950] The information processing device performs a reassessment based on the received assignment results and generates a report on the level of understanding of the entire class. The report generating device creates this report and sends it to the teacher terminal device to provide feedback to the teacher.

[0951] Specific examples

[0952] For example, consider a situation where students are learning new words during an English class. A first camera records the teacher's lesson content, and a second camera records the students' responses. An information processing device stores and analyzes the video data. A machine learning model device and an emotion recognition engine are used to evaluate whether the students understand the meaning of the words and what their emotional state is. An assignment generation device then generates individual review assignments and sends them to the student terminal device via a communication device. The students answer the assignments on their terminals, and the results are sent back to the server for re-evaluation and feedback.

[0953] Prompt Sentence Examples

[0954] Prompts to assess student understanding:

[0955] Use a facial recognition algorithm to analyze Student A's facial expressions within the most recent lesson data and score his / her level of understanding.

[0956] Prompts for analyzing emotional states:

[0957] Based on Student B's video data, please evaluate his emotional state in real time, such as enjoyment, interest, confusion, and fatigue.

[0958] In this way, the present invention can maximize the effectiveness of education by providing individual learning support that takes into account not only the student's level of understanding of the lesson content but also their emotional state.

[0959] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0960] Step 1:

[0961] Recording of lesson content and student responses

[0962] At the start of a lesson, the server checks whether the first and second image capture devices are running properly, and is then ready to record the lesson content and students' responses.

[0963] Input: Lesson start command, activation status of first and second imaging devices

[0964] Operation: The first camera records the teacher's lesson in real time and sends the video data to a server. At the same time, the second camera records the students' expressions and behavior and sends this video data to the server.

[0965] Output: Video data of teacher's lesson content, video data of students' reactions

[0966] Step 2:

[0967] Video data storage and quality check

[0968] The server temporarily stores the received video data, checking the quality and frame rate of the video data to ensure it is in a state suitable for optimal analysis.

[0969] Input: Video data of lesson content, video data of student reactions

[0970] Operation: Saving video data and checking the frame rate and quality

[0971] Output: High-quality video data stored

[0972] Step 3:

[0973] Analysis of video data and assessment of comprehension

[0974] The server then passes the stored video data to a machine learning model device, which uses a facial recognition algorithm to analyze the student's facial expressions and behavior and generate a score based on their level of understanding.

[0975] Input: High-quality video data

[0976] Action: Analyzing students' facial expressions and behaviors and scoring their understanding using a face recognition algorithm based on a machine learning model device

[0977] Output: Student comprehension score

[0978] Step 4:

[0979] Emotional state analysis

[0980] The server then passes the stored video data to an emotion recognition engine, which analyzes the student's emotional state, assessing emotions such as enjoyment, interest, confusion, and fatigue.

[0981] Input: High-quality video data

[0982] Action: Analyzing students' emotional states using an emotion recognition engine

[0983] Output: Student emotional state data

[0984] Step 5:

[0985] Generate individual assignments

[0986] The task generation device generates optimal preparation, review, and practice tasks for each student based on their comprehension scores and emotional state data.

[0987] Input: Student comprehension scores, student emotional state data

[0988] Action: Individual task generation according to comprehension level and emotional state

[0989] Output: Individual issue data

[0990] Step 6:

[0991] Submitting and viewing assignments

[0992] The created assignment is sent to the student terminal via the communication device, and the student terminal receives the assignment and displays it to the student.

[0993] Input: Individual issue data

[0994] Operation: Sending assignment data and displaying it on student terminal devices

[0995] Output: The displayed issues

[0996] Step 7:

[0997] Implementing the assignment and collecting the results

[0998] Students answer questions using their student devices and send the results to the server, which then receives the results.

[0999] Input: Viewed assignment

[1000] Operation: Student terminal device performs assignments, inputs results, and transmits results via communication device

[1001] Output: Student assignment answer result data

[1002] Step 8:

[1003] Review of results and feedback

[1004] The server analyzes the received answers and evaluates the level of understanding and areas for improvement. The report generator creates a report on the level of understanding of the entire class and sends it to the teacher's terminal device via the communication device.

[1005] Input: Student assignment answer result data

[1006] Actions: Analyze the results data, reassess the level of understanding, generate and send a feedback report

[1007] Output: Whole class comprehension report

[1008] Prompt Sentence Examples

[1009] "Use a facial recognition algorithm to analyze Student A's facial expressions within the most recent lesson data and score their level of understanding."

[1010] "Based on Student B's video data, please rate his emotional state in real time, such as enjoyment, interest, confusion, and fatigue."

[1011] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1012] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1013] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1014] [Third embodiment]

[1015] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1016] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1018] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1019] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1020] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1022] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1023] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1025] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1026] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1027] The present invention relates to an educational support system that can effectively record lesson content and grasp students' understanding in real time. This system is composed of multiple cameras, a server, an artificial intelligence model, an assignment generation unit, a transmission unit, student terminals, a display unit, a report generation unit, and a teacher terminal.

[1028] System program processing and explanation

[1029] Record of lesson content

[1030] At the start of a lesson, the server checks whether the first camera means (Camera 1) that records the lesson content and the second camera means (Camera 2) that records the students' reactions are running normally. When the lesson starts, Camera 1 records the teacher's lesson content in real time and sends the video data to the server. At the same time, Camera 2 records the expressions and actions of all the students in real time and also sends this video data to the server.

[1031] Video data storage and preparation

[1032] The server stores the video data of the lesson content and students' reactions received from the camera means in real time. This stored data will be used for later analysis, so it also checks the video quality and frame rate.

[1033] Data analysis and understanding assessment

[1034] The AI ​​model analyzes the video data stored on the server. A facial recognition algorithm is used to evaluate the student's level of understanding based on their facial expressions and behavior. The evaluation results are saved as a score and listed for each student.

[1035] Creating and submitting individual assignments

[1036] The assignment generation means generates preparation, review, and practice assignments suitable for each student based on the results of the assessment of comprehension by the artificial intelligence model means. The generated assignments are sent to each student's terminal via the transmission means.

[1037] Assignment and feedback of results

[1038] The student's device displays the received assignments and provides an interface for the student to complete them. After the student completes the assignment, the results are sent from the device to the server. The server reassessssments based on the received assignment results and generates a report on the overall class comprehension. This report is used by the teacher to optimize lesson plans.

[1039] Generate reports and provide feedback to teachers

[1040] The report generation tool compiles a summary of the class's overall understanding based on the results of the assignments and creates a detailed report. This report visualizes each student's strengths and weaknesses. The teacher's device receives and displays this report, allowing the teacher to grasp the overall situation in the class and plan appropriate measures for the next lesson.

[1041] Specific examples

[1042] 1. Conducting and recording lessons

[1043] The user (teacher) starts a math lesson. Camera 1 records the teacher's lesson content, and Camera 2 records the students' reactions. For example, when the teacher is explaining the basics of equations, Student A looks confused.

[1044] 2. Data analysis and understanding assessment

[1045] The server stores this video data, and the artificial intelligence model means judges the low level of understanding from the facial expression of Student A. Based on this, an assignment is created for Student A to review basic equation problems.

[1046] 3. Submitting and completing individual assignments

[1047] The assignment is sent to and displayed on Student A's device. Student A answers the assignment and sends the results to the server.

[1048] 4. Feedback on the results and reflecting them in lesson plans

[1049] The server receives Student A's answer and re-evaluates it. A report of the class's understanding is generated and sent to the teacher's terminal. Based on this, the teacher decides to allocate time to the basics of equations in the next lesson.

[1050] In this way, the present invention utilizes AI technology to reduce the burden on teachers while providing learning support that is tailored to each student.

[1051] The processing flow will be explained below.

[1052] Step 1:

[1053] At the start of a lesson, the server checks whether the first camera means (camera 1) for recording the lesson content and the second camera means (camera 2) for recording the students' reactions are running normally.

[1054] Step 2:

[1055] Camera 1 records the teacher's lesson content in real time and transmits the video data in streaming format to the server.

[1056] Step 3:

[1057] Camera 2 records the facial expressions and behavior of the entire classroom and each student in real time, and transmits the video data in streaming format to a server.

[1058] Step 4:

[1059] The server stores the received video data in real time and monitors the video quality and frame rate.

[1060] Step 5:

[1061] The AI ​​model analyzes the video data of the lesson content and students' reactions stored on the server. Specifically, it uses a facial recognition algorithm to identify students' facial expressions and behaviors and score each student's level of understanding.

[1062] Step 6:

[1063] The server stores the comprehension scores generated by the artificial intelligence model means and lists the learning needs of each student.

[1064] Step 7:

[1065] The assignment generation means generates optimal preparation, review, and practice assignments for each student based on the stored comprehension scores.

[1066] Step 8:

[1067] The transmission means transmits the generated individual assignments to the terminals of the students.

[1068] Step 9:

[1069] The student's device displays the received assignment and provides an interface through which the student can complete the assignment.

[1070] Step 10:

[1071] Students answer the assignments and send the results data from their devices to the server.

[1072] Step 11:

[1073] The server reassessed the students' understanding based on the received assignment results and generated a comprehension report for the entire class.

[1074] Step 12:

[1075] The report generating means transmits the generated comprehension report to the teacher's terminal.

[1076] Step 13:

[1077] The teacher's device will display a comprehension report, allowing the teacher to understand the learning status of the entire class.

[1078] Step 14:

[1079] Based on the comprehension reports, teachers decide what content and focus to cover in the next lesson.

[1080] Through this series of processing steps, the present invention realizes a system that provides learning support suited to each individual student and more effectively supports teachers' lesson plans.

[1081] Example 1

[1082] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1083] Conventional educational support systems have had difficulty accurately assessing each student's level of understanding in real time and providing individualized learning support. They also have limitations in comprehensively recording lesson content and effectively utilizing that data. This makes it difficult for teachers to grasp students' levels of understanding, leading to problems with efficient instruction.

[1084] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1085] In this invention, the server includes a first recording means for recording the content of the lesson, a second recording means for recording the responses of students, a data storage means for receiving and storing video data from the first recording means and the second recording means, an analysis means for analyzing the video data stored by the data storage means and evaluating the students' levels of comprehension, an assignment creation means for generating individual assignments based on the levels of comprehension evaluated by the analysis means, a transmission means for transmitting the assignments generated by the assignment creation means to student terminals, a display means for displaying the assignments received by the student terminals and having the students complete them, a transmission means for transmitting the results of the assignments completed by the student terminals to the data storage means, a report creation means for re-evaluating the results received by the data storage means and generating a report on the level of comprehension of the entire class, and a transmission means for transmitting the comprehension report generated by the report creation means to a teacher's terminal. This enables comprehensive recording of the content of the lesson and real-time evaluation and feedback of students' levels of comprehension.

[1086] The "primary recording means for recording the content of the class" is a device that records the lecture content and explanations of the instructor in charge of the class as video and audio.

[1087] The "second recording means for recording student responses" is a device that records students' facial expressions and behavior during class in real time as video.

[1088] The "data storage means" is a system or device for receiving the video data transmitted from the first recording means and the second recording means and storing the data for a long period of time.

[1089] The "analysis means" is a system that includes algorithms and software for analyzing the stored video data and assessing the student's level of understanding.

[1090] The "task creation means" is a system or device for automatically creating individual learning tasks based on the student's level of understanding evaluated by the analysis means.

[1091] The "transmission means" is a system or device for transmitting the generated assignments and comprehension reports to a specified terminal.

[1092] A "student device" is a device such as a computer or tablet used by a student that provides an interface for displaying and completing received assignments.

[1093] The "display means" is software or a system that displays the assignment received on the student terminal and provides an interface for the student to answer the assignment.

[1094] The "report creation means" is a system that compiles the level of understanding of the entire class based on the results of assignments sent from student devices and generates a comprehension report for the teacher.

[1095] A "teacher's terminal" is a device such as a computer or tablet used by a teacher to receive and display comprehension reports.

[1096] The present invention relates to an educational support system that can effectively record lesson content and grasp students' understanding in real time. This system is composed of multiple recording means, data storage means, analysis means, assignment creation means, transmission means, student terminals, display means, report creation means, and teacher terminals.

[1097] Hardware and Software Configuration

[1098] Specific hardware used includes camera devices (first and second recording means), servers (data storage means and analysis means), student tablets or PCs (student terminals), and faculty tablets or PCs (faculty terminals).Software includes video analysis algorithms (face recognition algorithms and facial expression analysis algorithms), AI models, assignment generation algorithms, and database management systems.

[1099] Data collection and storage

[1100] At the start of class, the server checks whether the first recording means (Camera 1), which records the teacher's lecture content, and the second recording means (Camera 2), which records the students' reactions, are operating normally. When class begins, Camera 1 records the teacher's lecture in real time and sends the data to the server. Camera 2 simultaneously records the expressions and movements of all the students and also sends this data to the server. The data storage means saves the received video data to a highly durable storage device.

[1101] Data analysis and understanding assessment

[1102] The saved video data is processed by an analysis means. The analysis means uses a facial recognition algorithm and an expression analysis algorithm to evaluate the student's level of comprehension based on their facial expressions and movements. This evaluation is quantified and recorded as a comprehension score for each student. The analysis results are saved in a database and used for subsequent processing.

[1103] Creating and submitting assignments

[1104] The assignment creation means generates individual preparation, review, and practice assignments based on the comprehension score obtained from the analysis means. The generated assignments are sent to each student's student device via the transmission means. For example, mathematics practice problems created using the generative AI model are distributed to student devices.

[1105] Assignment and feedback of results

[1106] The student terminals display the received assignments and provide an interface for students to answer. When students complete the assignments, their answers are automatically sent to the server. The server reevaluates the received answers and creates a report based on the overall class understanding. This report is generated by the report creation means and sent to the teacher terminal.

[1107] Specific examples

[1108] 1. Lesson Conduct and Recording:

[1109] A user (teacher) starts a math class at 9:00 a.m. Camera 1 records the teacher's lecture, and Camera 2 records the students' reactions. For example, when the teacher is explaining the basics of equations, Student A looks confused.

[1110] 2. Data analysis and comprehension assessment:

[1111] The server stores this video data, and the analysis means determines whether Student A has a low level of understanding based on his / her facial expressions, and records the result as a score. Data is generated to provide Student A with review assignments.

[1112] 3. Submit and complete the assignment:

[1113] The assignment creation means generates a review assignment suitable for Student A and sends it to Student A's student terminal. Student A's terminal displays the review math problems.

[1114] 4. Feedback on results and reflection on next class:

[1115] The server receives Student A's answers and re-evaluates them. A comprehension report is sent to the teacher's terminal, and the teacher uses it to plan the next lesson.

[1116] Prompt Sentence Examples

[1117] Below are some examples of prompts to give instructions to an AI generative model.

[1118] "To determine whether students are understanding during class, consider the following scenario: Camera 1 records a teacher teaching the basics of equations in a math class, while simultaneously recording the students' reactions with Camera 2. If it is determined that Student A has a confused expression, generate appropriate review assignments for Student A based on the results of the facial expression analysis."

[1119] In this way, the present invention utilizes AI technology to reduce the burden on teachers while providing learning support that is tailored to each student.

[1120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1121] Step 1: Start the lesson and check that the equipment is up and running

[1122] At the designated time, the server checks whether the lesson is about to begin. Here, it checks whether the first recording means (camera 1) and the second recording means (camera 2) are operating normally. Specifically, the server sends a PING command to each camera and monitors whether there is a response. If a normal response is confirmed, it determines that the lesson is ready to begin.

[1123] Input: lesson start time, camera status

[1124] Output: Confirmation that the device has started up normally

[1125] Step 2: Record lesson content and student reactions

[1126] When the class begins, the server has Camera 1 record the teacher's lecture in real time and transmits the video data to the server. At the same time, Camera 2 records the reactions of all students and transmits this to the server. The video data being recorded is received in real time and saved by the data saving means.

[1127] Input: Teacher lecture video, student reaction video

[1128] Output: Save real-time video data to the server

[1129] Step 3: Save the video data and check its quality

[1130] The server stores the received video data in a data storage device. At the same time, it checks the quality of the video data. Specifically, it automatically checks whether there are any missing frames and whether the audio data is synchronized. The results of this check are also saved and used to ensure the quality of the data.

[1131] Input: Received video data

[1132] Output: Saved video data, quality check results

[1133] Step 4: Analyze data and assess comprehension

[1134] The saved video data is processed by an analytical device. Specifically, facial recognition algorithms and facial expression analysis algorithms are used to quantify the student's level of comprehension based on their facial expressions and movements. The results of this analysis are recorded as a comprehension score for each student.

[1135] Input: Saved video data

[1136] Output: Comprehension score

[1137] Step 5: Generate assignments and send them to students

[1138] The assignment creation means generates individual assignments based on the comprehension scores. The generated assignments are sent to each student's student device via the transmission means. Based on the generation AI model, more basic questions are provided to students with low scores, and applied questions are provided to students with high scores.

[1139] Input: Comprehension score

[1140] Output: Individual assignment, Submit assignment

[1141] Step 6: Complete and submit the assignment

[1142] The student terminals display the received assignments and provide an interface for students to answer. When students complete the assignments, the answers are automatically sent to the server.

[1143] Input: Received assignment

[1144] Output: Send answer results

[1145] Step 7: Reassess results and provide feedback to teachers

[1146] The server reevaluates the received answers and reassess the level of understanding. Based on the reevaluation results, a report on the level of understanding of the entire class is generated. The report generation means creates this report and sends it to the teacher's terminal.

[1147] Input: Answer result

[1148] Output: Reassessed comprehension, Generate comprehension report

[1149] Step 8: Reflect on the next lesson plan

[1150] The teacher's device displays the received comprehension report and uses it as reference for planning the next lesson. Teachers can identify each student's strengths and weaknesses and optimize their lesson plans.

[1151] Input: Comprehension Report

[1152] Output: Optimized lesson plans

[1153] (Application example 1)

[1154] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1155] Modern education and retail operations require support tailored to diverse individual needs and levels of understanding. In educational environments, it is particularly important to assess each student's level of understanding in real time and provide appropriate learning content based on the results. Meanwhile, in brick-and-mortar stores, analyzing customer behavior and facial expressions and making appropriate product recommendations can be expected to boost purchasing motivation. Current systems face the challenge of being unable to efficiently respond to such diverse demands.

[1156] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1157] In this invention, the server includes a first image acquisition means for recording lesson content, a second image acquisition means for recording student responses, a data storage means for receiving and storing video data from the first image acquisition means and the second image acquisition means, an intelligence assessment means for analyzing the video data stored by the data storage means and evaluating the student's level of comprehension, a learning content generation means for generating individual learning content based on the level of comprehension assessed by the intelligence assessment means, a data transmission means for transmitting the learning content generated by the learning content generation means to a receiving device, a display device for displaying the learning content received by the receiving device and having the student execute it, a data transmission means for transmitting the results of the learning content executed by the receiving device to the data storage means, a report generation means for re-evaluating the results received by the data storage means and generating an overall comprehension report, and a data transmission means for transmitting the comprehension report generated by the report generation means to a reporting device. This enables support to be provided according to individual needs and levels of comprehension in educational and retail environments.

[1158] "Class content" is a term that refers to the content and themes that teachers teach to students in educational activities.

[1159] "Student reactions" is a term that refers to various behaviors that students show during class, such as facial expressions, movements, and actions, that indicate their understanding and interest in what they are learning.

[1160] "Image capture means" is a term that refers to an apparatus or system for capturing video data using a device such as a camera.

[1161] "Data storage means" is a term that refers to a device or system for temporarily or permanently storing acquired video data, analysis results, etc.

[1162] "Intelligence assessment means" is a term used to refer to a device or system that uses artificial intelligence to analyze data obtained and assess an individual's level of understanding and emotions.

[1163] "Learning content generation means" is a term that refers to a device or system for generating learning content optimized for each student based on the evaluation results from the intelligence evaluation means.

[1164] "Data transmission means" is a term that refers to an apparatus or system for transmitting generated learning content, analysis results, reports, etc. to a specific terminal or device.

[1165] "Receiving device" is a term that refers to a device or system for receiving data transmitted from a data transmitting means.

[1166] "Display device" is a term that refers to a device or system that displays data and learning content received by a receiving device so that the user can visually confirm it.

[1167] "Report generator" is a term that refers to a device or system for generating a comprehensive report based on intelligence assessment measures and learning results.

[1168] "Reporting device" is a term that refers to a device or system that ultimately reports generated reports or other information to a user.

[1169] To put this invention into practice, it is necessary to build systems for both education and retail use, and to be able to meet the needs of each environment. Specific examples of such systems are shown below.

[1170] System Overview

[1171] The system mainly includes the following components:

[1172] First image capture method (cameras that record lesson content and customer behavior)

[1173] Secondary image capture means (cameras that record students' reactions and customers' facial expressions)

[1174] Data storage means (server)

[1175] Intelligence evaluation method (analysis system using artificial intelligence model)

[1176] Learning content generation means (a system that generates individual tasks or product proposals)

[1177] Data transmission method (transmission server)

[1178] Receiving device (student device, store display, smartphone)

[1179] Display devices (terminals and displays)

[1180] Report generation means (systems that generate class-wide or customer behavior reports)

[1181] Reporting device (teacher terminal or store management system)

[1182] Hardware and Software

[1183] Hardware:

[1184] Camera (e.g. USB camera)

[1185] Server (data storage and analysis)

[1186] Smartphones, tablets, store displays, robots

[1187] software:

[1188] TensorFlow (a deep learning model for analyzing customer facial expressions)

[1189] OpenCV (acquisition and display of camera images)

[1190] REST API (data sending and receiving)

[1191] Data processing and calculation

[1192] The server processes and calculates the data in the following steps:

[1193] 1. Data Acquisition:

[1194] The first image acquisition means records the contents of the lesson and the behavior of the customer, and transmits the recorded images to the data storage means in real time.

[1195] A second image capturing means records the reactions of the students and the facial expressions of the customers and transmits them to the data storage means as well.

[1196] 2. Analysis and Evaluation:

[1197] The video data stored in the data storage means is analyzed using an intelligent evaluation means (e.g., an artificial intelligence model equipped with a face recognition algorithm) to evaluate the students' understanding and the customers' emotions.

[1198] 3. Data transmission:

[1199] Based on the results of the evaluation by the intelligence evaluation means, the learning content generation means generates appropriate assignments and product suggestions and transmits them to the receiving device.

[1200] 4. Interaction:

[1201] The assignments and product proposals received by the receiving device are displayed on the display device and are carried out by the students or customers.

[1202] 5. Feedback of results:

[1203] The results of the execution on the display device are again transmitted to the data storage means by the data transmission means, and the report generation means generates an understanding level report and a customer behavior report.

[1204] Specific examples

[1205] 1. Educational environment:

[1206] Consider a scenario in which a teacher begins a lesson. A first image acquisition means records the content of the lesson, and a second image acquisition means records the students' responses. An AI model analyzes the students' facial expressions, and if a particular student is struggling to understand, it sends that student an individual assignment. This assignment is displayed on the student's device, and the student answers it. The answers are sent to a server, and the teacher receives a report of the class's understanding on his or her device.

[1207] 2. Store environment:

[1208] Consider a scenario in which a customer enters a store. A first image capture means records the customer's behavior, and a second image capture means analyzes the customer's facial expressions. When the server detects the customer's expressions of interest or confusion, it generates an appropriate product suggestion and sends it to an in-store display or the customer's smartphone. An in-store robot interacts with the customer based on the suggestion (e.g., "Would you like this product?"). Ultimately, store staff optimize their response based on the report generated by the server.

[1209] Prompt Sentence Examples

[1210] For example, if a customer appears unsure, the following prompts can be fed into the generative AI model:

[1211] The customer's expression shows uncertainty. Which of the following products do you predict will most interest the customer?

[1212] This will improve the customer experience and streamline store operations.

[1213] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1214] Step 1:

[1215] Data Acquisition

[1216] Input: Real-time video data acquired by the first image acquisition means (camera 1) and the second image acquisition means (camera 2)

[1217] Output: Video data of lesson content, customer behavior, student reactions, and customer expressions sent to the server

[1218] Specific operation: The device activates Camera 1 and records the entire classroom or store. It then activates Camera 2 and records the facial expressions and behavior of individual students or customers. The captured video data is sent to the server in real time.

[1219] Step 2:

[1220] Data storage

[1221] Input: Real-time video data sent to the server

[1222] Output: Video data stored in a data storage device

[1223] Specific operation: The server stores the received video data in a data storage means (e.g., a database), and also checks the video quality and frame rate to ensure the quality of the stored data.

[1224] Step 3:

[1225] Data analysis and evaluation

[1226] Input: Video data stored in a data storage device

[1227] Output: Comprehension and emotional evaluation scores from intelligence assessment tools

[1228] Specific operation: The server's intelligent evaluation means (for example, an artificial intelligence model using a facial recognition algorithm) analyzes the stored video data. It evaluates the student's understanding and the customer's emotions and quantifies them as a score. This score is then listed for each student or customer.

[1229] Step 4:

[1230] Generate learning content and product suggestions

[1231] Input: Intelligence assessment score

[1232] Output: Individual tasks and product suggestions generated by the learning content generation means

[1233] Specific operation: Based on the score obtained by the intelligence evaluation means, the server uses a generative AI model to generate learning content, which generates individual assignments, review questions, or product suggestions suitable for the customer.

[1234] Step 5:

[1235] Data transmission

[1236] Input: Generated individual issues and product proposals

[1237] Output: Learning content and product suggestions sent to receiving devices (student terminals, store displays, smartphones)

[1238] Specific operation: The server transmits the assignments and product suggestions generated by the learning content generation means to each receiving device using the data transmission means.

[1239] Step 6:

[1240] Interaction

[1241] Input: Issues and product proposals displayed on the receiving device

[1242] Output: Answers and feedback

[1243] Specific operation: The receiving device (e.g., student terminal, smartphone, in-store display) displays the received assignment or product suggestion. The user (student or customer) then operates accordingly, entering an answer in the case of an assignment, or making a selection or purchasing action in the case of a product suggestion.

[1244] Step 7:

[1245] Feedback of results

[1246] Input: Answers and feedback from the receiving device

[1247] Output: Answer results and behavioral data fed back to the server

[1248] Specific operation: The terminal transmits the results of the user's operations (e.g., answers to assignments and product selection information) to the server. The server receives this and records it in the data storage means.

[1249] Step 8:

[1250] Report generation and delivery

[1251] Input: Recorded answer results and behavioral data

[1252] Output: Comprehension reports and customer behavior reports generated by the report generation means, and reports sent to the reporting device

[1253] Specific operation: Based on the received answer results and behavioral data, the server creates a comprehension report and a customer behavior report using the report generation means. The generated report is sent to a reporting device (e.g., a teacher terminal or a store management system) and displayed.

[1254] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1255] The present invention relates to an educational support system that provides effective learning support to each student by recording lesson content and student responses in real time and recognizing the emotional state of the student. This system is composed of multiple camera means, server means, artificial intelligence model means, emotion engine, assignment generation means, transmission means, student terminal means, display means, report generation means, and teacher terminal means.

[1256] System program processing and explanation

[1257] Record of lesson content

[1258] At the start of a lesson, the server checks whether the first camera means (Camera 1) that records the lesson content and the second camera means (Camera 2) that records the students' reactions are running properly. When the lesson starts, Camera 1 records the teacher's lesson content in real time and sends the video data to the server. At the same time, Camera 2 records the expressions and actions of all the students in real time and also sends this video data to the server.

[1259] Video data storage and preparation

[1260] The server stores the video data of the lesson content and students' reactions received from the camera means in real time. This stored data will be used for later analysis, so it also checks the video quality and frame rate.

[1261] Data analysis and understanding assessment

[1262] The AI ​​model analyzes video data of lesson content and student responses stored on a server. A facial recognition algorithm is used to assess students' comprehension based on their facial expressions and behavior. Furthermore, an emotion engine is incorporated to analyze students' emotional states. This emotion engine recognizes emotional states such as enjoyment, interest, confusion, and fatigue in real time, and reflects this data in the assessment of comprehension.

[1263] Creating and submitting individual assignments

[1264] The assignment generation means generates optimal preparation, review, and practice assignments for each student based on the level of understanding and emotional state generated by the artificial intelligence model means and the emotion engine. In particular, it presents assignments that reflect the analysis results of the emotion engine and take learning motivation into consideration. The generated assignments are sent to each student's device via the transmission means.

[1265] Assignment and feedback of results

[1266] The student's device displays the received assignment and provides an interface for the student to complete it. After the student completes the assignment, the results are sent from the device to the server. The server then reassessssments the results and generates a report on the overall class comprehension. This report is used by the teacher to optimize lesson plans.

[1267] Generate reports and provide feedback to teachers

[1268] The report generation tool compiles a summary of the class's overall understanding based on the results of the assignments and creates a detailed report. This report visualizes each student's strengths and weaknesses. The teacher's device receives and displays this report, allowing the teacher to grasp the overall situation in the class and plan appropriate measures for the next lesson.

[1269] Specific examples

[1270] 1. Conducting and recording lessons

[1271] The user (teacher) begins an English lesson. Camera 1 records the teacher's lesson, and Camera 2 records the students' reactions. For example, when the teacher explains the meaning of a new word, the camera captures the expressions of interest or confusion of a particular student.

[1272] 2. Data analysis and understanding assessment

[1273] The server stores these video data, and the artificial intelligence model means and emotion engine analyze the students' facial expressions and emotions, thereby determining whether a particular student has difficulty understanding new words.

[1274] 3. Submitting and completing individual assignments

[1275] Students can receive personalized review tasks on their devices that take into account their level of comprehension and emotional state, such as quiz-style tasks to practice the meaning of new words.

[1276] 4. Feedback on the results and reflecting them in lesson plans

[1277] The server receives the students' answers and re-evaluates them. A class-wide comprehension report is generated and sent to the teacher's device, allowing the teacher to determine which words and expressions need additional explanation in the next lesson.

[1278] In this way, the present invention utilizes AI and emotion recognition technology to provide optimal learning support tailored to each student's level of understanding and emotional state, thereby reducing the burden on teachers while increasing student motivation and creating an effective educational environment.

[1279] The processing flow will be explained below.

[1280] Step 1:

[1281] At the start of a lesson, the server checks whether the first camera means (camera 1) for recording the lesson content and the second camera means (camera 2) for recording the students' reactions are running normally.

[1282] Step 2:

[1283] Camera 1 records the teacher's lesson content in real time and transmits the video data in streaming format to the server.

[1284] Step 3:

[1285] Camera 2 records the facial expressions and behavior of the entire classroom and each student in real time, and transmits the video data in streaming format to a server.

[1286] Step 4:

[1287] The server stores the received video data in real time and monitors the video quality and frame rate.

[1288] Step 5:

[1289] The AI ​​model analyzes the video data of lesson content and student responses stored on the server. Specifically, it uses a facial recognition algorithm to identify students' facial expressions and behaviors and score each student's level of understanding.

[1290] Step 6:

[1291] The emotion engine recognizes students' emotional states (e.g., enjoyment, interest, confusion, fatigue) in real time based on the students' facial expression data acquired by the artificial intelligence model means, and the analysis results are reflected as additional information in the comprehension score.

[1292] Step 7:

[1293] The server stores the comprehension scores and emotion data generated by the artificial intelligence model means and emotion engine, and lists the learning needs of each student.

[1294] Step 8:

[1295] The task generation unit generates optimal preparation, review, and practice tasks for each student based on the stored comprehension scores and emotional data. In particular, tasks that increase the student's learning motivation are generated taking into account the student's emotional state.

[1296] Step 9:

[1297] The transmission means transmits the generated individual assignments to the terminals of the students.

[1298] Step 10:

[1299] The student's device displays the received assignment and provides an interface through which the student can complete the assignment.

[1300] Step 11:

[1301] Students answer the assignments and send the results data from their devices to the server.

[1302] Step 12:

[1303] The server reassessed the students' understanding based on the received assignment results and generated a comprehension report for the entire class.

[1304] Step 13:

[1305] The report generating means transmits the generated comprehension report to the teacher's terminal.

[1306] Step 14:

[1307] The teacher's device will display a comprehension report, allowing the teacher to understand the learning status of the entire class.

[1308] Step 15:

[1309] Based on the generated comprehension reports and student emotion data, the teacher can decide what to cover and what to focus on in the next lesson, for example, by planning to explain in detail a topic that many students found confusing.

[1310] Through this series of processing steps, the present invention combines AI technology and emotion recognition technology to realize a system that provides optimal learning support to each student while reducing the burden on teachers.

[1311] Example 2

[1312] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1313] Conventional educational systems lack efficiency and accuracy in recording lesson content, understanding student responses, and providing individualized instruction, making it difficult to provide individual learning support tailored to each student's level of understanding and emotional state. This also places a heavy burden on teachers, making it difficult to grasp the overall class level of understanding in real time and reflect this in the next lesson plan.

[1314] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a first imaging means for recording the content of the lesson, a second imaging means for recording the reactions of the learners, an information processing means for receiving and storing video data from the first imaging means and the second imaging means, an inference engine means for analyzing the video data stored by the information processing means and evaluating the level of understanding of the learners, a learning material generation means for generating individual learning materials based on the level of understanding evaluated by the inference engine means, a communication means for transmitting the learning materials generated by the learning material generation means to a learner terminal means, a display means for displaying the learning materials received by the learner terminal means and having the learner execute them, a communication means for transmitting the results of the learning materials executed by the learner terminal means to an information processing means, a report generation means for reevaluating based on the results received by the information processing means and generating a comprehension report for the entire group, and a communication means for transmitting the comprehension report generated by the report generation means to an educator terminal means. This will enable the provision of optimal learning support in real time based on each student's level of understanding and emotional state, reducing the burden on teachers while creating an effective educational environment.

[1315] "Capturing means" is a device for acquiring video data.

[1316] An "information processing means" is a device or system that receives, stores, and analyzes the captured video data.

[1317] "Inference engine means" refers to an algorithm or software for analyzing stored video data and assessing the learner's level of comprehension and emotional state.

[1318] The "learning material generating means" is a device or software for generating individual learning materials based on the evaluation results by the inference engine means.

[1319] A "communication means" is an interface for sending and receiving data, and is used to communicate via a network or data link.

[1320] "Student terminal means" refers to a terminal device used by a student, which displays learning materials and accepts input of answers.

[1321] The "display means" refers to a display or monitor for displaying learning materials on the learner terminal means.

[1322] The "report generation means" is a device or software for generating a comprehension report based on the learner's answers.

[1323] The "educator terminal means" is a terminal device used by an educator, which receives and displays the generated comprehension report.

[1324] The educational support system related to this invention provides individualized learning support based on the student's level of understanding and emotional state. This system requires a wide range of hardware and software to comprehensively record lesson content, record student responses, analyze data, generate individualized learning materials, provide feedback on the results, and generate reports.

[1325] Recording lesson content and student responses

[1326] The system is equipped with a first imaging means for recording the content of the lesson and a second imaging means for recording the reactions of students. High-resolution cameras are used for these imaging means. As a specific example, the first imaging means is a fixed camera installed at the front of the classroom, and the second imaging means is a wide-angle camera that can capture the entire student body. When the lesson begins, the server confirms that these cameras are activated and operating normally, and begins acquiring video data in real time.

[1327] Video data storage and preparation

[1328] The server receives the video data sent from the camera in real time and stores it in a database. At the same time, it checks the video quality and frame rate and stores the data in a format suitable for analysis.

[1329] Data analysis and understanding assessment

[1330] The server transmits the stored video data to an inference engine means. This inference engine means incorporates an artificial intelligence model including a facial recognition algorithm. The artificial intelligence model analyzes the students' facial expressions and behavior and quantifies their level of understanding. It also incorporates an emotion recognition engine to determine their emotional state, recognizing their emotional states such as interest, confusion, enjoyment, and fatigue in real time. Based on these results, each student's level of understanding is evaluated.

[1331] Creation and delivery of personalized learning materials

[1332] The learning material generation means generates optimal learning materials based on the data obtained from the inference engine means and the emotion recognition engine. For example, if a student is having difficulty understanding a new English word, a quiz-style assignment for vocabulary practice is generated for that student. The generated learning materials are transmitted to the learner terminal means of each student via the communication means.

[1333] Assignment and feedback of results

[1334] The learner's device displays the received learning materials and provides an interface for the learner to execute them. When the student completes the assignment, the answer is sent from the device to the server. The server re-evaluates the received answer and re-evaluates the overall class and individual comprehension.

[1335] Generate reports and provide feedback to teachers

[1336] The server generates a comprehension report for the entire group based on the answer results. The report generation means creates a detailed report and visualizes each student's strengths and weaknesses. The generated comprehension report is sent to the educator terminal means, where it can be viewed by the teacher. This allows the teacher to grasp the progress of the entire class and provide any additional explanations or individual guidance needed in the next lesson.

[1337] Specific examples

[1338] Prompt Sentence Examples

[1339] "We're starting English class. I'll explain some new words."

[1340] "Take the following quiz-style challenge. Choose the meaning of a new word."

[1341] "Student A seems confused about the meaning of new words."

[1342] "Further explanation will be required in the next class."

[1343] This allows the educational support system to utilize AI and emotion recognition technology to provide optimal learning support in real time according to each student's level of understanding and emotional state, reducing the burden on teachers and creating an effective educational environment.

[1344] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1345] Step 1: Record lesson content and student responses

[1346] 1.1 The server confirms that Camera 1 (for recording lesson content) and Camera 2 (for recording student responses) are running normally.

[1347] (Input): Camera startup status

[1348] (Output): Message confirming successful camera startup

[1349] 1.2 When the lesson begins, Camera 1 records the teacher's lesson content in real time.

[1350] (Input): Lesson start trigger

[1351] (Output): Video data of the teacher's lesson

[1352] 1.3 Camera 2 records students' reactions (facial expressions and behavior) in real time.

[1353] (Input): Lesson start trigger

[1354] (Output): Video data of students' reactions

[1355] 1.4 The video data acquired from Camera 1 and Camera 2 is sent to the server.

[1356] (Input): Video data of teacher's lesson content and students' responses

[1357] (Output): Message that video data has been sent to the server

[1358] Step 2: Saving and preparing your footage

[1359] 2.1 The server receives and stores the video data transmitted from Camera 1 and Camera 2 in real time.

[1360] (Input): Video data of teacher's lesson content and students' responses

[1361] (Output): Message confirming saving to database

[1362] 2.2 The server checks the video quality and frame rate of the stored data.

[1363] (Input): Saved video data

[1364] (Output): Video quality check results

[1365] Step 3: Analyze data and assess comprehension

[1366] 3.1 The server transmits the stored video data to the inference engine means.

[1367] (Input): Saved video data

[1368] (Output): Message that data has been sent to the inference engine

[1369] 3.2 The inference engine means uses a facial recognition algorithm to analyze the student's facial expressions and behavior.

[1370] (Input): Student reaction video data

[1371] (Output): Student facial expression and behavior analysis results

[1372] 3.3 The server sends the analysis results received from the inference engine means to the emotion engine to determine the emotional state of the student.

[1373] (Input): Facial expression and behavior analysis results

[1374] (Output): Student's emotional state assessment result

[1375] Step 4: Generate and send personalized learning materials

[1376] 4.1 The learning material generation means generates optimal learning materials based on the data obtained from the inference engine means and the emotion engine.

[1377] (Input): Comprehension and emotional state data

[1378] (Output): Individual learning materials

[1379] 4.2 The generated learning materials are transmitted to the learner terminal means of each student via communication means.

[1380] (Input): Individual learning materials

[1381] (Output): Message that the teaching materials have been sent to the student's device

[1382] Step 5: Implementing the task and providing feedback

[1383] 5.1 The learner's device displays the transmitted learning materials.

[1384] (Input): Individual learning materials

[1385] (Output): Display of learning materials

[1386] 5.2 Students work on the assignment and send their answers from their devices to the server.

[1387] (Input): Student's answer result

[1388] (Output): Message that the answer result has been sent to the server

[1389] 5.3 The server will re-evaluate the received answer results and re-evaluate the overall class and individual comprehension.

[1390] (Input): Student's answer result

[1391] (Output): Reassessment results (understanding data)

[1392] Step 6: Generate reports and provide feedback to teachers

[1393] 6.1 The server generates a group-wide comprehension report based on the answers.

[1394] (Input): Reassessment results (understanding data)

[1395] (Output): Comprehension report

[1396] 6.2 The report generator will produce detailed reports and visualize each student's strengths and weaknesses.

[1397] (Input): Comprehension Report

[1398] (Output):Detailed report

[1399] 6.3 The generated comprehension report will be sent to the educator terminal means so that the teacher can review it.

[1400] (Input):Detailed Report

[1401] (Output): Report sending completion message to teacher's device

[1402] Through the above processing steps, this system is able to efficiently record and analyze lesson content and student responses in real time, providing optimal learning support to each individual student.

[1403] (Application example 2)

[1404] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1405] Conventional educational support systems have difficulty in properly assessing students' understanding and learning progress, and in particular, they have been unable to provide sufficient learning support that takes into account the emotional state of each student. This can result in a lack of detailed learning support for each student, which can reduce learning effectiveness. Another issue is that teachers are unable to receive effective feedback when grasping the overall understanding level of the class and reflecting this in their next lesson plan.

[1406] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means: In this invention, the server includes a first image capturing device means for recording the content of the lesson, a second image capturing device means for recording the reactions of students, an information processing device means for receiving and storing video information from the first image capturing device and the second image capturing device, a machine learning model device means for analyzing the video information stored by the information processing device and evaluating the level of understanding of the students, an assignment generation device means for generating individual assignments based on the level of understanding evaluated by the machine learning model device, a communication device means for transmitting the assignments generated by the assignment generation device to student terminal devices, a display device means for displaying the assignments received by the student terminal devices and having the students perform them, a communication device means for transmitting the results of the assignments performed by the student terminal devices to the information processing device, a report generation device means for re-evaluating the results received by the information processing device and generating a comprehension report for the entire class, and a communication device means for transmitting the comprehension report generated by the report generation device to a teacher terminal device. This not only makes it possible to create individual tasks based on students' level of understanding and provide learning support that takes into account students' emotional state, but also allows the level of understanding of the entire class to be grasped, providing effective feedback that teachers can reflect in their next lesson plans.

[1407] The "first imaging device for recording the content of the lesson" is a video device for recording the progress of the lesson, and is a camera device for capturing the entire lesson, including what the teacher is saying and what he is writing on the blackboard.

[1408] The "second imaging device for recording students' reactions" is a video device for recording students' facial expressions and behavior, and a camera device for monitoring how students are reacting to lessons.

[1409] An "information processing device" is a computer system for receiving and storing video information that records lesson content and student responses, and is a device that analyzes and stores video data.

[1410] A "machine learning model device" is a data processing device that analyzes video data stored in an information processing device and evaluates students' level of understanding, and is an artificial intelligence device that has algorithms for facial recognition and behavioral analysis.

[1411] The "task generation device" is a device that automatically generates individual tasks based on the level of understanding assessed by the machine learning model device, and is a device for providing learning content adapted to each student.

[1412] The "communication device" is a network device for transmitting and receiving data between the information processing device, student terminal device, and teacher terminal device, and is a device for transmitting assignments and receiving results.

[1413] A "student terminal" is a device that receives and allows a student to complete assignments, and provides an interface for displaying and answering individual assignments.

[1414] The "display device" is a device for showing the assignment received by the student terminal device to the student, and is a monitor or screen for visually displaying the content of the assignment.

[1415] A "report generation device" is a device that compiles the level of understanding of the entire class based on the results of the assignments and creates a detailed report, generating materials that allow teachers to understand the learning situation of their class.

[1416] An "emotion recognition engine" is an algorithm that analyzes students' emotional state from their facial expressions and behavior and evaluates their motivation to learn, and is a technology that recognizes students' emotional changes in real time.

[1417] The present invention relates to an educational support system that analyzes a student's level of understanding and emotional state and provides individual learning support. The detailed configuration and operation of this system will be described below.

[1418] System configuration

[1419] This educational support system consists of the following main components:

[1420] 1. First imaging device

[1421] 2. Secondary imaging device

[1422] 3. Information Processing Device

[1423] 4. Machine Learning Model Device

[1424] 5. Task generation device

[1425] 6. Communications Equipment

[1426] 7. Student terminal devices

[1427] 8. Display device

[1428] 9. Report Generator

[1429] 10. Emotion Recognition Engine

[1430] System Operation

[1431] 1. Recording lesson content and student responses

[1432] When the lesson begins, the first camera records the teacher's lesson content in real time and transmits the video data to the information processing device. At the same time, the second camera records the students' expressions and behaviors and transmits the video data to the information processing device.

[1433] 2. Video data storage and analysis

[1434] The information processing device stores the received video data for later analysis. The quality and frame rate of the video are also checked to ensure optimal conditions are maintained.

[1435] The machine learning model uses the stored video data to assess the student's understanding, using data processing techniques such as facial recognition algorithms.

[1436] 3. Emotional state analysis

[1437] The emotion recognition engine analyzes students' emotional states from their facial expressions and behaviors, allowing real-time assessment of their emotional states, such as enjoyment, interest, confusion, and fatigue.

[1438] 4. Generating individual tasks

[1439] The assignment generator generates optimal preparatory, review, and practice assignments for each student based on data from the machine learning model device and emotion recognition engine.

[1440] In particular, taking into account emotional states provides tasks that are tailored to students' motivation.

[1441] 5. Submit and complete the assignment

[1442] The generated assignment is sent to the student terminal via the communication device, which displays the received assignment and provides an interface for the student to work on it.

[1443] When the student answers the assignment, the results are transmitted to the information processing device via the communication device.

[1444] 6. Review and feedback of results

[1445] The information processing device performs a reassessment based on the received assignment results and generates a report on the level of understanding of the entire class. The report generating device creates this report and sends it to the teacher terminal device to provide feedback to the teacher.

[1446] Specific examples

[1447] For example, consider a situation where students are learning new words during an English class. A first camera records the teacher's lesson content, and a second camera records the students' responses. An information processing device stores and analyzes the video data. A machine learning model device and an emotion recognition engine are used to evaluate whether the students understand the meaning of the words and what their emotional state is. An assignment generation device then generates individual review assignments and sends them to the student terminal device via a communication device. The students answer the assignments on their terminals, and the results are sent back to the server for re-evaluation and feedback.

[1448] Prompt Sentence Examples

[1449] Prompts to assess student understanding:

[1450] Use a facial recognition algorithm to analyze Student A's facial expressions within the most recent lesson data and score his / her level of understanding.

[1451] Prompts for analyzing emotional states:

[1452] Based on Student B's video data, please evaluate his emotional state in real time, such as enjoyment, interest, confusion, and fatigue.

[1453] In this way, the present invention can maximize the effectiveness of education by providing individual learning support that takes into account not only the student's level of understanding of the lesson content but also their emotional state.

[1454] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1455] Step 1:

[1456] Recording of lesson content and student responses

[1457] At the start of a lesson, the server checks whether the first and second image capture devices are running properly, and is then ready to record the lesson content and students' responses.

[1458] Input: Lesson start command, activation status of first and second imaging devices

[1459] Operation: The first camera records the teacher's lesson in real time and sends the video data to a server. At the same time, the second camera records the students' expressions and behavior and sends this video data to the server.

[1460] Output: Video data of teacher's lesson content, video data of students' reactions

[1461] Step 2:

[1462] Video data storage and quality check

[1463] The server temporarily stores the received video data, checking the quality and frame rate of the video data to ensure it is in a state suitable for optimal analysis.

[1464] Input: Video data of lesson content, video data of student reactions

[1465] Operation: Saving video data and checking the frame rate and quality

[1466] Output: High-quality video data stored

[1467] Step 3:

[1468] Analysis of video data and assessment of comprehension

[1469] The server then passes the stored video data to a machine learning model device, which uses a facial recognition algorithm to analyze the student's facial expressions and behavior and generate a score based on their level of understanding.

[1470] Input: High-quality video data

[1471] Action: Analyzing students' facial expressions and behaviors and scoring their understanding using a face recognition algorithm based on a machine learning model device

[1472] Output: Student comprehension score

[1473] Step 4:

[1474] Emotional state analysis

[1475] The server then passes the stored video data to an emotion recognition engine, which analyzes the student's emotional state, assessing emotions such as enjoyment, interest, confusion, and fatigue.

[1476] Input: High-quality video data

[1477] Action: Analyzing students' emotional states using an emotion recognition engine

[1478] Output: Student emotional state data

[1479] Step 5:

[1480] Generate individual assignments

[1481] The task generation device generates optimal preparation, review, and practice tasks for each student based on their comprehension scores and emotional state data.

[1482] Input: Student comprehension scores, student emotional state data

[1483] Action: Individual task generation according to comprehension level and emotional state

[1484] Output: Individual issue data

[1485] Step 6:

[1486] Submitting and viewing assignments

[1487] The created assignment is sent to the student terminal via the communication device, and the student terminal receives the assignment and displays it to the student.

[1488] Input: Individual issue data

[1489] Operation: Sending assignment data and displaying it on student terminal devices

[1490] Output: The displayed issues

[1491] Step 7:

[1492] Implementing the assignment and collecting the results

[1493] Students answer questions using their student devices and send the results to the server, which then receives the results.

[1494] Input: Viewed assignment

[1495] Operation: Student terminal device performs assignments, inputs results, and transmits results via communication device

[1496] Output: Student assignment answer result data

[1497] Step 8:

[1498] Review of results and feedback

[1499] The server analyzes the received answers and evaluates the level of understanding and areas for improvement. The report generator creates a report on the level of understanding of the entire class and sends it to the teacher's terminal device via the communication device.

[1500] Input: Student assignment answer result data

[1501] Actions: Analyze the results data, reassess the level of understanding, generate and send a feedback report

[1502] Output: Whole class comprehension report

[1503] Prompt Sentence Examples

[1504] "Use a facial recognition algorithm to analyze Student A's facial expressions within the most recent lesson data and score their level of understanding."

[1505] "Based on Student B's video data, please rate his emotional state in real time, such as enjoyment, interest, confusion, and fatigue."

[1506] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1507] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1508] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1509] [Fourth embodiment]

[1510] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1511] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1512] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1513] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1514] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1515] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1516] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1517] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1518] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1519] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1520] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1521] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1522] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1523] The present invention relates to an educational support system that can effectively record lesson content and grasp students' understanding in real time. This system is composed of multiple cameras, a server, an artificial intelligence model, an assignment generation unit, a transmission unit, student terminals, a display unit, a report generation unit, and a teacher terminal.

[1524] System program processing and explanation

[1525] Record of lesson content

[1526] At the start of a lesson, the server checks whether the first camera means (Camera 1) that records the lesson content and the second camera means (Camera 2) that records the students' reactions are running normally. When the lesson starts, Camera 1 records the teacher's lesson content in real time and sends the video data to the server. At the same time, Camera 2 records the expressions and actions of all the students in real time and also sends this video data to the server.

[1527] Video data storage and preparation

[1528] The server stores the video data of the lesson content and students' reactions received from the camera means in real time. This stored data will be used for later analysis, so it also checks the video quality and frame rate.

[1529] Data analysis and understanding assessment

[1530] The AI ​​model analyzes the video data stored on the server. A facial recognition algorithm is used to evaluate the student's level of understanding based on their facial expressions and behavior. The evaluation results are saved as a score and listed for each student.

[1531] Creating and submitting individual assignments

[1532] The assignment generation means generates preparation, review, and practice assignments suitable for each student based on the results of the assessment of comprehension by the artificial intelligence model means. The generated assignments are sent to each student's terminal via the transmission means.

[1533] Assignment and feedback of results

[1534] The student's device displays the received assignments and provides an interface for the student to complete them. After the student completes the assignment, the results are sent from the device to the server. The server reassessssments based on the received assignment results and generates a report on the overall class comprehension. This report is used by the teacher to optimize lesson plans.

[1535] Generate reports and provide feedback to teachers

[1536] The report generation tool compiles a summary of the class's overall understanding based on the results of the assignments and creates a detailed report. This report visualizes each student's strengths and weaknesses. The teacher's device receives and displays this report, allowing the teacher to grasp the overall situation in the class and plan appropriate measures for the next lesson.

[1537] Specific examples

[1538] 1. Conducting and recording lessons

[1539] The user (teacher) starts a math lesson. Camera 1 records the teacher's lesson content, and Camera 2 records the students' reactions. For example, when the teacher is explaining the basics of equations, Student A looks confused.

[1540] 2. Data analysis and understanding assessment

[1541] The server stores this video data, and the artificial intelligence model means judges the low level of understanding from the facial expression of Student A. Based on this, an assignment is created for Student A to review basic equation problems.

[1542] 3. Submitting and completing individual assignments

[1543] The assignment is sent to and displayed on Student A's device. Student A answers the assignment and sends the results to the server.

[1544] 4. Feedback on the results and reflecting them in lesson plans

[1545] The server receives Student A's answer and re-evaluates it. A report of the class's understanding is generated and sent to the teacher's terminal. Based on this, the teacher decides to allocate time to the basics of equations in the next lesson.

[1546] In this way, the present invention utilizes AI technology to reduce the burden on teachers while providing learning support that is tailored to each student.

[1547] The processing flow will be explained below.

[1548] Step 1:

[1549] At the start of a lesson, the server checks whether the first camera means (camera 1) for recording the lesson content and the second camera means (camera 2) for recording the students' reactions are running normally.

[1550] Step 2:

[1551] Camera 1 records the teacher's lesson content in real time and transmits the video data in streaming format to the server.

[1552] Step 3:

[1553] Camera 2 records the facial expressions and behavior of the entire classroom and each student in real time, and transmits the video data in streaming format to the server.

[1554] Step 4:

[1555] The server stores the received video data in real time and monitors the video quality and frame rate.

[1556] Step 5:

[1557] The AI ​​model analyzes the video data of the lesson content and students' reactions stored on the server. Specifically, it uses a facial recognition algorithm to identify students' facial expressions and behaviors and score each student's level of understanding.

[1558] Step 6:

[1559] The server stores the comprehension scores generated by the artificial intelligence model means and lists the learning needs of each student.

[1560] Step 7:

[1561] The assignment generation means generates optimal preparation, review, and practice assignments for each student based on the stored comprehension scores.

[1562] Step 8:

[1563] The transmission means transmits the generated individual assignments to the terminals of the students.

[1564] Step 9:

[1565] The student's device displays the received assignment and provides an interface through which the student can complete the assignment.

[1566] Step 10:

[1567] Students answer the assignments and send the results data from their devices to the server.

[1568] Step 11:

[1569] The server reassessed the students' understanding based on the received assignment results and generated a comprehension report for the entire class.

[1570] Step 12:

[1571] The report generating means transmits the generated comprehension report to the teacher's terminal.

[1572] Step 13:

[1573] The teacher's device will display a comprehension report, allowing the teacher to understand the learning status of the entire class.

[1574] Step 14:

[1575] Based on the comprehension reports, teachers decide what content and focus to cover in the next lesson.

[1576] Through this series of processing steps, the present invention realizes a system that provides learning support suited to each individual student and more effectively supports teachers' lesson plans.

[1577] Example 1

[1578] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1579] Conventional educational support systems have had difficulty accurately assessing each student's level of understanding in real time and providing individualized learning support. They also have limitations in comprehensively recording lesson content and effectively utilizing that data. This makes it difficult for teachers to grasp students' levels of understanding, leading to problems with efficient instruction.

[1580] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1581] In this invention, the server includes a first recording means for recording the content of the lesson, a second recording means for recording the responses of students, a data storage means for receiving and storing video data from the first recording means and the second recording means, an analysis means for analyzing the video data stored by the data storage means and evaluating the students' levels of comprehension, an assignment creation means for generating individual assignments based on the levels of comprehension evaluated by the analysis means, a transmission means for transmitting the assignments generated by the assignment creation means to student terminals, a display means for displaying the assignments received by the student terminals and having the students complete them, a transmission means for transmitting the results of the assignments completed by the student terminals to the data storage means, a report creation means for re-evaluating the results received by the data storage means and generating a report on the level of comprehension of the entire class, and a transmission means for transmitting the comprehension report generated by the report creation means to a teacher's terminal. This enables comprehensive recording of the content of the lesson and real-time evaluation and feedback of students' levels of comprehension.

[1582] The "primary recording means for recording the content of the class" is a device that records the lecture content and explanations of the instructor in charge of the class as video and audio.

[1583] The "second recording means for recording student responses" is a device that records students' facial expressions and behavior during class in real time as video.

[1584] The "data storage means" is a system or device for receiving the video data transmitted from the first recording means and the second recording means and storing the data for a long period of time.

[1585] The "analysis means" is a system that includes algorithms and software for analyzing the stored video data and assessing the student's level of understanding.

[1586] The "task creation means" is a system or device for automatically creating individual learning tasks based on the student's level of understanding evaluated by the analysis means.

[1587] The "transmission means" is a system or device for transmitting the generated assignments and comprehension reports to a specified terminal.

[1588] A "student device" is a device such as a computer or tablet used by a student that provides an interface for displaying and completing received assignments.

[1589] The "display means" is software or a system that displays the assignment received on the student terminal and provides an interface for the student to answer the assignment.

[1590] The "report creation means" is a system that compiles the level of understanding of the entire class based on the results of assignments sent from student devices and generates a comprehension report for the teacher.

[1591] A "teacher's terminal" is a device such as a computer or tablet used by a teacher to receive and display comprehension reports.

[1592] The present invention relates to an educational support system that can effectively record lesson content and grasp students' understanding in real time. This system is composed of multiple recording means, data storage means, analysis means, assignment creation means, transmission means, student terminals, display means, report creation means, and teacher terminals.

[1593] Hardware and Software Configuration

[1594] Specific hardware used includes camera devices (first and second recording means), servers (data storage means and analysis means), student tablets or PCs (student terminals), and faculty tablets or PCs (faculty terminals).Software includes video analysis algorithms (face recognition algorithms and facial expression analysis algorithms), AI models, assignment generation algorithms, and database management systems.

[1595] Data collection and storage

[1596] At the start of class, the server checks whether the first recording means (Camera 1), which records the teacher's lecture content, and the second recording means (Camera 2), which records the students' reactions, are operating normally. When class begins, Camera 1 records the teacher's lecture in real time and sends the data to the server. Camera 2 simultaneously records the expressions and movements of all the students and also sends this data to the server. The data storage means saves the received video data to a highly durable storage device.

[1597] Data analysis and understanding assessment

[1598] The saved video data is processed by an analysis means. The analysis means uses a facial recognition algorithm and an expression analysis algorithm to evaluate the student's level of comprehension based on their facial expressions and movements. This evaluation is quantified and recorded as a comprehension score for each student. The analysis results are saved in a database and used for subsequent processing.

[1599] Creating and submitting assignments

[1600] The assignment creation means generates individual preparation, review, and practice assignments based on the comprehension score obtained from the analysis means. The generated assignments are sent to each student's student device via the transmission means. For example, mathematics practice problems created using the generative AI model are distributed to student devices.

[1601] Assignment and feedback of results

[1602] The student terminals display the received assignments and provide an interface for students to answer. When students complete the assignments, their answers are automatically sent to the server. The server reevaluates the received answers and creates a report based on the overall class understanding. This report is generated by the report creation means and sent to the teacher terminal.

[1603] Specific examples

[1604] 1. Lesson Conduct and Recording:

[1605] A user (teacher) starts a math class at 9:00 a.m. Camera 1 records the teacher's lecture, and Camera 2 records the students' reactions. For example, when the teacher is explaining the basics of equations, Student A looks confused.

[1606] 2. Data analysis and comprehension assessment:

[1607] The server stores this video data, and the analysis means determines whether Student A has a low level of understanding based on his / her facial expressions, and records the result as a score. Data is generated to provide Student A with review assignments.

[1608] 3. Submit and complete the assignment:

[1609] The assignment creation means generates a review assignment suitable for Student A and sends it to Student A's student terminal. Student A's terminal displays the review math problems.

[1610] 4. Feedback on results and reflection on next class:

[1611] The server receives Student A's answers to the assignment and re-evaluates them. A comprehension report is sent to the teacher's terminal, and the teacher uses it to plan the next lesson.

[1612] Prompt Sentence Examples

[1613] Below are some examples of prompts to give instructions to an AI generative model.

[1614] "To determine whether students are understanding during class, consider the following scenario: Camera 1 records a teacher teaching the basics of equations in a math class, while simultaneously recording the students' reactions with Camera 2. If it is determined that Student A has a confused expression, generate appropriate review assignments for Student A based on the results of the facial expression analysis."

[1615] In this way, the present invention utilizes AI technology to reduce the burden on teachers while providing learning support that is tailored to each student.

[1616] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1617] Step 1: Start the lesson and check that the equipment is up and running

[1618] At the designated time, the server checks whether the lesson is about to begin. Here, it checks whether the first recording means (camera 1) and the second recording means (camera 2) are operating normally. Specifically, the server sends a PING command to each camera and monitors whether there is a response. If a normal response is confirmed, it determines that the lesson is ready to begin.

[1619] Input: lesson start time, camera status

[1620] Output: Confirmation that the device has started up normally

[1621] Step 2: Record lesson content and student reactions

[1622] When the class begins, the server has Camera 1 record the teacher's lecture in real time and transmits the video data to the server. At the same time, Camera 2 records the reactions of all students and transmits this to the server. The video data being recorded is received in real time and saved by the data saving means.

[1623] Input: Teacher lecture video, student reaction video

[1624] Output: Save real-time video data to the server

[1625] Step 3: Save the video data and check its quality

[1626] The server stores the received video data in a data storage device. At the same time, it checks the quality of the video data. Specifically, it automatically checks whether there are any missing frames and whether the audio data is synchronized. The results of this check are also saved and used to ensure the quality of the data.

[1627] Input: Received video data

[1628] Output: Saved video data, quality check results

[1629] Step 4: Analyze data and assess comprehension

[1630] The saved video data is processed by an analytical device. Specifically, facial recognition algorithms and facial expression analysis algorithms are used to quantify the student's level of comprehension based on their facial expressions and movements. The results of this analysis are recorded as a comprehension score for each student.

[1631] Input: Saved video data

[1632] Output: Comprehension score

[1633] Step 5: Generate assignments and send them to students

[1634] The assignment creation means generates individual assignments based on the comprehension scores. The generated assignments are sent to each student's student device via the transmission means. Based on the generation AI model, more basic questions are provided to students with low scores, and applied questions are provided to students with high scores.

[1635] Input: Comprehension score

[1636] Output: Individual assignment, Submit assignment

[1637] Step 6: Complete and submit the assignment

[1638] The student terminals display the received assignments and provide an interface for students to answer. When students complete the assignments, the answers are automatically sent to the server.

[1639] Input: Received assignment

[1640] Output: Send answer results

[1641] Step 7: Reassess results and provide feedback to teachers

[1642] The server reevaluates the received answers and reassess the level of understanding. Based on the reevaluation results, a report on the level of understanding of the entire class is generated. The report generation means creates this report and sends it to the teacher's terminal.

[1643] Input: Answer result

[1644] Output: Reassessed comprehension, Generate comprehension report

[1645] Step 8: Reflect on the next lesson plan

[1646] The teacher's device displays the received comprehension report and uses it as reference for planning the next lesson. Teachers can identify each student's strengths and weaknesses and optimize their lesson plans.

[1647] Input: Comprehension Report

[1648] Output: Optimized lesson plans

[1649] (Application example 1)

[1650] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1651] Modern education and retail operations require support tailored to diverse individual needs and levels of understanding. In educational environments, it is particularly important to assess each student's level of understanding in real time and provide appropriate learning content based on the results. Meanwhile, in brick-and-mortar stores, analyzing customer behavior and facial expressions and making appropriate product recommendations can be expected to boost purchasing motivation. Current systems face the challenge of being unable to efficiently respond to such diverse demands.

[1652] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1653] In this invention, the server includes a first image acquisition means for recording lesson content, a second image acquisition means for recording student responses, a data storage means for receiving and storing video data from the first image acquisition means and the second image acquisition means, an intelligence assessment means for analyzing the video data stored by the data storage means and evaluating the student's level of comprehension, a learning content generation means for generating individual learning content based on the level of comprehension assessed by the intelligence assessment means, a data transmission means for transmitting the learning content generated by the learning content generation means to a receiving device, a display device for displaying the learning content received by the receiving device and having the student execute it, a data transmission means for transmitting the results of the learning content executed by the receiving device to the data storage means, a report generation means for re-evaluating the results received by the data storage means and generating an overall comprehension report, and a data transmission means for transmitting the comprehension report generated by the report generation means to a reporting device. This enables support to be provided according to individual needs and levels of comprehension in educational and retail environments.

[1654] "Class content" is a term that refers to the content and themes that teachers teach to students in educational activities.

[1655] "Student reactions" is a term that refers to various behaviors that students show during class, such as facial expressions, movements, and actions, that indicate their understanding and interest in what they are learning.

[1656] "Image capture means" is a term that refers to an apparatus or system for capturing video data using a device such as a camera.

[1657] "Data storage means" is a term that refers to a device or system for temporarily or permanently storing acquired video data, analysis results, etc.

[1658] "Intelligence assessment means" is a term used to refer to a device or system that uses artificial intelligence to analyze data obtained and assess an individual's level of understanding and emotions.

[1659] "Learning content generation means" is a term that refers to a device or system for generating learning content optimized for each student based on the evaluation results from the intelligence evaluation means.

[1660] "Data transmission means" is a term that refers to an apparatus or system for transmitting generated learning content, analysis results, reports, etc. to a specific terminal or device.

[1661] "Receiving device" is a term that refers to a device or system for receiving data transmitted from a data transmitting means.

[1662] "Display device" is a term that refers to a device or system that displays data and learning content received by a receiving device so that the user can visually confirm it.

[1663] "Report generator" is a term that refers to a device or system for generating a comprehensive report based on intelligence assessment measures and learning results.

[1664] "Reporting device" is a term that refers to a device or system that ultimately reports generated reports or other information to a user.

[1665] To put this invention into practice, it is necessary to build systems for both education and retail use, and to be able to meet the needs of each environment. Specific examples of such systems are shown below.

[1666] System Overview

[1667] The system mainly includes the following components:

[1668] First image capture method (cameras that record lesson content and customer behavior)

[1669] Secondary image capture means (cameras that record students' reactions and customers' facial expressions)

[1670] Data storage means (server)

[1671] Intelligence evaluation method (analysis system using artificial intelligence model)

[1672] Learning content generation means (a system that generates individual tasks or product proposals)

[1673] Data transmission method (transmission server)

[1674] Receiving device (student device, store display, smartphone)

[1675] Display devices (terminals and displays)

[1676] Report generation means (systems that generate class-wide or customer behavior reports)

[1677] Reporting device (teacher terminal or store management system)

[1678] Hardware and Software

[1679] Hardware:

[1680] Camera (e.g. USB camera)

[1681] Server (data storage and analysis)

[1682] Smartphones, tablets, store displays, robots

[1683] software:

[1684] TensorFlow (a deep learning model for analyzing customer facial expressions)

[1685] OpenCV (acquisition and display of camera images)

[1686] REST API (data sending and receiving)

[1687] Data processing and calculation

[1688] The server processes and calculates the data in the following steps:

[1689] 1. Data Acquisition:

[1690] The first image acquisition means records the contents of the lesson and the behavior of the customer, and transmits the recorded images to the data storage means in real time.

[1691] A second image capturing means records the reactions of the students and the facial expressions of the customers and transmits them to the data storage means as well.

[1692] 2. Analysis and Evaluation:

[1693] The video data stored in the data storage means is analyzed using an intelligent evaluation means (e.g., an artificial intelligence model equipped with a face recognition algorithm) to evaluate the students' understanding and the customers' emotions.

[1694] 3. Data transmission:

[1695] Based on the results of the evaluation by the intelligence evaluation means, the learning content generation means generates appropriate assignments and product suggestions and transmits them to the receiving device.

[1696] 4. Interaction:

[1697] The assignments and product proposals received by the receiving device are displayed on the display device and are carried out by the students or customers.

[1698] 5. Feedback of results:

[1699] The results of the execution on the display device are again transmitted to the data storage means by the data transmission means, and the report generation means generates an understanding level report and a customer behavior report.

[1700] Specific examples

[1701] 1. Educational environment:

[1702] Consider a scenario in which a teacher begins a lesson. A first image acquisition means records the content of the lesson, and a second image acquisition means records the students' responses. An AI model analyzes the students' facial expressions, and if a particular student is struggling to understand, it sends that student an individual assignment. This assignment is displayed on the student's device, and the student answers it. The answers are sent to a server, and the teacher receives a report of the class's understanding on his or her device.

[1703] 2. Store environment:

[1704] Consider a scenario in which a customer enters a store. A first image capture means records the customer's behavior, and a second image capture means analyzes the customer's facial expressions. When the server detects the customer's expressions of interest or confusion, it generates an appropriate product suggestion and sends it to an in-store display or the customer's smartphone. An in-store robot interacts with the customer based on the suggestion (e.g., "Would you like this product?"). Ultimately, store staff optimize their response based on the report generated by the server.

[1705] Prompt Sentence Examples

[1706] For example, if a customer appears unsure, the following prompts can be fed into the generative AI model:

[1707] The customer's expression shows uncertainty. Which of the following products do you predict will most interest the customer?

[1708] This will improve the customer experience and streamline store operations.

[1709] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1710] Step 1:

[1711] Data Acquisition

[1712] Input: Real-time video data acquired by the first image acquisition means (camera 1) and the second image acquisition means (camera 2)

[1713] Output: Video data of lesson content, customer behavior, student reactions, and customer expressions sent to the server

[1714] Specific operation: The device activates Camera 1 and records the entire classroom or store. It then activates Camera 2 and records the facial expressions and behavior of individual students or customers. The captured video data is sent to the server in real time.

[1715] Step 2:

[1716] Data storage

[1717] Input: Real-time video data sent to the server

[1718] Output: Video data stored in a data storage device

[1719] Specific operation: The server stores the received video data in a data storage means (e.g., a database), and also checks the video quality and frame rate to ensure the quality of the stored data.

[1720] Step 3:

[1721] Data analysis and evaluation

[1722] Input: Video data stored in a data storage device

[1723] Output: Comprehension and emotional evaluation scores from intelligence assessment tools

[1724] Specific operation: The server's intelligent evaluation means (for example, an artificial intelligence model using a facial recognition algorithm) analyzes the stored video data. It evaluates the student's understanding and the customer's emotions and quantifies them as a score. This score is then listed for each student or customer.

[1725] Step 4:

[1726] Generate learning content and product suggestions

[1727] Input: Intelligence assessment score

[1728] Output: Individual tasks and product suggestions generated by the learning content generation means

[1729] Specific operation: Based on the score obtained by the intelligence evaluation means, the server uses a generative AI model to generate learning content, which generates individual assignments, review questions, or product suggestions suitable for the customer.

[1730] Step 5:

[1731] Data transmission

[1732] Input: Generated individual issues and product proposals

[1733] Output: Learning content and product suggestions sent to receiving devices (student terminals, store displays, smartphones)

[1734] Specific operation: The server transmits the assignments and product suggestions generated by the learning content generation means to each receiving device using the data transmission means.

[1735] Step 6:

[1736] Interaction

[1737] Input: Issues and product proposals displayed on the receiving device

[1738] Output: Answers and feedback

[1739] Specific operation: The receiving device (e.g., student terminal, smartphone, in-store display) displays the received assignment or product suggestion. The user (student or customer) then operates accordingly, entering an answer in the case of an assignment, or making a selection or purchasing action in the case of a product suggestion.

[1740] Step 7:

[1741] Feedback of results

[1742] Input: Answers and feedback from the receiving device

[1743] Output: Answer results and behavioral data fed back to the server

[1744] Specific operation: The terminal transmits the results of the user's operations (e.g., answers to assignments and product selection information) to the server. The server receives this and records it in the data storage means.

[1745] Step 8:

[1746] Report generation and delivery

[1747] Input: Recorded answer results and behavioral data

[1748] Output: Comprehension reports and customer behavior reports generated by the report generation means, and reports sent to the reporting device

[1749] Specific operation: Based on the received answer results and behavioral data, the server creates a comprehension report and a customer behavior report using the report generation means. The generated report is sent to a reporting device (e.g., a teacher terminal or a store management system) and displayed.

[1750] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1751] The present invention relates to an educational support system that provides effective learning support to each student by recording lesson content and student responses in real time and recognizing the emotional state of the student. This system is composed of multiple camera means, server means, artificial intelligence model means, emotion engine, assignment generation means, transmission means, student terminal means, display means, report generation means, and teacher terminal means.

[1752] System program processing and explanation

[1753] Record of lesson content

[1754] At the start of a lesson, the server checks whether the first camera means (Camera 1) that records the lesson content and the second camera means (Camera 2) that records the students' reactions are running properly. When the lesson starts, Camera 1 records the teacher's lesson content in real time and sends the video data to the server. At the same time, Camera 2 records the expressions and actions of all the students in real time and also sends this video data to the server.

[1755] Video data storage and preparation

[1756] The server stores the video data of the lesson content and students' reactions received from the camera means in real time. This stored data will be used for later analysis, so it also checks the video quality and frame rate.

[1757] Data analysis and understanding assessment

[1758] The AI ​​model analyzes video data of lesson content and student responses stored on a server. A facial recognition algorithm is used to assess students' comprehension based on their facial expressions and behavior. Furthermore, an emotion engine is incorporated to analyze students' emotional states. This emotion engine recognizes emotional states such as enjoyment, interest, confusion, and fatigue in real time, and reflects this data in the assessment of comprehension.

[1759] Creating and submitting individual assignments

[1760] The assignment generation means generates optimal preparation, review, and practice assignments for each student based on the level of understanding and emotional state generated by the artificial intelligence model means and the emotion engine. In particular, it presents assignments that reflect the analysis results of the emotion engine and take learning motivation into consideration. The generated assignments are sent to each student's device via the transmission means.

[1761] Assignment and feedback of results

[1762] The student's device displays the received assignment and provides an interface for the student to complete it. After the student completes the assignment, the results are sent from the device to the server. The server then reassessssments the results and generates a report on the overall class comprehension. This report is used by the teacher to optimize lesson plans.

[1763] Generate reports and provide feedback to teachers

[1764] The report generation tool compiles a summary of the class's overall understanding based on the results of the assignments and creates a detailed report. This report visualizes each student's strengths and weaknesses. The teacher's device receives and displays this report, allowing the teacher to grasp the overall situation in the class and plan appropriate measures for the next lesson.

[1765] Specific examples

[1766] 1. Conducting and recording lessons

[1767] The user (teacher) begins an English lesson. Camera 1 records the teacher's lesson, and Camera 2 records the students' reactions. For example, when the teacher explains the meaning of a new word, the camera captures the expressions of interest or confusion of a particular student.

[1768] 2. Data analysis and understanding assessment

[1769] The server stores these video data, and the artificial intelligence model means and emotion engine analyze the students' facial expressions and emotions, thereby determining whether a particular student has difficulty understanding new words.

[1770] 3. Submitting and completing individual assignments

[1771] Students can receive personalized review tasks on their devices that take into account their level of comprehension and emotional state, such as quiz-style tasks to practice the meaning of new words.

[1772] 4. Feedback on the results and reflecting them in lesson plans

[1773] The server receives the students' answers and re-evaluates them. A class-wide comprehension report is generated and sent to the teacher's device, allowing the teacher to determine which words and expressions need additional explanation in the next lesson.

[1774] In this way, the present invention utilizes AI and emotion recognition technology to provide optimal learning support tailored to each student's level of understanding and emotional state, thereby reducing the burden on teachers while increasing student motivation and creating an effective educational environment.

[1775] The processing flow will be explained below.

[1776] Step 1:

[1777] At the start of a lesson, the server checks whether the first camera means (camera 1) for recording the lesson content and the second camera means (camera 2) for recording the students' reactions are running normally.

[1778] Step 2:

[1779] Camera 1 records the teacher's lesson content in real time and transmits the video data in streaming format to the server.

[1780] Step 3:

[1781] Camera 2 records the facial expressions and behavior of the entire classroom and each student in real time, and transmits the video data in streaming format to a server.

[1782] Step 4:

[1783] The server stores the received video data in real time and monitors the video quality and frame rate.

[1784] Step 5:

[1785] The AI ​​model analyzes the video data of lesson content and student responses stored on the server. Specifically, it uses a facial recognition algorithm to identify students' facial expressions and behaviors and score each student's level of understanding.

[1786] Step 6:

[1787] The emotion engine recognizes students' emotional states (e.g., enjoyment, interest, confusion, fatigue) in real time based on the students' facial expression data acquired by the artificial intelligence model means, and the analysis results are reflected as additional information in the comprehension score.

[1788] Step 7:

[1789] The server stores the comprehension scores and emotion data generated by the artificial intelligence model means and emotion engine, and lists the learning needs of each student.

[1790] Step 8:

[1791] The task generation unit generates optimal preparation, review, and practice tasks for each student based on the stored comprehension scores and emotional data. In particular, tasks that increase the student's learning motivation are generated taking into account the student's emotional state.

[1792] Step 9:

[1793] The transmission means transmits the generated individual assignments to the terminals of the students.

[1794] Step 10:

[1795] The student's device displays the received assignment and provides an interface through which the student can complete the assignment.

[1796] Step 11:

[1797] Students answer the assignments and send the results data from their devices to the server.

[1798] Step 12:

[1799] The server reassessed the students' understanding based on the received assignment results and generated a comprehension report for the entire class.

[1800] Step 13:

[1801] The report generating means transmits the generated comprehension report to the teacher's terminal.

[1802] Step 14:

[1803] The teacher's device will display a comprehension report, allowing the teacher to understand the learning status of the entire class.

[1804] Step 15:

[1805] Based on the generated comprehension reports and student emotion data, the teacher can decide what to cover and what to focus on in the next lesson, for example, by planning to explain in detail a topic that many students found confusing.

[1806] Through this series of processing steps, the present invention combines AI technology and emotion recognition technology to realize a system that provides optimal learning support to each student while reducing the burden on teachers.

[1807] Example 2

[1808] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1809] Conventional educational systems lack efficiency and accuracy in recording lesson content, understanding student responses, and providing individualized instruction, making it difficult to provide individual learning support tailored to each student's level of understanding and emotional state. This also places a heavy burden on teachers, making it difficult to grasp the overall class level of understanding in real time and reflect this in the next lesson plan.

[1810] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a first imaging means for recording the content of the lesson, a second imaging means for recording the reactions of the learners, an information processing means for receiving and storing video data from the first imaging means and the second imaging means, an inference engine means for analyzing the video data stored by the information processing means and evaluating the level of understanding of the learners, a learning material generation means for generating individual learning materials based on the level of understanding evaluated by the inference engine means, a communication means for transmitting the learning materials generated by the learning material generation means to a learner terminal means, a display means for displaying the learning materials received by the learner terminal means and having the learner execute them, a communication means for transmitting the results of the learning materials executed by the learner terminal means to an information processing means, a report generation means for reevaluating based on the results received by the information processing means and generating a comprehension report for the entire group, and a communication means for transmitting the comprehension report generated by the report generation means to an educator terminal means. This will enable the provision of optimal learning support in real time based on each student's level of understanding and emotional state, reducing the burden on teachers while creating an effective educational environment.

[1811] "Capturing means" is a device for acquiring video data.

[1812] An "information processing means" is a device or system that receives, stores, and analyzes the captured video data.

[1813] "Inference engine means" refers to an algorithm or software for analyzing stored video data and assessing the learner's level of comprehension and emotional state.

[1814] The "learning material generating means" is a device or software for generating individual learning materials based on the evaluation results by the inference engine means.

[1815] A "communication means" is an interface for sending and receiving data, and is used to communicate via a network or data link.

[1816] "Student terminal means" refers to a terminal device used by a student, which displays learning materials and accepts input of answers.

[1817] The "display means" refers to a display or monitor for displaying learning materials on the learner terminal means.

[1818] The "report generation means" is a device or software for generating a comprehension report based on the learner's answers.

[1819] The "educator terminal means" is a terminal device used by an educator, which receives and displays the generated comprehension report.

[1820] The educational support system related to this invention provides individualized learning support based on the student's level of understanding and emotional state. This system requires a wide range of hardware and software to comprehensively record lesson content, record student responses, analyze data, generate individualized learning materials, provide feedback on the results, and generate reports.

[1821] Recording lesson content and student responses

[1822] The system is equipped with a first imaging means for recording the content of the lesson and a second imaging means for recording the reactions of students. High-resolution cameras are used for these imaging means. As a specific example, the first imaging means is a fixed camera installed at the front of the classroom, and the second imaging means is a wide-angle camera that can capture the entire student body. When the lesson begins, the server confirms that these cameras are activated and operating normally, and begins acquiring video data in real time.

[1823] Video data storage and preparation

[1824] The server receives the video data sent from the camera in real time and stores it in a database. At the same time, it checks the video quality and frame rate and stores the data in a format suitable for analysis.

[1825] Data analysis and understanding assessment

[1826] The server transmits the stored video data to an inference engine means. This inference engine means incorporates an artificial intelligence model including a facial recognition algorithm. The artificial intelligence model analyzes the students' facial expressions and behavior and quantifies their level of understanding. It also incorporates an emotion recognition engine to determine their emotional state, recognizing their emotional states such as interest, confusion, enjoyment, and fatigue in real time. Based on these results, each student's level of understanding is evaluated.

[1827] Creation and delivery of personalized learning materials

[1828] The learning material generation means generates optimal learning materials based on the data obtained from the inference engine means and the emotion recognition engine. For example, if a student is having difficulty understanding a new English word, a quiz-style assignment for vocabulary practice is generated for that student. The generated learning materials are transmitted to the learner terminal means of each student via the communication means.

[1829] Assignment and feedback of results

[1830] The learner's device displays the received learning materials and provides an interface for the learner to execute them. When the student completes the assignment, the answer is sent from the device to the server. The server re-evaluates the received answer and re-evaluates the overall class and individual comprehension.

[1831] Generate reports and provide feedback to teachers

[1832] The server generates a comprehension report for the entire group based on the answer results. The report generation means creates a detailed report and visualizes each student's strengths and weaknesses. The generated comprehension report is sent to the educator terminal means, where it can be viewed by the teacher. This allows the teacher to grasp the progress of the entire class and provide any additional explanations or individual guidance needed in the next lesson.

[1833] Specific examples

[1834] Prompt Sentence Examples

[1835] "We're starting English class. I'll explain some new words."

[1836] "Take the following quiz-style challenge. Choose the meaning of a new word."

[1837] "Student A seems confused about the meaning of new words."

[1838] "Further explanation will be required in the next class."

[1839] This allows the educational support system to utilize AI and emotion recognition technology to provide optimal learning support in real time according to each student's level of understanding and emotional state, reducing the burden on teachers and creating an effective educational environment.

[1840] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1841] Step 1: Record lesson content and student responses

[1842] 1.1 The server confirms that Camera 1 (for recording lesson content) and Camera 2 (for recording student responses) are running normally.

[1843] (Input): Camera startup status

[1844] (Output): Message confirming successful camera startup

[1845] 1.2 When the lesson begins, Camera 1 records the teacher's lesson content in real time.

[1846] (Input): Lesson start trigger

[1847] (Output): Video data of the teacher's lesson

[1848] 1.3 Camera 2 records students' reactions (facial expressions and behavior) in real time.

[1849] (Input): Lesson start trigger

[1850] (Output): Video data of students' reactions

[1851] 1.4 The video data acquired from Camera 1 and Camera 2 is sent to the server.

[1852] (Input): Video data of teacher's lesson content and students' responses

[1853] (Output): Message that video data has been sent to the server

[1854] Step 2: Saving and preparing your footage

[1855] 2.1 The server receives and stores the video data transmitted from Camera 1 and Camera 2 in real time.

[1856] (Input): Video data of teacher's lesson content and students' responses

[1857] (Output): Message confirming saving to database

[1858] 2.2 The server checks the video quality and frame rate of the stored data.

[1859] (Input): Saved video data

[1860] (Output): Video quality check results

[1861] Step 3: Analyze data and assess comprehension

[1862] 3.1 The server transmits the stored video data to the inference engine means.

[1863] (Input): Saved video data

[1864] (Output): Message that data has been sent to the inference engine

[1865] 3.2 The inference engine means uses a facial recognition algorithm to analyze the student's facial expressions and behavior.

[1866] (Input): Student reaction video data

[1867] (Output): Student facial expression and behavior analysis results

[1868] 3.3 The server sends the analysis results received from the inference engine means to the emotion engine to determine the emotional state of the student.

[1869] (Input): Facial expression and behavior analysis results

[1870] (Output): Student's emotional state assessment result

[1871] Step 4: Generate and send personalized learning materials

[1872] 4.1 The learning material generation means generates optimal learning materials based on the data obtained from the inference engine means and the emotion engine.

[1873] (Input): Comprehension and emotional state data

[1874] (Output): Individual learning materials

[1875] 4.2 The generated learning materials are transmitted to the learner terminal means of each student via communication means.

[1876] (Input): Individual learning materials

[1877] (Output): Message that the teaching materials have been sent to the student's device

[1878] Step 5: Implementing the task and providing feedback

[1879] 5.1 The learner's device displays the transmitted learning materials.

[1880] (Input): Individual learning materials

[1881] (Output): Display of learning materials

[1882] 5.2 Students work on the assignment and send their answers from their devices to the server.

[1883] (Input): Student's answer result

[1884] (Output): Message that the answer result has been sent to the server

[1885] 5.3 The server will re-evaluate the received answer results and re-evaluate the overall class and individual comprehension.

[1886] (Input): Student's answer result

[1887] (Output): Reassessment results (understanding data)

[1888] Step 6: Generate reports and provide feedback to teachers

[1889] 6.1 The server generates a group-wide comprehension report based on the answers.

[1890] (Input): Reassessment results (understanding data)

[1891] (Output): Comprehension report

[1892] 6.2 The report generator will produce detailed reports and visualize each student's strengths and weaknesses.

[1893] (Input): Comprehension Report

[1894] (Output):Detailed report

[1895] 6.3 The generated comprehension report will be sent to the educator terminal means so that the teacher can review it.

[1896] (Input):Detailed Report

[1897] (Output): Report sending completion message to teacher's device

[1898] Through the above processing steps, this system is able to efficiently record and analyze lesson content and student responses in real time, providing optimal learning support to each individual student.

[1899] (Application example 2)

[1900] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1901] Conventional educational support systems have difficulty in properly assessing students' understanding and learning progress, and in particular, they have been unable to provide sufficient learning support that takes into account the emotional state of each student. This can result in a lack of detailed learning support for each student, which can reduce learning effectiveness. Another issue is that teachers are unable to receive effective feedback when grasping the overall understanding level of the class and reflecting this in their next lesson plan.

[1902] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means: In this invention, the server includes a first image capturing device means for recording the content of the lesson, a second image capturing device means for recording the reactions of students, an information processing device means for receiving and storing video information from the first image capturing device and the second image capturing device, a machine learning model device means for analyzing the video information stored by the information processing device and evaluating the level of understanding of the students, an assignment generation device means for generating individual assignments based on the level of understanding evaluated by the machine learning model device, a communication device means for transmitting the assignments generated by the assignment generation device to student terminal devices, a display device means for displaying the assignments received by the student terminal devices and having the students perform them, a communication device means for transmitting the results of the assignments performed by the student terminal devices to the information processing device, a report generation device means for re-evaluating the results received by the information processing device and generating a comprehension report for the entire class, and a communication device means for transmitting the comprehension report generated by the report generation device to a teacher terminal device. This not only makes it possible to create individual tasks based on students' level of understanding and provide learning support that takes into account students' emotional state, but also allows the level of understanding of the entire class to be grasped, providing effective feedback that teachers can reflect in their next lesson plans.

[1903] The "first imaging device for recording the content of the lesson" is a video device for recording the progress of the lesson, and is a camera device for capturing the entire lesson, including what the teacher is saying and what he is writing on the blackboard.

[1904] The "second imaging device for recording students' reactions" is a video device for recording students' facial expressions and behavior, and a camera device for monitoring how students are reacting to lessons.

[1905] An "information processing device" is a computer system for receiving and storing video information that records lesson content and student responses, and is a device that analyzes and stores video data.

[1906] A "machine learning model device" is a data processing device that analyzes video data stored in an information processing device and evaluates students' level of understanding, and is an artificial intelligence device that has algorithms for facial recognition and behavioral analysis.

[1907] The "task generation device" is a device that automatically generates individual tasks based on the level of understanding assessed by the machine learning model device, and is a device for providing learning content adapted to each student.

[1908] The "communication device" is a network device for transmitting and receiving data between the information processing device, student terminal device, and teacher terminal device, and is a device for transmitting assignments and rece...

Claims

1. a first camera means for recording the content of the lesson; a second camera means for recording student responses; a server means for receiving and storing video data from the first camera means and the second camera means; an artificial intelligence model means for analyzing the video data stored by the server means and evaluating the students' understanding; a task generation means for generating individual tasks based on the level of understanding evaluated by the artificial intelligence model means; a transmission means for transmitting the assignment generated by the assignment generation means to a student terminal means; a display means for displaying the assignment received by the student terminal means and having the student carry it out; a transmitting means for transmitting the results of the assignments executed by the student terminal means to a server means; a report generating means for re-evaluating the results received by the server means and generating a report on the level of understanding of the entire class; The system further includes a transmitting means for transmitting the comprehension report generated by the report generating means to a teacher terminal means.

2. 2. The system according to claim 1, wherein said first camera means and said second camera means transmit video data to server means in a streaming format in real time.

3. 2. The system according to claim 1, wherein said artificial intelligence model means analyzes the expressions and actions of students using a face recognition algorithm and scores their level of understanding.

Images