system

Abstract

Provide a system. 【Solution means】 Means for receiving video data of students, Means for converting the received video data into an analyzable format, Means for extracting feature data from the expressions and actions of students, Means for analyzing the extracted feature data to evaluate the understanding and concentration of students, Means for generating feedback to teachers based on the analysis results, Means for notifying teachers in real time when the understanding and concentration of students fall below the standard, Means for generating a comprehensive evaluation report of the class, Means for proposing training content to support the improvement of teachers' guidance skills, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a classroom, it is difficult to objectively evaluate the understanding and concentration of students in real time. Therefore, teachers lack means to adjust the progress of the class on the spot, and it is difficult to maximize the learning effect of students. Also, it is impossible to obtain effective feedback after the class, and there is a limit to improving the teaching skills of teachers, which is an issue.

Means for Solving the Problems

[0005] This invention includes means for receiving video data of students in the classroom in real time and converting it into an analyzable format. It extracts feature data from students' facial expressions and movements and analyzes it using a deep learning algorithm to evaluate their comprehension and concentration levels. Furthermore, it generates feedback for teachers based on the analysis results and provides real-time notifications if students' comprehension or concentration levels do not meet the required standards. The system also includes means for generating a comprehensive evaluation report after the lesson and suggesting training content aimed at improving teachers' instructional skills. This makes it possible to improve the quality of lessons and maximize student learning effectiveness.

[0006] "Video data" refers to data in a format that can be stored or transmitted as a digital signal, obtained by a camera from visual information including students' facial expressions and movements.

[0007] An "analyzable format" refers to a format in which AI agents or analytical software can process the data and extract specific characteristics or patterns.

[0008] "Feature data" refers to specific elements and indicators necessary for analysis, extracted from students' facial expressions and movements.

[0009] "Comprehension level" is an indicator of how accurately students understand the lesson content.

[0010] "Concentration level" is an indicator of how much attention students are paying to the lesson.

[0011] A "deep learning algorithm" is a method that uses multi-layered artificial neural networks to automatically learn and analyze useful features from data.

[0012] "Feedback" refers to specific suggestions and points based on analysis results that teachers receive to improve their lessons.

[0013] "Real-time notification" is a method of immediately transmitting information to teachers based on the results of data analysis.

[0014] The "Comprehensive Evaluation Report" is a document that summarizes the results of an analysis of the entire lesson, including student comprehension levels, concentration levels, and areas for improvement in the lesson.

[0015] "Training content" refers to learning materials and programs proposed with the aim of improving teachers' instructional skills. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] The system for implementing this invention consists of a camera installed in the classroom, a server, and a teacher's terminal. The camera captures the facial expressions and movements of students in the classroom in real time and transmits the video data to the server. The server receives this video data and converts it into an analyzable format. From the converted data, an AI agent analyzes the students' facial expressions and posture and extracts feature data.

[0038] The server uses deep learning algorithms to analyze this feature data and evaluate students' comprehension and concentration levels. The evaluation results are recorded as numerical values ​​and indicators. The server also generates feedback based on the analysis results and data to improve teachers' teaching skills. This feedback provides specific areas for improvement regarding the progress of the lesson and is delivered to the teacher's terminal in real time.

[0039] Furthermore, the server immediately sends an alert to the teacher's terminal if students' comprehension or concentration levels fall below set standards. This alert function allows teachers to adjust the lesson progress on the spot as needed.

[0040] After the lesson ends, the server generates a comprehensive evaluation report based on the data collected during the lesson. This report includes detailed analysis results to help teachers improve their lessons and suggests teaching methods that will aid in improvement. The server also recommends appropriate training content to enhance teachers' teaching skills. This allows teachers to continuously improve their skills.

[0041] As a concrete example, if changes in student A's facial expression or eye movements are observed during a lesson, the camera transmits this information to the server in real time. The server uses an AI agent to analyze the sudden decrease in student A's concentration and sends an alert to the teacher's terminal stating, "Student A's concentration level is decreasing," prompting the teacher to take immediate action. Based on this alert, the teacher can improve the effectiveness of the lesson by providing appropriate guidance or engaging with the student.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The server receives video data in real time from cameras in the classroom. The video data includes information about students' facial expressions and posture.

[0045] Step 2:

[0046] The server converts the received video data into an analyzable format that the AI ​​agent can process. This conversion process includes adjusting the image resolution and removing noise.

[0047] Step 3:

[0048] The server uses an AI agent to analyze the students' facial expressions and movements from the converted data and extracts feature data. This feature data includes elements such as facial expressions and eye movements.

[0049] Step 4:

[0050] The server uses deep learning algorithms to analyze the extracted feature data. This analysis evaluates and quantifies students' comprehension and concentration levels.

[0051] Step 5:

[0052] The server generates feedback data for teachers based on the analysis results. This feedback includes suggestions for improving lessons and effective teaching methods.

[0053] Step 6:

[0054] The server sends a real-time alert to the teacher's terminal if students' comprehension or concentration levels fall below pre-set criteria. This alert allows the teacher to adjust the lesson progress on the spot.

[0055] Step 7:

[0056] After the lesson ends, the server generates a comprehensive evaluation report based on the lesson data. This report includes the progression of students' understanding and an overall evaluation of the lesson.

[0057] Step 8:

[0058] The server suggests training content aimed at improving teachers' teaching skills based on lesson results and feedback data. This allows teachers, as users, to further enhance their skills.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] In today's educational environment, it is difficult for teachers to grasp students' understanding and concentration levels in real time and adjust the pace of lessons accordingly. Furthermore, there is a lack of concrete feedback and training programs to improve teachers' own teaching skills. As a result, the quality of lessons and learning effectiveness are not maximized.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes means for receiving video data of learners in an educational environment, means for converting the received video data into analyzable information, and means for extracting characteristic information from the learners' nonverbal behavior. This makes it possible to evaluate the learners' knowledge acquisition level and level of engagement in real time and to provide educators with immediate improvement measures and training resources.

[0064] "Educational environment" refers to the physical or online space or situation provided for learners to acquire knowledge and skills.

[0065] A "learner" refers to a person who is the subject of acquiring knowledge and skills in an educational environment.

[0066] "Video data" refers to dynamic or static visual information collected by cameras or other video acquisition devices.

[0067] "Analyzable information" refers to information that has been formatted in a way that allows it to be processed and analyzed for a specific purpose.

[0068] "Nonverbal behavior" refers to actions that express intentions or emotions without using language, such as facial expressions or physical movements.

[0069] "Characteristic information" refers to important parameters and indicators extracted from nonverbal behavior for a particular analysis.

[0070] "Knowledge acquisition level" refers to an indicator that shows the extent to which learners understand and remember the content of the lessons.

[0071] "Degree of concentration" refers to an indicator that shows the extent to which learners are concentrating and paying attention to their learning activities.

[0072] An "educator" refers to a person who is responsible for providing guidance and support to learners in a learning environment.

[0073] "Improvement measures" refer to plans or actions taken to make a particular problem better.

[0074] "Training resources" refer to educational materials and programs provided to improve the skills and knowledge of educators.

[0075] The system for implementing this invention evaluates learners' understanding and engagement in an educational environment in real time and provides appropriate feedback and improvement measures to educators. The system consists of multiple elements.

[0076] The server receives video data from learners using video acquisition devices in classrooms or online learning environments. These devices include common video cameras and webcams, and the video data is transmitted to the server using streaming protocols. The server uses commonly used software libraries to encode this data and convert it into a format suitable for analysis. A specific example is image processing libraries such as OpenCV.

[0077] The server then analyzes the received video data using a generative AI model. Deep learning technology is employed for the analysis, extracting feature information from the learner's nonverbal behavior (such as facial expressions and posture). Regarding the AI ​​model, frameworks such as TENSORFLOW® and PyTorch are selected as needed.

[0078] The server uses this characteristic information to quantify learners' knowledge acquisition and engagement levels, and transmits the results to educators' terminals in real time. Based on this information, educators can make necessary improvements and adjust teaching methods in real time. In addition, along with the analysis results, the system supports the improvement of teaching skills by recommending optimal learning resources (such as training content) to educators.

[0079] As a concrete example, if student A's concentration level decreases during a lesson, the server immediately detects this and sends a warning message to the educator's terminal stating, "Student A's concentration level is decreasing." Based on this, the educator can provide appropriate guidance to the student on the spot, optimizing the learning effect.

[0080] Furthermore, an example of a prompt for a generative AI model is, "Please tell me how to analyze changes in learner concentration levels during class in real time and provide immediate feedback." By using this prompt, the system can suggest the optimal data processing and analysis methods.

[0081] In this way, the system operates in a manner that supports real-time educational improvement.

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The server receives video data of learners transmitted from cameras in the educational environment. This input data includes video from different angles and distances depending on the camera's position. Specifically, the server continuously captures video data using a streaming protocol and stores each frame in a buffer for real-time processing.

[0085] Step 2:

[0086] The server converts the received video data into analyzable information. This conversion process involves extracting each frame from the video data and preprocessing it using an image processing library (e.g., OpenCV). Specifically, noise is removed and the resolution is adjusted as needed. The output of this step is image data formatted for easy analysis.

[0087] Step 3:

[0088] The server provides pre-processed image data as input to the AI ​​agent, which then analyzes the data using a deep learning model (e.g., TensorFlow or PyTorch). This data analysis extracts feature information from the learner's nonverbal behavior, such as facial muscle movements and posture. Specifically, it performs calculations such as face recognition and joint position estimation within each image. The output is numerical data indicating the learner's level of understanding and engagement.

[0089] Step 4:

[0090] The server transmits the results of the analysis based on the characteristic information to the educators' terminals in real time, providing suggestions for improvement and feedback. In this step, the analysis results are output in the form of graphs and numerical data via an interface to visualize them. Based on this, educators can immediately take concrete steps to adjust the lesson content.

[0091] Step 5:

[0092] The server immediately notifies the educator's terminal of a warning if a learner's knowledge acquisition level or level of engagement falls below a pre-set threshold. This warning is displayed on the terminal as a specific message, prompting the educator to take corrective action.

[0093] Step 6:

[0094] After the lesson ends, the server generates a comprehensive evaluation report based on the collected data. This report summarizes the analyzed data to show learner performance and areas for improvement in the lesson. Specifically, the report is automatically output in text report format or as a visualized dashboard and distributed to educators.

[0095] Step 7:

[0096] The server recommends appropriate training content to improve the teaching skills of educators. In this step, based on the generated report data, an educational analysis algorithm is used to select effective learning resources and take specific actions to notify users. This allows educators, as users, to continuously improve their skills.

[0097] (Application Example 1)

[0098] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] In educational settings, it is essential for instructors to be able to grasp students' attention and comprehension levels in real time and provide effective educational support. Current educational systems sometimes make it difficult for instructors to intuitively assess students' comprehension and respond quickly. Furthermore, it is necessary to continuously support the skill development of instructors. Solving these challenges is the objective of this invention.

[0100] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0101] In this invention, the server includes means for receiving image information of a person, means for converting the received image information into an analyzable format, and means for extracting feature information from the person's facial expressions and movements. This enables the instructor to evaluate the person's level of attention and understanding in real time and provide appropriate feedback and educational support.

[0102] "Personal image information" refers to information that visually records an individual's physical characteristics and actions.

[0103] "Means of converting to an analyzable format" refers to the process of changing received image information into a data format suitable for analysis.

[0104] "Feature information" refers to specific data extracted from image information, such as an individual's facial expressions and body posture.

[0105] "Attention level" is an indicator that shows how much attention an individual is paying to a subject.

[0106] "Comprehension level" is an indicator that shows how well an individual understands the subject matter.

[0107] An "instructor" is a person whose job is to guide educational activities.

[0108] "Means for generating feedback" refers to methods for providing information to the instructor based on the analysis results.

[0109] A "Comprehensive Evaluation Report" is a document that summarizes the overall evaluation of educational activities.

[0110] "Educational materials" are learning content designed to support the improvement of instructors' teaching skills.

[0111] A "deep learning algorithm" is a machine learning technique used to perform highly accurate predictions and analyses through the analysis of complex data.

[0112] In its embodiment, this system primarily consists of a server, a camera for acquiring image information of people, and a terminal used by the instructor. The server converts the received image data into an analyzable format, and then analyzes the collected feature information using a deep learning algorithm. Based on the analysis results, the server can evaluate the level of attention and understanding of the person.

[0113] The evaluation results are provided as real-time feedback to the instructor's terminal, and if necessary, cautionary notifications are sent to the instructor. Furthermore, a comprehensive evaluation report is generated after the activity is completed, and the instructor receives suggestions that will help improve the teaching. In this process, image processing libraries such as OpenCV and deep learning frameworks such as TensorFlow and PyTorch are used.

[0114] As a concrete example, during a class, a camera observes a person's facial expressions in real time, and if the server uses an AI agent to determine that the person's level of attention is decreasing, a notification saying "Attention level is decreasing" is sent to the instructor's terminal. This allows the instructor to take immediate action. Furthermore, when utilizing the generative AI model, a prompt message such as "We want to evaluate the person's level of attention and understanding in real time and provide feedback that will be useful for educational support" is used.

[0115] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0116] Step 1:

[0117] The server receives real-time video data from cameras in the classroom. At this time, the camera's video data is the input, and this data is sent to the server.

[0118] Step 2:

[0119] The server converts the received video data into a format that can be analyzed. The input is video data received from the camera, and the output is data in a format that can be processed by a deep learning algorithm. This conversion is performed using OpenCV, an image processing software.

[0120] Step 3:

[0121] The server extracts feature information based on the converted data. The input here is data converted into an analyzable format, and the output is feature information related to facial expressions and posture. Deep learning techniques using TensorFlow and PyTorch are utilized for feature extraction.

[0122] Step 4:

[0123] The server analyzes the extracted feature information and evaluates the level of attention and understanding of the person. The input is the extracted feature information, and the output is numerical data representing the level of attention and understanding as evaluation results. A deep learning algorithm is used for this evaluation.

[0124] Step 5:

[0125] The server generates and provides feedback to the instructor's terminal based on the evaluation results. The input is the evaluation result, and the output is the feedback message. To generate the feedback, the message is constructed according to predetermined criteria.

[0126] Step 6:

[0127] The server sends a real-time alert notification to the instructor's terminal if attention or comprehension levels fall below set criteria. The input is the reduced evaluation result, and the output is a warning message. The alert is delivered to the instructor through the notification system.

[0128] Step 7:

[0129] After the lesson ends, the server generates a comprehensive evaluation report of the activity and provides it to the instructor. The input is evaluation data collected during the lesson, and the output is a comprehensive evaluation report that includes suggestions for improvement. This report also includes suggestions for educational materials that will help improve the instructor's teaching skills.

[0130] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0131] This invention incorporates a new emotion engine to analyze the emotional state of students in the classroom. The entire system consists of a group of cameras installed in the classroom, a server, a teacher's terminal, and the emotion engine. The cameras in the classroom capture students' facial expressions and movements in real time and transmit the video data to the server. This data is processed by an AI agent and the emotion engine.

[0132] The server first converts the received video data into an analyzable format. This conversion includes adjusting the image resolution and removing noise necessary for analysis. After conversion, the AI ​​agent extracts feature data from the students' facial expressions and postures. Based on this feature data, the emotion engine identifies the students' emotional states. For example, it individually recognizes smiles and confused expressions and records them as emotion data.

[0133] The server further uses deep learning algorithms to analyze students' comprehension, concentration, and emotional data. This analysis is quantified and recorded along a timeline, allowing for a comprehensive evaluation of students' comprehension and emotional changes.

[0134] Based on the analysis results, the server generates feedback for the teacher. This feedback includes suggestions for improvements to the lesson content and teaching methods tailored to the emotional state of specific students. In particular, if a student's emotional state indicates resistance to learning, focused feedback is provided to the teacher's terminal.

[0135] The real-time alert function sends an alert to the teacher's terminal if the server analyzes that a student's concentration level or emotional state falls below a pre-set standard, prompting a quick response.

[0136] After the lesson ends, the server generates a comprehensive evaluation report based on the lesson data. This report includes detailed analysis of students' understanding and emotional states, as well as an overall evaluation of the lesson. Furthermore, the server recommends appropriate training content to improve teachers' teaching skills.

[0137] As a concrete example, if anxiety is detected from student B's facial expression during class, the emotion engine sends that information to the server. The server analyzes this data and notifies the teacher's terminal with an alert stating, "Student B is feeling anxious." This alert allows the teacher to quickly take concrete actions to alleviate student B's anxiety.

[0138] The following describes the processing flow.

[0139] Step 1:

[0140] The server receives video data in real time from cameras in the classroom. The video data includes detailed information about students' facial expressions and movements.

[0141] Step 2:

[0142] The server converts the received video data into an analyzable format. This data conversion includes noise reduction and frame extraction for face recognition.

[0143] Step 3:

[0144] The server processes the converted data using an AI agent to extract feature data from the students' facial expressions and movements. This feature data includes things like smiles, confusion, and eye movements.

[0145] Step 4:

[0146] The server uses an emotion engine to identify a student's emotional state from specific facial expressions and behavioral patterns. For example, a smile is recognized as indicating happiness, while a furrowed brow suggests confusion.

[0147] Step 5:

[0148] The server uses deep learning algorithms to analyze feature data and sentiment data. This analysis also quantifies learning states such as comprehension and concentration levels.

[0149] Step 6:

[0150] The server generates feedback data for teachers based on the analysis results. This feedback includes suggestions for teaching methods and lesson improvements that take into account the emotional state of students.

[0151] Step 7:

[0152] The server sends real-time alerts to teachers' terminals if students' concentration levels or emotional states fall below set criteria. These alerts immediately provide teachers with information about students who require attention.

[0153] Step 8:

[0154] After the lesson ends, the server generates a comprehensive evaluation report based on the data collected during the lesson. This report details the progression of understanding, changes in emotional state, and areas for improvement in instruction.

[0155] Step 9:

[0156] The server suggests training content aimed at improving teachers' teaching skills based on lesson results and feedback. This allows teachers, as users, to gain concrete ways to improve themselves from individual lessons.

[0157] (Example 2)

[0158] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0159] In educational settings, there is a need to accurately grasp students' learning progress and comprehension levels in real time, and to propose appropriate teaching methods based on that information. Furthermore, there is a need for methods to evaluate how students' emotional states affect learning and to conduct more effective education. However, conventional methods have not adequately analyzed video and emotional data, placing a significant burden on educators.

[0160] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0161] In this invention, the server includes means for acquiring video data of students, means for identifying their emotional state, and means for generating feedback for educators. This makes it possible to analyze students' comprehension and concentration levels in real time and suggest appropriate teaching methods.

[0162] "Student video data" refers to visual information that records students' activities and facial expressions within the classroom.

[0163] "Means of converting to a data format" refers to a processing device that formats received video data into a format suitable for analysis.

[0164] "Methods for extracting feature data" refers to algorithms for extracting changes in students' facial expressions and posture as numerical values ​​or categories.

[0165] "Means for identifying emotional states" refers to a system that uses extracted feature data to determine students' emotional responses.

[0166] "Means of analysis" refers to processors used to scientifically evaluate students' learning progress and concentration levels.

[0167] "Means of generating feedback" refers to the function of formulating guidance advice and improvement measures for educators based on the analysis results.

[0168] "Means of providing real-time notifications" refers to a system that immediately alerts educators when a student's status information falls below a certain standard.

[0169] "Means for generating a comprehensive evaluation report" refers to a tool that creates a report summarizing the analysis results of the entire course.

[0170] "Means of proposing learning content" refers to a system that provides appropriate training programs aimed at improving the skills of educators.

[0171] This system is designed to analyze students' emotional states in educational settings. The system primarily consists of cameras, a server, educator terminals, and an emotion analysis engine. Each device is connected via a network, allowing for real-time data exchange.

[0172] First, a camera captures images of students in the classroom and sends the video data to a server. The camera used at this time not only acquires high-resolution video but also has the ability to capture data at an appropriate frame rate and send it to the server quickly.

[0173] The server converts the received video data into an analyzable format. This conversion process uses image processing software to standardize the data resolution and remove noise. Furthermore, the AI ​​agent extracts features from the students' facial expressions and posture data and associates them with their emotional states. A deep learning model is involved in this process, enabling highly accurate emotion identification.

[0174] The converted and analyzed data is quantified and recorded by the server along with detailed learning progress. The analysis results are displayed to educators on their devices as real-time feedback. This feedback includes student comprehension and concentration levels, as well as suggestions for appropriate teaching methods.

[0175] For example, if the emotion engine detects that student B is feeling anxious during class, the server will immediately display an alert on the educator's terminal stating, "Student B is feeling anxious," allowing the educator to take prompt and appropriate action.

[0176] As a concrete example of a prompt to a generating AI model, it is presented as follows: "Please describe the outline of a system that analyzes students' anxiety and concentration levels in real time based on their facial expression data in the classroom and notifies educators as needed."

[0177] This technology is useful for improving students' learning efficiency and for educators to provide individualized instruction.

[0178] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0179] Step 1:

[0180] The server acquires video data of students from cameras installed in the classroom. The cameras capture high-resolution video in real time and send it to the server. The input is video of students, and the output is in an analyzable video file format.

[0181] Step 2:

[0182] The server preprocesses the received video data. Specifically, it generates clean data by appropriately adjusting the image resolution and performing noise filtering. The input is video of students transmitted from the camera, and the output is denoised image data.

[0183] Step 3:

[0184] The server uses an AI agent to extract specific facial expression and posture features from pre-processed image data. This process employs machine learning models and advanced pattern recognition techniques. The input is pre-processed image data, and the output is numerical data representing the students' facial features.

[0185] Step 4:

[0186] The server analyzes feature data using an emotion engine to identify the student's emotional state. A deep learning algorithm is used to convert facial expression characteristic values ​​into emotion categories. The input is numerical characteristic data, and the output is the identified emotional state.

[0187] Step 5:

[0188] The server analyzes comprehension and concentration levels based on emotional states. This analysis integrates multiple data points, including emotional data, to assess the overall learning situation. The input is emotional state data, and the output is quantified comprehension and concentration levels.

[0189] Step 6:

[0190] The server generates and sends feedback to the educator's terminal based on the analysis results. This feedback is designed to include specific teaching actions and improvement suggestions. The input is the analysis results, and the output is a feedback message for the educator.

[0191] (Application Example 2)

[0192] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0193] In homes and educational settings, it is crucial to understand learners' emotional states and concentration levels in real time and provide appropriate feedback. However, conventional systems lack real-time capabilities and sufficient emotion-based feedback, making effective learning support difficult. Solving this problem is essential.

[0194] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0195] In this invention, the server includes means for receiving video information, means for converting the received video information into an analyzable format, and means for extracting characteristic information from the subject's facial expressions and movements. This enables accurate understanding of the learner's emotions and level of concentration, and allows for real-time responses and feedback.

[0196] "Visual information" refers to visual data that captures the subject's facial expressions and movements.

[0197] "Means of converting into an analyzable format" refers to the process of converting received video information into a format suitable for emotion recognition and concentration level evaluation.

[0198] "Feature information" refers to important data extracted from a subject's facial expressions and movements, used to determine their emotions and level of concentration.

[0199] A "means for evaluating comprehension and concentration" is a system that uses extracted characteristic information to determine the depth of understanding and the state of concentration during the learning or work being performed.

[0200] "Means of generating feedback for instructors" refers to the process of creating information based on analysis results to provide areas for improvement and advice regarding learning and instruction.

[0201] The "means of real-time notification" refer to a function that immediately informs instructors if, based on the evaluation results, the current situation needs to be resolved or improved.

[0202] "Methods for generating a comprehensive evaluation report of educational activities" refers to the process of analyzing data accumulated throughout the entire activity in question and putting a comprehensive evaluation into writing.

[0203] "A means of proposing training content that supports the improvement of leaders' leadership abilities" refers to a system that proposes the most suitable educational resources for leaders to acquire more effective teaching methods.

[0204] "Means for generating voice output based on emotional information" refers to a function that generates and transmits voice feedback corresponding to the emotional state of the target.

[0205] The system for realizing this application consists of robots designed to support home education and digital devices in the learning environment. The system is configured and operated as follows:

[0206] The server first receives video information. This video information is captured by a camera built into the robot and includes the learner's facial expressions and movements. The server converts the received data into a format that can be analyzed. Here, noise reduction and image resolution adjustment can be performed, and image processing libraries such as OpenCV can be used.

[0207] The system extracts the subject's facial expressions and movements as feature information from the received video data. Deep learning frameworks such as TensorFlow can be used for this purpose. By inputting the extracted feature information into an AI model, the learner's emotional state and level of concentration can be evaluated.

[0208] The evaluation results are processed as real-time feedback and communicated to the learner via audio output and screen display. This allows learners to immediately receive advice such as, "Continue studying with focus."

[0209] Furthermore, the server can generate a comprehensive evaluation report of educational activities. This allows for an analysis of overall learning progress and comprehension, providing detailed reports to instructors and parents.

[0210] For example, if a learner is found to be losing focus while solving a math problem, the system can output an encouraging voice message such as, "Let's try a little harder!" An example of a prompt to the generative AI model might be, "Output an example of voice feedback that will help a child maintain a sense of security and focus while learning."

[0211] In this way, the system provides real-time support that responds to the learner's emotions, offering a more effective learning environment.

[0212] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0213] Step 1:

[0214] The server receives video information from the robot's built-in camera. The input at this stage is real-time video data, including the students' facial expressions and movements. By receiving this video data, the server prepares for the next analysis process.

[0215] Step 2:

[0216] The server converts the received video information into an analyzable format by applying noise reduction and resolution adjustments. The input is the video data obtained in step 1, and the output is clear data processed using an image processing library such as OpenCV. This improves the accuracy of the analysis.

[0217] Step 3:

[0218] The server extracts the subject's facial expressions and movements as feature information from clear video footage. At this stage, feature vectors are generated using a deep learning framework such as TensorFlow. The input is the processed video data obtained in step 2, and the output is feature data suitable for analysis by the AI ​​model.

[0219] Step 4:

[0220] The server uses an AI model to evaluate the learner's emotional state and concentration level based on the extracted feature information. The input is the feature information obtained in step 3, and the output is analyzed data showing the learner's comprehension, concentration level, and emotional state. This data forms the basis for generating feedback based on the evaluation.

[0221] Step 5:

[0222] Based on the evaluation results, the server generates real-time audio or visual feedback and transmits it to the learner via the device. For example, if concentration wavers, it can output a message such as "Try a little harder." The input is the analysis result data obtained in step 4, and the output is a feedback message tailored to the learner.

[0223] Step 6:

[0224] The server aggregates evaluation data over a long period and automatically generates a comprehensive evaluation report of educational activities. The input is analysis data accumulated from multiple sessions, and the output is a detailed evaluation report provided to instructors and parents. This allows for a grasp of learners' progress and achievements.

[0225] Step 7:

[0226] Based on the analysis results, the server recommends training content for instructors and provides a means to improve their teaching skills. The input is long-term analysis data, and the output is a suggestion of educational resources for instructors. This operation allows instructors to learn more effective teaching methods.

[0227] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0228] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0229] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0230] [Second Embodiment]

[0231] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0232] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0233] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0234] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0235] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0236] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0237] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0238] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0239] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0240] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0241] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0242] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0243] The system for implementing this invention consists of a camera installed in the classroom, a server, and a teacher's terminal. The camera captures the facial expressions and movements of students in the classroom in real time and transmits the video data to the server. The server receives this video data and converts it into an analyzable format. From the converted data, an AI agent analyzes the students' facial expressions and posture and extracts feature data.

[0244] The server uses deep learning algorithms to analyze this feature data and evaluate students' comprehension and concentration levels. The evaluation results are recorded as numerical values ​​and indicators. The server also generates feedback based on the analysis results and data to improve teachers' teaching skills. This feedback provides specific areas for improvement regarding the progress of the lesson and is delivered to the teacher's terminal in real time.

[0245] Furthermore, the server immediately sends an alert to the teacher's terminal if students' comprehension or concentration levels fall below set standards. This alert function allows teachers to adjust the lesson progress on the spot as needed.

[0246] After the lesson ends, the server generates a comprehensive evaluation report based on the data collected during the lesson. This report includes detailed analysis results to help teachers improve their lessons and suggests teaching methods that will aid in improvement. The server also recommends appropriate training content to enhance teachers' teaching skills. This allows teachers to continuously improve their skills.

[0247] As a concrete example, if changes in student A's facial expression or eye movements are observed during a lesson, the camera transmits this information to the server in real time. The server uses an AI agent to analyze the sudden decrease in student A's concentration and sends an alert to the teacher's terminal stating, "Student A's concentration level is decreasing," prompting the teacher to take immediate action. Based on this alert, the teacher can improve the effectiveness of the lesson by providing appropriate guidance or engaging with the student.

[0248] The following describes the processing flow.

[0249] Step 1:

[0250] The server receives video data in real time from cameras in the classroom. The video data includes information about students' facial expressions and posture.

[0251] Step 2:

[0252] The server converts the received video data into an analyzable format that the AI ​​agent can process. This conversion process includes adjusting the image resolution and removing noise.

[0253] Step 3:

[0254] The server uses an AI agent to analyze the students' facial expressions and movements from the converted data and extracts feature data. This feature data includes elements such as facial expressions and eye movements.

[0255] Step 4:

[0256] The server uses deep learning algorithms to analyze the extracted feature data. This analysis evaluates and quantifies students' comprehension and concentration levels.

[0257] Step 5:

[0258] The server generates feedback data for teachers based on the analysis results. This feedback includes suggestions for improving lessons and effective teaching methods.

[0259] Step 6:

[0260] The server sends a real-time alert to the teacher's terminal if students' comprehension or concentration levels fall below pre-set criteria. This alert allows the teacher to adjust the lesson progress on the spot.

[0261] Step 7:

[0262] After the lesson ends, the server generates a comprehensive evaluation report based on the lesson data. This report includes the progression of students' understanding and an overall evaluation of the lesson.

[0263] Step 8:

[0264] The server suggests training content aimed at improving teachers' teaching skills based on lesson results and feedback data. This allows teachers, as users, to further enhance their skills.

[0265] (Example 1)

[0266] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0267] In today's educational environment, it is difficult for teachers to grasp students' understanding and concentration levels in real time and adjust the pace of lessons accordingly. Furthermore, there is a lack of concrete feedback and training programs to improve teachers' own teaching skills. As a result, the quality of lessons and learning effectiveness are not maximized.

[0268] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0269] In this invention, the server includes means for receiving video data of learners in an educational environment, means for converting the received video data into analyzable information, and means for extracting characteristic information from the learners' nonverbal behavior. This makes it possible to evaluate the learners' knowledge acquisition level and level of engagement in real time and to provide educators with immediate improvement measures and training resources.

[0270] "Educational environment" refers to the physical or online space or situation provided for learners to acquire knowledge and skills.

[0271] A "learner" refers to a person who is the subject of acquiring knowledge and skills in an educational environment.

[0272] "Video data" refers to dynamic or static visual information collected by cameras or other video acquisition devices.

[0273] "Analyzable information" refers to information that has been formatted in a way that allows it to be processed and analyzed for a specific purpose.

[0274] "Nonverbal behavior" refers to actions that express intentions or emotions without using language, such as facial expressions or physical movements.

[0275] "Characteristic information" refers to important parameters and indicators extracted from nonverbal behavior for a particular analysis.

[0276] "Knowledge acquisition level" refers to an indicator that shows the extent to which learners understand and remember the content of the lessons.

[0277] "Degree of concentration" refers to an indicator that shows the extent to which learners are concentrating and paying attention to their learning activities.

[0278] An "educator" refers to a person who is responsible for providing guidance and support to learners in a learning environment.

[0279] "Improvement measures" refer to plans or actions taken to make a particular problem better.

[0280] "Training resources" refer to educational materials and programs provided to improve the skills and knowledge of educators.

[0281] The system for implementing this invention evaluates learners' understanding and engagement in an educational environment in real time and provides appropriate feedback and improvement measures to educators. The system consists of multiple elements.

[0282] The server receives video data of learners from video acquisition devices used in classrooms or online education settings. This video acquisition device includes common video cameras and webcams, and the video data is transmitted to the server using a streaming protocol. The server utilizes commonly used software libraries as a technology to encode this data and convert it into a format that can be analyzed. Specific examples include image processing libraries such as OpenCV.

[0283] Based on the subsequently received video data, the server analyzes the data using a generative AI model. For the analysis, deep learning techniques are adopted to extract feature information from the non-verbal behaviors (such as expressions and postures) of the learners. Regarding the AI model, frameworks such as TensorFlow and PyTorch are selected as needed.

[0284] The server quantifies the knowledge acquisition level and concentration of the learners using this feature information and transmits the results to the terminals of the education staff in real-time. Based on this information, the education staff can make necessary improvements and adjust the teaching methods in real-time. Also, in conjunction with the analysis results, the system recommends the optimal learning resources (such as training content) to the education staff to support the improvement of teaching skills.

[0285] As a specific example, when the concentration of learner A decreases during a class, the server immediately detects this and sends a warning "The concentration of learner A is decreasing" to the terminal of the education staff. Based on this, the education staff can provide appropriate guidance to the learner on the spot and optimize the learning effect.

[0286] Furthermore, as an example of a prompt sentence for the generative AI model, there is "Please teach me a method to analyze the change in the concentration of learners during a class in real-time and provide immediate feedback." By using this prompt sentence, the system can present optimal data processing and analysis methods.

[0287] In this way, the system operates in a manner that supports real-time educational improvement.

[0288] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0289] Step 1:

[0290] The server receives video data of learners transmitted from cameras in the educational environment. This input data includes video from different angles and distances depending on the camera's position. Specifically, the server continuously captures video data using a streaming protocol and stores each frame in a buffer for real-time processing.

[0291] Step 2:

[0292] The server converts the received video data into analyzable information. This conversion process involves extracting each frame from the video data and preprocessing it using an image processing library (e.g., OpenCV). Specifically, noise is removed and the resolution is adjusted as needed. The output of this step is image data formatted for easy analysis.

[0293] Step 3:

[0294] The server provides pre-processed image data as input to the AI ​​agent, which then analyzes the data using a deep learning model (e.g., TensorFlow or PyTorch). This data analysis extracts feature information from the learner's nonverbal behavior, such as facial muscle movements and posture. Specifically, it performs calculations such as face recognition and joint position estimation within each image. The output is numerical data indicating the learner's level of understanding and engagement.

[0295] Step 4:

[0296] The server transmits the results of the analysis based on the characteristic information to the educators' terminals in real time, providing suggestions for improvement and feedback. In this step, the analysis results are output in the form of graphs and numerical data via an interface to visualize them. Based on this, educators can immediately take concrete steps to adjust the lesson content.

[0297] Step 5:

[0298] The server immediately notifies the educator's terminal of a warning if a learner's knowledge acquisition level or level of engagement falls below a pre-set threshold. This warning is displayed on the terminal as a specific message, prompting the educator to take corrective action.

[0299] Step 6:

[0300] After the lesson ends, the server generates a comprehensive evaluation report based on the collected data. This report summarizes the analyzed data to show learner performance and areas for improvement in the lesson. Specifically, the report is automatically output in text report format or as a visualized dashboard and distributed to educators.

[0301] Step 7:

[0302] The server recommends appropriate training content to improve the teaching skills of educators. In this step, based on the generated report data, an educational analysis algorithm is used to select effective learning resources and take specific actions to notify users. This allows educators, as users, to continuously improve their skills.

[0303] (Application Example 1)

[0304] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0305] In an educational environment, it is required that an instructor grasps the attention level and understanding level of a person in real time and provides effective educational support. In the current educational system, it may be difficult for an instructor to intuitively judge the understanding level of a person and respond quickly. Furthermore, it is also necessary to continuously support the improvement of the instructor's skills. Solving these problems is the object of this invention.

[0306] The specific processing by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is realized by the following respective means.

[0307] In this invention, the server includes means for receiving image information of a person, means for converting the received image information into an analyzable format, and means for extracting feature information from the expression and movement of the person. Thereby, it becomes possible for an instructor to evaluate the attention level and understanding level of a person in real time and provide appropriate feedback and educational support.

[0308] "Image information of a person" is information that visually records the external features and movements of an individual.

[0309] "Means for converting into an analyzable format" is a process of changing the received image information into a data format suitable for analysis.

[0310] "Feature information" is specific data such as the facial expression and body posture of an individual extracted from the image information.

[0311] "Attention level" is an index indicating how concentrated an individual is on the subject.

[0312] "Understanding level" is an index indicating how well an individual understands the content targeted.

[0313] "Instructor" is a person having the task of guiding educational activities.

[0314] "Means for generating feedback" is a method of providing information to an instructor based on the analysis result.

[0315] A "Comprehensive Evaluation Report" is a document that summarizes the overall evaluation of educational activities.

[0316] "Educational materials" are learning content designed to support the improvement of instructors' teaching skills.

[0317] A "deep learning algorithm" is a machine learning technique used to perform highly accurate predictions and analyses through the analysis of complex data.

[0318] In its embodiment, this system primarily consists of a server, a camera for acquiring image information of people, and a terminal used by the instructor. The server converts the received image data into an analyzable format, and then analyzes the collected feature information using a deep learning algorithm. Based on the analysis results, the server can evaluate the level of attention and understanding of the person.

[0319] The evaluation results are provided as real-time feedback to the instructor's terminal, and if necessary, cautionary notifications are sent to the instructor. Furthermore, a comprehensive evaluation report is generated after the activity is completed, and the instructor receives suggestions that will help improve the teaching. In this process, image processing libraries such as OpenCV and deep learning frameworks such as TensorFlow and PyTorch are used.

[0320] As a concrete example, during a class, a camera observes a person's facial expressions in real time, and if the server uses an AI agent to determine that the person's level of attention is decreasing, a notification saying "Attention level is decreasing" is sent to the instructor's terminal. This allows the instructor to take immediate action. Furthermore, when utilizing the generative AI model, a prompt message such as "We want to evaluate the person's level of attention and understanding in real time and provide feedback that will be useful for educational support" is used.

[0321] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0322] Step 1:

[0323] The server receives real-time video data from cameras in the classroom. At this time, the camera's video data is the input, and this data is sent to the server.

[0324] Step 2:

[0325] The server converts the received video data into a format that can be analyzed. The input is video data received from the camera, and the output is data in a format that can be processed by a deep learning algorithm. This conversion is performed using OpenCV, an image processing software.

[0326] Step 3:

[0327] The server extracts feature information based on the converted data. The input here is data converted into an analyzable format, and the output is feature information related to facial expressions and posture. Deep learning techniques using TensorFlow and PyTorch are utilized for feature extraction.

[0328] Step 4:

[0329] The server analyzes the extracted feature information and evaluates the level of attention and understanding of the person. The input is the extracted feature information, and the output is numerical data representing the level of attention and understanding as evaluation results. A deep learning algorithm is used for this evaluation.

[0330] Step 5:

[0331] The server generates and provides feedback to the instructor's terminal based on the evaluation results. The input is the evaluation result, and the output is the feedback message. To generate the feedback, the message is constructed according to predetermined criteria.

[0332] Step 6:

[0333] The server sends a real-time alert notification to the instructor's terminal if attention or comprehension levels fall below set criteria. The input is the reduced evaluation result, and the output is a warning message. The alert is delivered to the instructor through the notification system.

[0334] Step 7:

[0335] After the lesson ends, the server generates a comprehensive evaluation report of the activity and provides it to the instructor. The input is evaluation data collected during the lesson, and the output is a comprehensive evaluation report that includes suggestions for improvement. This report also includes suggestions for educational materials that will help improve the instructor's teaching skills.

[0336] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0337] This invention incorporates a new emotion engine to analyze the emotional state of students in the classroom. The entire system consists of a group of cameras installed in the classroom, a server, a teacher's terminal, and the emotion engine. The cameras in the classroom capture students' facial expressions and movements in real time and transmit the video data to the server. This data is processed by an AI agent and the emotion engine.

[0338] The server first converts the received video data into an analyzable format. This conversion includes adjusting the image resolution and removing noise necessary for analysis. After conversion, the AI ​​agent extracts feature data from the students' facial expressions and postures. Based on this feature data, the emotion engine identifies the students' emotional states. For example, it individually recognizes smiles and confused expressions and records them as emotion data.

[0339] The server further uses deep learning algorithms to analyze students' comprehension, concentration, and emotional data. This analysis is quantified and recorded along a timeline, allowing for a comprehensive evaluation of students' comprehension and emotional changes.

[0340] Based on the analysis results, the server generates feedback for the teacher. This feedback includes suggestions for improvements to the lesson content and teaching methods tailored to the emotional state of specific students. In particular, if a student's emotional state indicates resistance to learning, focused feedback is provided to the teacher's terminal.

[0341] The real-time alert function sends an alert to the teacher's terminal if the server analyzes that a student's concentration level or emotional state falls below a pre-set standard, prompting a quick response.

[0342] After the lesson ends, the server generates a comprehensive evaluation report based on the lesson data. This report includes detailed analysis of students' understanding and emotional states, as well as an overall evaluation of the lesson. Furthermore, the server recommends appropriate training content to improve teachers' teaching skills.

[0343] As a concrete example, if anxiety is detected from student B's facial expression during class, the emotion engine sends that information to the server. The server analyzes this data and notifies the teacher's terminal with an alert stating, "Student B is feeling anxious." This alert allows the teacher to quickly take concrete actions to alleviate student B's anxiety.

[0344] The following describes the processing flow.

[0345] Step 1:

[0346] The server receives video data in real time from cameras in the classroom. The video data includes detailed information about students' facial expressions and movements.

[0347] Step 2:

[0348] The server converts the received video data into an analyzable format. This data conversion includes noise reduction and frame extraction for face recognition.

[0349] Step 3:

[0350] The server processes the converted data using an AI agent to extract feature data from the students' facial expressions and movements. This feature data includes things like smiles, confusion, and eye movements.

[0351] Step 4:

[0352] The server uses an emotion engine to identify a student's emotional state from specific facial expressions and behavioral patterns. For example, a smile is recognized as indicating happiness, while a furrowed brow suggests confusion.

[0353] Step 5:

[0354] The server uses deep learning algorithms to analyze feature data and sentiment data. This analysis also quantifies learning states such as comprehension and concentration levels.

[0355] Step 6:

[0356] The server generates feedback data for teachers based on the analysis results. This feedback includes suggestions for teaching methods and lesson improvements that take into account the emotional state of students.

[0357] Step 7:

[0358] The server sends real-time alerts to teachers' terminals if students' concentration levels or emotional states fall below set criteria. These alerts immediately provide teachers with information about students who require attention.

[0359] Step 8:

[0360] After the lesson ends, the server generates a comprehensive evaluation report based on the data collected during the lesson. This report details the progression of understanding, changes in emotional state, and areas for improvement in instruction.

[0361] Step 9:

[0362] The server suggests training content aimed at improving teachers' teaching skills based on lesson results and feedback. This allows teachers, as users, to gain concrete ways to improve themselves from individual lessons.

[0363] (Example 2)

[0364] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0365] In educational settings, there is a need to accurately grasp students' learning progress and comprehension levels in real time, and to propose appropriate teaching methods based on that information. Furthermore, there is a need for methods to evaluate how students' emotional states affect learning and to conduct more effective education. However, conventional methods have not adequately analyzed video and emotional data, placing a significant burden on educators.

[0366] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0367] In this invention, the server includes means for acquiring video data of students, means for identifying their emotional state, and means for generating feedback for educators. This makes it possible to analyze students' comprehension and concentration levels in real time and suggest appropriate teaching methods.

[0368] "Student video data" refers to visual information that records students' activities and facial expressions within the classroom.

[0369] "Means of converting to a data format" refers to a processing device that formats received video data into a format suitable for analysis.

[0370] "Methods for extracting feature data" refers to algorithms for extracting changes in students' facial expressions and posture as numerical values ​​or categories.

[0371] "Means for identifying emotional states" refers to a system that uses extracted feature data to determine students' emotional responses.

[0372] "Means of analysis" refers to processors used to scientifically evaluate students' learning progress and concentration levels.

[0373] "Means of generating feedback" refers to the function of formulating guidance advice and improvement measures for educators based on the analysis results.

[0374] "Means of providing real-time notifications" refers to a system that immediately alerts educators when a student's status information falls below a certain standard.

[0375] "Means for generating a comprehensive evaluation report" refers to a tool that creates a report summarizing the analysis results of the entire course.

[0376] "Means of proposing learning content" refers to a system that provides appropriate training programs aimed at improving the skills of educators.

[0377] This system is designed to analyze students' emotional states in educational settings. The system primarily consists of cameras, a server, educator terminals, and an emotion analysis engine. Each device is connected via a network, allowing for real-time data exchange.

[0378] First, a camera captures images of students in the classroom and sends the video data to a server. The camera used at this time not only acquires high-resolution video but also has the ability to capture data at an appropriate frame rate and send it to the server quickly.

[0379] The server converts the received video data into an analyzable format. This conversion process uses image processing software to standardize the data resolution and remove noise. Furthermore, the AI ​​agent extracts features from the students' facial expressions and posture data and associates them with their emotional states. A deep learning model is involved in this process, enabling highly accurate emotion identification.

[0380] The converted and analyzed data is quantified and recorded by the server along with detailed learning progress. The analysis results are displayed to educators on their devices as real-time feedback. This feedback includes student comprehension and concentration levels, as well as suggestions for appropriate teaching methods.

[0381] For example, if the emotion engine detects that student B is feeling anxious during class, the server will immediately display an alert on the educator's terminal stating, "Student B is feeling anxious," allowing the educator to take prompt and appropriate action.

[0382] As a concrete example of a prompt to a generating AI model, it is presented as follows: "Please describe the outline of a system that analyzes students' anxiety and concentration levels in real time based on their facial expression data in the classroom and notifies educators as needed."

[0383] This technology is useful for improving students' learning efficiency and for educators to provide individualized instruction.

[0384] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0385] Step 1:

[0386] The server acquires video data of students from cameras installed in the classroom. The cameras capture high-resolution video in real time and send it to the server. The input is video of students, and the output is in an analyzable video file format.

[0387] Step 2:

[0388] The server preprocesses the received video data. Specifically, it generates clean data by appropriately adjusting the image resolution and performing noise filtering. The input is video of students transmitted from the camera, and the output is denoised image data.

[0389] Step 3:

[0390] The server uses an AI agent to extract specific facial expression and posture features from pre-processed image data. This process employs machine learning models and advanced pattern recognition techniques. The input is pre-processed image data, and the output is numerical data representing the students' facial features.

[0391] Step 4:

[0392] The server analyzes feature data using an emotion engine to identify the student's emotional state. A deep learning algorithm is used to convert facial expression characteristic values ​​into emotion categories. The input is numerical characteristic data, and the output is the identified emotional state.

[0393] Step 5:

[0394] The server analyzes comprehension and concentration levels based on emotional states. This analysis integrates multiple data points, including emotional data, to assess the overall learning situation. The input is emotional state data, and the output is quantified comprehension and concentration levels.

[0395] Step 6:

[0396] The server generates and sends feedback to the educator's terminal based on the analysis results. This feedback is designed to include specific teaching actions and improvement suggestions. The input is the analysis results, and the output is a feedback message for the educator.

[0397] (Application Example 2)

[0398] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0399] In homes and educational settings, it is crucial to understand learners' emotional states and concentration levels in real time and provide appropriate feedback. However, conventional systems lack real-time capabilities and sufficient emotion-based feedback, making effective learning support difficult. Solving this problem is essential.

[0400] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0401] In this invention, the server includes means for receiving video information, means for converting the received video information into an analyzable format, and means for extracting characteristic information from the subject's facial expressions and movements. This enables accurate understanding of the learner's emotions and level of concentration, and allows for real-time responses and feedback.

[0402] "Visual information" refers to visual data that captures the subject's facial expressions and movements.

[0403] "Means of converting into an analyzable format" refers to the process of converting received video information into a format suitable for emotion recognition and concentration level evaluation.

[0404] "Feature information" refers to important data extracted from a subject's facial expressions and movements, used to determine their emotions and level of concentration.

[0405] A "means for evaluating comprehension and concentration" is a system that uses extracted characteristic information to determine the depth of understanding and the state of concentration during the learning or work being performed.

[0406] "Means of generating feedback for instructors" refers to the process of creating information based on analysis results to provide areas for improvement and advice regarding learning and instruction.

[0407] The "means of real-time notification" refer to a function that immediately informs instructors if, based on the evaluation results, the current situation needs to be resolved or improved.

[0408] "Methods for generating a comprehensive evaluation report of educational activities" refers to the process of analyzing data accumulated throughout the entire activity in question and putting a comprehensive evaluation into writing.

[0409] "A means of proposing training content that supports the improvement of leaders' leadership abilities" refers to a system that proposes the most suitable educational resources for leaders to acquire more effective teaching methods.

[0410] "Means for generating voice output based on emotional information" refers to a function that generates and transmits voice feedback corresponding to the emotional state of the target.

[0411] The system for realizing this application consists of robots designed to support home education and digital devices in the learning environment. The system is configured and operated as follows:

[0412] The server first receives video information. This video information is captured by a camera built into the robot and includes the learner's facial expressions and movements. The server converts the received data into a format that can be analyzed. Here, noise reduction and image resolution adjustment can be performed, and image processing libraries such as OpenCV can be used.

[0413] The system extracts the subject's facial expressions and movements as feature information from the received video data. Deep learning frameworks such as TensorFlow can be used for this purpose. By inputting the extracted feature information into an AI model, the learner's emotional state and level of concentration can be evaluated.

[0414] The evaluation results are processed as real-time feedback and communicated to the learner via audio output and screen display. This allows learners to immediately receive advice such as, "Continue studying with focus."

[0415] Furthermore, the server can generate a comprehensive evaluation report of educational activities. This allows for an analysis of overall learning progress and comprehension, providing detailed reports to instructors and parents.

[0416] For example, if a learner is found to be losing focus while solving a math problem, the system can output an encouraging voice message such as, "Let's try a little harder!" An example of a prompt to the generative AI model might be, "Output an example of voice feedback that will help a child maintain a sense of security and focus while learning."

[0417] In this way, the system provides real-time support that responds to the learner's emotions, offering a more effective learning environment.

[0418] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0419] Step 1:

[0420] The server receives video information from the robot's built-in camera. The input at this stage is real-time video data, including the students' facial expressions and movements. By receiving this video data, the server prepares for the next analysis process.

[0421] Step 2:

[0422] The server converts the received video information into an analyzable format by applying noise reduction and resolution adjustments. The input is the video data obtained in step 1, and the output is clear data processed using an image processing library such as OpenCV. This improves the accuracy of the analysis.

[0423] Step 3:

[0424] The server extracts the subject's facial expressions and movements as feature information from clear video footage. At this stage, feature vectors are generated using a deep learning framework such as TensorFlow. The input is the processed video data obtained in step 2, and the output is feature data suitable for analysis by the AI ​​model.

[0425] Step 4:

[0426] The server uses an AI model to evaluate the learner's emotional state and concentration level based on the extracted feature information. The input is the feature information obtained in step 3, and the output is analyzed data showing the learner's comprehension, concentration level, and emotional state. This data forms the basis for generating feedback based on the evaluation.

[0427] Step 5:

[0428] Based on the evaluation results, the server generates real-time audio or visual feedback and transmits it to the learner via the device. For example, if concentration wavers, it can output a message such as "Try a little harder." The input is the analysis result data obtained in step 4, and the output is a feedback message tailored to the learner.

[0429] Step 6:

[0430] The server aggregates evaluation data over a long period and automatically generates a comprehensive evaluation report of educational activities. The input is analysis data accumulated from multiple sessions, and the output is a detailed evaluation report provided to instructors and parents. This allows for a grasp of learners' progress and achievements.

[0431] Step 7:

[0432] Based on the analysis results, the server recommends training content for instructors and provides a means to improve their teaching skills. The input is long-term analysis data, and the output is a suggestion of educational resources for instructors. This operation allows instructors to learn more effective teaching methods.

[0433] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0434] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0435] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0436] [Third Embodiment]

[0437] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0438] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0439] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0440] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0441] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0442] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0443] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0444] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0445] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0446] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0447] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0448] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0449] The system for implementing this invention consists of a camera installed in the classroom, a server, and a teacher's terminal. The camera captures the facial expressions and movements of students in the classroom in real time and transmits the video data to the server. The server receives this video data and converts it into an analyzable format. From the converted data, an AI agent analyzes the students' facial expressions and posture and extracts feature data.

[0450] The server uses deep learning algorithms to analyze this feature data and evaluate students' comprehension and concentration levels. The evaluation results are recorded as numerical values ​​and indicators. The server also generates feedback based on the analysis results and data to improve teachers' teaching skills. This feedback provides specific areas for improvement regarding the progress of the lesson and is delivered to the teacher's terminal in real time.

[0451] Furthermore, the server immediately sends an alert to the teacher's terminal if students' comprehension or concentration levels fall below set standards. This alert function allows teachers to adjust the lesson progress on the spot as needed.

[0452] After the lesson ends, the server generates a comprehensive evaluation report based on the data collected during the lesson. This report includes detailed analysis results to help teachers improve their lessons and suggests teaching methods that will aid in improvement. The server also recommends appropriate training content to enhance teachers' teaching skills. This allows teachers to continuously improve their skills.

[0453] As a concrete example, if changes in student A's facial expression or eye movements are observed during a lesson, the camera transmits this information to the server in real time. The server uses an AI agent to analyze the sudden decrease in student A's concentration and sends an alert to the teacher's terminal stating, "Student A's concentration level is decreasing," prompting the teacher to take immediate action. Based on this alert, the teacher can improve the effectiveness of the lesson by providing appropriate guidance or engaging with the student.

[0454] The following describes the processing flow.

[0455] Step 1:

[0456] The server receives video data in real time from cameras in the classroom. The video data includes information about students' facial expressions and posture.

[0457] Step 2:

[0458] The server converts the received video data into an analyzable format that the AI ​​agent can process. This conversion process includes adjusting the image resolution and removing noise.

[0459] Step 3:

[0460] The server uses an AI agent to analyze the students' facial expressions and movements from the converted data and extracts feature data. This feature data includes elements such as facial expressions and eye movements.

[0461] Step 4:

[0462] The server uses deep learning algorithms to analyze the extracted feature data. This analysis evaluates and quantifies students' comprehension and concentration levels.

[0463] Step 5:

[0464] The server generates feedback data for teachers based on the analysis results. This feedback includes suggestions for improving lessons and effective teaching methods.

[0465] Step 6:

[0466] The server sends a real-time alert to the teacher's terminal if students' comprehension or concentration levels fall below pre-set criteria. This alert allows the teacher to adjust the lesson progress on the spot.

[0467] Step 7:

[0468] After the lesson ends, the server generates a comprehensive evaluation report based on the lesson data. This report includes the progression of students' understanding and an overall evaluation of the lesson.

[0469] Step 8:

[0470] The server suggests training content aimed at improving teachers' teaching skills based on lesson results and feedback data. This allows teachers, as users, to further enhance their skills.

[0471] (Example 1)

[0472] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0473] In today's educational environment, it is difficult for teachers to grasp students' understanding and concentration levels in real time and adjust the pace of lessons accordingly. Furthermore, there is a lack of concrete feedback and training programs to improve teachers' own teaching skills. As a result, the quality of lessons and learning effectiveness are not maximized.

[0474] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0475] In this invention, the server includes means for receiving video data of learners in an educational environment, means for converting the received video data into analyzable information, and means for extracting characteristic information from the learners' nonverbal behavior. This makes it possible to evaluate the learners' knowledge acquisition level and level of engagement in real time and to provide educators with immediate improvement measures and training resources.

[0476] "Educational environment" refers to the physical or online space or situation provided for learners to acquire knowledge and skills.

[0477] A "learner" refers to a person who is the subject of acquiring knowledge and skills in an educational environment.

[0478] "Video data" refers to dynamic or static visual information collected by cameras or other video acquisition devices.

[0479] "Analyzable information" refers to information that has been formatted in a way that allows it to be processed and analyzed for a specific purpose.

[0480] "Nonverbal behavior" refers to actions that express intentions or emotions without using language, such as facial expressions or physical movements.

[0481] "Characteristic information" refers to important parameters and indicators extracted from nonverbal behavior for a particular analysis.

[0482] "Knowledge acquisition level" refers to an indicator that shows the extent to which learners understand and remember the content of the lessons.

[0483] "Degree of concentration" refers to an indicator that shows the extent to which learners are concentrating and paying attention to their learning activities.

[0484] An "educator" refers to a person who is responsible for providing guidance and support to learners in a learning environment.

[0485] "Improvement measures" refer to plans or actions taken to make a particular problem better.

[0486] "Training resources" refer to educational materials and programs provided to improve the skills and knowledge of educators.

[0487] The system for implementing this invention evaluates learners' understanding and engagement in an educational environment in real time and provides appropriate feedback and improvement measures to educators. The system consists of multiple elements.

[0488] The server receives video data from learners using video acquisition devices in classrooms or online learning environments. These devices include common video cameras and webcams, and the video data is transmitted to the server using streaming protocols. The server uses commonly used software libraries to encode this data and convert it into a format suitable for analysis. A specific example is image processing libraries such as OpenCV.

[0489] The server then analyzes the received video data using a generative AI model. Deep learning technology is employed for the analysis, extracting feature information from the learner's nonverbal behavior (such as facial expressions and posture). Regarding the AI ​​model, frameworks such as TensorFlow and PyTorch are selected as needed.

[0490] The server uses this characteristic information to quantify learners' knowledge acquisition and engagement levels, and transmits the results to educators' terminals in real time. Based on this information, educators can make necessary improvements and adjust teaching methods in real time. In addition, along with the analysis results, the system supports the improvement of teaching skills by recommending optimal learning resources (such as training content) to educators.

[0491] As a concrete example, if student A's concentration level decreases during a lesson, the server immediately detects this and sends a warning message to the educator's terminal stating, "Student A's concentration level is decreasing." Based on this, the educator can provide appropriate guidance to the student on the spot, optimizing the learning effect.

[0492] Furthermore, an example of a prompt for a generative AI model is, "Please tell me how to analyze changes in learner concentration levels during class in real time and provide immediate feedback." By using this prompt, the system can suggest the optimal data processing and analysis methods.

[0493] In this way, the system operates in a manner that supports real-time educational improvement.

[0494] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0495] Step 1:

[0496] The server receives video data of learners transmitted from cameras in the educational environment. This input data includes video from different angles and distances depending on the camera's position. Specifically, the server continuously captures video data using a streaming protocol and stores each frame in a buffer for real-time processing.

[0497] Step 2:

[0498] The server converts the received video data into analyzable information. This conversion process involves extracting each frame from the video data and preprocessing it using an image processing library (e.g., OpenCV). Specifically, noise is removed and the resolution is adjusted as needed. The output of this step is image data formatted for easy analysis.

[0499] Step 3:

[0500] The server provides pre-processed image data as input to the AI ​​agent, which then analyzes the data using a deep learning model (e.g., TensorFlow or PyTorch). This data analysis extracts feature information from the learner's nonverbal behavior, such as facial muscle movements and posture. Specifically, it performs calculations such as face recognition and joint position estimation within each image. The output is numerical data indicating the learner's level of understanding and engagement.

[0501] Step 4:

[0502] The server transmits the results of the analysis based on the characteristic information to the educators' terminals in real time, providing suggestions for improvement and feedback. In this step, the analysis results are output in the form of graphs and numerical data via an interface to visualize them. Based on this, educators can immediately take concrete steps to adjust the lesson content.

[0503] Step 5:

[0504] The server immediately notifies the educator's terminal of a warning if a learner's knowledge acquisition level or level of engagement falls below a pre-set threshold. This warning is displayed on the terminal as a specific message, prompting the educator to take corrective action.

[0505] Step 6:

[0506] After the lesson ends, the server generates a comprehensive evaluation report based on the collected data. This report summarizes the analyzed data to show learner performance and areas for improvement in the lesson. Specifically, the report is automatically output in text report format or as a visualized dashboard and distributed to educators.

[0507] Step 7:

[0508] The server recommends appropriate training content to improve the teaching skills of educators. In this step, based on the generated report data, an educational analysis algorithm is used to select effective learning resources and take specific actions to notify users. This allows educators, as users, to continuously improve their skills.

[0509] (Application Example 1)

[0510] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0511] In educational settings, it is essential for instructors to be able to grasp students' attention and comprehension levels in real time and provide effective educational support. Current educational systems sometimes make it difficult for instructors to intuitively assess students' comprehension and respond quickly. Furthermore, it is necessary to continuously support the skill development of instructors. Solving these challenges is the objective of this invention.

[0512] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0513] In this invention, the server includes means for receiving image information of a person, means for converting the received image information into an analyzable format, and means for extracting feature information from the person's facial expressions and movements. This enables the instructor to evaluate the person's level of attention and understanding in real time and provide appropriate feedback and educational support.

[0514] "Personal image information" refers to information that visually records an individual's physical characteristics and actions.

[0515] "Means of converting to an analyzable format" refers to the process of changing received image information into a data format suitable for analysis.

[0516] "Feature information" refers to specific data extracted from image information, such as an individual's facial expressions and body posture.

[0517] "Attention level" is an indicator that shows how much attention an individual is paying to a subject.

[0518] "Comprehension level" is an indicator that shows how well an individual understands the subject matter.

[0519] An "instructor" is a person whose job is to guide educational activities.

[0520] "Means for generating feedback" refers to methods for providing information to the instructor based on the analysis results.

[0521] A "Comprehensive Evaluation Report" is a document that summarizes the overall evaluation of educational activities.

[0522] "Educational materials" are learning content designed to support the improvement of instructors' teaching skills.

[0523] A "deep learning algorithm" is a machine learning technique used to perform highly accurate predictions and analyses through the analysis of complex data.

[0524] In its embodiment, this system primarily consists of a server, a camera for acquiring image information of people, and a terminal used by the instructor. The server converts the received image data into an analyzable format, and then analyzes the collected feature information using a deep learning algorithm. Based on the analysis results, the server can evaluate the level of attention and understanding of the person.

[0525] The evaluation results are provided as real-time feedback to the instructor's terminal, and if necessary, cautionary notifications are sent to the instructor. Furthermore, a comprehensive evaluation report is generated after the activity is completed, and the instructor receives suggestions that will help improve the teaching. In this process, image processing libraries such as OpenCV and deep learning frameworks such as TensorFlow and PyTorch are used.

[0526] As a concrete example, during a class, a camera observes a person's facial expressions in real time, and if the server uses an AI agent to determine that the person's level of attention is decreasing, a notification saying "Attention level is decreasing" is sent to the instructor's terminal. This allows the instructor to take immediate action. Furthermore, when utilizing the generative AI model, a prompt message such as "We want to evaluate the person's level of attention and understanding in real time and provide feedback that will be useful for educational support" is used.

[0527] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0528] Step 1:

[0529] The server receives real-time video data from cameras in the classroom. At this time, the camera's video data is the input, and this data is sent to the server.

[0530] Step 2:

[0531] The server converts the received video data into a format that can be analyzed. The input is video data received from the camera, and the output is data in a format that can be processed by a deep learning algorithm. This conversion is performed using OpenCV, an image processing software.

[0532] Step 3:

[0533] The server extracts feature information based on the converted data. The input here is data converted into an analyzable format, and the output is feature information related to facial expressions and posture. Deep learning techniques using TensorFlow and PyTorch are utilized for feature extraction.

[0534] Step 4:

[0535] The server analyzes the extracted feature information and evaluates the level of attention and understanding of the person. The input is the extracted feature information, and the output is numerical data representing the level of attention and understanding as evaluation results. A deep learning algorithm is used for this evaluation.

[0536] Step 5:

[0537] The server generates and provides feedback to the instructor's terminal based on the evaluation results. The input is the evaluation result, and the output is the feedback message. To generate the feedback, the message is constructed according to predetermined criteria.

[0538] Step 6:

[0539] The server sends a real-time alert notification to the instructor's terminal if attention or comprehension levels fall below set criteria. The input is the reduced evaluation result, and the output is a warning message. The alert is delivered to the instructor through the notification system.

[0540] Step 7:

[0541] After the lesson ends, the server generates a comprehensive evaluation report of the activity and provides it to the instructor. The input is evaluation data collected during the lesson, and the output is a comprehensive evaluation report that includes suggestions for improvement. This report also includes suggestions for educational materials that will help improve the instructor's teaching skills.

[0542] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0543] This invention incorporates a new emotion engine to analyze the emotional state of students in the classroom. The entire system consists of a group of cameras installed in the classroom, a server, a teacher's terminal, and the emotion engine. The cameras in the classroom capture students' facial expressions and movements in real time and transmit the video data to the server. This data is processed by an AI agent and the emotion engine.

[0544] The server first converts the received video data into an analyzable format. This conversion includes adjusting the image resolution and removing noise necessary for analysis. After conversion, the AI ​​agent extracts feature data from the students' facial expressions and postures. Based on this feature data, the emotion engine identifies the students' emotional states. For example, it individually recognizes smiles and confused expressions and records them as emotion data.

[0545] The server further uses deep learning algorithms to analyze students' comprehension, concentration, and emotional data. This analysis is quantified and recorded along a timeline, allowing for a comprehensive evaluation of students' comprehension and emotional changes.

[0546] Based on the analysis results, the server generates feedback for the teacher. This feedback includes suggestions for improvements to the lesson content and teaching methods tailored to the emotional state of specific students. In particular, if a student's emotional state indicates resistance to learning, focused feedback is provided to the teacher's terminal.

[0547] The real-time alert function sends an alert to the teacher's terminal if the server analyzes that a student's concentration level or emotional state falls below a pre-set standard, prompting a quick response.

[0548] After the lesson ends, the server generates a comprehensive evaluation report based on the lesson data. This report includes detailed analysis of students' understanding and emotional states, as well as an overall evaluation of the lesson. Furthermore, the server recommends appropriate training content to improve teachers' teaching skills.

[0549] As a concrete example, if anxiety is detected from student B's facial expression during class, the emotion engine sends that information to the server. The server analyzes this data and notifies the teacher's terminal with an alert stating, "Student B is feeling anxious." This alert allows the teacher to quickly take concrete actions to alleviate student B's anxiety.

[0550] The following describes the processing flow.

[0551] Step 1:

[0552] The server receives video data in real time from cameras in the classroom. The video data includes detailed information about students' facial expressions and movements.

[0553] Step 2:

[0554] The server converts the received video data into an analyzable format. This data conversion includes noise reduction and frame extraction for face recognition.

[0555] Step 3:

[0556] The server processes the converted data using an AI agent to extract feature data from the students' facial expressions and movements. This feature data includes things like smiles, confusion, and eye movements.

[0557] Step 4:

[0558] The server uses an emotion engine to identify a student's emotional state from specific facial expressions and behavioral patterns. For example, a smile is recognized as indicating happiness, while a furrowed brow suggests confusion.

[0559] Step 5:

[0560] The server uses deep learning algorithms to analyze feature data and sentiment data. This analysis also quantifies learning states such as comprehension and concentration levels.

[0561] Step 6:

[0562] The server generates feedback data for teachers based on the analysis results. This feedback includes suggestions for teaching methods and lesson improvements that take into account the emotional state of students.

[0563] Step 7:

[0564] The server sends real-time alerts to teachers' terminals if students' concentration levels or emotional states fall below set criteria. These alerts immediately provide teachers with information about students who require attention.

[0565] Step 8:

[0566] After the lesson ends, the server generates a comprehensive evaluation report based on the data collected during the lesson. This report details the progression of understanding, changes in emotional state, and areas for improvement in instruction.

[0567] Step 9:

[0568] The server suggests training content aimed at improving teachers' teaching skills based on lesson results and feedback. This allows teachers, as users, to gain concrete ways to improve themselves from individual lessons.

[0569] (Example 2)

[0570] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0571] In educational settings, there is a need to accurately grasp students' learning progress and comprehension levels in real time, and to propose appropriate teaching methods based on that information. Furthermore, there is a need for methods to evaluate how students' emotional states affect learning and to conduct more effective education. However, conventional methods have not adequately analyzed video and emotional data, placing a significant burden on educators.

[0572] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0573] In this invention, the server includes means for acquiring video data of students, means for identifying their emotional state, and means for generating feedback for educators. This makes it possible to analyze students' comprehension and concentration levels in real time and suggest appropriate teaching methods.

[0574] "Student video data" refers to visual information that records students' activities and facial expressions within the classroom.

[0575] "Means of converting to a data format" refers to a processing device that formats received video data into a format suitable for analysis.

[0576] "Methods for extracting feature data" refers to algorithms for extracting changes in students' facial expressions and posture as numerical values ​​or categories.

[0577] "Means for identifying emotional states" refers to a system that uses extracted feature data to determine students' emotional responses.

[0578] "Means of analysis" refers to processors used to scientifically evaluate students' learning progress and concentration levels.

[0579] "Means of generating feedback" refers to the function of formulating guidance advice and improvement measures for educators based on the analysis results.

[0580] "Means of providing real-time notifications" refers to a system that immediately alerts educators when a student's status information falls below a certain standard.

[0581] "Means for generating a comprehensive evaluation report" refers to a tool that creates a report summarizing the analysis results of the entire course.

[0582] "Means of proposing learning content" refers to a system that provides appropriate training programs aimed at improving the skills of educators.

[0583] This system is designed to analyze students' emotional states in educational settings. The system primarily consists of cameras, a server, educator terminals, and an emotion analysis engine. Each device is connected via a network, allowing for real-time data exchange.

[0584] First, a camera captures images of students in the classroom and sends the video data to a server. The camera used at this time not only acquires high-resolution video but also has the ability to capture data at an appropriate frame rate and send it to the server quickly.

[0585] The server converts the received video data into an analyzable format. This conversion process uses image processing software to standardize the data resolution and remove noise. Furthermore, the AI ​​agent extracts features from the students' facial expressions and posture data and associates them with their emotional states. A deep learning model is involved in this process, enabling highly accurate emotion identification.

[0586] The converted and analyzed data is quantified and recorded by the server along with detailed learning progress. The analysis results are displayed to educators on their devices as real-time feedback. This feedback includes student comprehension and concentration levels, as well as suggestions for appropriate teaching methods.

[0587] For example, if the emotion engine detects that student B is feeling anxious during class, the server will immediately display an alert on the educator's terminal stating, "Student B is feeling anxious," allowing the educator to take prompt and appropriate action.

[0588] As a concrete example of a prompt to a generating AI model, it is presented as follows: "Please describe the outline of a system that analyzes students' anxiety and concentration levels in real time based on their facial expression data in the classroom and notifies educators as needed."

[0589] This technology is useful for improving students' learning efficiency and for educators to provide individualized instruction.

[0590] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0591] Step 1:

[0592] The server acquires video data of students from cameras installed in the classroom. The cameras capture high-resolution video in real time and send it to the server. The input is video of students, and the output is in an analyzable video file format.

[0593] Step 2:

[0594] The server preprocesses the received video data. Specifically, it generates clean data by appropriately adjusting the image resolution and performing noise filtering. The input is video of students transmitted from the camera, and the output is denoised image data.

[0595] Step 3:

[0596] The server uses an AI agent to extract specific facial expression and posture features from pre-processed image data. This process employs machine learning models and advanced pattern recognition techniques. The input is pre-processed image data, and the output is numerical data representing the students' facial features.

[0597] Step 4:

[0598] The server analyzes feature data using an emotion engine to identify the student's emotional state. A deep learning algorithm is used to convert facial expression characteristic values ​​into emotion categories. The input is numerical characteristic data, and the output is the identified emotional state.

[0599] Step 5:

[0600] The server analyzes comprehension and concentration levels based on emotional states. This analysis integrates multiple data points, including emotional data, to assess the overall learning situation. The input is emotional state data, and the output is quantified comprehension and concentration levels.

[0601] Step 6:

[0602] The server generates and sends feedback to the educator's terminal based on the analysis results. This feedback is designed to include specific teaching actions and improvement suggestions. The input is the analysis results, and the output is a feedback message for the educator.

[0603] (Application Example 2)

[0604] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0605] In homes and educational settings, it is crucial to understand learners' emotional states and concentration levels in real time and provide appropriate feedback. However, conventional systems lack real-time capabilities and sufficient emotion-based feedback, making effective learning support difficult. Solving this problem is essential.

[0606] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0607] In this invention, the server includes means for receiving video information, means for converting the received video information into an analyzable format, and means for extracting characteristic information from the subject's facial expressions and movements. This enables accurate understanding of the learner's emotions and level of concentration, and allows for real-time responses and feedback.

[0608] "Visual information" refers to visual data that captures the subject's facial expressions and movements.

[0609] "Means of converting into an analyzable format" refers to the process of converting received video information into a format suitable for emotion recognition and concentration level evaluation.

[0610] "Feature information" refers to important data extracted from a subject's facial expressions and movements, used to determine their emotions and level of concentration.

[0611] A "means for evaluating comprehension and concentration" is a system that uses extracted characteristic information to determine the depth of understanding and the state of concentration during the learning or work being performed.

[0612] "Means of generating feedback for instructors" refers to the process of creating information based on analysis results to provide areas for improvement and advice regarding learning and instruction.

[0613] The "means of real-time notification" refer to a function that immediately informs instructors if, based on the evaluation results, the current situation needs to be resolved or improved.

[0614] "Methods for generating a comprehensive evaluation report of educational activities" refers to the process of analyzing data accumulated throughout the entire activity in question and putting a comprehensive evaluation into writing.

[0615] "A means of proposing training content that supports the improvement of leaders' leadership abilities" refers to a system that proposes the most suitable educational resources for leaders to acquire more effective teaching methods.

[0616] "Means for generating voice output based on emotional information" refers to a function that generates and transmits voice feedback corresponding to the emotional state of the target.

[0617] The system for realizing this application consists of robots designed to support home education and digital devices in the learning environment. The system is configured and operated as follows:

[0618] The server first receives video information. This video information is captured by a camera built into the robot and includes the learner's facial expressions and movements. The server converts the received data into a format that can be analyzed. Here, noise reduction and image resolution adjustment can be performed, and image processing libraries such as OpenCV can be used.

[0619] The system extracts the subject's facial expressions and movements as feature information from the received video data. Deep learning frameworks such as TensorFlow can be used for this purpose. By inputting the extracted feature information into an AI model, the learner's emotional state and level of concentration can be evaluated.

[0620] The evaluation results are processed as real-time feedback and communicated to the learner via audio output and screen display. This allows learners to immediately receive advice such as, "Continue studying with focus."

[0621] Furthermore, the server can generate a comprehensive evaluation report of educational activities. This allows for an analysis of overall learning progress and comprehension, providing detailed reports to instructors and parents.

[0622] For example, if a learner is found to be losing focus while solving a math problem, the system can output an encouraging voice message such as, "Let's try a little harder!" An example of a prompt to the generative AI model might be, "Output an example of voice feedback that will help a child maintain a sense of security and focus while learning."

[0623] In this way, the system provides real-time support that responds to the learner's emotions, offering a more effective learning environment.

[0624] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0625] Step 1:

[0626] The server receives video information from the robot's built-in camera. The input at this stage is real-time video data, including the students' facial expressions and movements. By receiving this video data, the server prepares for the next analysis process.

[0627] Step 2:

[0628] The server converts the received video information into an analyzable format by applying noise reduction and resolution adjustments. The input is the video data obtained in step 1, and the output is clear data processed using an image processing library such as OpenCV. This improves the accuracy of the analysis.

[0629] Step 3:

[0630] The server extracts the subject's facial expressions and movements as feature information from clear video footage. At this stage, feature vectors are generated using a deep learning framework such as TensorFlow. The input is the processed video data obtained in step 2, and the output is feature data suitable for analysis by the AI ​​model.

[0631] Step 4:

[0632] The server uses an AI model to evaluate the learner's emotional state and concentration level based on the extracted feature information. The input is the feature information obtained in step 3, and the output is analyzed data showing the learner's comprehension, concentration level, and emotional state. This data forms the basis for generating feedback based on the evaluation.

[0633] Step 5:

[0634] Based on the evaluation results, the server generates real-time audio or visual feedback and transmits it to the learner via the device. For example, if concentration wavers, it can output a message such as "Try a little harder." The input is the analysis result data obtained in step 4, and the output is a feedback message tailored to the learner.

[0635] Step 6:

[0636] The server aggregates evaluation data over a long period and automatically generates a comprehensive evaluation report of educational activities. The input is analysis data accumulated from multiple sessions, and the output is a detailed evaluation report provided to instructors and parents. This allows for a grasp of learners' progress and achievements.

[0637] Step 7:

[0638] Based on the analysis results, the server recommends training content for instructors and provides a means to improve their teaching skills. The input is long-term analysis data, and the output is a suggestion of educational resources for instructors. This operation allows instructors to learn more effective teaching methods.

[0639] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0640] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0641] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0642] [Fourth Embodiment]

[0643] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0644] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0645] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0646] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0647] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0648] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0649] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0650] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0651] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0652] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0653] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0654] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0655] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0656] The system for implementing this invention consists of a camera installed in the classroom, a server, and a teacher's terminal. The camera captures the facial expressions and movements of students in the classroom in real time and transmits the video data to the server. The server receives this video data and converts it into an analyzable format. From the converted data, an AI agent analyzes the students' facial expressions and posture and extracts feature data.

[0657] The server uses deep learning algorithms to analyze this feature data and evaluate students' comprehension and concentration levels. The evaluation results are recorded as numerical values ​​and indicators. The server also generates feedback based on the analysis results and data to improve teachers' teaching skills. This feedback provides specific areas for improvement regarding the progress of the lesson and is delivered to the teacher's terminal in real time.

[0658] Furthermore, the server immediately sends an alert to the teacher's terminal if students' comprehension or concentration levels fall below set standards. This alert function allows teachers to adjust the lesson progress on the spot as needed.

[0659] After the lesson ends, the server generates a comprehensive evaluation report based on the data collected during the lesson. This report includes detailed analysis results to help teachers improve their lessons and suggests teaching methods that will aid in improvement. The server also recommends appropriate training content to enhance teachers' teaching skills. This allows teachers to continuously improve their skills.

[0660] As a concrete example, if changes in student A's facial expression or eye movements are observed during a lesson, the camera transmits this information to the server in real time. The server uses an AI agent to analyze the sudden decrease in student A's concentration and sends an alert to the teacher's terminal stating, "Student A's concentration level is decreasing," prompting the teacher to take immediate action. Based on this alert, the teacher can improve the effectiveness of the lesson by providing appropriate guidance or engaging with the student.

[0661] The following describes the processing flow.

[0662] Step 1:

[0663] The server receives video data in real time from cameras in the classroom. The video data includes information about students' facial expressions and posture.

[0664] Step 2:

[0665] The server converts the received video data into an analyzable format that the AI ​​agent can process. This conversion process includes adjusting the image resolution and removing noise.

[0666] Step 3:

[0667] The server uses an AI agent to analyze the students' facial expressions and movements from the converted data and extracts feature data. This feature data includes elements such as facial expressions and eye movements.

[0668] Step 4:

[0669] The server uses deep learning algorithms to analyze the extracted feature data. This analysis evaluates and quantifies students' comprehension and concentration levels.

[0670] Step 5:

[0671] The server generates feedback data for teachers based on the analysis results. This feedback includes suggestions for improving lessons and effective teaching methods.

[0672] Step 6:

[0673] The server sends a real-time alert to the teacher's terminal if students' comprehension or concentration levels fall below pre-set criteria. This alert allows the teacher to adjust the lesson progress on the spot.

[0674] Step 7:

[0675] After the lesson ends, the server generates a comprehensive evaluation report based on the lesson data. This report includes the progression of students' understanding and an overall evaluation of the lesson.

[0676] Step 8:

[0677] The server suggests training content aimed at improving teachers' teaching skills based on lesson results and feedback data. This allows teachers, as users, to further enhance their skills.

[0678] (Example 1)

[0679] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0680] In today's educational environment, it is difficult for teachers to grasp students' understanding and concentration levels in real time and adjust the pace of lessons accordingly. Furthermore, there is a lack of concrete feedback and training programs to improve teachers' own teaching skills. As a result, the quality of lessons and learning effectiveness are not maximized.

[0681] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0682] In this invention, the server includes means for receiving video data of learners in an educational environment, means for converting the received video data into analyzable information, and means for extracting characteristic information from the learners' nonverbal behavior. This makes it possible to evaluate the learners' knowledge acquisition level and level of engagement in real time and to provide educators with immediate improvement measures and training resources.

[0683] "Educational environment" refers to the physical or online space or situation provided for learners to acquire knowledge and skills.

[0684] A "learner" refers to a person who is the subject of acquiring knowledge and skills in an educational environment.

[0685] "Video data" refers to dynamic or static visual information collected by cameras or other video acquisition devices.

[0686] "Analyzable information" refers to information that has been formatted in a way that allows it to be processed and analyzed for a specific purpose.

[0687] "Nonverbal behavior" refers to actions that express intentions or emotions without using language, such as facial expressions or physical movements.

[0688] "Characteristic information" refers to important parameters and indicators extracted from nonverbal behavior for a particular analysis.

[0689] "Knowledge acquisition level" refers to an indicator that shows the extent to which learners understand and remember the content of the lessons.

[0690] "Degree of concentration" refers to an indicator that shows the extent to which learners are concentrating and paying attention to their learning activities.

[0691] An "educator" refers to a person who is responsible for providing guidance and support to learners in a learning environment.

[0692] "Improvement measures" refer to plans or actions taken to make a particular problem better.

[0693] "Training resources" refer to educational materials and programs provided to improve the skills and knowledge of educators.

[0694] The system for implementing this invention evaluates learners' understanding and engagement in an educational environment in real time and provides appropriate feedback and improvement measures to educators. The system consists of multiple elements.

[0695] The server receives video data from learners using video acquisition devices in classrooms or online learning environments. These devices include common video cameras and webcams, and the video data is transmitted to the server using streaming protocols. The server uses commonly used software libraries to encode this data and convert it into a format suitable for analysis. A specific example is image processing libraries such as OpenCV.

[0696] The server then analyzes the received video data using a generative AI model. Deep learning technology is employed for the analysis, extracting feature information from the learner's nonverbal behavior (such as facial expressions and posture). Regarding the AI ​​model, frameworks such as TensorFlow and PyTorch are selected as needed.

[0697] The server uses this characteristic information to quantify learners' knowledge acquisition and engagement levels, and transmits the results to educators' terminals in real time. Based on this information, educators can make necessary improvements and adjust teaching methods in real time. In addition, along with the analysis results, the system supports the improvement of teaching skills by recommending optimal learning resources (such as training content) to educators.

[0698] As a concrete example, if student A's concentration level decreases during a lesson, the server immediately detects this and sends a warning message to the educator's terminal stating, "Student A's concentration level is decreasing." Based on this, the educator can provide appropriate guidance to the student on the spot, optimizing the learning effect.

[0699] Furthermore, an example of a prompt for a generative AI model is, "Please tell me how to analyze changes in learner concentration levels during class in real time and provide immediate feedback." By using this prompt, the system can suggest the optimal data processing and analysis methods.

[0700] In this way, the system operates in a manner that supports real-time educational improvement.

[0701] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0702] Step 1:

[0703] The server receives video data of learners transmitted from cameras in the educational environment. This input data includes video from different angles and distances depending on the camera's position. Specifically, the server continuously captures video data using a streaming protocol and stores each frame in a buffer for real-time processing.

[0704] Step 2:

[0705] The server converts the received video data into analyzable information. This conversion process involves extracting each frame from the video data and preprocessing it using an image processing library (e.g., OpenCV). Specifically, noise is removed and the resolution is adjusted as needed. The output of this step is image data formatted for easy analysis.

[0706] Step 3:

[0707] The server provides pre-processed image data as input to the AI ​​agent, which then analyzes the data using a deep learning model (e.g., TensorFlow or PyTorch). This data analysis extracts feature information from the learner's nonverbal behavior, such as facial muscle movements and posture. Specifically, it performs calculations such as face recognition and joint position estimation within each image. The output is numerical data indicating the learner's level of understanding and engagement.

[0708] Step 4:

[0709] The server transmits the results of the analysis based on the characteristic information to the educators' terminals in real time, providing suggestions for improvement and feedback. In this step, the analysis results are output in the form of graphs and numerical data via an interface to visualize them. Based on this, educators can immediately take concrete steps to adjust the lesson content.

[0710] Step 5:

[0711] The server immediately notifies the educator's terminal of a warning if a learner's knowledge acquisition level or level of engagement falls below a pre-set threshold. This warning is displayed on the terminal as a specific message, prompting the educator to take corrective action.

[0712] Step 6:

[0713] After the lesson ends, the server generates a comprehensive evaluation report based on the collected data. This report summarizes the analyzed data to show learner performance and areas for improvement in the lesson. Specifically, the report is automatically output in text report format or as a visualized dashboard and distributed to educators.

[0714] Step 7:

[0715] The server recommends appropriate training content to improve the teaching skills of educators. In this step, based on the generated report data, an educational analysis algorithm is used to select effective learning resources and take specific actions to notify users. This allows educators, as users, to continuously improve their skills.

[0716] (Application Example 1)

[0717] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0718] In educational settings, it is essential for instructors to be able to grasp students' attention and comprehension levels in real time and provide effective educational support. Current educational systems sometimes make it difficult for instructors to intuitively assess students' comprehension and respond quickly. Furthermore, it is necessary to continuously support the skill development of instructors. Solving these challenges is the objective of this invention.

[0719] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0720] In this invention, the server includes means for receiving image information of a person, means for converting the received image information into an analyzable format, and means for extracting feature information from the person's facial expressions and movements. This enables the instructor to evaluate the person's level of attention and understanding in real time and provide appropriate feedback and educational support.

[0721] "Personal image information" refers to information that visually records an individual's physical characteristics and actions.

[0722] "Means of converting to an analyzable format" refers to the process of changing received image information into a data format suitable for analysis.

[0723] "Feature information" refers to specific data extracted from image information, such as an individual's facial expressions and body posture.

[0724] "Attention level" is an indicator that shows how much attention an individual is paying to a subject.

[0725] "Comprehension level" is an indicator that shows how well an individual understands the subject matter.

[0726] An "instructor" is a person whose job is to guide educational activities.

[0727] "Means for generating feedback" refers to methods for providing information to the instructor based on the analysis results.

[0728] A "Comprehensive Evaluation Report" is a document that summarizes the overall evaluation of educational activities.

[0729] "Educational materials" are learning content designed to support the improvement of instructors' teaching skills.

[0730] A "deep learning algorithm" is a machine learning technique used to perform highly accurate predictions and analyses through the analysis of complex data.

[0731] In its embodiment, this system primarily consists of a server, a camera for acquiring image information of people, and a terminal used by the instructor. The server converts the received image data into an analyzable format, and then analyzes the collected feature information using a deep learning algorithm. Based on the analysis results, the server can evaluate the level of attention and understanding of the person.

[0732] The evaluation results are provided as real-time feedback to the instructor's terminal, and if necessary, cautionary notifications are sent to the instructor. Furthermore, a comprehensive evaluation report is generated after the activity is completed, and the instructor receives suggestions that will help improve the teaching. In this process, image processing libraries such as OpenCV and deep learning frameworks such as TensorFlow and PyTorch are used.

[0733] As a concrete example, during a class, a camera observes a person's facial expressions in real time, and if the server uses an AI agent to determine that the person's level of attention is decreasing, a notification saying "Attention level is decreasing" is sent to the instructor's terminal. This allows the instructor to take immediate action. Furthermore, when utilizing the generative AI model, a prompt message such as "We want to evaluate the person's level of attention and understanding in real time and provide feedback that will be useful for educational support" is used.

[0734] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0735] Step 1:

[0736] The server receives real-time video data from cameras in the classroom. At this time, the camera's video data is the input, and this data is sent to the server.

[0737] Step 2:

[0738] The server converts the received video data into a format that can be analyzed. The input is video data received from the camera, and the output is data in a format that can be processed by a deep learning algorithm. This conversion is performed using OpenCV, an image processing software.

[0739] Step 3:

[0740] The server extracts feature information based on the converted data. The input here is data converted into an analyzable format, and the output is feature information related to facial expressions and posture. Deep learning techniques using TensorFlow and PyTorch are utilized for feature extraction.

[0741] Step 4:

[0742] The server analyzes the extracted feature information and evaluates the level of attention and understanding of the person. The input is the extracted feature information, and the output is numerical data representing the level of attention and understanding as evaluation results. A deep learning algorithm is used for this evaluation.

[0743] Step 5:

[0744] The server generates and provides feedback to the instructor's terminal based on the evaluation results. The input is the evaluation result, and the output is the feedback message. To generate the feedback, the message is constructed according to predetermined criteria.

[0745] Step 6:

[0746] The server sends a real-time alert notification to the instructor's terminal if attention or comprehension levels fall below set criteria. The input is the reduced evaluation result, and the output is a warning message. The alert is delivered to the instructor through the notification system.

[0747] Step 7:

[0748] After the lesson ends, the server generates a comprehensive evaluation report of the activity and provides it to the instructor. The input is evaluation data collected during the lesson, and the output is a comprehensive evaluation report that includes suggestions for improvement. This report also includes suggestions for educational materials that will help improve the instructor's teaching skills.

[0749] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0750] This invention incorporates a new emotion engine to analyze the emotional state of students in the classroom. The entire system consists of a group of cameras installed in the classroom, a server, a teacher's terminal, and the emotion engine. The cameras in the classroom capture students' facial expressions and movements in real time and transmit the video data to the server. This data is processed by an AI agent and the emotion engine.

[0751] The server first converts the received video data into an analyzable format. This conversion includes adjusting the image resolution and removing noise necessary for analysis. After conversion, the AI ​​agent extracts feature data from the students' facial expressions and postures. Based on this feature data, the emotion engine identifies the students' emotional states. For example, it individually recognizes smiles and confused expressions and records them as emotion data.

[0752] The server further uses deep learning algorithms to analyze students' comprehension, concentration, and emotional data. This analysis is quantified and recorded along a timeline, allowing for a comprehensive evaluation of students' comprehension and emotional changes.

[0753] Based on the analysis results, the server generates feedback for the teacher. This feedback includes suggestions for improvements to the lesson content and teaching methods tailored to the emotional state of specific students. In particular, if a student's emotional state indicates resistance to learning, focused feedback is provided to the teacher's terminal.

[0754] The real-time alert function sends an alert to the teacher's terminal if the server analyzes that a student's concentration level or emotional state falls below a pre-set standard, prompting a quick response.

[0755] After the lesson ends, the server generates a comprehensive evaluation report based on the lesson data. This report includes detailed analysis of students' understanding and emotional states, as well as an overall evaluation of the lesson. Furthermore, the server recommends appropriate training content to improve teachers' teaching skills.

[0756] As a concrete example, if anxiety is detected from student B's facial expression during class, the emotion engine sends that information to the server. The server analyzes this data and notifies the teacher's terminal with an alert stating, "Student B is feeling anxious." This alert allows the teacher to quickly take concrete actions to alleviate student B's anxiety.

[0757] The following describes the processing flow.

[0758] Step 1:

[0759] The server receives video data in real time from cameras in the classroom. The video data includes detailed information about students' facial expressions and movements.

[0760] Step 2:

[0761] The server converts the received video data into an analyzable format. This data conversion includes noise reduction and frame extraction for face recognition.

[0762] Step 3:

[0763] The server processes the converted data using an AI agent to extract feature data from the students' facial expressions and movements. This feature data includes things like smiles, confusion, and eye movements.

[0764] Step 4:

[0765] The server uses an emotion engine to identify a student's emotional state from specific facial expressions and behavioral patterns. For example, a smile is recognized as indicating happiness, while a furrowed brow suggests confusion.

[0766] Step 5:

[0767] The server uses deep learning algorithms to analyze feature data and sentiment data. This analysis also quantifies learning states such as comprehension and concentration levels.

[0768] Step 6:

[0769] The server generates feedback data for teachers based on the analysis results. This feedback includes suggestions for teaching methods and lesson improvements that take into account the emotional state of students.

[0770] Step 7:

[0771] The server sends real-time alerts to teachers' terminals if students' concentration levels or emotional states fall below set criteria. These alerts immediately provide teachers with information about students who require attention.

[0772] Step 8:

[0773] After the lesson ends, the server generates a comprehensive evaluation report based on the data collected during the lesson. This report details the progression of understanding, changes in emotional state, and areas for improvement in instruction.

[0774] Step 9:

[0775] The server suggests training content aimed at improving teachers' teaching skills based on lesson results and feedback. This allows teachers, as users, to gain concrete ways to improve themselves from individual lessons.

[0776] (Example 2)

[0777] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0778] In educational settings, there is a need to accurately grasp students' learning progress and comprehension levels in real time, and to propose appropriate teaching methods based on that information. Furthermore, there is a need for methods to evaluate how students' emotional states affect learning and to conduct more effective education. However, conventional methods have not adequately analyzed video and emotional data, placing a significant burden on educators.

[0779] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0780] In this invention, the server includes means for acquiring video data of students, means for identifying their emotional state, and means for generating feedback for educators. This makes it possible to analyze students' comprehension and concentration levels in real time and suggest appropriate teaching methods.

[0781] "Student video data" refers to visual information that records students' activities and facial expressions within the classroom.

[0782] "Means of converting to a data format" refers to a processing device that formats received video data into a format suitable for analysis.

[0783] "Methods for extracting feature data" refers to algorithms for extracting changes in students' facial expressions and posture as numerical values ​​or categories.

[0784] "Means for identifying emotional states" refers to a system that uses extracted feature data to determine students' emotional responses.

[0785] "Means of analysis" refers to processors used to scientifically evaluate students' learning progress and concentration levels.

[0786] "Means of generating feedback" refers to the function of formulating guidance advice and improvement measures for educators based on the analysis results.

[0787] "Means of providing real-time notifications" refers to a system that immediately alerts educators when a student's status information falls below a certain standard.

[0788] "Means for generating a comprehensive evaluation report" refers to a tool that creates a report summarizing the analysis results of the entire course.

[0789] "Means of proposing learning content" refers to a system that provides appropriate training programs aimed at improving the skills of educators.

[0790] This system is designed to analyze students' emotional states in educational settings. The system primarily consists of cameras, a server, educator terminals, and an emotion analysis engine. Each device is connected via a network, allowing for real-time data exchange.

[0791] First, a camera captures images of students in the classroom and sends the video data to a server. The camera used at this time not only acquires high-resolution video but also has the ability to capture data at an appropriate frame rate and send it to the server quickly.

[0792] The server converts the received video data into an analyzable format. This conversion process uses image processing software to standardize the data resolution and remove noise. Furthermore, the AI ​​agent extracts features from the students' facial expressions and posture data and associates them with their emotional states. A deep learning model is involved in this process, enabling highly accurate emotion identification.

[0793] The converted and analyzed data is quantified and recorded by the server along with detailed learning progress. The analysis results are displayed to educators on their devices as real-time feedback. This feedback includes student comprehension and concentration levels, as well as suggestions for appropriate teaching methods.

[0794] For example, if the emotion engine detects that student B is feeling anxious during class, the server will immediately display an alert on the educator's terminal stating, "Student B is feeling anxious," allowing the educator to take prompt and appropriate action.

[0795] As a concrete example of a prompt to a generating AI model, it is presented as follows: "Please describe the outline of a system that analyzes students' anxiety and concentration levels in real time based on their facial expression data in the classroom and notifies educators as needed."

[0796] This technology is useful for improving students' learning efficiency and for educators to provide individualized instruction.

[0797] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0798] Step 1:

[0799] The server acquires video data of students from cameras installed in the classroom. The cameras capture high-resolution video in real time and send it to the server. The input is video of students, and the output is in an analyzable video file format.

[0800] Step 2:

[0801] The server preprocesses the received video data. Specifically, it generates clean data by appropriately adjusting the image resolution and performing noise filtering. The input is video of students transmitted from the camera, and the output is denoised image data.

[0802] Step 3:

[0803] The server uses an AI agent to extract specific facial expression and posture features from pre-processed image data. This process employs machine learning models and advanced pattern recognition techniques. The input is pre-processed image data, and the output is numerical data representing the students' facial features.

[0804] Step 4:

[0805] The server analyzes feature data using an emotion engine to identify the student's emotional state. A deep learning algorithm is used to convert facial expression characteristic values ​​into emotion categories. The input is numerical characteristic data, and the output is the identified emotional state.

[0806] Step 5:

[0807] The server analyzes comprehension and concentration levels based on emotional states. This analysis integrates multiple data points, including emotional data, to assess the overall learning situation. The input is emotional state data, and the output is quantified comprehension and concentration levels.

[0808] Step 6:

[0809] The server generates and sends feedback to the educator's terminal based on the analysis results. This feedback is designed to include specific teaching actions and improvement suggestions. The input is the analysis results, and the output is a feedback message for the educator.

[0810] (Application Example 2)

[0811] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0812] In homes and educational settings, it is crucial to understand learners' emotional states and concentration levels in real time and provide appropriate feedback. However, conventional systems lack real-time capabilities and sufficient emotion-based feedback, making effective learning support difficult. Solving this problem is essential.

[0813] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0814] In this invention, the server includes means for receiving video information, means for converting the received video information into an analyzable format, and means for extracting characteristic information from the subject's facial expressions and movements. This enables accurate understanding of the learner's emotions and level of concentration, and allows for real-time responses and feedback.

[0815] "Visual information" refers to visual data that captures the subject's facial expressions and movements.

[0816] "Means of converting into an analyzable format" refers to the process of converting received video information into a format suitable for emotion recognition and concentration level evaluation.

[0817] "Feature information" refers to important data extracted from a subject's facial expressions and movements, used to determine their emotions and level of concentration.

[0818] A "means for evaluating comprehension and concentration" is a system that uses extracted characteristic information to determine the depth of understanding and the state of concentration during the learning or work being performed.

[0819] "Means of generating feedback for instructors" refers to the process of creating information based on analysis results to provide areas for improvement and advice regarding learning and instruction.

[0820] The "means of real-time notification" refer to a function that immediately informs instructors if, based on the evaluation results, the current situation needs to be resolved or improved.

[0821] "Methods for generating a comprehensive evaluation report of educational activities" refers to the process of analyzing data accumulated throughout the entire activity in question and putting a comprehensive evaluation into writing.

[0822] "A means of proposing training content that supports the improvement of leaders' leadership abilities" refers to a system that proposes the most suitable educational resources for leaders to acquire more effective teaching methods.

[0823] "Means for generating voice output based on emotional information" refers to a function that generates and transmits voice feedback corresponding to the emotional state of the target.

[0824] The system for realizing this application consists of robots designed to support home education and digital devices in the learning environment. The system is configured and operated as follows:

[0825] The server first receives video information. This video information is captured by a camera built into the robot and includes the learner's facial expressions and movements. The server converts the received data into a format that can be analyzed. Here, noise reduction and image resolution adjustment can be performed, and image processing libraries such as OpenCV can be used.

[0826] The system extracts the subject's facial expressions and movements as feature information from the received video data. Deep learning frameworks such as TensorFlow can be used for this purpose. By inputting the extracted feature information into an AI model, the learner's emotional state and level of concentration can be evaluated.

[0827] The evaluation results are processed as real-time feedback and communicated to the learner via audio output and screen display. This allows learners to immediately receive advice such as, "Continue studying with focus."

[0828] Furthermore, the server can generate a comprehensive evaluation report of educational activities. This allows for an analysis of overall learning progress and comprehension, providing detailed reports to instructors and parents.

[0829] For example, if a learner is found to be losing focus while solving a math problem, the system can output an encouraging voice message such as, "Let's try a little harder!" An example of a prompt to the generative AI model might be, "Output an example of voice feedback that will help a child maintain a sense of security and focus while learning."

[0830] In this way, the system provides real-time support that responds to the learner's emotions, offering a more effective learning environment.

[0831] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0832] Step 1:

[0833] The server receives video information from the robot's built-in camera. The input at this stage is real-time video data, including the students' facial expressions and movements. By receiving this video data, the server prepares for the next analysis process.

[0834] Step 2:

[0835] The server converts the received video information into an analyzable format by applying noise reduction and resolution adjustments. The input is the video data obtained in step 1, and the output is clear data processed using an image processing library such as OpenCV. This improves the accuracy of the analysis.

[0836] Step 3:

[0837] The server extracts the subject's facial expressions and movements as feature information from clear video footage. At this stage, feature vectors are generated using a deep learning framework such as TensorFlow. The input is the processed video data obtained in step 2, and the output is feature data suitable for analysis by the AI ​​model.

[0838] Step 4:

[0839] The server uses an AI model to evaluate the learner's emotional state and concentration level based on the extracted feature information. The input is the feature information obtained in step 3, and the output is analyzed data showing the learner's comprehension, concentration level, and emotional state. This data forms the basis for generating feedback based on the evaluation.

[0840] Step 5:

[0841] Based on the evaluation results, the server generates real-time audio or visual feedback and transmits it to the learner via the device. For example, if concentration wavers, it can output a message such as "Try a little harder." The input is the analysis result data obtained in step 4, and the output is a feedback message tailored to the learner.

[0842] Step 6:

[0843] The server aggregates evaluation data over a long period and automatically generates a comprehensive evaluation report of educational activities. The input is analysis data accumulated from multiple sessions, and the output is a detailed evaluation report provided to instructors and parents. This allows for a grasp of learners' progress and achievements.

[0844] Step 7:

[0845] Based on the analysis results, the server recommends training content for instructors and provides a means to improve their teaching skills. The input is long-term analysis data, and the output is a suggestion of educational resources for instructors. This operation allows instructors to learn more effective teaching methods.

[0846] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0847] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0848] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0849] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0850] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0851] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0852] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0853] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0854] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0855] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0856] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0857] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0858] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0859] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0860] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0861] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0862] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0863] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0864] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0865] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0866] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0867] The following is further disclosed regarding the embodiments described above.

[0868] (Claim 1)

[0869] A means of receiving student video data,

[0870] A means for converting received video data into an analyzable format,

[0871] A method for extracting characteristic data from students' facial expressions and movements,

[0872] A method for analyzing extracted feature data to evaluate students' comprehension and concentration levels,

[0873] A means of generating feedback for teachers based on the analysis results,

[0874] A means of notifying teachers in real time if students' comprehension or concentration levels fall below the standard,

[0875] A means of generating a comprehensive evaluation report for the course,

[0876] A means of proposing training content that supports the improvement of teachers' instructional skills,

[0877] A system that includes this.

[0878] (Claim 2)

[0879] The system according to claim 1, comprising means for quantifying and recording students' levels of understanding and concentration.

[0880] (Claim 3)

[0881] The system according to claim 1, comprising means for using a deep learning algorithm for generating feedback.

[0882] "Example 1"

[0883] (Claim 1)

[0884] Means for receiving learners' video data in an educational environment,

[0885] A means for converting received video data into analyzable information,

[0886] A means of extracting characteristic information from learners' nonverbal behaviors,

[0887] A means of analyzing extracted feature information to evaluate learners' knowledge acquisition and level of engagement,

[0888] A means of providing improvement measures to educators based on the analysis results,

[0889] A means of immediately sending a warning to educators when learners' knowledge acquisition or level of engagement falls below a certain standard,

[0890] A means of generating a comprehensive evaluation document for classroom activities,

[0891] A means of recommending learning resources that support the improvement of teaching skills among educators,

[0892] A system that includes this.

[0893] (Claim 2)

[0894] The system according to claim 1, comprising means for quantitatively recording the learner's level of knowledge acquisition and level of concentration.

[0895] (Claim 3)

[0896] The system according to claim 1, comprising means for utilizing artificial intelligence technology to provide improvement measures.

[0897] "Application Example 1"

[0898] (Claim 1)

[0899] A means for receiving image information of a person,

[0900] A means for converting received image information into an analyzable format,

[0901] A method for extracting characteristic information from a person's facial expressions and movements,

[0902] A method for analyzing extracted feature information to evaluate the level of attention and understanding of a person,

[0903] A means for generating feedback to the instructor based on the analysis results,

[0904] A means of notifying the instructor in real time if a person's level of attention or understanding falls below a certain standard,

[0905] A means of generating a comprehensive evaluation report of the activities,

[0906] A means of proposing educational materials that support the improvement of instructors' training skills,

[0907] A means of suggesting the use of visual teaching materials to instructors,

[0908] A system that includes this.

[0909] (Claim 2)

[0910] The system according to claim 1, comprising means for quantifying and recording the level of attention and understanding of a person.

[0911] (Claim 3)

[0912] The system according to claim 1, comprising means for using a deep learning algorithm for generating feedback.

[0913] "Example 2 of combining an emotion engine"

[0914] (Claim 1)

[0915] Methods for acquiring student video data,

[0916] A means for converting received video data into an analyzable data format,

[0917] A method for extracting characteristic data from students' facial expressions and postures,

[0918] A means of identifying students' emotional states based on extracted feature data,

[0919] A method for analyzing students' comprehension and concentration levels using identified emotion data,

[0920] A means of generating feedback for educators based on the analysis results,

[0921] A means of notifying educators in real time if a student's level of understanding or concentration falls below a predetermined standard,

[0922] A means of generating a comprehensive evaluation report for a course,

[0923] A means of proposing learning content to support the improvement of educators' teaching skills,

[0924] A means of detecting and warning educators about changes in emotional states that require attention,

[0925] A system that includes this.

[0926] (Claim 2)

[0927] The system according to claim 1, comprising means for quantifying and recording data relating to the emotional state of students.

[0928] (Claim 3)

[0929] The system according to claim 1, comprising means for using a machine learning algorithm in generating feedback.

[0930] "Application example 2 when combining with an emotional engine"

[0931] (Claim 1)

[0932] A means of receiving video information,

[0933] A means for converting received video information into an analyzable format,

[0934] A means of extracting characteristic information from the subject's facial expressions and movements,

[0935] A means of analyzing extracted feature information to evaluate the level of understanding and concentration of the subject,

[0936] A means of generating feedback for instructors based on the analysis results,

[0937] A means of notifying instructors in real time if the subject's level of understanding or concentration falls below the standard,

[0938] A means of generating a comprehensive evaluation report of educational activities,

[0939] A means of proposing training content that supports the improvement of leaders' leadership skills,

[0940] A means for generating audio output based on the target's emotional information,

[0941] A system that includes this.

[0942] (Claim 2)

[0943] The system according to claim 1, comprising means for quantifying and recording the level of understanding and concentration of the subject.

[0944] (Claim 3)

[0945] The system according to claim 1, comprising means for using a deep learning algorithm for generating feedback. [Explanation of symbols]

[0946] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of receiving student video data, A means for converting received video data into an analyzable format, A method for extracting characteristic data from students' facial expressions and movements, A method for analyzing extracted feature data to evaluate students' comprehension and concentration levels, A means of generating feedback for teachers based on the analysis results, A means of notifying teachers in real time if students' comprehension or concentration levels fall below the standard, A means of generating a comprehensive evaluation report for the course, A means of proposing training content that supports the improvement of teachers' instructional skills, A system that includes this.

2. The system according to claim 1, comprising means for quantifying and recording students' levels of understanding and concentration.

3. The system according to claim 1, comprising means for using a deep learning algorithm for generating feedback.

Images