system

Abstract

We provide the system. [Solution] Information gathering methods to assist in evaluation at educational institutions, A data analysis means for analyzing the data acquired by the aforementioned information gathering means, An evaluation generation means that generates a fair evaluation based on the output of the data analysis means, A feedback providing means that provides feedback based on the evaluation information generated by the evaluation generation means, A real-time evaluation display system that assesses concentration and comprehension in real time and notifies instructors and administrators of the provided feedback information, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the educational field, there is a problem that teachers are overwhelmed with busy work and have limited time to interact with each student. Therefore, in evaluations such as students' internal assessment scores, there is a lack of objectivity and fairness, and cases where it depends on the subjectivity of individual teachers have become an issue.

Means for Solving the Problems

[0005] This invention provides a system to assist in evaluation in educational institutions. This system acquires video information from the classroom and image data of students' answers using a data collection means, and analyzes this information using an analysis means. Furthermore, it generates evaluations such as internal assessment scores based on unified standards using an evaluation means, and provides the formed evaluation information to students using a feedback means, thereby supporting teachers' work and enabling fair and objective evaluation.

[0006] "Educational institutions" is a general term for places that provide education, such as schools, vocational schools, and universities.

[0007] "Evaluation" is the act of scoring and judging students' learning outcomes and behavior, and expressing these judgments as scores or comments.

[0008] "Data collection means" refers to hardware and software methods and devices for acquiring information, including cameras and sensors.

[0009] "Analysis means" refers to methods and systems for analyzing collected data and extracting information according to a specific purpose.

[0010] An "evaluation tool" is a function that expresses students' performance as numerical values ​​or comments using unified standards based on analyzed data.

[0011] "Feedback methods" refer to techniques and systems for communicating evaluation results to students and teachers, and are provided in the form of reports or notifications.

[0012] A "video acquisition module" refers to a function or device that includes a part that acquires visual information using a camera or sensor.

[0013] A "facial expression analysis module" is a system that analyzes a person's facial expressions from video data to infer their emotions, attention level, and other characteristics. [Brief explanation of the drawing]

[0014] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This embodiment provides a system for assisting student evaluation in educational institutions. Specifically, it is a network-based system including data collection means, analysis means, evaluation means, and feedback means.

[0036] As a means of data collection, AI cameras and scanners will be installed in classrooms to acquire video data from lessons and digital data of students' answer sheets and notebooks. The terminals will transmit this data to a server in real time.

[0037] In the analysis process, the server first performs facial expression analysis based on video information. This determines each student's level of concentration and emotional state. Furthermore, the scanned answer sheet image data is converted into text data using OCR technology. This conversion digitizes the students' answers, enabling accurate analysis.

[0038] The evaluation process is performed by a server, and based on the obtained analysis data, a unified evaluation standard is applied to calculate the internal assessment score. This evaluation standard is predetermined within the educational institution, reducing variability in evaluations among teachers.

[0039] The feedback system automatically aggregates evaluation results and generates detailed feedback on each student's performance. This feedback information can be viewed by the user (teacher) via their device, and supplementary comments can be added as needed. Finally, the feedback is provided to students and their parents via email or in printed form.

[0040] As a concrete example, during a math class, the server analyzes each student's video feed to measure their level of concentration. After the test, the answer sheets are scanned, and their correctness is determined while their internal assessment scores are calculated. This process allows users to evaluate students' learning outcomes more objectively and efficiently.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The device activates an AI camera to record video data from the classroom. The camera is configured to recognize each student's face and capture their image individually. The recorded data is sent to the server in real time.

[0044] Step 2:

[0045] The device scans answer sheets and notebooks, generating image data. This image data is then transferred to a server for OCR (Optical Character Recognition) processing.

[0046] Step 3:

[0047] The server processes the received video data through a facial expression analysis program. The program then generates data to estimate and evaluate students' concentration levels and emotional states based on changes in their facial expressions.

[0048] Step 4:

[0049] The server uses OCR to extract text from image data. It converts handwritten characters and answers into a format that can be analyzed as digital text.

[0050] Step 5:

[0051] The server uses the analyzed text data to evaluate students' answers. It analyzes the accuracy rate and trends in incorrect answers, and scores them based on predetermined criteria.

[0052] Step 6:

[0053] The server calculates the internal assessment score by aggregating each student's evaluation. This includes data on concentration levels during class, aiming for a fair and comprehensive assessment.

[0054] Step 7:

[0055] The server generates the final evaluation and feedback report. The report includes detailed information about each student's strengths and areas for improvement.

[0056] Step 8:

[0057] Users can improve the quality of their feedback by reviewing feedback reports on their devices and adding comments and additional information as needed.

[0058] Step 9:

[0059] The server sends the final feedback report to the student and their parents. The report is provided via email or in paper format.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] In modern educational institutions, the evaluation of individual students is often subjective, making fairness and accuracy of assessment a challenge. Furthermore, the time and effort required for data collection and analysis are increasing, making efficient feedback difficult in educational settings. There is a need to address these challenges and realize a more objective and rapid student evaluation system.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes information gathering means, analysis means, evaluation means, and feedback means. This makes it possible to objectively evaluate students' concentration levels and emotional states based on the collected data and quickly calculate fair evaluation scores. In addition, the feedback means provides each individual with detailed feedback including points for improvement, thereby improving the efficiency and fairness of evaluation in educational settings.

[0065] "Information gathering means" refers to devices used in educational institutions to acquire data such as video data from classes and student answer sheets, and includes AI cameras and scanners.

[0066] "Analysis methods" refer to technologies that process collected data to determine students' levels of concentration and emotional states, and utilize facial recognition technology and OCR technology.

[0067] "Evaluation methods" refer to the process of calculating student evaluation scores based on analyzed data and according to pre-set criteria.

[0068] A "feedback mechanism" is a function that aggregates evaluation results and generates and provides detailed feedback, including areas for improvement, to each student.

[0069] An "evaluation score" is a numerical representation of a student's academic performance, calculated using fair and consistent criteria based on various indicators obtained through analytical methods.

[0070] "Facial expression recognition technology" is a technique that analyzes facial expressions from students' video data and quantifies their emotions and level of concentration.

[0071] "OCR technology" is a technology that mechanically reads character information within scanned image data and converts it into text data.

[0072] This invention provides a system for objectively and efficiently evaluating the academic performance of each student in an educational institution. This system includes information gathering means, analysis means, evaluation means, and feedback means.

[0073] The system uses AI cameras and scanners installed in the classroom to collect video data from lessons, as well as students' answer sheets and notebooks, in digital format. Specifically, the AI ​​cameras record each student's activities as video in real time, and the scanners acquire answer sheets as image data.

[0074] The server analyzes the received video data using facial recognition technology to determine the students' level of concentration and emotional state. A general-purpose facial recognition library is used for this purpose. Additionally, OCR technology is applied to the image data of the answer sheets, converting it into text data. The open-source Tesseract is a possible OCR engine for this purpose.

[0075] Based on the analyzed text data and facial expression analysis data, the server calculates an evaluation score according to predefined evaluation criteria. These criteria are predetermined by the educational institution, ensuring consistency in evaluations among teachers.

[0076] The feedback system automatically generates detailed feedback for each student based on their evaluation results, which users can view on their own devices. Teachers can add additional comments to the feedback as needed. The final feedback is communicated to students and parents via email or printed materials.

[0077] As a concrete example, in a math class, a device captures video of students, a server uses that data to measure their concentration level, and test papers are analyzed using OCR to provide accurate evaluations. This process allows users to quickly and fairly grasp each student's learning outcomes.

[0078] As an example of a prompt to the generative AI model, you can use: "Please describe in detail the steps of how the educational system collects student data and assesses their level of concentration through facial expression analysis."

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The device uses AI cameras and scanners in the classroom to collect video data from lessons, as well as digital data of students' answer sheets and notebooks. Specifically, the camera captures each student in real time, and the scanner digitizes the answer sheets. This input data is sent directly to the server for pre-processing for analysis. The output consists of video data and image data.

[0082] Step 2:

[0083] The server processes the received video data and uses a facial recognition algorithm to analyze students' concentration levels and emotional states. In this process, each student's facial expression is extracted from the video data and quantified, such as smiles or confusion. The input at this stage is video data, and the output is quantified data on concentration levels and emotional states.

[0084] Step 3:

[0085] The server converts the image data of the answer sheet obtained from the scanner into text data using OCR technology. Specifically, an OCR engine such as Tesseract identifies the characters in the image and converts them into text. The input at this stage is image data, and the output is text data. As a result, the answer content is obtained as digital data.

[0086] Step 4:

[0087] The server uses the concentration level, emotional state data, and text data obtained in the previous step to calculate the internal assessment score according to pre-defined evaluation criteria. These criteria are based on the student's concentration level and the accuracy of their answers, ensuring fair evaluation. The input is the analyzed numerical data, and the output is the evaluation score.

[0088] Step 5:

[0089] The server generates a feedback report based on the calculated evaluation score. Specifically, the feedback includes not only the score, but also a graph of concentration levels and areas for improvement. This information can be viewed by the user on their terminal, and additional comments can be added as needed. The input is the evaluation score, and the output is a detailed feedback report.

[0090] Step 6:

[0091] Ultimately, the server provides feedback to students and parents via email or in print. Specifically, it generates a feedback report in PDF format and sends it to the registered email address. Alternatively, it may provide printed feedback. The input is the feedback report, and the output is the distributed feedback results.

[0092] (Application Example 1)

[0093] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0094] To improve the effectiveness of learning in public educational facilities for citizens, it is necessary to provide an environment where instructors and administrators can monitor students' concentration and comprehension levels in real time and adjust lecture content on the spot. However, conventional systems have made it difficult to grasp students' status immediately, making it challenging to maximize learning effectiveness. Therefore, the challenge is to provide a system that can evaluate students' learning progress in real time and provide appropriate feedback.

[0095] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0096] In this invention, the server includes information gathering means, data analysis means, evaluation generation means, and real-time evaluation display means. This allows instructors and operators to evaluate the level of concentration and understanding of students in real time and flexibly adjust the lecture content as needed.

[0097] "Information gathering means" refers to a system used in educational institutions to acquire video information and related data of students.

[0098] A "data analysis system" is a system that processes data obtained through information gathering methods to analyze the level of concentration and comprehension of participants.

[0099] An "evaluation generation method" is a system that creates a fair evaluation of a student's learning progress based on analyzed data.

[0100] A "real-time evaluation display system" is a system used to immediately notify instructors and administrators of the evaluation results obtained by the evaluation generation system, and to adjust the lecture content based on those results.

[0101] The system implementing this invention mainly consists of three elements: a server, a terminal, and a user. First, the server includes information gathering means, data analysis means, evaluation generation means, and real-time evaluation display means. The server receives video data of participants acquired via terminals such as smartphones and smart glasses installed in the facility in real time through the network. The information gathering means analyzes the participants' facial expressions using the OpenCV facial recognition library and quantifies their level of concentration and comprehension. Furthermore, it efficiently analyzes the content of participants' answers by converting materials and answer sheets into digital data using the Tesseract OCR engine.

[0102] The data analysis method utilizes a data analysis platform such as TENSORFLOW® to evaluate students' learning progress based on collected data. This enables the generation of fair and objective evaluations of students. Furthermore, the evaluation generation method creates feedback based on the generated evaluations and displays the results in real time on terminals held by instructors and administrators. This allows instructors and administrators to adjust the lecture content on the spot according to the students' progress.

[0103] As a concrete example, in an IT skills training seminar held at a public library, the participants' facial expressions were analyzed in real time, and if their concentration level decreased, the instructor was notified, allowing for adjustments such as returning the lecture content to basic material. In this way, the learning effect for all participants can be improved.

[0104] An example of a prompt for a generated AI model is: "We want to design a system that analyzes video data acquired by an AI camera to evaluate and notify citizens of their level of concentration and understanding in real time, and use this to adjust the content of lectures."

[0105] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0106] Step 1:

[0107] The device acquires video information of the participant using its camera. This prepares the image data necessary for facial recognition as input data.

[0108] Step 2:

[0109] The terminal transmits the image data it has acquired to the server via the network. The transmitted image data becomes the server's input data.

[0110] Step 3:

[0111] The server uses the OpenCV library to detect the participants' faces and perform facial expression analysis. This generates data that quantifies and outputs the participants' concentration levels and emotional states based on their facial expressions.

[0112] Step 4:

[0113] The server uses the OCR engine Tesseract to convert image data of documents sent from the terminal into text data. This allows the student's answers to be obtained as digital information.

[0114] Step 5:

[0115] The server uses TensorFlow to analyze quantified concentration data and answer text data. Through this analysis, it evaluates the students' understanding and learning progress, and outputs evaluation data.

[0116] Step 6:

[0117] The server generates feedback information based on evaluation data and notifies instructors and administrators in real time. Based on the generated feedback information, instructors and administrators can adjust the content of their lectures.

[0118] Step 7:

[0119] By allowing users to check information notified via their devices and modifying the lecture content and pace as needed, the system improves the learning effectiveness for participants.

[0120] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0121] This embodiment is a system that effectively assists student assessment in educational institutions, and by incorporating a new emotion engine, it provides deeper insights. The main components of the system are data collection means, analysis means, evaluation means, feedback means, and emotion engine.

[0122] As a data collection method, AI cameras and microphones are installed in the classroom to acquire video and audio data. The terminals collect this data individually and transmit it to the server in real time.

[0123] In the analysis system, the server uses video data to perform facial expression analysis and determine the students' concentration levels and emotions. In particular, by combining facial expression analysis with an emotion engine, the emotional state of the students is analyzed from multiple perspectives. Furthermore, emotions are inferred from the tone of the students' voices and speaking style through audio data analysis.

[0124] The evaluation method involves comprehensively analyzing data collected by the server, taking into account the output of the emotion engine, to score students' understanding and concentration levels. In addition to traditional academic performance assessments, it also provides indicators of emotional stability and motivation.

[0125] The feedback mechanism is a process in which the server generates feedback tailored to each student. This feedback can also utilize the output of the emotion engine to include specific suggestions for improvement and encouraging messages.

[0126] The emotion engine is a crucial component of the entire system, capable of evaluating the emotional state of individual students in detail through the analysis of audio and video data. This engine provides information to evaluation and feedback methods, meeting diverse needs in learning instruction.

[0127] As a concrete example, during a math lesson, the server collects facial expression analysis and audio data using an AI camera. An emotion engine then analyzes this data and detects if a student may be showing signs of frustration. This information is used by the evaluation system to adjust grades, and feedback is generated recommending that the student "consider providing additional support in math." This process allows educators to receive a comprehensive evaluation that includes emotional information.

[0128] The following describes the processing flow.

[0129] Step 1:

[0130] The device activates an AI camera and microphone to collect video and audio data of students in the classroom. The AI ​​camera has the ability to recognize each student's face and capture individual video footage. The microphone also captures the tone of each student's voice and speaking style.

[0131] Step 2:

[0132] The terminal compresses the data it collects and sends it to the server in real time. Here, the data undergoes signal processing to prepare it for efficient analysis.

[0133] Step 3:

[0134] The server processes the received video data through a facial expression analysis module to analyze changes in each student's facial expressions. Basic emotions such as smiles and confusion are recognized here.

[0135] Step 4:

[0136] The server feeds the audio data into an emotion analysis module, which infers the student's emotions from their voice tone and volume. This allows it to capture subtle emotional changes such as frustration or a sense of relief.

[0137] Step 5:

[0138] The server aggregates the analysis results from the emotion engine and synthesizes the acquired data to evaluate the student's current emotional state. By combining multiple data points, it performs more accurate emotion recognition.

[0139] Step 6:

[0140] The server utilizes the emotional information it receives in the evaluation process and reflects it in the assessment of students' academic performance. In addition to concentration and comprehension, emotional state is also taken into consideration to calculate an overall score.

[0141] Step 7:

[0142] Based on the evaluation results and sentiment analysis generated by the server, individual feedback is created for each student. Specific learning advice and emotional support are recommended.

[0143] Step 8:

[0144] The user (teacher) uses their device to review feedback reports provided by the server and uses them to develop teaching strategies for students. They can also revise the feedback and add supplementary comments as needed.

[0145] Step 9:

[0146] The server sends a final feedback report to the student and their parents. This allows the student to receive detailed information about their academic performance and emotional state.

[0147] (Example 2)

[0148] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0149] Traditional educational assessment systems struggle to accurately evaluate students' comprehension and concentration levels. In particular, these systems are heavily reliant on quantitative data and fail to consider emotional factors, resulting in a lack of comprehensive assessment that includes students' emotional states. Consequently, providing individualized instruction and feedback to each student becomes difficult.

[0150] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0151] In this invention, the server includes: information acquisition means for assisting evaluation in educational institutions; analysis means for analyzing the student's voice and video information acquired by the information acquisition means and estimating the student's emotional state; and integrated evaluation means for generating an evaluation that scores the student's level of comprehension and concentration based on the output of the analysis means. This makes it possible to perform a comprehensive evaluation that includes the student's emotional state. As a result, educators can provide instruction and feedback that takes into account the emotional elements of each student, thereby improving the quality of education.

[0152] "Information acquisition means" refers to devices or methods for collecting audio and video information of students within an educational institution.

[0153] "Analysis means" refers to a technology or process for comprehensively analyzing a student's emotional state based on acquired audio and video information.

[0154] An "integrated evaluation tool" is a function that uses the output of an analysis tool to quantify students' level of understanding and concentration, and to generate a fair evaluation.

[0155] A "feedback generation method" is a method or system for specifically proposing instructional content and improvement measures for students based on evaluation data obtained through an integrated evaluation method.

[0156] "Emotional state" refers to the psychological and emotional condition detected from a student's facial expressions and voice.

[0157] "Multifaceted analysis" refers to a detailed analysis of data using multiple perspectives and methods.

[0158] This invention provides a method for effectively implementing a student evaluation system in educational institutions. The following describes how each element of the system is specifically implemented.

[0159] Regarding the means of obtaining information

[0160] The terminals will use audio and video acquisition devices installed in the classroom. A high-sensitivity microphone will be used as the audio acquisition device, and an AI camera will be installed as the video acquisition device.

[0161] The terminal collects audio and video data from these devices in real time and transmits it to the server via the network.

[0162] Regarding the means of analysis

[0163] The server analyzes the received audio and video data. For the video data, libraries such as OpenCV are used to extract students' facial expressions as feature points, and an emotion engine is used to analyze their emotional state from multiple perspectives.

[0164] The system performs spectral analysis on the audio data to infer the students' emotions. This allows users to understand the students' level of concentration and motivation.

[0165] Regarding integrated evaluation means and feedback generation means

[0166] The server performs an integrated assessment based on the analysis results, quantifying students' understanding and concentration levels. In addition to traditional academic performance assessments, it provides emotional indicators.

[0167] The server uses a generative AI model to generate personalized feedback. This feedback can include advice and encouragement to improve students' motivation to learn.

[0168] As a concrete example, during a math lesson, a device collects video and audio from the classroom, and the server analyzes this data to detect when a student is struggling with a problem based on their deep thought or tone of voice. Based on this information, the server generates specific feedback such as, "Reviewing would be effective for further understanding." This entire process is automated and in real time using AI technology, enabling educators to teach more efficiently through the system.

[0169] An example of an input prompt for a generative AI model is, "Describe the specific algorithmic steps for assessing a student's emotional state and providing personalized feedback." This allows the generative AI model to receive guidance on how to appropriately generate feedback messages.

[0170] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0171] Step 1:

[0172] The terminal collects data using audio and video acquisition devices installed in the classroom.

[0173] Input: Real-time audio and video information from students.

[0174] Data processing: The device digitizes audio from a high-sensitivity microphone and divides video acquired by an AI camera into video frames.

[0175] Output: A data stream that compresses real-time audio and video data and sends it to a server via the network.

[0176] Specific operation: The terminal compresses video data in H.264 format and transmits audio data in MP3 format.

[0177] Step 2:

[0178] The server analyzes the received audio and video data.

[0179] Input: Audio and video data transmitted in real time.

[0180] Data processing: For video data, the server uses the OpenCV library to extract facial feature points and analyzes facial expressions using an emotion engine. For audio data, spectral analysis is performed to extract vocal cord patterns.

[0181] Output: Student emotional state data after analysis.

[0182] Specific operation: The server analyzes the frequency components of the audio data to evaluate the tone and pitch of the student's voice. From the video data, it analyzes eyebrow movements and mouth shape.

[0183] Step 3:

[0184] The server integrates the analysis results and performs an integrated evaluation.

[0185] Input: Emotional state data based on facial expression analysis and voice analysis.

[0186] Data processing: The server quantifies students' understanding and concentration levels by weighting the analysis results, and performs an integrated evaluation by also considering the output of the emotion engine.

[0187] Output: Evaluation data that quantifies students' level of understanding and concentration.

[0188] Specific operation: The server converts emotional state data into quantitative parameters and uses them to calculate the student's concentration score.

[0189] Step 4:

[0190] The server generates feedback based on the evaluation data.

[0191] Input: Student evaluation data obtained through integrated evaluation methods.

[0192] Data processing: The server uses a generative AI model to generate specific feedback messages that are appropriate for the student's evaluation data.

[0193] Output: Customized feedback messages for students.

[0194] Specific operation: The server automatically generates text containing study methods and advice tailored to each student's individual needs.

[0195] (Application Example 2)

[0196] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0197] In educational institutions and homes, optimizing learners' learning effectiveness and providing appropriate instruction tailored to their progress requires considering not only traditional performance evaluations but also psychological aspects such as learners' emotions and motivation. However, efficiently and in real time, it is difficult to grasp this information, posing a major obstacle to educators making decisions regarding learning support.

[0198] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0199] In this invention, the server includes data acquisition means for assisting evaluation in educational institutions, analysis means for analyzing information acquired by the data acquisition means, and evaluation generation means for generating fair evaluations based on the output of the analysis means. This enables a multifaceted analysis of learners' learning status and emotional state, allowing for comprehensive evaluation and appropriate feedback.

[0200] An "educational institution" is an organization or facility established for learners to acquire knowledge and skills.

[0201] "Data acquisition methods to support assessment" refer to tools and equipment used to collect various data related to learners.

[0202] "Analysis means" refers to processes or systems for processing acquired data and analyzing the learner's state and learning progress.

[0203] An "evaluation generation method" is a function that generates numerical values ​​or indicators of learners' learning effectiveness and motivation based on the analysis results.

[0204] A "feedback provision method" is a system for delivering improvement suggestions and advice to learners and educators based on evaluation results.

[0205] "Emotion analysis methods" are techniques for inferring a learner's emotional state from data and clarifying the specific type and intensity of their emotions.

[0206] A "video acquisition device" is a device installed to physically collect video data from the environment.

[0207] The "facial expression data analysis module" is a software component that analyzes facial feature points and other elements within video data to estimate emotions and attention levels.

[0208] The system of this invention provides a multifaceted evaluation of learners' learning progress in educational institutions and homes. The system mainly consists of data acquisition means, analysis means, evaluation generation means, feedback provision means, and sentiment analysis means.

[0209] The device uses an AI camera and microphone to collect video and audio data from learners and transmits it to a server in real time. This data acquisition method records all of the learner's actions in the environment.

[0210] The server uses the open-source facial expression analysis library "OpenFace" and the audio analysis library "pyAudioAnalysis" to analyze video and audio data. It evaluates the learner's level of concentration and emotional state through changes in facial expressions and tone of voice.

[0211] Based on the output of the analysis means, the server numerically evaluates the learner's learning and emotional state through the evaluation generation means. This enables a comprehensive evaluation that includes not only academic ability but also motivation and emotional appropriateness.

[0212] The feedback system generates feedback for learners and guardians based on the evaluations received. This process takes into account the output of the sentiment analysis system, providing specific suggestions for improvement and encouraging messages.

[0213] As a concrete example, a home learning support robot collects facial expression data while a child is doing their homework, and a server analyzes it. Based on the analysis, if the robot determines that the child may be showing frustration with a math problem, it will offer words of encouragement such as, "You're doing very well! Let me know if there's anything I can do to help."

[0214] Example of an input prompt for a generative AI model: "Analyze the learner's current emotional state based on their facial expression and voice data. If stress or lack of concentration is indicated, generate an appropriate encouraging comment."

[0215] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0216] Step 1:

[0217] The device uses an AI camera and microphone to collect the learner's facial expressions and voice. In this step, video and audio data of the environment are obtained as input. The obtained data is preprocessed, such as noise reduction and brightness adjustment, and then sent to the server.

[0218] Step 2:

[0219] The server analyzes the received video data using the "OpenFace" library. It receives pre-processed video data as input, extracts facial feature points, and uses these to estimate the learner's emotional state from their facial expressions. The output of this step is the learner's emotional data.

[0220] Step 3:

[0221] In parallel, the server analyzes the audio data using the "pyAudioAnalysis" library. It receives pre-processed audio data as input, analyzes the tone and speed of the voice, and evaluates the learner's emotions. The output of this step is emotion data derived from the audio.

[0222] Step 4:

[0223] The server integrates the outputs from steps 2 and 3 and uses an evaluation generation mechanism to quantify the learner's comprehension and concentration levels. It receives emotional data as input and analyzes it comprehensively to calculate the learner's overall emotional evaluation. The output of this step is a comprehensive emotional evaluation score.

[0224] Step 5:

[0225] The feedback system generates feedback for learners based on their evaluation scores. It considers the evaluation scores as input and creates appropriate improvement suggestions and encouraging messages for the learners. The output of this step is the feedback message.

[0226] Step 6:

[0227] The user reviews the feedback provided through the device, and the learning support robot provides that feedback to the learner verbally as needed. Based on the outputted feedback, the learner can try to implement improvements.

[0228] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0229] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0230] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0231] [Second Embodiment]

[0232] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0233] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0234] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0235] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0236] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0237] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0238] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0239] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0240] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0241] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0242] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0243] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0244] This embodiment provides a system for assisting student evaluation in educational institutions. Specifically, it is a network-based system including data collection means, analysis means, evaluation means, and feedback means.

[0245] As a means of data collection, AI cameras and scanners will be installed in classrooms to acquire video data from lessons and digital data of students' answer sheets and notebooks. The terminals will transmit this data to a server in real time.

[0246] In the analysis process, the server first performs facial expression analysis based on video information. This determines each student's level of concentration and emotional state. Furthermore, the scanned answer sheet image data is converted into text data using OCR technology. This conversion digitizes the students' answers, enabling accurate analysis.

[0247] The evaluation process is performed by a server, and based on the obtained analysis data, a unified evaluation standard is applied to calculate the internal assessment score. This evaluation standard is predetermined within the educational institution, reducing variability in evaluations among teachers.

[0248] The feedback system automatically aggregates evaluation results and generates detailed feedback on each student's performance. This feedback information can be viewed by the user (teacher) via their device, and supplementary comments can be added as needed. Finally, the feedback is provided to students and their parents via email or in printed form.

[0249] As a concrete example, during a math class, the server analyzes each student's video feed to measure their level of concentration. After the test, the answer sheets are scanned, and their correctness is determined while their internal assessment scores are calculated. This process allows users to evaluate students' learning outcomes more objectively and efficiently.

[0250] The following describes the processing flow.

[0251] Step 1:

[0252] The device activates an AI camera to record video data from the classroom. The camera is configured to recognize each student's face and capture their image individually. The recorded data is sent to the server in real time.

[0253] Step 2:

[0254] The device scans answer sheets and notebooks, generating image data. This image data is then transferred to a server for OCR (Optical Character Recognition) processing.

[0255] Step 3:

[0256] The server processes the received video data through a facial expression analysis program. The program then generates data to estimate and evaluate students' concentration levels and emotional states based on changes in their facial expressions.

[0257] Step 4:

[0258] The server uses OCR to extract text from image data. It converts handwritten characters and answers into a format that can be analyzed as digital text.

[0259] Step 5:

[0260] The server uses the analyzed text data to evaluate students' answers. It analyzes the accuracy rate and trends in incorrect answers, and scores them based on predetermined criteria.

[0261] Step 6:

[0262] The server calculates the internal assessment score by aggregating each student's evaluation. This includes data on concentration levels during class, aiming for a fair and comprehensive assessment.

[0263] Step 7:

[0264] The server generates the final evaluation and feedback report. The report includes detailed information about each student's strengths and areas for improvement.

[0265] Step 8:

[0266] Users can improve the quality of their feedback by reviewing feedback reports on their devices and adding comments and additional information as needed.

[0267] Step 9:

[0268] The server sends the final feedback report to the student and their parents. The report is provided via email or in paper format.

[0269] (Example 1)

[0270] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0271] In modern educational institutions, the evaluation of individual students is often subjective, making fairness and accuracy of assessment a challenge. Furthermore, the time and effort required for data collection and analysis are increasing, making efficient feedback difficult in educational settings. There is a need to address these challenges and realize a more objective and rapid student evaluation system.

[0272] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0273] In this invention, the server includes information gathering means, analysis means, evaluation means, and feedback means. This makes it possible to objectively evaluate students' concentration levels and emotional states based on the collected data and quickly calculate fair evaluation scores. In addition, the feedback means provides each individual with detailed feedback including points for improvement, thereby improving the efficiency and fairness of evaluation in educational settings.

[0274] "Information gathering means" refers to devices used in educational institutions to acquire data such as video data from classes and student answer sheets, and includes AI cameras and scanners.

[0275] "Analysis methods" refer to technologies that process collected data to determine students' levels of concentration and emotional states, and utilize facial recognition technology and OCR technology.

[0276] "Evaluation methods" refer to the process of calculating student evaluation scores based on analyzed data and according to pre-set criteria.

[0277] A "feedback mechanism" is a function that aggregates evaluation results and generates and provides detailed feedback, including areas for improvement, to each student.

[0278] An "evaluation score" is a numerical representation of a student's academic performance, calculated using fair and consistent criteria based on various indicators obtained through analytical methods.

[0279] "Facial expression recognition technology" is a technique that analyzes facial expressions from students' video data and quantifies their emotions and level of concentration.

[0280] "OCR technology" is a technology that mechanically reads character information within scanned image data and converts it into text data.

[0281] This invention provides a system for objectively and efficiently evaluating the academic performance of each student in an educational institution. This system includes information gathering means, analysis means, evaluation means, and feedback means.

[0282] The system uses AI cameras and scanners installed in the classroom to collect video data from lessons, as well as students' answer sheets and notebooks, in digital format. Specifically, the AI ​​cameras record each student's activities as video in real time, and the scanners acquire answer sheets as image data.

[0283] The server analyzes the received video data using facial recognition technology to determine the students' level of concentration and emotional state. A general-purpose facial recognition library is used for this purpose. Additionally, OCR technology is applied to the image data of the answer sheets, converting it into text data. The open-source Tesseract is a possible OCR engine for this purpose.

[0284] Based on the parsed text data and facial expression analysis data, the server calculates an evaluation score according to the set evaluation criteria. These evaluation criteria are determined in advance by educational institutions, thus ensuring the consistency of evaluations among teachers.

[0285] In the feedback means, the server automatically generates detailed feedback based on the evaluation results for each student, and the user can view this through their own terminal. If necessary, teachers can enter additional comments in the feedback. The final feedback is transmitted to students and guardians via email or printed materials.

[0286] As a specific example, in a math class, the terminal captures the video of the students, the server uses this data to measure the concentration level, and the test answers are analyzed by OCR for accurate evaluation. Through this process, users can quickly and fairly understand the learning achievements of each student.

[0287] As an example of a prompt for the generative AI model, "Please explain the detailed steps of how an educational system collects students' data and evaluates concentration levels through facial expression analysis." can be used.

[0288] The flow of the specific process in Example 1 will be described using FIG. 11.

[0289] Step 1:

[0290] The terminal uses the AI cameras and scanners in the classroom to collect video data during the class and digital data of the students' answer sheets and notes. As specific operations, the camera captures each student in real time, and the scanner digitizes the answer sheets. This input data is directly sent to the server for preprocessing for analysis. As output, video data and image data are obtained.

[0291] Step 2:

[0292] The server processes the received video data and uses a facial recognition algorithm to analyze students' concentration levels and emotional states. In this process, each student's facial expression is extracted from the video data and quantified, such as smiles or confusion. The input at this stage is video data, and the output is quantified data on concentration levels and emotional states.

[0293] Step 3:

[0294] The server converts the image data of the answer sheet obtained from the scanner into text data using OCR technology. Specifically, an OCR engine such as Tesseract identifies the characters in the image and converts them into text. The input at this stage is image data, and the output is text data. As a result, the answer content is obtained as digital data.

[0295] Step 4:

[0296] The server uses the concentration level, emotional state data, and text data obtained in the previous step to calculate the internal assessment score according to pre-defined evaluation criteria. These criteria are based on the student's concentration level and the accuracy of their answers, ensuring fair evaluation. The input is the analyzed numerical data, and the output is the evaluation score.

[0297] Step 5:

[0298] The server generates a feedback report based on the calculated evaluation score. Specifically, the feedback includes not only the score, but also a graph of concentration levels and areas for improvement. This information can be viewed by the user on their terminal, and additional comments can be added as needed. The input is the evaluation score, and the output is a detailed feedback report.

[0299] Step 6:

[0300] Finally, the server provides feedback to students and guardians as emails or printed materials. Specifically, it generates a feedback report in PDF format and sends it to the registered email address. Or it may provide printed feedback. The input is the feedback report, and the output is the distributed feedback result.

[0301] (Application Example 1)

[0302] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0303] In order to improve the learning effect in public education facilities for citizens, it is required to provide an environment where the concentration and understanding of the attendees can be grasped in real time, and the lecturer and operator can adjust the lecture content on the spot. However, in the conventional system, it is difficult to immediately grasp the state of the attendees, and it is difficult to maximize the learning effect. Therefore, it is an issue to provide a system that can evaluate the learning situation of the attendees in real time and provide appropriate feedback.

[0304] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0305] In this invention, the server includes an information collection means, a data analysis means, an evaluation generation means, and a real-time evaluation display means. As a result, the lecturer and operator can evaluate the concentration and understanding of the attendees in real time and flexibly adjust the lecture content as needed.

[0306] The "information collection means" is a system that acquires video information and related data of the attendees in an educational facility.

[0307] The "data analysis means" is a system that processes the data obtained by the information collection means and analyzes the concentration and understanding of the attendees.

[0308] An "evaluation generation method" is a system that creates a fair evaluation of a student's learning progress based on analyzed data.

[0309] A "real-time evaluation display system" is a system used to immediately notify instructors and administrators of the evaluation results obtained by the evaluation generation system, and to adjust the lecture content based on those results.

[0310] The system implementing this invention mainly consists of three elements: a server, a terminal, and a user. First, the server includes information gathering means, data analysis means, evaluation generation means, and real-time evaluation display means. The server receives video data of participants acquired via terminals such as smartphones and smart glasses installed in the facility in real time through the network. The information gathering means analyzes the participants' facial expressions using the OpenCV facial recognition library and quantifies their level of concentration and comprehension. Furthermore, it efficiently analyzes the content of participants' answers by converting materials and answer sheets into digital data using the Tesseract OCR engine.

[0311] The data analysis method utilizes a data analysis platform such as TensorFlow to evaluate students' learning progress based on collected data. This enables the generation of fair and objective evaluations of students. Furthermore, the evaluation generation method creates feedback based on the generated evaluations and displays the results in real time on terminals held by instructors and administrators. This allows instructors and administrators to adjust the lecture content on the spot according to the students' progress.

[0312] As a concrete example, in an IT skills training seminar held at a public library, the participants' facial expressions were analyzed in real time, and if their concentration level decreased, the instructor was notified, allowing for adjustments such as returning the lecture content to basic material. In this way, the learning effect for all participants can be improved.

[0313] An example of a prompt for a generated AI model is: "We want to design a system that analyzes video data acquired by an AI camera to evaluate and notify citizens of their level of concentration and understanding in real time, and use this to adjust the content of lectures."

[0314] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0315] Step 1:

[0316] The device acquires video information of the participant using its camera. This prepares the image data necessary for facial recognition as input data.

[0317] Step 2:

[0318] The terminal transmits the image data it has acquired to the server via the network. The transmitted image data becomes the server's input data.

[0319] Step 3:

[0320] The server uses the OpenCV library to detect the participants' faces and perform facial expression analysis. This generates data that quantifies and outputs the participants' concentration levels and emotional states based on their facial expressions.

[0321] Step 4:

[0322] The server uses the OCR engine Tesseract to convert image data of documents sent from the terminal into text data. This allows the student's answers to be obtained as digital information.

[0323] Step 5:

[0324] The server uses TensorFlow to analyze quantified concentration data and answer text data. Through this analysis, it evaluates the students' understanding and learning progress, and outputs evaluation data.

[0325] Step 6:

[0326] The server generates feedback information based on evaluation data and notifies instructors and administrators in real time. Based on the generated feedback information, instructors and administrators can adjust the content of their lectures.

[0327] Step 7:

[0328] By allowing users to check information notified via their devices and modifying the lecture content and pace as needed, the system improves the learning effectiveness for participants.

[0329] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0330] This embodiment is a system that effectively assists student assessment in educational institutions, and by incorporating a new emotion engine, it provides deeper insights. The main components of the system are data collection means, analysis means, evaluation means, feedback means, and emotion engine.

[0331] As a data collection method, AI cameras and microphones are installed in the classroom to acquire video and audio data. The terminals collect this data individually and transmit it to the server in real time.

[0332] In the analysis system, the server uses video data to perform facial expression analysis and determine the students' concentration levels and emotions. In particular, by combining facial expression analysis with an emotion engine, the emotional state of the students is analyzed from multiple perspectives. Furthermore, emotions are inferred from the tone of the students' voices and speaking style through audio data analysis.

[0333] The evaluation method involves comprehensively analyzing data collected by the server, taking into account the output of the emotion engine, to score students' understanding and concentration levels. In addition to traditional academic performance assessments, it also provides indicators of emotional stability and motivation.

[0334] The feedback mechanism is a process in which the server generates feedback tailored to each student. This feedback can also utilize the output of the emotion engine to include specific suggestions for improvement and encouraging messages.

[0335] The emotion engine is a crucial component of the entire system, capable of evaluating the emotional state of individual students in detail through the analysis of audio and video data. This engine provides information to evaluation and feedback methods, meeting diverse needs in learning instruction.

[0336] As a concrete example, during a math lesson, the server collects facial expression analysis and audio data using an AI camera. An emotion engine then analyzes this data and detects if a student may be showing signs of frustration. This information is used by the evaluation system to adjust grades, and feedback is generated recommending that the student "consider providing additional support in math." This process allows educators to receive a comprehensive evaluation that includes emotional information.

[0337] The following describes the processing flow.

[0338] Step 1:

[0339] The device activates an AI camera and microphone to collect video and audio data of students in the classroom. The AI ​​camera has the ability to recognize each student's face and capture individual video footage. The microphone also captures the tone of each student's voice and speaking style.

[0340] Step 2:

[0341] The terminal compresses the data it collects and sends it to the server in real time. Here, the data undergoes signal processing to prepare it for efficient analysis.

[0342] Step 3:

[0343] The server processes the received video data through a facial expression analysis module to analyze changes in each student's facial expressions. Basic emotions such as smiles and confusion are recognized here.

[0344] Step 4:

[0345] The server feeds the audio data into an emotion analysis module, which infers the student's emotions from their voice tone and volume. This allows it to capture subtle emotional changes such as frustration or a sense of relief.

[0346] Step 5:

[0347] The server aggregates the analysis results from the emotion engine and synthesizes the acquired data to evaluate the student's current emotional state. By combining multiple data points, it performs more accurate emotion recognition.

[0348] Step 6:

[0349] The server utilizes the emotional information it receives in the evaluation process and reflects it in the assessment of students' academic performance. In addition to concentration and comprehension, emotional state is also taken into consideration to calculate an overall score.

[0350] Step 7:

[0351] Based on the evaluation results and sentiment analysis generated by the server, individual feedback is created for each student. Specific learning advice and emotional support are recommended.

[0352] Step 8:

[0353] The user (teacher) uses their device to review feedback reports provided by the server and uses them to develop teaching strategies for students. They can also revise the feedback and add supplementary comments as needed.

[0354] Step 9:

[0355] The server sends a final feedback report to the student and their parents. This allows the student to receive detailed information about their academic performance and emotional state.

[0356] (Example 2)

[0357] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0358] Traditional educational assessment systems struggle to accurately evaluate students' comprehension and concentration levels. In particular, these systems are heavily reliant on quantitative data and fail to consider emotional factors, resulting in a lack of comprehensive assessment that includes students' emotional states. Consequently, providing individualized instruction and feedback to each student becomes difficult.

[0359] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0360] In this invention, the server includes: information acquisition means for assisting evaluation in educational institutions; analysis means for analyzing the student's voice and video information acquired by the information acquisition means and estimating the student's emotional state; and integrated evaluation means for generating an evaluation that scores the student's level of comprehension and concentration based on the output of the analysis means. This makes it possible to perform a comprehensive evaluation that includes the student's emotional state. As a result, educators can provide instruction and feedback that takes into account the emotional elements of each student, thereby improving the quality of education.

[0361] "Information acquisition means" refers to devices or methods for collecting audio and video information of students within an educational institution.

[0362] "Analysis means" refers to a technology or process for comprehensively analyzing a student's emotional state based on acquired audio and video information.

[0363] An "integrated evaluation tool" is a function that uses the output of an analysis tool to quantify students' level of understanding and concentration, and to generate a fair evaluation.

[0364] A "feedback generation method" is a method or system for specifically proposing instructional content and improvement measures for students based on evaluation data obtained through an integrated evaluation method.

[0365] "Emotional state" refers to the psychological and emotional condition detected from a student's facial expressions and voice.

[0366] "Multifaceted analysis" refers to a detailed analysis of data using multiple perspectives and methods.

[0367] This invention provides a method for effectively implementing a student evaluation system in educational institutions. The following describes how each element of the system is specifically implemented.

[0368] Regarding the means of obtaining information

[0369] The terminals will use audio and video acquisition devices installed in the classroom. A high-sensitivity microphone will be used as the audio acquisition device, and an AI camera will be installed as the video acquisition device.

[0370] The terminal collects audio and video data from these devices in real time and transmits it to the server via the network.

[0371] Regarding the means of analysis

[0372] The server analyzes the received audio and video data. For the video data, libraries such as OpenCV are used to extract students' facial expressions as feature points, and an emotion engine is used to analyze their emotional state from multiple perspectives.

[0373] The system performs spectral analysis on the audio data to infer the students' emotions. This allows users to understand the students' level of concentration and motivation.

[0374] Regarding integrated evaluation means and feedback generation means

[0375] The server performs an integrated assessment based on the analysis results, quantifying students' understanding and concentration levels. In addition to traditional academic performance assessments, it provides emotional indicators.

[0376] The server uses a generative AI model to generate personalized feedback. This feedback can include advice and encouragement to improve students' motivation to learn.

[0377] As a concrete example, during a math lesson, a device collects video and audio from the classroom, and the server analyzes this data to detect when a student is struggling with a problem based on their deep thought or tone of voice. Based on this information, the server generates specific feedback such as, "Reviewing would be effective for further understanding." This entire process is automated and in real time using AI technology, enabling educators to teach more efficiently through the system.

[0378] An example of an input prompt for a generative AI model is, "Describe the specific algorithmic steps for assessing a student's emotional state and providing personalized feedback." This allows the generative AI model to receive guidance on how to appropriately generate feedback messages.

[0379] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0380] Step 1:

[0381] The terminal collects data using audio and video acquisition devices installed in the classroom.

[0382] Input: Real-time audio and video information from students.

[0383] Data processing: The device digitizes audio from a high-sensitivity microphone and divides video acquired by an AI camera into video frames.

[0384] Output: A data stream that compresses real-time audio and video data and sends it to a server via the network.

[0385] Specific operation: The terminal compresses video data in H.264 format and transmits audio data in MP3 format.

[0386] Step 2:

[0387] The server analyzes the received audio and video data.

[0388] Input: Audio and video data transmitted in real time.

[0389] Data processing: For video data, the server uses the OpenCV library to extract facial feature points and analyzes facial expressions using an emotion engine. For audio data, spectral analysis is performed to extract vocal cord patterns.

[0390] Output: Student emotional state data after analysis.

[0391] Specific operation: The server analyzes the frequency components of the audio data to evaluate the tone and pitch of the student's voice. From the video data, it analyzes eyebrow movements and mouth shape.

[0392] Step 3:

[0393] The server integrates the analysis results and performs an integrated evaluation.

[0394] Input: Emotional state data based on facial expression analysis and voice analysis.

[0395] Data processing: The server quantifies students' understanding and concentration levels by weighting the analysis results, and performs an integrated evaluation by also considering the output of the emotion engine.

[0396] Output: Evaluation data that quantifies students' level of understanding and concentration.

[0397] Specific operation: The server converts emotional state data into quantitative parameters and uses them to calculate the student's concentration score.

[0398] Step 4:

[0399] The server generates feedback based on the evaluation data.

[0400] Input: Student evaluation data obtained through integrated evaluation methods.

[0401] Data processing: The server uses a generative AI model to generate specific feedback messages that are appropriate for the student's evaluation data.

[0402] Output: Customized feedback messages for students.

[0403] Specific operation: The server automatically generates text containing study methods and advice tailored to each student's individual needs.

[0404] (Application Example 2)

[0405] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0406] In educational institutions and homes, optimizing learners' learning effectiveness and providing appropriate instruction tailored to their progress requires considering not only traditional performance evaluations but also psychological aspects such as learners' emotions and motivation. However, efficiently and in real time, it is difficult to grasp this information, posing a major obstacle to educators making decisions regarding learning support.

[0407] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0408] In this invention, the server includes data acquisition means for assisting evaluation in educational institutions, analysis means for analyzing information acquired by the data acquisition means, and evaluation generation means for generating fair evaluations based on the output of the analysis means. This enables a multifaceted analysis of learners' learning status and emotional state, allowing for comprehensive evaluation and appropriate feedback.

[0409] An "educational institution" is an organization or facility established for learners to acquire knowledge and skills.

[0410] "Data acquisition methods to support assessment" refer to tools and equipment used to collect various data related to learners.

[0411] "Analysis means" refers to processes or systems for processing acquired data and analyzing the learner's state and learning progress.

[0412] An "evaluation generation method" is a function that generates numerical values ​​or indicators of learners' learning effectiveness and motivation based on the analysis results.

[0413] A "feedback provision method" is a system for delivering improvement suggestions and advice to learners and educators based on evaluation results.

[0414] "Emotion analysis methods" are techniques for inferring a learner's emotional state from data and clarifying the specific type and intensity of their emotions.

[0415] A "video acquisition device" is a device installed to physically collect video data from the environment.

[0416] The "facial expression data analysis module" is a software component that analyzes facial feature points and other elements within video data to estimate emotions and attention levels.

[0417] The system of this invention provides a multifaceted evaluation of learners' learning progress in educational institutions and homes. The system mainly consists of data acquisition means, analysis means, evaluation generation means, feedback provision means, and sentiment analysis means.

[0418] The device uses an AI camera and microphone to collect video and audio data from learners and transmits it to a server in real time. This data acquisition method records all of the learner's actions in the environment.

[0419] The server uses the open-source facial expression analysis library "OpenFace" and the audio analysis library "pyAudioAnalysis" to analyze video and audio data. It evaluates the learner's level of concentration and emotional state through changes in facial expressions and tone of voice.

[0420] Based on the output of the analysis means, the server numerically evaluates the learner's learning and emotional state through the evaluation generation means. This enables a comprehensive evaluation that includes not only academic ability but also motivation and emotional appropriateness.

[0421] The feedback system generates feedback for learners and guardians based on the evaluations received. This process takes into account the output of the sentiment analysis system, providing specific suggestions for improvement and encouraging messages.

[0422] As a concrete example, a home learning support robot collects facial expression data while a child is doing their homework, and a server analyzes it. Based on the analysis, if the robot determines that the child may be showing frustration with a math problem, it will offer words of encouragement such as, "You're doing very well! Let me know if there's anything I can do to help."

[0423] Example of an input prompt for a generative AI model: "Analyze the learner's current emotional state based on their facial expression and voice data. If stress or lack of concentration is indicated, generate an appropriate encouraging comment."

[0424] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0425] Step 1:

[0426] The device uses an AI camera and microphone to collect the learner's facial expressions and voice. In this step, video and audio data of the environment are obtained as input. The obtained data is preprocessed, such as noise reduction and brightness adjustment, and then sent to the server.

[0427] Step 2:

[0428] The server analyzes the received video data using the "OpenFace" library. It receives pre-processed video data as input, extracts facial feature points, and uses these to estimate the learner's emotional state from their facial expressions. The output of this step is the learner's emotional data.

[0429] Step 3:

[0430] In parallel, the server analyzes the audio data using the "pyAudioAnalysis" library. It receives pre-processed audio data as input, analyzes the tone and speed of the voice, and evaluates the learner's emotions. The output of this step is emotion data derived from the audio.

[0431] Step 4:

[0432] The server integrates the outputs from steps 2 and 3 and uses an evaluation generation mechanism to quantify the learner's comprehension and concentration levels. It receives emotional data as input and analyzes it comprehensively to calculate the learner's overall emotional evaluation. The output of this step is a comprehensive emotional evaluation score.

[0433] Step 5:

[0434] The feedback system generates feedback for learners based on their evaluation scores. It considers the evaluation scores as input and creates appropriate improvement suggestions and encouraging messages for the learners. The output of this step is the feedback message.

[0435] Step 6:

[0436] The user reviews the feedback provided through the device, and the learning support robot provides that feedback to the learner verbally as needed. Based on the outputted feedback, the learner can try to implement improvements.

[0437] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0438] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0439] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0440] [Third Embodiment]

[0441] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0442] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0443] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0444] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0445] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0446] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0447] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0448] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0449] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0450] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0451] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0452] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0453] This embodiment provides a system for assisting student evaluation in educational institutions. Specifically, it is a network-based system including data collection means, analysis means, evaluation means, and feedback means.

[0454] As a means of data collection, AI cameras and scanners will be installed in classrooms to acquire video data from lessons and digital data of students' answer sheets and notebooks. The terminals will transmit this data to a server in real time.

[0455] In the analysis process, the server first performs facial expression analysis based on video information. This determines each student's level of concentration and emotional state. Furthermore, the scanned answer sheet image data is converted into text data using OCR technology. This conversion digitizes the students' answers, enabling accurate analysis.

[0456] The evaluation process is performed by a server, and based on the obtained analysis data, a unified evaluation standard is applied to calculate the internal assessment score. This evaluation standard is predetermined within the educational institution, reducing variability in evaluations among teachers.

[0457] The feedback system automatically aggregates evaluation results and generates detailed feedback on each student's performance. This feedback information can be viewed by the user (teacher) via their device, and supplementary comments can be added as needed. Finally, the feedback is provided to students and their parents via email or in printed form.

[0458] As a concrete example, during a math class, the server analyzes each student's video feed to measure their level of concentration. After the test, the answer sheets are scanned, and their correctness is determined while their internal assessment scores are calculated. This process allows users to evaluate students' learning outcomes more objectively and efficiently.

[0459] The following describes the processing flow.

[0460] Step 1:

[0461] The device activates an AI camera to record video data from the classroom. The camera is configured to recognize each student's face and capture their image individually. The recorded data is sent to the server in real time.

[0462] Step 2:

[0463] The device scans answer sheets and notebooks, generating image data. This image data is then transferred to a server for OCR (Optical Character Recognition) processing.

[0464] Step 3:

[0465] The server processes the received video data through a facial expression analysis program. The program then generates data to estimate and evaluate students' concentration levels and emotional states based on changes in their facial expressions.

[0466] Step 4:

[0467] The server uses OCR to extract text from image data. It converts handwritten characters and answers into a format that can be analyzed as digital text.

[0468] Step 5:

[0469] The server uses the analyzed text data to evaluate students' answers. It analyzes the accuracy rate and trends in incorrect answers, and scores them based on predetermined criteria.

[0470] Step 6:

[0471] The server calculates the internal assessment score by aggregating each student's evaluation. This includes data on concentration levels during class, aiming for a fair and comprehensive assessment.

[0472] Step 7:

[0473] The server generates the final evaluation and feedback report. The report includes detailed information about each student's strengths and areas for improvement.

[0474] Step 8:

[0475] Users can improve the quality of their feedback by reviewing feedback reports on their devices and adding comments and additional information as needed.

[0476] Step 9:

[0477] The server sends the final feedback report to the student and their parents. The report is provided via email or in paper format.

[0478] (Example 1)

[0479] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0480] In modern educational institutions, the evaluation of individual students is often subjective, making fairness and accuracy of assessment a challenge. Furthermore, the time and effort required for data collection and analysis are increasing, making efficient feedback difficult in educational settings. There is a need to address these challenges and realize a more objective and rapid student evaluation system.

[0481] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0482] In this invention, the server includes information gathering means, analysis means, evaluation means, and feedback means. This makes it possible to objectively evaluate students' concentration levels and emotional states based on the collected data and quickly calculate fair evaluation scores. In addition, the feedback means provides each individual with detailed feedback including points for improvement, thereby improving the efficiency and fairness of evaluation in educational settings.

[0483] "Information gathering means" refers to devices used in educational institutions to acquire data such as video data from classes and student answer sheets, and includes AI cameras and scanners.

[0484] "Analysis methods" refer to technologies that process collected data to determine students' levels of concentration and emotional states, and utilize facial recognition technology and OCR technology.

[0485] "Evaluation methods" refer to the process of calculating student evaluation scores based on analyzed data and according to pre-set criteria.

[0486] A "feedback mechanism" is a function that aggregates evaluation results and generates and provides detailed feedback, including areas for improvement, to each student.

[0487] An "evaluation score" is a numerical representation of a student's academic performance, calculated using fair and consistent criteria based on various indicators obtained through analytical methods.

[0488] "Facial expression recognition technology" is a technique that analyzes facial expressions from students' video data and quantifies their emotions and level of concentration.

[0489] "OCR technology" is a technology that mechanically reads character information within scanned image data and converts it into text data.

[0490] This invention provides a system for objectively and efficiently evaluating the academic performance of each student in an educational institution. This system includes information gathering means, analysis means, evaluation means, and feedback means.

[0491] The system uses AI cameras and scanners installed in the classroom to collect video data from lessons, as well as students' answer sheets and notebooks, in digital format. Specifically, the AI ​​cameras record each student's activities as video in real time, and the scanners acquire answer sheets as image data.

[0492] The server analyzes the received video data using facial recognition technology to determine the students' level of concentration and emotional state. A general-purpose facial recognition library is used for this purpose. Additionally, OCR technology is applied to the image data of the answer sheets, converting it into text data. The open-source Tesseract is a possible OCR engine for this purpose.

[0493] Based on the analyzed text data and facial expression analysis data, the server calculates an evaluation score according to predefined evaluation criteria. These criteria are predetermined by the educational institution, ensuring consistency in evaluations among teachers.

[0494] The feedback system automatically generates detailed feedback for each student based on their evaluation results, which users can view on their own devices. Teachers can add additional comments to the feedback as needed. The final feedback is communicated to students and parents via email or printed materials.

[0495] As a concrete example, in a math class, a device captures video of students, a server uses that data to measure their concentration level, and test papers are analyzed using OCR to provide accurate evaluations. This process allows users to quickly and fairly grasp each student's learning outcomes.

[0496] As an example of a prompt to the generative AI model, you can use: "Please describe in detail the steps of how the educational system collects student data and assesses their level of concentration through facial expression analysis."

[0497] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0498] Step 1:

[0499] The device uses AI cameras and scanners in the classroom to collect video data from lessons, as well as digital data of students' answer sheets and notebooks. Specifically, the camera captures each student in real time, and the scanner digitizes the answer sheets. This input data is sent directly to the server for pre-processing for analysis. The output consists of video data and image data.

[0500] Step 2:

[0501] The server processes the received video data and uses a facial recognition algorithm to analyze students' concentration levels and emotional states. In this process, each student's facial expression is extracted from the video data and quantified, such as smiles or confusion. The input at this stage is video data, and the output is quantified data on concentration levels and emotional states.

[0502] Step 3:

[0503] The server converts the image data of the answer sheet obtained from the scanner into text data using OCR technology. Specifically, an OCR engine such as Tesseract identifies the characters in the image and converts them into text. The input at this stage is image data, and the output is text data. As a result, the answer content is obtained as digital data.

[0504] Step 4:

[0505] The server uses the concentration level, emotional state data, and text data obtained in the previous step to calculate the internal assessment score according to pre-defined evaluation criteria. These criteria are based on the student's concentration level and the accuracy of their answers, ensuring fair evaluation. The input is the analyzed numerical data, and the output is the evaluation score.

[0506] Step 5:

[0507] The server generates a feedback report based on the calculated evaluation score. Specifically, the feedback includes not only the score, but also a graph of concentration levels and areas for improvement. This information can be viewed by the user on their terminal, and additional comments can be added as needed. The input is the evaluation score, and the output is a detailed feedback report.

[0508] Step 6:

[0509] Ultimately, the server provides feedback to students and parents via email or in print. Specifically, it generates a feedback report in PDF format and sends it to the registered email address. Alternatively, it may provide printed feedback. The input is the feedback report, and the output is the distributed feedback results.

[0510] (Application Example 1)

[0511] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0512] To improve the effectiveness of learning in public educational facilities for citizens, it is necessary to provide an environment where instructors and administrators can monitor students' concentration and comprehension levels in real time and adjust lecture content on the spot. However, conventional systems have made it difficult to grasp students' status immediately, making it challenging to maximize learning effectiveness. Therefore, the challenge is to provide a system that can evaluate students' learning progress in real time and provide appropriate feedback.

[0513] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0514] In this invention, the server includes information gathering means, data analysis means, evaluation generation means, and real-time evaluation display means. This allows instructors and operators to evaluate the level of concentration and understanding of students in real time and flexibly adjust the lecture content as needed.

[0515] "Information gathering means" refers to a system used in educational institutions to acquire video information and related data of students.

[0516] A "data analysis system" is a system that processes data obtained through information gathering methods to analyze the level of concentration and comprehension of participants.

[0517] An "evaluation generation method" is a system that creates a fair evaluation of a student's learning progress based on analyzed data.

[0518] A "real-time evaluation display system" is a system used to immediately notify instructors and administrators of the evaluation results obtained by the evaluation generation system, and to adjust the lecture content based on those results.

[0519] The system implementing this invention mainly consists of three elements: a server, a terminal, and a user. First, the server includes information gathering means, data analysis means, evaluation generation means, and real-time evaluation display means. The server receives video data of participants acquired via terminals such as smartphones and smart glasses installed in the facility in real time through the network. The information gathering means analyzes the participants' facial expressions using the OpenCV facial recognition library and quantifies their level of concentration and comprehension. Furthermore, it efficiently analyzes the content of participants' answers by converting materials and answer sheets into digital data using the Tesseract OCR engine.

[0520] The data analysis method utilizes a data analysis platform such as TensorFlow to evaluate students' learning progress based on collected data. This enables the generation of fair and objective evaluations of students. Furthermore, the evaluation generation method creates feedback based on the generated evaluations and displays the results in real time on terminals held by instructors and administrators. This allows instructors and administrators to adjust the lecture content on the spot according to the students' progress.

[0521] As a concrete example, in an IT skills training seminar held at a public library, the participants' facial expressions were analyzed in real time, and if their concentration level decreased, the instructor was notified, allowing for adjustments such as returning the lecture content to basic material. In this way, the learning effect for all participants can be improved.

[0522] An example of a prompt for a generated AI model is: "We want to design a system that analyzes video data acquired by an AI camera to evaluate and notify citizens of their level of concentration and understanding in real time, and use this to adjust the content of lectures."

[0523] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0524] Step 1:

[0525] The device acquires video information of the participant using its camera. This prepares the image data necessary for facial recognition as input data.

[0526] Step 2:

[0527] The terminal transmits the image data it has acquired to the server via the network. The transmitted image data becomes the server's input data.

[0528] Step 3:

[0529] The server uses the OpenCV library to detect the participants' faces and perform facial expression analysis. This generates data that quantifies and outputs the participants' concentration levels and emotional states based on their facial expressions.

[0530] Step 4:

[0531] The server uses the OCR engine Tesseract to convert image data of documents sent from the terminal into text data. This allows the student's answers to be obtained as digital information.

[0532] Step 5:

[0533] The server uses TensorFlow to analyze quantified concentration data and answer text data. Through this analysis, it evaluates the students' understanding and learning progress, and outputs evaluation data.

[0534] Step 6:

[0535] The server generates feedback information based on evaluation data and notifies instructors and administrators in real time. Based on the generated feedback information, instructors and administrators can adjust the content of their lectures.

[0536] Step 7:

[0537] By allowing users to check information notified via their devices and modifying the lecture content and pace as needed, the system improves the learning effectiveness for participants.

[0538] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0539] This embodiment is a system that effectively assists student assessment in educational institutions, and by incorporating a new emotion engine, it provides deeper insights. The main components of the system are data collection means, analysis means, evaluation means, feedback means, and emotion engine.

[0540] As a data collection method, AI cameras and microphones are installed in the classroom to acquire video and audio data. The terminals collect this data individually and transmit it to the server in real time.

[0541] In the analysis system, the server uses video data to perform facial expression analysis and determine the students' concentration levels and emotions. In particular, by combining facial expression analysis with an emotion engine, the emotional state of the students is analyzed from multiple perspectives. Furthermore, emotions are inferred from the tone of the students' voices and speaking style through audio data analysis.

[0542] The evaluation method involves comprehensively analyzing data collected by the server, taking into account the output of the emotion engine, to score students' understanding and concentration levels. In addition to traditional academic performance assessments, it also provides indicators of emotional stability and motivation.

[0543] The feedback mechanism is a process in which the server generates feedback tailored to each student. This feedback can also utilize the output of the emotion engine to include specific suggestions for improvement and encouraging messages.

[0544] The emotion engine is a crucial component of the entire system, capable of evaluating the emotional state of individual students in detail through the analysis of audio and video data. This engine provides information to evaluation and feedback methods, meeting diverse needs in learning instruction.

[0545] As a concrete example, during a math lesson, the server collects facial expression analysis and audio data using an AI camera. An emotion engine then analyzes this data and detects if a student may be showing signs of frustration. This information is used by the evaluation system to adjust grades, and feedback is generated recommending that the student "consider providing additional support in math." This process allows educators to receive a comprehensive evaluation that includes emotional information.

[0546] The following describes the processing flow.

[0547] Step 1:

[0548] The device activates an AI camera and microphone to collect video and audio data of students in the classroom. The AI ​​camera has the ability to recognize each student's face and capture individual video footage. The microphone also captures the tone of each student's voice and speaking style.

[0549] Step 2:

[0550] The terminal compresses the data it collects and sends it to the server in real time. Here, the data undergoes signal processing to prepare it for efficient analysis.

[0551] Step 3:

[0552] The server processes the received video data through a facial expression analysis module to analyze changes in each student's facial expressions. Basic emotions such as smiles and confusion are recognized here.

[0553] Step 4:

[0554] The server feeds the audio data into an emotion analysis module, which infers the student's emotions from their voice tone and volume. This allows it to capture subtle emotional changes such as frustration or a sense of relief.

[0555] Step 5:

[0556] The server aggregates the analysis results from the emotion engine and synthesizes the acquired data to evaluate the student's current emotional state. By combining multiple data points, it performs more accurate emotion recognition.

[0557] Step 6:

[0558] The server utilizes the emotional information it receives in the evaluation process and reflects it in the assessment of students' academic performance. In addition to concentration and comprehension, emotional state is also taken into consideration to calculate an overall score.

[0559] Step 7:

[0560] Based on the evaluation results and sentiment analysis generated by the server, individual feedback is created for each student. Specific learning advice and emotional support are recommended.

[0561] Step 8:

[0562] The user (teacher) uses their device to review feedback reports provided by the server and uses them to develop teaching strategies for students. They can also revise the feedback and add supplementary comments as needed.

[0563] Step 9:

[0564] The server sends a final feedback report to the student and their parents. This allows the student to receive detailed information about their academic performance and emotional state.

[0565] (Example 2)

[0566] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0567] Traditional educational assessment systems struggle to accurately evaluate students' comprehension and concentration levels. In particular, these systems are heavily reliant on quantitative data and fail to consider emotional factors, resulting in a lack of comprehensive assessment that includes students' emotional states. Consequently, providing individualized instruction and feedback to each student becomes difficult.

[0568] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0569] In this invention, the server includes: information acquisition means for assisting evaluation in educational institutions; analysis means for analyzing the student's voice and video information acquired by the information acquisition means and estimating the student's emotional state; and integrated evaluation means for generating an evaluation that scores the student's level of comprehension and concentration based on the output of the analysis means. This makes it possible to perform a comprehensive evaluation that includes the student's emotional state. As a result, educators can provide instruction and feedback that takes into account the emotional elements of each student, thereby improving the quality of education.

[0570] "Information acquisition means" refers to devices or methods for collecting audio and video information of students within an educational institution.

[0571] "Analysis means" refers to a technology or process for comprehensively analyzing a student's emotional state based on acquired audio and video information.

[0572] An "integrated evaluation tool" is a function that uses the output of an analysis tool to quantify students' level of understanding and concentration, and to generate a fair evaluation.

[0573] A "feedback generation method" is a method or system for specifically proposing instructional content and improvement measures for students based on evaluation data obtained through an integrated evaluation method.

[0574] "Emotional state" refers to the psychological and emotional condition detected from a student's facial expressions and voice.

[0575] "Multifaceted analysis" refers to a detailed analysis of data using multiple perspectives and methods.

[0576] This invention provides a method for effectively implementing a student evaluation system in educational institutions. The following describes how each element of the system is specifically implemented.

[0577] Regarding the means of obtaining information

[0578] The terminals will use audio and video acquisition devices installed in the classroom. A high-sensitivity microphone will be used as the audio acquisition device, and an AI camera will be installed as the video acquisition device.

[0579] The terminal collects audio and video data from these devices in real time and transmits it to the server via the network.

[0580] Regarding the means of analysis

[0581] The server analyzes the received audio and video data. For the video data, libraries such as OpenCV are used to extract students' facial expressions as feature points, and an emotion engine is used to analyze their emotional state from multiple perspectives.

[0582] The system performs spectral analysis on the audio data to infer the students' emotions. This allows users to understand the students' level of concentration and motivation.

[0583] Regarding integrated evaluation means and feedback generation means

[0584] The server performs an integrated assessment based on the analysis results, quantifying students' understanding and concentration levels. In addition to traditional academic performance assessments, it provides emotional indicators.

[0585] The server uses a generative AI model to generate personalized feedback. This feedback can include advice and encouragement to improve students' motivation to learn.

[0586] As a concrete example, during a math lesson, a device collects video and audio from the classroom, and the server analyzes this data to detect when a student is struggling with a problem based on their deep thought or tone of voice. Based on this information, the server generates specific feedback such as, "Reviewing would be effective for further understanding." This entire process is automated and in real time using AI technology, enabling educators to teach more efficiently through the system.

[0587] An example of an input prompt for a generative AI model is, "Describe the specific algorithmic steps for assessing a student's emotional state and providing personalized feedback." This allows the generative AI model to receive guidance on how to appropriately generate feedback messages.

[0588] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0589] Step 1:

[0590] The terminal collects data using audio and video acquisition devices installed in the classroom.

[0591] Input: Real-time audio and video information from students.

[0592] Data processing: The device digitizes audio from a high-sensitivity microphone and divides video acquired by an AI camera into video frames.

[0593] Output: A data stream that compresses real-time audio and video data and sends it to a server via the network.

[0594] Specific operation: The terminal compresses video data in H.264 format and transmits audio data in MP3 format.

[0595] Step 2:

[0596] The server analyzes the received audio and video data.

[0597] Input: Audio and video data transmitted in real time.

[0598] Data processing: For video data, the server uses the OpenCV library to extract facial feature points and analyzes facial expressions using an emotion engine. For audio data, spectral analysis is performed to extract vocal cord patterns.

[0599] Output: Student emotional state data after analysis.

[0600] Specific operation: The server analyzes the frequency components of the audio data to evaluate the tone and pitch of the student's voice. From the video data, it analyzes eyebrow movements and mouth shape.

[0601] Step 3:

[0602] The server integrates the analysis results and performs an integrated evaluation.

[0603] Input: Emotional state data based on facial expression analysis and voice analysis.

[0604] Data processing: The server quantifies students' understanding and concentration levels by weighting the analysis results, and performs an integrated evaluation by also considering the output of the emotion engine.

[0605] Output: Evaluation data that quantifies students' level of understanding and concentration.

[0606] Specific operation: The server converts emotional state data into quantitative parameters and uses them to calculate the student's concentration score.

[0607] Step 4:

[0608] The server generates feedback based on the evaluation data.

[0609] Input: Student evaluation data obtained through integrated evaluation methods.

[0610] Data processing: The server uses a generative AI model to generate specific feedback messages that are appropriate for the student's evaluation data.

[0611] Output: Customized feedback messages for students.

[0612] Specific operation: The server automatically generates text containing study methods and advice tailored to each student's individual needs.

[0613] (Application Example 2)

[0614] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0615] In educational institutions and homes, optimizing learners' learning effectiveness and providing appropriate instruction tailored to their progress requires considering not only traditional performance evaluations but also psychological aspects such as learners' emotions and motivation. However, efficiently and in real time, it is difficult to grasp this information, posing a major obstacle to educators making decisions regarding learning support.

[0616] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0617] In this invention, the server includes data acquisition means for assisting evaluation in educational institutions, analysis means for analyzing information acquired by the data acquisition means, and evaluation generation means for generating fair evaluations based on the output of the analysis means. This enables a multifaceted analysis of learners' learning status and emotional state, allowing for comprehensive evaluation and appropriate feedback.

[0618] An "educational institution" is an organization or facility established for learners to acquire knowledge and skills.

[0619] "Data acquisition methods to support assessment" refer to tools and equipment used to collect various data related to learners.

[0620] "Analysis means" refers to processes or systems for processing acquired data and analyzing the learner's state and learning progress.

[0621] An "evaluation generation method" is a function that generates numerical values ​​or indicators of learners' learning effectiveness and motivation based on the analysis results.

[0622] A "feedback provision method" is a system for delivering improvement suggestions and advice to learners and educators based on evaluation results.

[0623] "Emotion analysis methods" are techniques for inferring a learner's emotional state from data and clarifying the specific type and intensity of their emotions.

[0624] A "video acquisition device" is a device installed to physically collect video data from the environment.

[0625] The "facial expression data analysis module" is a software component that analyzes facial feature points and other elements within video data to estimate emotions and attention levels.

[0626] The system of this invention provides a multifaceted evaluation of learners' learning progress in educational institutions and homes. The system mainly consists of data acquisition means, analysis means, evaluation generation means, feedback provision means, and sentiment analysis means.

[0627] The device uses an AI camera and microphone to collect video and audio data from learners and transmits it to a server in real time. This data acquisition method records all of the learner's actions in the environment.

[0628] The server uses the open-source facial expression analysis library "OpenFace" and the audio analysis library "pyAudioAnalysis" to analyze video and audio data. It evaluates the learner's level of concentration and emotional state through changes in facial expressions and tone of voice.

[0629] Based on the output of the analysis means, the server numerically evaluates the learner's learning and emotional state through the evaluation generation means. This enables a comprehensive evaluation that includes not only academic ability but also motivation and emotional appropriateness.

[0630] The feedback system generates feedback for learners and guardians based on the evaluations received. This process takes into account the output of the sentiment analysis system, providing specific suggestions for improvement and encouraging messages.

[0631] As a concrete example, a home learning support robot collects facial expression data while a child is doing their homework, and a server analyzes it. Based on the analysis, if the robot determines that the child may be showing frustration with a math problem, it will offer words of encouragement such as, "You're doing very well! Let me know if there's anything I can do to help."

[0632] Example of an input prompt for a generative AI model: "Analyze the learner's current emotional state based on their facial expression and voice data. If stress or lack of concentration is indicated, generate an appropriate encouraging comment."

[0633] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0634] Step 1:

[0635] The device uses an AI camera and microphone to collect the learner's facial expressions and voice. In this step, video and audio data of the environment are obtained as input. The obtained data is preprocessed, such as noise reduction and brightness adjustment, and then sent to the server.

[0636] Step 2:

[0637] The server analyzes the received video data using the "OpenFace" library. It receives pre-processed video data as input, extracts facial feature points, and uses these to estimate the learner's emotional state from their facial expressions. The output of this step is the learner's emotional data.

[0638] Step 3:

[0639] In parallel, the server analyzes the audio data using the "pyAudioAnalysis" library. It receives pre-processed audio data as input, analyzes the tone and speed of the voice, and evaluates the learner's emotions. The output of this step is emotion data derived from the audio.

[0640] Step 4:

[0641] The server integrates the outputs from steps 2 and 3 and uses an evaluation generation mechanism to quantify the learner's comprehension and concentration levels. It receives emotional data as input and analyzes it comprehensively to calculate the learner's overall emotional evaluation. The output of this step is a comprehensive emotional evaluation score.

[0642] Step 5:

[0643] The feedback system generates feedback for learners based on their evaluation scores. It considers the evaluation scores as input and creates appropriate improvement suggestions and encouraging messages for the learners. The output of this step is the feedback message.

[0644] Step 6:

[0645] The user reviews the feedback provided through the device, and the learning support robot provides that feedback to the learner verbally as needed. Based on the outputted feedback, the learner can try to implement improvements.

[0646] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0647] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0648] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0649] [Fourth Embodiment]

[0650] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0651] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0652] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0653] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0654] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0655] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0656] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0657] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0658] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0659] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0660] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0661] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0662] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0663] This embodiment provides a system for assisting student evaluation in educational institutions. Specifically, it is a network-based system including data collection means, analysis means, evaluation means, and feedback means.

[0664] As a means of data collection, AI cameras and scanners will be installed in classrooms to acquire video data from lessons and digital data of students' answer sheets and notebooks. The terminals will transmit this data to a server in real time.

[0665] In the analysis process, the server first performs facial expression analysis based on video information. This determines each student's level of concentration and emotional state. Furthermore, the scanned answer sheet image data is converted into text data using OCR technology. This conversion digitizes the students' answers, enabling accurate analysis.

[0666] The evaluation process is performed by a server, and based on the obtained analysis data, a unified evaluation standard is applied to calculate the internal assessment score. This evaluation standard is predetermined within the educational institution, reducing variability in evaluations among teachers.

[0667] The feedback system automatically aggregates evaluation results and generates detailed feedback on each student's performance. This feedback information can be viewed by the user (teacher) via their device, and supplementary comments can be added as needed. Finally, the feedback is provided to students and their parents via email or in printed form.

[0668] As a concrete example, during a math class, the server analyzes each student's video feed to measure their level of concentration. After the test, the answer sheets are scanned, and their correctness is determined while their internal assessment scores are calculated. This process allows users to evaluate students' learning outcomes more objectively and efficiently.

[0669] The following describes the processing flow.

[0670] Step 1:

[0671] The device activates an AI camera to record video data from the classroom. The camera is configured to recognize each student's face and capture their image individually. The recorded data is sent to the server in real time.

[0672] Step 2:

[0673] The device scans answer sheets and notebooks, generating image data. This image data is then transferred to a server for OCR (Optical Character Recognition) processing.

[0674] Step 3:

[0675] The server processes the received video data through a facial expression analysis program. The program then generates data to estimate and evaluate students' concentration levels and emotional states based on changes in their facial expressions.

[0676] Step 4:

[0677] The server uses OCR to extract text from image data. It converts handwritten characters and answers into a format that can be analyzed as digital text.

[0678] Step 5:

[0679] The server uses the analyzed text data to evaluate students' answers. It analyzes the accuracy rate and trends in incorrect answers, and scores them based on predetermined criteria.

[0680] Step 6:

[0681] The server calculates the internal assessment score by aggregating each student's evaluation. This includes data on concentration levels during class, aiming for a fair and comprehensive assessment.

[0682] Step 7:

[0683] The server generates the final evaluation and feedback report. The report includes detailed information about each student's strengths and areas for improvement.

[0684] Step 8:

[0685] Users can improve the quality of their feedback by reviewing feedback reports on their devices and adding comments and additional information as needed.

[0686] Step 9:

[0687] The server sends the final feedback report to the student and their parents. The report is provided via email or in paper format.

[0688] (Example 1)

[0689] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0690] In modern educational institutions, the evaluation of individual students is often subjective, making fairness and accuracy of assessment a challenge. Furthermore, the time and effort required for data collection and analysis are increasing, making efficient feedback difficult in educational settings. There is a need to address these challenges and realize a more objective and rapid student evaluation system.

[0691] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0692] In this invention, the server includes information gathering means, analysis means, evaluation means, and feedback means. This makes it possible to objectively evaluate students' concentration levels and emotional states based on the collected data and quickly calculate fair evaluation scores. In addition, the feedback means provides each individual with detailed feedback including points for improvement, thereby improving the efficiency and fairness of evaluation in educational settings.

[0693] "Information gathering means" refers to devices used in educational institutions to acquire data such as video data from classes and student answer sheets, and includes AI cameras and scanners.

[0694] "Analysis methods" refer to technologies that process collected data to determine students' levels of concentration and emotional states, and utilize facial recognition technology and OCR technology.

[0695] "Evaluation methods" refer to the process of calculating student evaluation scores based on analyzed data and according to pre-set criteria.

[0696] A "feedback mechanism" is a function that aggregates evaluation results and generates and provides detailed feedback, including areas for improvement, to each student.

[0697] An "evaluation score" is a numerical representation of a student's academic performance, calculated using fair and consistent criteria based on various indicators obtained through analytical methods.

[0698] "Facial expression recognition technology" is a technique that analyzes facial expressions from students' video data and quantifies their emotions and level of concentration.

[0699] "OCR technology" is a technology that mechanically reads character information within scanned image data and converts it into text data.

[0700] This invention provides a system for objectively and efficiently evaluating the academic performance of each student in an educational institution. This system includes information gathering means, analysis means, evaluation means, and feedback means.

[0701] The system uses AI cameras and scanners installed in the classroom to collect video data from lessons, as well as students' answer sheets and notebooks, in digital format. Specifically, the AI ​​cameras record each student's activities as video in real time, and the scanners acquire answer sheets as image data.

[0702] The server analyzes the received video data using facial recognition technology to determine the students' level of concentration and emotional state. A general-purpose facial recognition library is used for this purpose. Additionally, OCR technology is applied to the image data of the answer sheets, converting it into text data. The open-source Tesseract is a possible OCR engine for this purpose.

[0703] Based on the analyzed text data and facial expression analysis data, the server calculates an evaluation score according to predefined evaluation criteria. These criteria are predetermined by the educational institution, ensuring consistency in evaluations among teachers.

[0704] The feedback system automatically generates detailed feedback for each student based on their evaluation results, which users can view on their own devices. Teachers can add additional comments to the feedback as needed. The final feedback is communicated to students and parents via email or printed materials.

[0705] As a concrete example, in a math class, a device captures video of students, a server uses that data to measure their concentration level, and test papers are analyzed using OCR to provide accurate evaluations. This process allows users to quickly and fairly grasp each student's learning outcomes.

[0706] As an example of a prompt to the generative AI model, you can use: "Please describe in detail the steps of how the educational system collects student data and assesses their level of concentration through facial expression analysis."

[0707] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0708] Step 1:

[0709] The device uses AI cameras and scanners in the classroom to collect video data from lessons, as well as digital data of students' answer sheets and notebooks. Specifically, the camera captures each student in real time, and the scanner digitizes the answer sheets. This input data is sent directly to the server for pre-processing for analysis. The output consists of video data and image data.

[0710] Step 2:

[0711] The server processes the received video data and uses a facial recognition algorithm to analyze students' concentration levels and emotional states. In this process, each student's facial expression is extracted from the video data and quantified, such as smiles or confusion. The input at this stage is video data, and the output is quantified data on concentration levels and emotional states.

[0712] Step 3:

[0713] The server converts the image data of the answer sheet obtained from the scanner into text data using OCR technology. Specifically, an OCR engine such as Tesseract identifies the characters in the image and converts them into text. The input at this stage is image data, and the output is text data. As a result, the answer content is obtained as digital data.

[0714] Step 4:

[0715] The server uses the concentration level, emotional state data, and text data obtained in the previous step to calculate the internal assessment score according to pre-defined evaluation criteria. These criteria are based on the student's concentration level and the accuracy of their answers, ensuring fair evaluation. The input is the analyzed numerical data, and the output is the evaluation score.

[0716] Step 5:

[0717] The server generates a feedback report based on the calculated evaluation score. Specifically, the feedback includes not only the score, but also a graph of concentration levels and areas for improvement. This information can be viewed by the user on their terminal, and additional comments can be added as needed. The input is the evaluation score, and the output is a detailed feedback report.

[0718] Step 6:

[0719] Ultimately, the server provides feedback to students and parents via email or in print. Specifically, it generates a feedback report in PDF format and sends it to the registered email address. Alternatively, it may provide printed feedback. The input is the feedback report, and the output is the distributed feedback results.

[0720] (Application Example 1)

[0721] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0722] To improve the effectiveness of learning in public educational facilities for citizens, it is necessary to provide an environment where instructors and administrators can monitor students' concentration and comprehension levels in real time and adjust lecture content on the spot. However, conventional systems have made it difficult to grasp students' status immediately, making it challenging to maximize learning effectiveness. Therefore, the challenge is to provide a system that can evaluate students' learning progress in real time and provide appropriate feedback.

[0723] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0724] In this invention, the server includes information gathering means, data analysis means, evaluation generation means, and real-time evaluation display means. This allows instructors and operators to evaluate the level of concentration and understanding of students in real time and flexibly adjust the lecture content as needed.

[0725] "Information gathering means" refers to a system used in educational institutions to acquire video information and related data of students.

[0726] A "data analysis system" is a system that processes data obtained through information gathering methods to analyze the level of concentration and comprehension of participants.

[0727] An "evaluation generation method" is a system that creates a fair evaluation of a student's learning progress based on analyzed data.

[0728] A "real-time evaluation display system" is a system used to immediately notify instructors and administrators of the evaluation results obtained by the evaluation generation system, and to adjust the lecture content based on those results.

[0729] The system implementing this invention mainly consists of three elements: a server, a terminal, and a user. First, the server includes information gathering means, data analysis means, evaluation generation means, and real-time evaluation display means. The server receives video data of participants acquired via terminals such as smartphones and smart glasses installed in the facility in real time through the network. The information gathering means analyzes the participants' facial expressions using the OpenCV facial recognition library and quantifies their level of concentration and comprehension. Furthermore, it efficiently analyzes the content of participants' answers by converting materials and answer sheets into digital data using the Tesseract OCR engine.

[0730] The data analysis method utilizes a data analysis platform such as TensorFlow to evaluate students' learning progress based on collected data. This enables the generation of fair and objective evaluations of students. Furthermore, the evaluation generation method creates feedback based on the generated evaluations and displays the results in real time on terminals held by instructors and administrators. This allows instructors and administrators to adjust the lecture content on the spot according to the students' progress.

[0731] As a concrete example, in an IT skills training seminar held at a public library, the participants' facial expressions were analyzed in real time, and if their concentration level decreased, the instructor was notified, allowing for adjustments such as returning the lecture content to basic material. In this way, the learning effect for all participants can be improved.

[0732] An example of a prompt for a generated AI model is: "We want to design a system that analyzes video data acquired by an AI camera to evaluate and notify citizens of their level of concentration and understanding in real time, and use this to adjust the content of lectures."

[0733] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0734] Step 1:

[0735] The device acquires video information of the participant using its camera. This prepares the image data necessary for facial recognition as input data.

[0736] Step 2:

[0737] The terminal transmits the image data it has acquired to the server via the network. The transmitted image data becomes the server's input data.

[0738] Step 3:

[0739] The server uses the OpenCV library to detect the participants' faces and perform facial expression analysis. This generates data that quantifies and outputs the participants' concentration levels and emotional states based on their facial expressions.

[0740] Step 4:

[0741] The server uses the OCR engine Tesseract to convert image data of documents sent from the terminal into text data. This allows the student's answers to be obtained as digital information.

[0742] Step 5:

[0743] The server uses TensorFlow to analyze quantified concentration data and answer text data. Through this analysis, it evaluates the students' understanding and learning progress, and outputs evaluation data.

[0744] Step 6:

[0745] The server generates feedback information based on evaluation data and notifies instructors and administrators in real time. Based on the generated feedback information, instructors and administrators can adjust the content of their lectures.

[0746] Step 7:

[0747] By allowing users to check information notified via their devices and modifying the lecture content and pace as needed, the system improves the learning effectiveness for participants.

[0748] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0749] This embodiment is a system that effectively assists student assessment in educational institutions, and by incorporating a new emotion engine, it provides deeper insights. The main components of the system are data collection means, analysis means, evaluation means, feedback means, and emotion engine.

[0750] As a data collection method, AI cameras and microphones are installed in the classroom to acquire video and audio data. The terminals collect this data individually and transmit it to the server in real time.

[0751] In the analysis system, the server uses video data to perform facial expression analysis and determine the students' concentration levels and emotions. In particular, by combining facial expression analysis with an emotion engine, the emotional state of the students is analyzed from multiple perspectives. Furthermore, emotions are inferred from the tone of the students' voices and speaking style through audio data analysis.

[0752] The evaluation method involves comprehensively analyzing data collected by the server, taking into account the output of the emotion engine, to score students' understanding and concentration levels. In addition to traditional academic performance assessments, it also provides indicators of emotional stability and motivation.

[0753] The feedback mechanism is a process in which the server generates feedback tailored to each student. This feedback can also utilize the output of the emotion engine to include specific suggestions for improvement and encouraging messages.

[0754] The emotion engine is a crucial component of the entire system, capable of evaluating the emotional state of individual students in detail through the analysis of audio and video data. This engine provides information to evaluation and feedback methods, meeting diverse needs in learning instruction.

[0755] As a concrete example, during a math lesson, the server collects facial expression analysis and audio data using an AI camera. An emotion engine then analyzes this data and detects if a student may be showing signs of frustration. This information is used by the evaluation system to adjust grades, and feedback is generated recommending that the student "consider providing additional support in math." This process allows educators to receive a comprehensive evaluation that includes emotional information.

[0756] The following describes the processing flow.

[0757] Step 1:

[0758] The device activates an AI camera and microphone to collect video and audio data of students in the classroom. The AI ​​camera has the ability to recognize each student's face and capture individual video footage. The microphone also captures the tone of each student's voice and speaking style.

[0759] Step 2:

[0760] The terminal compresses the data it collects and sends it to the server in real time. Here, the data undergoes signal processing to prepare it for efficient analysis.

[0761] Step 3:

[0762] The server processes the received video data through a facial expression analysis module to analyze changes in each student's facial expressions. Basic emotions such as smiles and confusion are recognized here.

[0763] Step 4:

[0764] The server feeds the audio data into an emotion analysis module, which infers the student's emotions from their voice tone and volume. This allows it to capture subtle emotional changes such as frustration or a sense of relief.

[0765] Step 5:

[0766] The server aggregates the analysis results from the emotion engine and synthesizes the acquired data to evaluate the student's current emotional state. By combining multiple data points, it performs more accurate emotion recognition.

[0767] Step 6:

[0768] The server utilizes the emotional information it receives in the evaluation process and reflects it in the assessment of students' academic performance. In addition to concentration and comprehension, emotional state is also taken into consideration to calculate an overall score.

[0769] Step 7:

[0770] Based on the evaluation results and sentiment analysis generated by the server, individual feedback is created for each student. Specific learning advice and emotional support are recommended.

[0771] Step 8:

[0772] The user (teacher) uses their device to review feedback reports provided by the server and uses them to develop teaching strategies for students. They can also revise the feedback and add supplementary comments as needed.

[0773] Step 9:

[0774] The server sends a final feedback report to the student and their parents. This allows the student to receive detailed information about their academic performance and emotional state.

[0775] (Example 2)

[0776] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0777] Traditional educational assessment systems struggle to accurately evaluate students' comprehension and concentration levels. In particular, these systems are heavily reliant on quantitative data and fail to consider emotional factors, resulting in a lack of comprehensive assessment that includes students' emotional states. Consequently, providing individualized instruction and feedback to each student becomes difficult.

[0778] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0779] In this invention, the server includes: information acquisition means for assisting evaluation in educational institutions; analysis means for analyzing the student's voice and video information acquired by the information acquisition means and estimating the student's emotional state; and integrated evaluation means for generating an evaluation that scores the student's level of comprehension and concentration based on the output of the analysis means. This makes it possible to perform a comprehensive evaluation that includes the student's emotional state. As a result, educators can provide instruction and feedback that takes into account the emotional elements of each student, thereby improving the quality of education.

[0780] "Information acquisition means" refers to devices or methods for collecting audio and video information of students within an educational institution.

[0781] "Analysis means" refers to a technology or process for comprehensively analyzing a student's emotional state based on acquired audio and video information.

[0782] An "integrated evaluation tool" is a function that uses the output of an analysis tool to quantify students' level of understanding and concentration, and to generate a fair evaluation.

[0783] A "feedback generation method" is a method or system for specifically proposing instructional content and improvement measures for students based on evaluation data obtained through an integrated evaluation method.

[0784] "Emotional state" refers to the psychological and emotional condition detected from a student's facial expressions and voice.

[0785] "Multifaceted analysis" refers to a detailed analysis of data using multiple perspectives and methods.

[0786] This invention provides a method for effectively implementing a student evaluation system in educational institutions. The following describes how each element of the system is specifically implemented.

[0787] Regarding the means of obtaining information

[0788] The terminals will use audio and video acquisition devices installed in the classroom. A high-sensitivity microphone will be used as the audio acquisition device, and an AI camera will be installed as the video acquisition device.

[0789] The terminal collects audio and video data from these devices in real time and transmits it to the server via the network.

[0790] Regarding the means of analysis

[0791] The server analyzes the received audio and video data. For the video data, libraries such as OpenCV are used to extract students' facial expressions as feature points, and an emotion engine is used to analyze their emotional state from multiple perspectives.

[0792] The system performs spectral analysis on the audio data to infer the students' emotions. This allows users to understand the students' level of concentration and motivation.

[0793] Regarding integrated evaluation means and feedback generation means

[0794] The server performs an integrated assessment based on the analysis results, quantifying students' understanding and concentration levels. In addition to traditional academic performance assessments, it provides emotional indicators.

[0795] The server uses a generative AI model to generate personalized feedback. This feedback can include advice and encouragement to improve students' motivation to learn.

[0796] As a concrete example, during a math lesson, a device collects video and audio from the classroom, and the server analyzes this data to detect when a student is struggling with a problem based on their deep thought or tone of voice. Based on this information, the server generates specific feedback such as, "Reviewing would be effective for further understanding." This entire process is automated and in real time using AI technology, enabling educators to teach more efficiently through the system.

[0797] An example of an input prompt for a generative AI model is, "Describe the specific algorithmic steps for assessing a student's emotional state and providing personalized feedback." This allows the generative AI model to receive guidance on how to appropriately generate feedback messages.

[0798] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0799] Step 1:

[0800] The terminal collects data using audio and video acquisition devices installed in the classroom.

[0801] Input: Real-time audio and video information from students.

[0802] Data processing: The device digitizes audio from a high-sensitivity microphone and divides video acquired by an AI camera into video frames.

[0803] Output: A data stream that compresses real-time audio and video data and sends it to a server via the network.

[0804] Specific operation: The terminal compresses video data in H.264 format and transmits audio data in MP3 format.

[0805] Step 2:

[0806] The server analyzes the received audio and video data.

[0807] Input: Audio and video data transmitted in real time.

[0808] Data processing: For video data, the server uses the OpenCV library to extract facial feature points and analyzes facial expressions using an emotion engine. For audio data, spectral analysis is performed to extract vocal cord patterns.

[0809] Output: Student emotional state data after analysis.

[0810] Specific operation: The server analyzes the frequency components of the audio data to evaluate the tone and pitch of the student's voice. From the video data, it analyzes eyebrow movements and mouth shape.

[0811] Step 3:

[0812] The server integrates the analysis results and performs an integrated evaluation.

[0813] Input: Emotional state data based on facial expression analysis and voice analysis.

[0814] Data processing: The server quantifies students' understanding and concentration levels by weighting the analysis results, and performs an integrated evaluation by also considering the output of the emotion engine.

[0815] Output: Evaluation data that quantifies students' level of understanding and concentration.

[0816] Specific operation: The server converts emotional state data into quantitative parameters and uses them to calculate the student's concentration score.

[0817] Step 4:

[0818] The server generates feedback based on the evaluation data.

[0819] Input: Student evaluation data obtained through integrated evaluation methods.

[0820] Data processing: The server uses a generative AI model to generate specific feedback messages that are appropriate for the student's evaluation data.

[0821] Output: Customized feedback messages for students.

[0822] Specific operation: The server automatically generates text containing study methods and advice tailored to each student's individual needs.

[0823] (Application Example 2)

[0824] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0825] In educational institutions and homes, optimizing learners' learning effectiveness and providing appropriate instruction tailored to their progress requires considering not only traditional performance evaluations but also psychological aspects such as learners' emotions and motivation. However, efficiently and in real time, it is difficult to grasp this information, posing a major obstacle to educators making decisions regarding learning support.

[0826] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0827] In this invention, the server includes data acquisition means for assisting evaluation in educational institutions, analysis means for analyzing information acquired by the data acquisition means, and evaluation generation means for generating fair evaluations based on the output of the analysis means. This enables a multifaceted analysis of learners' learning status and emotional state, allowing for comprehensive evaluation and appropriate feedback.

[0828] An "educational institution" is an organization or facility established for learners to acquire knowledge and skills.

[0829] "Data acquisition methods to support assessment" refer to tools and equipment used to collect various data related to learners.

[0830] "Analysis means" refers to processes or systems for processing acquired data and analyzing the learner's state and learning progress.

[0831] An "evaluation generation method" is a function that generates numerical values ​​or indicators of learners' learning effectiveness and motivation based on the analysis results.

[0832] A "feedback provision method" is a system for delivering improvement suggestions and advice to learners and educators based on evaluation results.

[0833] "Emotion analysis methods" are techniques for inferring a learner's emotional state from data and clarifying the specific type and intensity of their emotions.

[0834] A "video acquisition device" is a device installed to physically collect video data from the environment.

[0835] The "facial expression data analysis module" is a software component that analyzes facial feature points and other elements within video data to estimate emotions and attention levels.

[0836] The system of this invention provides a multifaceted evaluation of learners' learning progress in educational institutions and homes. The system mainly consists of data acquisition means, analysis means, evaluation generation means, feedback provision means, and sentiment analysis means.

[0837] The device uses an AI camera and microphone to collect video and audio data from learners and transmits it to a server in real time. This data acquisition method records all of the learner's actions in the environment.

[0838] The server uses the open-source facial expression analysis library "OpenFace" and the audio analysis library "pyAudioAnalysis" to analyze video and audio data. It evaluates the learner's level of concentration and emotional state through changes in facial expressions and tone of voice.

[0839] Based on the output of the analysis means, the server numerically evaluates the learner's learning and emotional state through the evaluation generation means. This enables a comprehensive evaluation that includes not only academic ability but also motivation and emotional appropriateness.

[0840] The feedback system generates feedback for learners and guardians based on the evaluations received. This process takes into account the output of the sentiment analysis system, providing specific suggestions for improvement and encouraging messages.

[0841] As a concrete example, a home learning support robot collects facial expression data while a child is doing their homework, and a server analyzes it. Based on the analysis, if the robot determines that the child may be showing frustration with a math problem, it will offer words of encouragement such as, "You're doing very well! Let me know if there's anything I can do to help."

[0842] Example of an input prompt for a generative AI model: "Analyze the learner's current emotional state based on their facial expression and voice data. If stress or lack of concentration is indicated, generate an appropriate encouraging comment."

[0843] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0844] Step 1:

[0845] The device uses an AI camera and microphone to collect the learner's facial expressions and voice. In this step, video and audio data of the environment are obtained as input. The obtained data is preprocessed, such as noise reduction and brightness adjustment, and then sent to the server.

[0846] Step 2:

[0847] The server analyzes the received video data using the "OpenFace" library. It receives pre-processed video data as input, extracts facial feature points, and uses these to estimate the learner's emotional state from their facial expressions. The output of this step is the learner's emotional data.

[0848] Step 3:

[0849] In parallel, the server analyzes the audio data using the "pyAudioAnalysis" library. It receives pre-processed audio data as input, analyzes the tone and speed of the voice, and evaluates the learner's emotions. The output of this step is emotion data derived from the audio.

[0850] Step 4:

[0851] The server integrates the outputs from steps 2 and 3 and uses an evaluation generation mechanism to quantify the learner's comprehension and concentration levels. It receives emotional data as input and analyzes it comprehensively to calculate the learner's overall emotional evaluation. The output of this step is a comprehensive emotional evaluation score.

[0852] Step 5:

[0853] The feedback system generates feedback for learners based on their evaluation scores. It considers the evaluation scores as input and creates appropriate improvement suggestions and encouraging messages for the learners. The output of this step is the feedback message.

[0854] Step 6:

[0855] The user reviews the feedback provided through the device, and the learning support robot provides that feedback to the learner verbally as needed. Based on the outputted feedback, the learner can try to implement improvements.

[0856] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0857] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0858] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0859] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0860] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0861] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0862] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0863] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0864] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0865] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0866] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0867] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0868] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0869] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0870] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0871] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0872] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0873] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0874] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0875] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0876] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0877] The following is further disclosed regarding the embodiments described above.

[0878] (Claim 1)

[0879] Data collection methods to assist in evaluation in educational institutions,

[0880] An analysis means for analyzing the information acquired by the data collection means,

[0881] An evaluation means that generates a fair evaluation based on the output of the analysis means,

[0882] A feedback means that provides feedback based on evaluation information generated by the evaluation means,

[0883] A system that includes this.

[0884] (Claim 2)

[0885] The system according to claim 1, characterized in that the data collection means includes a video acquisition module that acquires video information of the classroom.

[0886] (Claim 3)

[0887] The system according to claim 1, characterized in that the analysis means includes a facial expression analysis module for analyzing the level of concentration of students from the acquired information.

[0888] "Example 1"

[0889] (Claim 1)

[0890] Information gathering methods to assist in evaluation in educational institutions,

[0891] An analysis means that analyzes the data acquired by the aforementioned information gathering means and uses facial recognition technology to determine the individual level of concentration and emotions,

[0892] Based on the aforementioned analysis means and OCR technology, an evaluation means generates text data and calculates a fair evaluation score.

[0893] A feedback means that aggregates the evaluation information generated by the aforementioned evaluation means and provides feedback to each individual,

[0894] A system that includes this.

[0895] (Claim 2)

[0896] The system according to claim 1, characterized in that the information gathering means includes an image acquisition device for acquiring visual information in a classroom environment.

[0897] (Claim 3)

[0898] The system according to claim 1, characterized in that the analysis means includes a facial expression analysis device for analyzing the learner's level of concentration from the acquired data.

[0899] "Application Example 1"

[0900] (Claim 1)

[0901] Information gathering methods to assist in evaluation at educational institutions,

[0902] A data analysis means for analyzing the data acquired by the aforementioned information gathering means,

[0903] An evaluation generation means that generates a fair evaluation based on the output of the data analysis means,

[0904] A feedback providing means that provides feedback based on the evaluation information generated by the evaluation generation means,

[0905] A real-time evaluation display system that assesses concentration and comprehension in real time and notifies instructors and administrators of the provided feedback information,

[0906] A system that includes this.

[0907] (Claim 2)

[0908] The system according to claim 1, characterized in that the information gathering means includes a video acquisition system for acquiring video information within the facility.

[0909] (Claim 3)

[0910] The system according to claim 1, characterized in that the data analysis means includes a face recognition analysis system for analyzing the concentration level of participants from the acquired information.

[0911] "Example 2 of combining an emotion engine"

[0912] (Claim 1)

[0913] Information acquisition methods to assist in evaluation in educational institutions,

[0914] An analysis means for analyzing the student's voice and video information obtained by the aforementioned information acquisition means and estimating the student's emotional state,

[0915] An integrated evaluation means that generates an evaluation that scores the student's level of understanding and concentration based on the output of the analysis means,

[0916] A feedback generation means that provides finely adjusted feedback based on the evaluation information generated by the integrated evaluation means,

[0917] A system that includes this.

[0918] (Claim 2)

[0919] The system according to claim 1, characterized in that the information acquisition means includes an audio acquisition module and a video acquisition module for acquiring audio and video information within a classroom.

[0920] (Claim 3)

[0921] The system according to claim 1, characterized in that the analysis means includes an emotion analysis module for comprehensively analyzing the emotional state of students from acquired audio and video information.

[0922] "Application example 2 when combining with an emotional engine"

[0923] (Claim 1)

[0924] Data acquisition methods to assist in evaluation in educational institutions,

[0925] An analysis means for analyzing the information acquired by the data acquisition means,

[0926] An evaluation generation means that generates a fair evaluation based on the output of the analysis means,

[0927] A feedback providing means that provides feedback based on the evaluation information generated by the evaluation generation means,

[0928] An emotion analysis tool that analyzes the learner's emotional state and suggests improvement plans,

[0929] A system that includes this.

[0930] (Claim 2)

[0931] The system according to claim 1, characterized in that the data acquisition means includes a video acquisition device for acquiring video information within the environment.

[0932] (Claim 3)

[0933] The system according to claim 1, characterized in that the analysis means includes a facial expression data analysis module for analyzing the learner's level of attention from the acquired information. [Explanation of Symbols]

[0934] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Information gathering methods to assist in evaluation at educational institutions, A data analysis means for analyzing the data acquired by the aforementioned information gathering means, An evaluation generation means that generates a fair evaluation based on the output of the data analysis means, A feedback providing means that provides feedback based on the evaluation information generated by the evaluation generation means, A real-time evaluation display system that assesses concentration and comprehension in real time and notifies instructors and administrators of the provided feedback information, A system that includes this.

2. The system according to claim 1, characterized in that the information gathering means includes a video acquisition system for acquiring video information within the facility.

3. The system according to claim 1, characterized in that the data analysis means includes a face recognition analysis system for analyzing the concentration level of participants from the acquired information.

Images