System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving and pre-processing lesson content; means for operating a generative AI model for automatically generating lesson plans and test questions based on the pre-processed content; means for transmitting the generated lesson plans and test questions to terminals of educational institutions; means for collecting lesson videos through cameras installed in classrooms; means for automatically generating objective evaluations of students; and means for transmitting the generated evaluation results to the terminals of the educational institutions.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's educational environment, teachers' overtime hours are a problem. In particular, teachers spend a lot of time on tasks such as lesson preparation, test creation, and lesson evaluation, leaving them with insufficient time to communicate with students. Furthermore, differences in evaluation criteria between teachers can be problematic, resulting in a lack of consistency and objectivity in evaluations. The present invention aims to solve these issues and ensure consistency and objectivity in evaluations while improving teacher work efficiency. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by the following means.

[0006] A means of receiving and preprocessing lesson content data

[0007] A means to run a generative AI model to automatically generate lesson plans and test questions based on preprocessed data

[0008] A means of sending generated lesson plans and test questions to educational institution devices

[0009] A means of collecting video footage from classrooms via cameras installed in the classroom

[0010] A means of analyzing collected video data and automatically generating objective evaluations of students

[0011] A means of transmitting the generated assessment results to the educational institution's terminal

[0012] These measures allow teachers to significantly reduce the time it takes to prepare lessons and create tests, improving the efficiency and objectivity of assessment work. Specifically, lesson plans and test questions are automatically generated, and teachers only need to review and revise them. Furthermore, generative AI can automatically generate objective evaluations of students using video data from lessons, reducing the variation in assessment criteria between teachers. This allows teachers to spend more time communicating with students and provide higher-quality education.

[0013] "Course content data" refers to information about teaching materials, textbooks, and related materials used by educational institutions.

[0014] "Preprocessing" refers to a series of operations that transform raw data into a form that is easier for generative AI to process.

[0015] A "generative AI model" is a type of artificial intelligence that uses specific algorithms to automatically generate lesson plans, test questions, and other information from data.

[0016] "Lesson Plan" means a detailed plan for a lesson, including the lesson agenda, materials to be used, exercises, supporting visual materials, etc.

[0017] "Test Questions" refers to a set of questions used to measure comprehension of course content.

[0018] "Educational institution terminals" refers to digital devices such as computers and tablets used by faculty members belonging to educational institutions.

[0019] "Camera" means a video camera device used to capture images in the classroom and record what is happening during the lesson.

[0020] "Class footage" refers to video data recorded by a camera during class.

[0021] "Analysis" refers to the process of analyzing collected data and extracting and understanding useful information from it.

[0022] "Objective evaluation" refers to an evaluation that is conducted fairly based on certain standards, eliminating subjective bias.

[0023] "Evaluation results" refers to the collective evaluation of each student derived based on data analyzed by the generative AI.

[0024] "Verification and correction" refers to a series of operations in which teachers review the generated data and results and make changes as necessary. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0026] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0029] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0030] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0031] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0033] [First embodiment]

[0034] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0035] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0036] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0038] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0041] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0043] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0044] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0045] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] The present invention relates to a system that automates lesson preparation and evaluation tasks by utilizing generative AI, with the aim of improving the efficiency of teachers' work.

[0047] System configuration overview

[0048] The system includes the following major components:

[0049] server

[0050] Terminal

[0051] User (teacher)

[0052] Class content data

[0053] Cameras that collect video footage from lessons

[0054] Generative AI Models

[0055] A natural language explanation of the program's processing

[0056] 1. Receiving and preprocessing the teaching material data

[0057] Server: Receives educational material data (e.g., a mathematics textbook for second-year junior high school students) from the educational institution's database. The received data is preprocessed and converted into a format that is easy for the generative AI to process.

[0058] 2. Automatic generation of lesson plans and test questions

[0059] Server: Runs a generative AI model based on the preprocessed data to automatically generate a lesson plan for "Introduction to Linear Equations." The lesson plan includes the lesson progress, materials to be used, practice problems, and supplementary visual materials. The generative AI model also generates test questions, including "basic and applied problems involving linear equations."

[0060] 3. Notification to instructors and corrections

[0061] Device: The generated lesson plans and test questions are sent to the teacher's device, where the teacher can review the plans and questions and make any necessary corrections.

[0062] 4. Recording lessons and collecting data

[0063] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The collected video data is temporarily stored.

[0064] 5. Analysis of video data

[0065] Server: Analyzes collected video data using AI to assess students' comments, attitudes, and progress on assignments. Generates an objective evaluation based on each student's evaluation indicators (level of participation, understanding, cooperation, etc.).

[0066] 6. Sending evaluation results and feedback to instructors

[0067] Terminal: The generated evaluation results are sent to the teacher's terminal. The teacher checks the evaluation results on the terminal and makes corrections as necessary. The corrected evaluation results are saved as the final version.

[0068] Users: Teachers communicate assessment results through feedback to students and engage in individual communication.

[0069] Specific examples

[0070] For example, consider a situation in which a teacher introduces linear equations to second-year junior high school students in a mathematics class.

[0071] 1. Server: Receives the chapters and sections on linear equations from a second-year junior high school mathematics textbook and performs preprocessing.

[0072] 2. Server: Automatically generates lesson plans and test questions based on the preprocessed data and sends them to the teacher's device.

[0073] 3. On the device: The teacher reviews the generated lesson plans and test questions and makes any necessary adjustments.

[0074] 4. Server: During the lesson, the camera collects video footage and stores it on the server.

[0075] 5. Server: The AI ​​analyzes the video data and generates an objective evaluation based on the student's level of participation and understanding.

[0076] 6. Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can check and correct them and provide feedback to the students.

[0077] This system allows teachers to significantly reduce the time they spend preparing lessons and grading, and allows them to spend more time communicating with students. This system simultaneously improves the efficiency of teachers' work and the consistency and objectivity of grading.

[0078] The processing flow will be explained below.

[0079] Step 1:

[0080] Server: Receives educational material data (e.g., a mathematics textbook for eighth-grade students) from the educational institution's database. The received data contains information in various formats, such as text, charts, and images. Preprocessing is performed to unify this information into a consistent format. Specifically, this includes text analysis, data cleansing, and format conversion.

[0081] Step 2:

[0082] Server: The generative AI model is run based on the preprocessed data. The lesson plan generation module is activated to generate a lesson plan for "Introduction to Linear Equations." This lesson plan automatically summarizes the lesson content, materials to be used, practice questions, and supporting visual materials. In addition, the test question generation module is activated to generate test questions including "Basic and applied problems for linear equations."

[0083] Step 3:

[0084] Server: Sends the generated lesson plans and test questions to the educational institution's terminals. The data is categorized and tagged for easy access by teachers.

[0085] Step 4:

[0086] Device: Lesson plans and test questions are sent to the teacher's device. The teacher can review the content and make adjustments as necessary. Specifically, they can fine-tune the lesson content, add materials, increase or decrease the number of practice questions, etc.

[0087] Step 5:

[0088] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The video data is first saved to local storage and then uploaded to a cloud server.

[0089] Step 6:

[0090] Server: A generative AI model runs to analyze the collected video data. The analysis module records student speech using voice recognition and analyzes video to determine student attitudes and progress on assignments. This generates an objective evaluation based on each student's evaluation indicators (level of participation, understanding, cooperation, etc.).

[0091] Step 7:

[0092] Server: The generated evaluation results are sent to the teacher's terminal. The evaluation results are organized for each individual student and presented in a visually easy-to-understand format.

[0093] Step 8:

[0094] Terminal: The teacher reviews the evaluation results and makes corrections as necessary. When reviewing the evaluation results, the teacher makes adjustments based on their own observations to determine whether the evaluations automatically generated by the system are appropriate.

[0095] Step 9:

[0096] Users: Teachers can provide feedback on revised assessment results to students and communicate their grades through individual communication, allowing students to accurately understand their strengths and weaknesses and use this information to guide their next learning steps.

[0097] These are the specific steps in the program. These processes will improve the efficiency of teachers' work and provide appropriate feedback to students.

[0098] Example 1

[0099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] Traditionally, teachers' work required a lot of time for lesson preparation and student evaluation, which could lead to inconsistencies in the quality of education. Furthermore, student evaluations were often subjective and inconsistent. This prevented teachers from concentrating on the essential aspects of education. Furthermore, systems that were inefficient in processing data on lesson content and collecting and analyzing lesson footage were problematic.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0102] In this invention, the server includes: means for receiving and preprocessing lesson content data; means for operating a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data; means for transmitting the generated lesson plans and test questions to a terminal of the educational institution; means for collecting lesson videos using a camera installed in the classroom; means for analyzing the collected video data and automatically generating objective evaluations of students; means for transmitting the generated evaluation results to a terminal of the educational institution; means for teachers to confirm and correct the evaluation results received on the terminal; means for the generative AI model to receive teaching material data and preprocess using a text analysis tool; means for operating the generative AI model using prompt statements; means for converting the preprocessed data into JSON format; and means for analyzing the video data using a machine learning algorithm and generating scores based on each student's evaluation indicators. This improves the efficiency of lesson preparation and evaluation work, allowing teachers to focus on the essence of education and ensuring consistency and objectivity in student evaluations.

[0103] "Class content data" refers to teaching materials, textbooks, lecture notes, slides, and other educational materials used by educational institutions in classes.

[0104] "Preprocessing" refers to the process of structuring the received educational data and converting it into a format that is easy for the generative AI model to process, and includes text analysis and the removal of unnecessary data.

[0105] A "generative AI model" refers to an artificial intelligence model that uses machine learning algorithms to automatically generate lesson plans and test questions.

[0106] A "lesson plan" refers to an educational guideline that includes a rough outline of a lesson, such as the content of the lesson, the materials to be used, exercises, and supporting visual materials.

[0107] "Test Questions" refers to exam questions automatically generated by a generative AI model to assess students' understanding and knowledge.

[0108] "Educational institution devices" refers to electronic devices, such as computers and tablets, used within an educational institution to review or correct lesson plans and test questions.

[0109] "Filming equipment" refers to cameras and video recording devices installed in classrooms to collect footage of lessons.

[0110] "Video data" refers to video information recorded by a camera during class, and is the subject of analysis.

[0111] "Objective evaluation" refers to student evaluation indicators automatically generated by a generative AI model based on students' comments, attitudes, and progress on assignments.

[0112] "Evaluation results" refers to the student's evaluation scores and feedback information generated based on the analysis of video data by the generative AI model.

[0113] "Means for checking and correcting on a device" refers to the ability for teachers to check the lesson plans and evaluation results generated using the educational institution's devices and correct the content as necessary.

[0114] "Text analysis tools" refers to software and algorithms used in the preprocessing of educational data to analyze text data and extract specific information.

[0115] A "prompt sentence" is an input sentence used to give instructions to a generative AI model, and refers to a sentence that clearly indicates the specific content and purpose.

[0116] The "JSON format" is a lightweight data format used for data exchange and storage, and is suitable for describing preprocessed data because it has a structure that is easily readable by both humans and computers.

[0117] "Machine learning algorithms" refer to artificial intelligence techniques that learn patterns based on data and make predictions and classifications.

[0118] This invention relates to a system that automates lesson preparation and assessment tasks by utilizing generative AI, aiming to improve the efficiency of teachers' work. This system receives and preprocesses lesson content data, then automatically generates lesson plans and test questions using a generative AI model, and further analyzes classroom video data to objectively assess students.

[0119] The main components of the system consist of a server, terminals, users (teachers), data on lesson content, cameras that collect lesson footage, and generative AI models.

[0120] The server connects to the educational institution's database and receives data on lesson content. For example, when retrieving data from a junior high school second-year mathematics textbook, it receives the teaching material data from a MySQL database. The received data is preprocessed in XML format, and only the necessary information is extracted using a text analysis tool (e.g., SpaCy). During this process, the data is converted to JSON format, which is easy for the generative AI model to handle.

[0121] Based on the preprocessed data, the server runs a generative AI model (e.g., GPT-4). A prompt (e.g., "Based on a second-year junior high school mathematics textbook, please generate a lesson plan and test questions for the introduction of linear equations") is entered, and a lesson plan and test questions are generated. The generated lesson plan includes the lesson progress, materials to be used, practice questions, and supporting visual materials. At the same time, the generative AI model also generates a variety of test questions, from basic to applied.

[0122] The generated lesson plans and test questions are sent to the teacher's device. The teacher's device is an electronic device such as a laptop or tablet, and the teacher uses a dedicated application (e.g., a teacher support app) to review and revise the plans and questions. Using dedicated functions, the teacher can change the slide layout and revise the questions. The order of the tasks can also be changed using drag and drop.

[0123] At the start of a lesson, a camera installed in the classroom automatically starts up and records the lesson video in real time. The camera is installed in a position that can overlook every corner of the classroom, and a series of video data is streamed in real time to a server. The data is temporarily stored and later divided into specific frame intervals for analysis by a generative AI model.

[0124] The server analyzes the collected video data using a generative AI model (e.g., Viso.ai). From the analysis, a machine learning algorithm is used to determine students' comments, attitudes, and progress on assignments. Facial recognition and facial expression analysis are performed to measure each student's level of understanding and concentration. In addition, text analysis is performed on the number of times a student's hand is raised and the content of their comments, and participation is reflected as part of the evaluation. The analysis results are converted into a score based on each student's evaluation indicators (e.g., participation, understanding, and cooperation).

[0125] The generated evaluation results are sent to the teacher's device. The teacher can then view the evaluation results in a dedicated application on the device and check each student's score and analysis results. If necessary, the teacher can manually correct the evaluation and add comments. This result is saved as the final version and provided to each student as feedback.

[0126] Teachers can plan individual consultations and supplementary lessons based on students' levels of understanding and participation. They can also communicate the results of the assessment to students through a dedicated application and provide specific advice based on the results.

[0127] In this way, the present invention allows teachers to improve the efficiency of lesson preparation and assessment work, and to focus on communication with students. Furthermore, by improving the consistency and objectivity of assessment, students' learning progress can be accurately grasped.

[0128] For example, here is a prompt for a teacher to "introduce linear equations" to an eighth grade math class:

[0129] "Generate a lesson plan and test questions for 'Introduction to Linear Equations' based on an eighth grade mathematics textbook. The lesson plan should include the lesson progression, materials to be used, practice problems, and supporting visual materials. The test questions should include both basic and applied problems."

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Step 1:

[0132] The server receives data on lesson content from the educational institution's database. For example, data on a junior high school second-year mathematics textbook is retrieved from a MySQL database. Based on the received data, the server uses a text analysis tool (e.g., SpaCy) to extract the necessary information and remove unnecessary parts. This preprocessing converts the data into a format (JSON) that is easy for the generative AI to handle.

[0133] Input: Course content data stored in the institution's database

[0134] Output: Preprocessed data in JSON format

[0135] Step 2:

[0136] The server runs a generative AI model (e.g., GPT-4) based on the preprocessed data. It generates a lesson plan and test questions based on a prompt ("Generate a lesson plan and test questions for the introduction of linear equations based on a second-year junior high school mathematics textbook."). The generated results include the lesson plan, materials to be used, practice questions, and supporting visual materials.

[0137] Input: Preprocessed JSON format data, prompt statement

[0138] Output: Auto-generated lesson plans and test questions

[0139] Step 3:

[0140] The server then sends the generated lesson plans and test questions to the educational institution's terminals, which can be electronic devices such as teachers' laptops or tablets. After sending, teachers can use a dedicated application to review these materials and make any necessary corrections. For example, they can change the layout of slides or modify the content of test questions.

[0141] Input: Auto-generated lesson plans and test questions

[0142] Output: Lesson plans and test questions sent to the teacher's device

[0143] Step 4:

[0144] When a lesson starts, the server automatically activates the cameras installed in the classrooms. These cameras record the lesson video in real time and stream it to the server, where the video data is temporarily stored.

[0145] Input: Real-time classroom video

[0146] Output: Video data of lessons stored on the server

[0147] Step 5:

[0148] The server analyzes the collected video data using a generative AI model. Machine learning algorithms (e.g., Viso.ai) are used to analyze the video data and determine student speech, behavior, and task performance. This analysis generates a score based on facial recognition and facial expression analysis to assess student engagement and comprehension.

[0149] Input: Video data of lessons stored on the server

[0150] Output: Student evaluation scores

[0151] Step 6:

[0152] The evaluation results generated by the server are then sent back to the teacher's device. The teacher can then check the results using a dedicated application on the device and manually correct them as necessary. The corrected evaluation results are saved as the final version and provided as feedback to the students.

[0153] Input: Generated student assessment scores

[0154] Output: Evaluation results sent to the teacher's terminal, evaluation results corrected by the teacher

[0155] Step 7:

[0156] The user (teacher) provides feedback on the evaluation results to students through a dedicated application. Based on the evaluation results, the user can plan individual interviews and supplementary lessons and provide specific advice to students.

[0157] Input: Evaluation results corrected by the teacher

[0158] Output: Individual feedback and advice to students

[0159] (Application example 1)

[0160] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0161] In conventional educational systems, teachers expend a great deal of effort preparing lessons and evaluating students. Furthermore, even in online classes, there are few means for evaluating students' understanding and participation in real time and providing feedback. This increases the burden on teachers and leads to issues such as a lack of communication with students. This invention aims to solve these issues and improve the efficiency of teachers' work and the quality of lessons.

[0162] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0163] In this invention, the server includes means for receiving and preprocessing data on lesson content, means for operating a generative AI model for automatically generating lesson plans and test questions based on the preprocessed data, means for transmitting the generated lesson plans and test questions to a terminal of the educational institution, means for collecting lesson video using a camera installed in the classroom, means for analyzing the collected video data and automatically generating objective evaluations of students, means for transmitting the generated evaluation results to a terminal of the educational institution, means for providing feedback to students according to their learning progress using the generative AI model, means for distributing online lesson content based on the automatically generated lesson plans and test questions, and means for recording and evaluating lessons in real time via an application installed on a smartphone or tablet. This reduces the workload of teachers and enables them to grasp students' levels of understanding and participation in real time and provide feedback.

[0164] "Course content data" refers to a set of information related to teaching materials and courses provided by educational institutions.

[0165] "Preprocessing" refers to a series of operations that prepare lesson content data into a format that is easy for the generative AI model to handle.

[0166] "Generative AI models" refer to algorithms or software that use artificial intelligence technology to automatically generate lesson plans and test questions.

[0167] A "lesson plan" is a detailed plan that includes the lesson content, materials to be used, exercises, supporting visual materials, etc.

[0168] "Test questions" refer to questions created by generative AI models to assess comprehension and proficiency of lesson content.

[0169] "Educational institution device" refers to an electronic device, such as a computer or tablet, owned by an educational institution.

[0170] "Camera" refers to video recording equipment installed in the classroom to collect footage during lessons.

[0171] "Video data" refers to video files and related information captured by a camera during class.

[0172] "Objective evaluation" refers to a quantitative evaluation calculated by a generative AI model based on students' comments, attitudes, and progress on assignments.

[0173] "Students" refers to students who participate in classes.

[0174] "Learning progress" refers to the degree to which a participant has progressed in the knowledge and skills they have acquired through classes and learning materials.

[0175] "Feedback" refers to advice and suggestions provided by a generative AI model based on a student's learning progress and level of understanding.

[0176] The present invention is a system for automating lesson preparation and evaluation tasks, and detailed embodiments thereof are described below. The system includes, as its main components, a server, terminals, users (teachers), data on lesson content, a camera for collecting lesson footage, and a generative AI model.

[0177] First, the server receives data on lesson content from the educational institution's database. For example, data from a mathematics textbook for second-year junior high school students. Next, the received data is preprocessed and converted into a format that is easy for the generative AI model to process. This ensures that the teaching material data is properly structured.

[0178] The server runs a generative AI model based on the preprocessed data to automatically generate lesson plans and test questions. For example, let's take a lesson plan for "Introduction to Linear Equations." This lesson plan includes the lesson progress, materials to be used, practice questions, and supporting visual materials. The generative AI model also generates test questions, including "basic and applied problems involving linear equations."

[0179] The generated lesson plans and test questions are sent to the educational institution's terminals, where teachers can review them and make corrections as necessary.

[0180] During lessons, cameras installed in the classroom are turned on and capture video footage of the lesson in real time. The server temporarily stores the collected video data and then analyzes it using a generative AI model. Specifically, it assesses students' comments, attitudes, and progress on assignments, and automatically generates an objective evaluation. Evaluation criteria include participation, understanding, and cooperation.

[0181] The generated evaluation results are sent to the teacher's device. The teacher reviews the results and makes corrections as necessary. The corrected evaluation results are saved as the final version and feedback is provided to the students.

[0182] Additionally, feedback is automatically provided using a generative AI model according to the student's learning progress, allowing appropriate learning support to be provided to each individual student.

[0183] The system also has a function for distributing online lesson content. It automatically generates lesson plans and test questions, enabling effective online lessons for students. It also allows lessons to be recorded and evaluated in real time via an application installed on a smartphone or tablet.

[0184] For example, if you are teaching second-year junior high school maths classes about "Introduction to Linear Equations," you might input the following prompts into the generative AI model:

[0185] Example prompt sentence:

[0186] "Generate lesson plans and test questions for an introduction to linear equations based on eighth grade mathematics materials. Include concrete examples, visual aids, and practice problems in your lesson plans."

[0187] This system will enable teachers to significantly reduce the time they spend preparing lessons and assessing them, allowing them to spend more time communicating with students, resulting in improved quality of education and more efficient teacher work.

[0188] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0189] Step 1:

[0190] The server receives data on lesson content from the educational institution's database. For example, it retrieves data on a mathematics textbook for second-year junior high school students. The input here is the educational institution's database, and the output is the received teaching material data. Specifically, it accesses the database through an API and downloads the necessary teaching material data.

[0191] Step 2:

[0192] The server preprocesses the received teaching material data. Preprocessing involves cleaning and tokenizing the text data, converting it into a format that is easy for the generative AI model to process. The input here is the received teaching material data, and the output is the preprocessed teaching material data. Specifically, it converts unstructured data into an appropriate format and removes unnecessary characters and errors.

[0193] Step 3:

[0194] The server runs a generative AI model based on the preprocessed data to automatically generate lesson plans and test questions. The input is the preprocessed teaching material data, and the output is the generated lesson plans and test questions. Specifically, a natural language processing algorithm is used to generate prompt sentences based on the specified theme.

[0195] Step 4:

[0196] The server sends the generated lesson plans and test questions to the educational institution's terminals. The input here is the generated lesson plans and test questions, and the output is the data displayed on the teacher's terminal. Specifically, the data is securely sent via the network so that the teacher can access it.

[0197] Step 5:

[0198] The camera installed in the classroom is activated and collects the lesson video in real time. The input here is the video of the lesson scene, and the output is the collected video data. Specifically, the camera device is activated remotely and recording begins.

[0199] Step 6:

[0200] The server temporarily stores the collected video data and then analyzes it using a generative AI model. The input here is the collected video data, and the output is the analysis results and an objective evaluation of the student. Specifically, video recognition technology is used to analyze the student's comments and behavior and calculate evaluation indicators.

[0201] Step 7:

[0202] The server sends the generated evaluation results to the educational institution's terminal. The input here is the analysis results and the student's objective evaluation, and the output is the evaluation results displayed on the teacher's terminal. Specifically, the evaluation results are sent via the network so that the teacher can check and correct them.

[0203] Step 8:

[0204] Teachers check the evaluation results through their terminals and make corrections as necessary. The input here is the generated evaluation results, and the output is the final evaluation results after corrections. Specifically, teachers check the evaluation results using the terminal interface and make corrections using a dedicated editing tool.

[0205] Step 9:

[0206] The server uses a generative AI model to provide feedback based on the student's learning progress. The input is the student's learning progress data, and the output is the feedback content. Specifically, it generates individual advice and supplementary learning materials based on test results and participation status.

[0207] Step 10:

[0208] The server distributes online lesson content based on the automatically generated lesson plans and test questions. The input here is the generated lesson plans and test questions, and the output is the content of the online lesson. Specifically, the content of the lesson is provided in a live streaming or video-on-demand format, allowing students to access it.

[0209] Step 11:

[0210] Lessons are recorded in real time and evaluated through an application installed on a smartphone or tablet. The input is real-time lesson footage, and the output is the recorded data and its evaluation results. Specifically, lessons are recorded using the mobile device's camera and analyzed using a generative AI model.

[0211] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0212] This invention relates to a system that aims to improve the efficiency of teachers' work by automating lesson preparation and evaluation tasks by utilizing generative AI and an emotion engine, and further improving the quality of evaluations through student emotion recognition.

[0213] System configuration overview

[0214] The system includes the following major components:

[0215] server

[0216] Terminal

[0217] User (teacher)

[0218] Class content data

[0219] Cameras that collect video footage from lessons

[0220] Generative AI Models

[0221] Emotion Engine

[0222] A natural language explanation of the program's processing

[0223] 1. Receiving and preprocessing the teaching material data

[0224] Server: Receives educational material data (e.g., a mathematics textbook for eighth-grade students) from the educational institution's database. The received data contains information in various formats, such as text, charts, and images. Preprocessing is performed to unify this information into a consistent format. Specifically, this includes text analysis, data cleansing, and format conversion.

[0225] 2. Automatic generation of lesson plans and test questions

[0226] Server: The generative AI model is run based on the preprocessed data. The lesson plan generation module is activated to generate a lesson plan for "Introduction to Linear Equations." This lesson plan automatically summarizes the lesson content, materials to be used, practice questions, and supporting visual materials. In addition, the test question generation module is activated to generate test questions including "Basic and applied problems for linear equations."

[0227] 3. Notification to instructors and corrections

[0228] Device: The generated lesson plans and test questions are sent to the teacher's device, where the teacher can review the plans and questions and make any necessary corrections.

[0229] 4. Recording lessons and collecting data

[0230] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The video data is temporarily saved in local storage and then uploaded to the cloud server.

[0231] 5. Video Data Analysis and Emotion Recognition

[0232] Server: A generative AI model runs to analyze the collected video data. The analysis module uses voice recognition to record what students say and analyzes video to determine their attitudes and progress on assignments. Furthermore, an emotion engine runs, analyzing students' facial expressions and tone of voice to recognize emotions, such as joy, sadness, anger, and surprise.

[0233] 6. Generating evaluation results and feedback

[0234] Server: Generates objective evaluations based on each student's evaluation indicators (e.g., participation, understanding, cooperation) and emotion recognition results. The generated evaluation results are organized for each individual student and presented in a visually easy-to-understand format.

[0235] Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can review them and make corrections as necessary. Because the evaluation also includes emotion recognition information, the teacher can assess the student's mental state while still understanding it.

[0236] Users: Teachers can provide feedback on revised assessment results to students and communicate their grades through individual communication, allowing students to accurately understand their strengths and weaknesses and use this information to guide their next learning steps.

[0237] Specific examples

[0238] For example, consider a situation in which a teacher introduces linear equations to second-year junior high school students in a mathematics class.

[0239] 1. Server: Receives the chapters and sections on linear equations from a second-year junior high school mathematics textbook and performs preprocessing.

[0240] 2. Server: Automatically generates lesson plans and test questions based on the preprocessed data and sends them to the teacher's device.

[0241] 3. Terminal: Teachers review lesson plans and test questions and make adjustments as needed.

[0242] 4. Server: During the lesson, the camera collects video footage and stores it on the server.

[0243] 5. Server: Analyzes video data using generative AI and an emotion engine to generate objective evaluations based on students' emotions, comments, and attitudes.

[0244] 6. Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can check and correct them.

[0245] 7. User: Teachers communicate assessment results through feedback to students and engage in individual communication.

[0246] The above is a specific "Mode for carrying out the invention." This system allows teachers to significantly reduce the time they spend preparing lessons and assessing them, allowing them to spend more time communicating with students. By combining emotion recognition, more comprehensive assessments are possible, improving students' learning experiences.

[0247] The processing flow will be explained below.

[0248] Step 1:

[0249] Server: Receives educational material data (e.g., a mathematics textbook for second-year junior high school students) from the educational institution's database. The received data contains information in a variety of formats, including text, charts, and images. Preprocessing is performed to unify this information into a consistent format. Specifically, text analysis, various format conversions, and data cleansing are performed.

[0250] Step 2:

[0251] Server: The generative AI model is run based on the preprocessed data. The lesson plan generation module is activated to generate a lesson plan for "Introduction to Linear Equations." This lesson plan automatically summarizes the lesson content, materials to be used, practice questions, and supporting visual materials. In addition, the test question generation module is activated to generate test questions including "Basic and applied problems for linear equations."

[0252] Step 3:

[0253] Server: Generated lesson plans and test questions are sent to the educational institution's terminals. The data is categorized and tagged for easy access by teachers.

[0254] Step 4:

[0255] Devices: Lesson plans and test questions are sent to teachers' devices for review. Teachers can review the content and make adjustments as needed. Specifically, they can fine-tune the lesson content, add materials, increase or decrease the number of practice questions, etc.

[0256] Step 5:

[0257] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The video data is saved in local storage and then uploaded to a cloud server.

[0258] Step 6:

[0259] Server: The collected video data is analyzed using a generative AI model and emotion engine. The generative AI model records students' comments using voice recognition and determines their attitudes and progress on assignments through video analysis. The emotion engine analyzes students' facial expressions and tone of voice to recognize their emotions. For example, emotions such as joy, sadness, anger, and surprise can be recognized.

[0260] Step 7:

[0261] Server: Generates objective evaluations based on each student's evaluation indicators (e.g., participation, understanding, cooperation) and emotion recognition results. The generated evaluation results are organized for each individual student and presented in a visually easy-to-understand format.

[0262] Step 8:

[0263] Terminal: The evaluation results are sent to the teacher's terminal. The teacher can review them and make corrections as necessary. Emotion recognition information is also included, so the teacher can evaluate the student while understanding their mental state.

[0264] Step 9:

[0265] User: Teachers provide feedback to students on the revised assessment results. Students are informed of their grades through individual communication, allowing them to accurately understand their strengths and weaknesses. This feedback helps students take the next step in their studies.

[0266] These are the specific processing steps of the program, which will enable teachers to significantly reduce the time required for lesson preparation and assessment, while also enabling comprehensive assessment that takes students' emotions into account.

[0267] Example 2

[0268] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0269] Teachers at educational institutions spend a great deal of time and effort preparing lessons and conducting assessments. It is particularly difficult to objectively assess students' progress and emotions while understanding their learning progress. They also need to provide students with prompt and accurate feedback. Furthermore, if video data from lessons could be used to assess students' learning status, more efficient educational support would be possible. However, current educational systems lack such functionality. Therefore, a new system is needed to improve teacher efficiency and comprehensively assess students.

[0270] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0271] In this invention, the server includes means for receiving and preprocessing lesson content data, means for operating a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data, means for transmitting the generated lesson plans and test questions to an educational institution's terminal, means for reviewing and correcting the generated lesson plans and test questions on the educational institution's terminal, means for collecting lesson video using a video capture device installed in the classroom, means for analyzing the collected video data and automatically generating objective evaluations of students using the generative AI model and an emotion recognition algorithm, means for transmitting the generated evaluation results to the educational institution's terminal and reviewing and correcting the evaluation results on the educational institution's terminal, and means for providing feedback to students from the educational institution's terminal. This reduces teachers' lesson preparation time and enables accurate evaluation of students' learning progress and emotions. It also allows teachers to quickly review and correct the evaluation results and provide quick and accurate feedback to students.

[0272] "Course content data" means information related to a course offered by an educational institution, including textbooks, slides, visual materials, audio files, video files, and other formats.

[0273] "Preprocessing" refers to a series of steps taken to convert received data into a consistent format, including parsing text, removing unnecessary information, and standardizing formats.

[0274] A "generative AI model" is a model that uses artificial intelligence algorithms to automatically generate lesson plans and test questions from basic data. Specifically, machine learning and neural networks are often applied.

[0275] A "lesson plan" is a written plan prepared by a teacher to conduct a lesson, and includes the content of the lesson, teaching materials to be used, practice questions, supplementary visual materials, etc.

[0276] "Test questions" are questions that are asked to assess a student's level of understanding and learning progress, and include multiple choice questions, essay questions, applied questions, etc.

[0277] "Educational institution device" refers to an electronic device, such as a computer, tablet, or smartphone, owned by an educational institution and used by faculty to review, modify, or access lesson plans and test questions.

[0278] "Filming equipment" refers to devices such as cameras and recording devices installed in classrooms to collect footage of lessons.

[0279] "Generative AI model and emotion recognition algorithm" refers to an algorithm that uses AI technology to analyze students' facial expressions and tone of voice to recognize their emotions, which can then be used to evaluate their learning attitudes and emotions.

[0280] "Objective assessment of students" means evaluating students' learning progress, level of understanding, participation, etc. based on data and algorithms, without relying on the teacher's subjective opinion.

[0281] "Feedback" refers to the act of teachers giving students grades and advice based on the results of their assessments, allowing students to understand their own strengths and weaknesses and use this information to guide their next steps in learning.

[0282] MODE FOR CARRYING OUT THE INVENTION

[0283] This invention aims to improve the work efficiency of teachers in educational institutions by automating lesson preparation and evaluation tasks. Furthermore, it aims to improve the quality of evaluations through student emotion recognition. This system mainly consists of three elements: a server, a terminal, and a user (teacher).

[0284] Receiving and preprocessing teaching material data

[0285] Server: Receives course content data from the educational institution's database. The received data contains a variety of formats, including text, charts, and images, and undergoes preprocessing to unify them into a consistent format. Preprocessing includes text analysis, data cleansing, and format conversion.

[0286] Automatic generation of lesson plans and test questions

[0287] Server: Based on the preprocessed data, a generative AI model is run to automatically generate lesson plans and test questions. The generative AI model uses machine learning algorithms, and algorithms optimized for each subject area are implemented. The generated lesson plans include the lesson progress, materials to be used, practice questions, and supporting visual materials. In addition, a test question generation module automatically generates multiple-choice and essay questions.

[0288] Instructor notification and corrections

[0289] Devices: The generated lesson plans and test questions are sent to the educational institution's devices. Teachers can review them on their devices and make corrections as needed. Corrections can be easily made through the device's editing interface.

[0290] Recording lessons and collecting data

[0291] Server: During lessons, the camera installed in the classroom is turned on and captures video of the lesson in real time. The collected video data is temporarily saved in local storage and then uploaded to the cloud server.

[0292] Video data analysis and emotion recognition

[0293] Server: Runs generative AI models and emotion recognition algorithms to analyze the collected video data. The analysis module uses voice recognition technology to record students' speech and video analysis technology to evaluate their attitudes and progress on assignments. It also uses emotion recognition algorithms to analyze students' facial expressions and tone of voice to recognize their emotions.

[0294] Generation of evaluation results and feedback

[0295] Server: Generates objective evaluations based on evaluation indicators such as student participation, understanding, and cooperation, as well as emotion recognition results. The evaluation results are organized in a visually easy-to-understand format.

[0296] Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can review them and make corrections as necessary. Emotion recognition information is also included, so the teacher can evaluate the student while understanding their mental state.

[0297] Users: Teachers can provide feedback on revised assessment results to students and communicate their grades through individual communication, allowing students to accurately identify their strengths and weaknesses and use them to guide their next learning steps.

[0298] Specific examples

[0299] For example, consider a lesson on "Introduction to Linear Equations" in mathematics for second-year junior high school students. First, the server extracts the chapters and sections on linear equations from the second-year junior high school mathematics textbook and performs preprocessing. Next, a generative AI model is run based on the preprocessed data, automatically generating lesson plans and test questions. The generated plans and questions are sent to the teacher's device, who reviews and modifies them. After that, cameras installed during the lesson collect footage of the lesson, and the server analyzes the data. The analysis results are objectively evaluated based on the students' emotions and comments, and the evaluation results are sent to the teacher's device. The teacher reviews and modifies these evaluation results and provides feedback to the students.

[0300] Prompt Sentence Examples

[0301] "Generate a lesson plan for introducing linear equations.

[0302] Lessons should include basic conceptual explanations, examples, exercises, and supporting visual materials.

[0303] The system of this invention can significantly reduce the time required for teachers to prepare and evaluate lessons, and improve the learning experience for students. By combining it with emotion recognition, more comprehensive and objective evaluation can be achieved, supporting the growth of each student.

[0304] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0305] Program processing flow

[0306] Step 1: Receiving and preprocessing the teaching material data

[0307] server:

[0308] Input: Course content data obtained from an educational institution's database (e.g., an XML file of a second-year junior high school mathematics textbook)

[0309] Processing: Parse the data and convert it into a consistent format. This involves text analysis, removal of unnecessary tags, and standardization of formats.

[0310] Output: Preprocessed data (JSON formatted lesson content data)

[0311] Specific behavior:

[0312] The server accesses the educational institution's database and retrieves the data for the specified course content. If the data is in XML format, it first parses it to extract the textbook content into text format. It then removes unnecessary tags and noise and converts it into a consistent JSON format.

[0313] Step 2: Automatically generate lesson plans and test questions

[0314] server:

[0315] Input: Preprocessed lesson content data (JSON format)

[0316] Processing: Run generative AI models to generate lesson plans and test questions, using machine learning algorithms and domain-specific optimized models.

[0317] Output: Auto-generated lesson plans and test questions (JSON or text format)

[0318] Specific behavior:

[0319] The server inputs the preprocessed lesson content data into a generative AI model. Using advanced machine learning algorithms such as the GPT-4 model, it automatically generates a lesson plan for "Introduction to Linear Equations." This plan includes teaching materials, a progress schedule, practice questions, and supplementary materials for teachers to use. A test question generation module also runs simultaneously, generating multiple-choice and essay questions to assess comprehension.

[0320] Step 3: Notify instructor and make corrections

[0321] Device:

[0322] Input: Generated lesson plans and test questions (JSON or text format)

[0323] Processing: The teacher will review these on the system and make corrections as necessary.

[0324] Output: Modified lesson plans and test questions (JSON or text format)

[0325] Specific behavior:

[0326] The generated lesson plans and test questions are sent to the teacher's device, where the teacher can review the content on the device's display screen and make any necessary changes using the editing interface, such as changing the format of a particular question or adjusting the lesson schedule.

[0327] Step 4: Recording lessons and collecting data

[0328] server:

[0329] Input: Video data of lessons collected in real time

[0330] Processing: During lessons, video footage is collected using video recording equipment installed in the classroom. The video data is temporarily stored in local storage and later uploaded to a cloud server.

[0331] Output: Lesson video data stored in cloud storage

[0332] Specific behavior:

[0333] The camera automatically starts recording as soon as the lesson begins, capturing the entire lesson. The recorded video is temporarily saved in local storage. After the lesson ends, the video data is uploaded to cloud storage by the server.

[0334] Step 5: Video data analysis and emotion recognition

[0335] server:

[0336] Input: Video data from lessons stored in cloud storage

[0337] Processing: Using generative AI models and emotion recognition algorithms, the video data is analyzed to assess student speech, attitudes, and emotions.

[0338] Output: Student objective evaluation indicators (evaluation data in JSON format)

[0339] Specific behavior:

[0340] The server retrieves the video data from cloud storage and launches the analysis module. It uses a speech recognition API to transcribe what students say, applies a posture analysis model to evaluate their attitude and progress on assignments, and uses a facial expression analysis API to recognize emotions from students' facial expressions and tone of voice.

[0341] Step 6: Generating evaluation results and feedback

[0342] server:

[0343] Input: Student objective evaluation indicators (evaluation data in JSON format)

[0344] Processing: Generate objective evaluations based on each student's participation, understanding, cooperation, and other evaluation indicators, as well as emotion recognition results. Organize the results in a visually easy-to-understand format.

[0345] Output: Evaluation results (visualized evaluation report)

[0346] Specific behavior:

[0347] The system uses k-means clustering to classify student assessment data and visualize it in a dashboard format. Assessment reports are organized for each student and are created in a visually easy-to-understand format, including supporting materials.

[0348] Device:

[0349] Input: Evaluation results (visualized evaluation report)

[0350] Processing: The teacher reviews the evaluation results and makes corrections if necessary.

[0351] Output: Corrected evaluation results (final evaluation report)

[0352] Specific behavior:

[0353] Teachers can open the evaluation report on their devices, make any necessary corrections, review the results, and add specific comments about students' strengths and weaknesses.

[0354] User:

[0355] Input: Corrected evaluation results (final evaluation report)

[0356] Processing: Teachers provide feedback to students and communicate the results of the assessment.

[0357] Output: Assessment feedback that students can understand

[0358] Specific behavior:

[0359] Teachers meet with students individually to explain the results of their assessments in detail, allowing students to understand their strengths and weaknesses and use the feedback to plan their future learning.

[0360] This system not only significantly reduces the time required for teachers to prepare and evaluate lessons, but also enables more comprehensive and objective evaluation, improving students' learning experiences and supporting the growth of each individual student.

[0361] (Application example 2)

[0362] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0363] In traditional education systems, teachers have to expend a great deal of effort preparing lessons and evaluating students, making efficient lesson management difficult and potentially resulting in a decline in the quality of education. Furthermore, in brick-and-mortar stores, it is difficult for employees to immediately grasp and respond to customer emotions, hindering customer satisfaction. To solve these issues, it is necessary to utilize generative AI models and emotion engines to automate lesson preparation and evaluation, and further improve the quality of customer service in brick-and-mortar stores.

[0364] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0365] In this invention, the server includes means for receiving and preprocessing lesson content data, means for running a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data, means for transmitting the generated lesson plans and test questions to a terminal at the educational institution, means for collecting lesson video from a camera installed in the classroom, means for analyzing the collected video data and automatically generating objective evaluations of students, means for transmitting the generated evaluation results to a terminal at the educational institution, means for application to a customer service robot that recognizes customer emotions in real time and automatically adjusts responses, means for automatically adjusting customer service scripts and generating messages based on the emotion recognition results, and means for accumulating emotion data and utilizing it for future customer service. This enables the efficiency of teachers' work, improvement of the quality of education, and improvement of the quality of customer service in physical stores.

[0366] "Class content data" refers to text data, image data, video data, and other related information that constitute the content of classes provided by educational institutions.

[0367] "Preprocessing" refers to the process of data cleansing and format conversion to analyze received lesson content data and convert it into a consistent format.

[0368] A "generative AI model" is an algorithm and its execution environment that uses artificial intelligence to automatically generate new data based on specified purposes or themes.

[0369] A "lesson plan" is a written plan that summarizes the content of the lesson, the materials to be used, practice questions, supplementary visual materials, etc.

[0370] A "test" is a set of questions or tasks designed to assess a student's understanding or achievement.

[0371] "Educational institution terminals" refer to electronic devices such as computers and tablets used by faculty and administrators affiliated with educational institutions.

[0372] "Camera installed in the classroom" refers to a recording device placed to collect video and audio from the classroom in real time.

[0373] "Video data" refers to video files collected by a camera and any accompanying audio and metadata.

[0374] "Objective evaluation" refers to the results of an objective analysis of a student's behavior, comments, emotions, etc.

[0375] "Means for recognizing customer emotions in real time" refers to technology that uses cameras and voice recognition devices to analyze a customer's facial expressions and tone of voice to instantly determine their emotional state.

[0376] "Application to customer service robots that automatically adjust responses" refers to an applied technology in which the robot instantly changes its customer service scripts and behavior based on the results of emotion recognition, providing the optimal response for each individual customer.

[0377] A "customer service script" refers to a series of lines and action steps that a customer service robot speaks to customers.

[0378] "Emotion data" refers to the inferred emotions obtained through facial expression and voice analysis of customers, as well as related metadata.

[0379] "Storage" is the process of continually collecting certain data and storing it for later use.

[0380] "Methods to utilize this information in future customer service" refers to analyzing accumulated emotional data and providing optimal customer service based on the customer's past reactions.

[0381] This invention is an integrated system for improving the efficiency of teachers' work and the quality of customer service in brick-and-mortar stores. It mainly consists of a server, terminals, and users (teachers or store staff).

[0382] Main components of the system

[0383] 1. Server

[0384] The server performs the following main functions:

[0385] Receiving and preprocessing data for lesson content:

[0386] The server receives data on lesson content from the educational institution's database. The received data includes a variety of formats, such as text data, image data, and video data. To unify this data into a consistent format, preprocessing such as text analysis, data cleansing, and format conversion is performed.

[0387] Automatic generation of lesson plans and test questions:

[0388] Based on the preprocessed data, the generative AI model is run. The lesson plan generation module automatically generates a plan for a specific lesson content, including the lesson progress, materials to be used, practice questions, visual materials, etc. Furthermore, the test question generation module runs and generates test questions corresponding to the lesson content.

[0389] Send generated data:

[0390] The generated lesson plans and test questions are sent to the educational institution's terminals, where teachers can review them and make corrections as needed.

[0391] Video data collection and analysis:

[0392] Cameras installed in classrooms are used to record lessons in progress. The collected video data is first stored in local storage and then uploaded to a cloud server. An analysis module analyzes this video data and automatically evaluates students' behavior and comments. An emotion engine is also used to recognize students' emotions and improve the quality of the evaluation.

[0393] Application in physical stores

[0394] Customer Emotion Recognition:

[0395] Using cameras and a voice recognition system installed in the store, the system analyzes customers' facial expressions and tone of voice in real time to recognize their emotions. The emotion engine then recognizes their emotions, such as joy, sadness, and surprise.

[0396] Automatic adjustment of customer service scripts:

[0397] The robot's script is adjusted in real time based on the results of emotion recognition. For example, if a customer has a troubled expression, the robot will respond with, "Are you looking for something?"

[0398] Accumulating customer data and utilizing it for the next customer visit:

[0399] The collected emotional data is stored on a server, and this data is used in future visits to provide the optimal response based on the customer's past reactions.

[0400] Hardware and Software Used

[0401] Hardware: Camera, robot (Raspberry Pi or NVIDIA Jetson)

[0402] Software: OpenCV, DeepFace, GPT-3 (emotion recognition and sentence generation)

[0403] Specific examples

[0404] For example, if a customer walks into a brick-and-mortar store with a tired expression, the camera and voice recognition system will recognize that emotion, and the customer service robot will say, "We're happy to help you if you need anything."

[0405] Example prompt sentence:

[0406] Input prompt: "Generate a response message when the customer has a confused look on their face."

[0407] Example generated: "Looking for something? Let us help."

[0408] This will enable teachers to improve their work efficiency and the quality of their education, and store staff to increase customer satisfaction.

[0409] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0410] Step 1:

[0411] Receiving and preprocessing lesson content data

[0412] The server receives lesson content data from the educational institution's database. This includes text data, image data, video data, etc. The server performs preprocessing such as text analysis, data cleansing, and format conversion to unify the received data into a consistent format. The input is lesson content data from the educational institution's database, and the output is preprocessed data in a unified format.

[0413] Step 2:

[0414] Automatic generation of lesson plans and test questions

[0415] The server runs a generative AI model based on the preprocessed data. The lesson plan generation module generates lesson plans for specific lesson content, and the test question generation module generates corresponding test questions. The input is the preprocessed data, and the output is the generated lesson plans and test questions. This includes data analysis and model generation.

[0416] Step 3:

[0417] Sending generated data

[0418] The server transmits the generated lesson plans and test questions to the educational institution's terminal. The input is the generated lesson plans and test questions, and the output is the data transmitted to the teacher's terminal. A data transmission protocol is used.

[0419] Step 4:

[0420] Collecting video data from within the classroom

[0421] The server collects video footage from classrooms via cameras installed in the classrooms. This video data is stored in local storage and then uploaded to a cloud server. The input is real-time video footage from classroom cameras, and the output is video data stored on the cloud server. This includes video collection and uploading.

[0422] Step 5:

[0423] Video data analysis and emotion recognition

[0424] The server runs a generative AI model to analyze video data stored on a cloud server. It records student speech using voice recognition, determines student attitudes and progress on assignments using video analysis, and analyzes student emotions (e.g., joy, sadness, anger, surprise) using an emotion engine. The input is the collected video data, and the output is the analysis results and emotion recognition results. It includes data analysis and emotion evaluation.

[0425] Step 6:

[0426] Generating and sending evaluation results

[0427] The server generates an objective evaluation based on each student's evaluation indicators (level of participation, understanding, cooperation, etc.) and the emotion recognition results, and sends it to the educational institution's terminal. The input is the analysis results and emotion recognition results, and the output is the generated evaluation results. The evaluation results are provided in a visually easy-to-understand format.

[0428] Step 7:

[0429] Customer Emotion Recognition

[0430] The device uses cameras and voice recognition systems installed in physical stores to analyze customers' facial expressions and tone of voice in real time, and then activates an emotion engine to determine their emotions. The input is data from the in-store cameras and voice recognition system, and the output is analyzed emotion data. Emotion data analysis is included.

[0431] Step 8:

[0432] Automatic adjustment of customer service scripts

[0433] The terminal adjusts the customer service robot's script in real time based on the emotion recognition results and generates an appropriate message. For example, it responds to a customer with a troubled expression with "Are you looking for something?" The input is the emotion recognition result, and the output is the generated customer service message. This process includes script generation and adjustment.

[0434] Step 9:

[0435] Accumulation of customer data and next use

[0436] The server stores the collected emotion data and uses it for future customer service. The input is each customer's emotion data, and the output is the stored customer data. This includes saving the data and preparing it for the next use.

[0437] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0438] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0439] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0440] [Second embodiment]

[0441] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0442] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0443] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0444] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0445] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0446] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0447] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0448] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0449] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0450] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0451] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0452] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0453] The present invention relates to a system that automates lesson preparation and evaluation tasks by utilizing generative AI, with the aim of improving the efficiency of teachers' work.

[0454] System configuration overview

[0455] The system includes the following major components:

[0456] server

[0457] Terminal

[0458] User (teacher)

[0459] Class content data

[0460] Cameras that collect video footage from lessons

[0461] Generative AI Models

[0462] A natural language explanation of the program's processing

[0463] 1. Receiving and preprocessing the teaching material data

[0464] Server: Receives educational material data (e.g., a mathematics textbook for second-year junior high school students) from the educational institution's database. The received data is preprocessed and converted into a format that is easy for the generative AI to process.

[0465] 2. Automatic generation of lesson plans and test questions

[0466] Server: Runs a generative AI model based on the preprocessed data to automatically generate a lesson plan for "Introduction to Linear Equations." The lesson plan includes the lesson progress, materials to be used, practice problems, and supplementary visual materials. The generative AI model also generates test questions, including "basic and applied problems involving linear equations."

[0467] 3. Notification to instructors and corrections

[0468] Device: The generated lesson plans and test questions are sent to the teacher's device, where the teacher can review the plans and questions and make any necessary corrections.

[0469] 4. Recording lessons and collecting data

[0470] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The collected video data is temporarily stored.

[0471] 5. Analysis of video data

[0472] Server: Analyzes collected video data using AI to assess students' comments, attitudes, and progress on assignments. Generates an objective evaluation based on each student's evaluation indicators (level of participation, understanding, cooperation, etc.).

[0473] 6. Sending evaluation results and feedback to instructors

[0474] Terminal: The generated evaluation results are sent to the teacher's terminal. The teacher checks the evaluation results on the terminal and makes corrections as necessary. The corrected evaluation results are saved as the final version.

[0475] Users: Teachers communicate assessment results through feedback to students and engage in individual communication.

[0476] Specific examples

[0477] For example, consider a situation in which a teacher introduces linear equations to second-year junior high school students in a mathematics class.

[0478] 1. Server: Receives the chapters and sections on linear equations from a second-year junior high school mathematics textbook and performs preprocessing.

[0479] 2. Server: Automatically generates lesson plans and test questions based on the preprocessed data and sends them to the teacher's device.

[0480] 3. On the device: The teacher reviews the generated lesson plans and test questions and makes any necessary adjustments.

[0481] 4. Server: During the lesson, the camera collects video footage and stores it on the server.

[0482] 5. Server: The AI ​​analyzes the video data and generates an objective evaluation based on the student's level of participation and understanding.

[0483] 6. Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can check and correct them and provide feedback to the students.

[0484] This system allows teachers to significantly reduce the time they spend preparing lessons and grading, and allows them to spend more time communicating with students. This system simultaneously improves the efficiency of teachers' work and the consistency and objectivity of grading.

[0485] The processing flow will be explained below.

[0486] Step 1:

[0487] Server: Receives educational material data (e.g., a mathematics textbook for eighth-grade students) from the educational institution's database. The received data contains information in various formats, such as text, charts, and images. Preprocessing is performed to unify this information into a consistent format. Specifically, this includes text analysis, data cleansing, and format conversion.

[0488] Step 2:

[0489] Server: The generative AI model is run based on the preprocessed data. The lesson plan generation module is activated to generate a lesson plan for "Introduction to Linear Equations." This lesson plan automatically summarizes the lesson content, materials to be used, practice questions, and supporting visual materials. In addition, the test question generation module is activated to generate test questions including "Basic and applied problems for linear equations."

[0490] Step 3:

[0491] Server: Sends the generated lesson plans and test questions to the educational institution's terminals. The data is categorized and tagged for easy access by teachers.

[0492] Step 4:

[0493] Device: Lesson plans and test questions are sent to the teacher's device. The teacher can review the content and make adjustments as necessary. Specifically, they can fine-tune the lesson content, add materials, increase or decrease the number of practice questions, etc.

[0494] Step 5:

[0495] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The video data is first saved to local storage and then uploaded to a cloud server.

[0496] Step 6:

[0497] Server: A generative AI model runs to analyze the collected video data. The analysis module records student speech using voice recognition and analyzes video to determine student attitudes and progress on assignments. This generates an objective evaluation based on each student's evaluation indicators (level of participation, understanding, cooperation, etc.).

[0498] Step 7:

[0499] Server: The generated evaluation results are sent to the teacher's terminal. The evaluation results are organized for each individual student and presented in a visually easy-to-understand format.

[0500] Step 8:

[0501] Terminal: The teacher reviews the evaluation results and makes corrections as necessary. When reviewing the evaluation results, the teacher makes adjustments based on their own observations to determine whether the evaluations automatically generated by the system are appropriate.

[0502] Step 9:

[0503] Users: Teachers can provide feedback on revised assessment results to students and communicate their grades through individual communication, allowing students to accurately understand their strengths and weaknesses and use this information to guide their next learning steps.

[0504] These are the specific steps in the program. These processes will improve the efficiency of teachers' work and provide appropriate feedback to students.

[0505] Example 1

[0506] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0507] Traditionally, teachers' work required a lot of time for lesson preparation and student evaluation, which could lead to inconsistencies in the quality of education. Furthermore, student evaluations were often subjective and inconsistent. This prevented teachers from concentrating on the essential aspects of education. Furthermore, systems that were inefficient in processing data on lesson content and collecting and analyzing lesson footage were problematic.

[0508] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0509] In this invention, the server includes: means for receiving and preprocessing lesson content data; means for operating a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data; means for transmitting the generated lesson plans and test questions to a terminal of the educational institution; means for collecting lesson videos using a camera installed in the classroom; means for analyzing the collected video data and automatically generating objective evaluations of students; means for transmitting the generated evaluation results to a terminal of the educational institution; means for teachers to confirm and correct the evaluation results received on the terminal; means for the generative AI model to receive teaching material data and preprocess using a text analysis tool; means for operating the generative AI model using prompt statements; means for converting the preprocessed data into JSON format; and means for analyzing the video data using a machine learning algorithm and generating scores based on each student's evaluation indicators. This improves the efficiency of lesson preparation and evaluation work, allowing teachers to focus on the essence of education and ensuring consistency and objectivity in student evaluations.

[0510] "Class content data" refers to teaching materials, textbooks, lecture notes, slides, and other educational materials used by educational institutions in classes.

[0511] "Preprocessing" refers to the process of structuring the received educational data and converting it into a format that is easy for the generative AI model to process, and includes text analysis and the removal of unnecessary data.

[0512] A "generative AI model" refers to an artificial intelligence model that uses machine learning algorithms to automatically generate lesson plans and test questions.

[0513] A "lesson plan" refers to an educational guideline that includes a rough outline of a lesson, such as the content of the lesson, the materials to be used, exercises, and supporting visual materials.

[0514] "Test Questions" refers to exam questions automatically generated by a generative AI model to assess students' understanding and knowledge.

[0515] "Educational institution devices" refers to electronic devices, such as computers and tablets, used within an educational institution to review or correct lesson plans and test questions.

[0516] "Filming equipment" refers to cameras and video recording devices installed in classrooms to collect footage of lessons.

[0517] "Video data" refers to video information recorded by a camera during class, and is the subject of analysis.

[0518] "Objective evaluation" refers to student evaluation indicators automatically generated by a generative AI model based on students' comments, attitudes, and progress on assignments.

[0519] "Evaluation results" refers to the student's evaluation scores and feedback information generated based on the analysis of video data by the generative AI model.

[0520] "Means for checking and correcting on a device" refers to the ability for teachers to check the lesson plans and evaluation results generated using the educational institution's devices and correct the content as necessary.

[0521] "Text analysis tools" refers to software and algorithms used in the preprocessing of educational data to analyze text data and extract specific information.

[0522] A "prompt sentence" is an input sentence used to give instructions to a generative AI model, and refers to a sentence that clearly indicates the specific content and purpose.

[0523] The "JSON format" is a lightweight data format used for data exchange and storage, and is suitable for describing preprocessed data because it has a structure that is easily readable by both humans and computers.

[0524] "Machine learning algorithms" refer to artificial intelligence techniques that learn patterns based on data and make predictions and classifications.

[0525] This invention relates to a system that automates lesson preparation and assessment tasks by utilizing generative AI, aiming to improve the efficiency of teachers' work. This system receives and preprocesses lesson content data, then automatically generates lesson plans and test questions using a generative AI model, and further analyzes classroom video data to objectively assess students.

[0526] The main components of the system consist of a server, terminals, users (teachers), data on lesson content, cameras that collect lesson footage, and generative AI models.

[0527] The server connects to the educational institution's database and receives data on lesson content. For example, when retrieving data from a junior high school second-year mathematics textbook, it receives the teaching material data from a MySQL database. The received data is preprocessed in XML format, and only the necessary information is extracted using a text analysis tool (e.g., SpaCy). During this process, the data is converted to JSON format, which is easy for the generative AI model to handle.

[0528] Based on the preprocessed data, the server runs a generative AI model (e.g., GPT-4). A prompt (e.g., "Based on a second-year junior high school mathematics textbook, please generate a lesson plan and test questions for the introduction of linear equations") is entered, and a lesson plan and test questions are generated. The generated lesson plan includes the lesson progress, materials to be used, practice questions, and supporting visual materials. At the same time, the generative AI model also generates a variety of test questions, from basic to applied.

[0529] The generated lesson plans and test questions are sent to the teacher's device. The teacher's device is an electronic device such as a laptop or tablet, and the teacher uses a dedicated application (e.g., a teacher support app) to review and revise the plans and questions. Using dedicated functions, the teacher can change the slide layout and revise the questions. The order of the tasks can also be changed using drag and drop.

[0530] At the start of a lesson, a camera installed in the classroom automatically starts up and records the lesson video in real time. The camera is installed in a position that can overlook every corner of the classroom, and a series of video data is streamed in real time to a server. The data is temporarily stored and later divided into specific frame intervals for analysis by a generative AI model.

[0531] The server analyzes the collected video data using a generative AI model (e.g., Viso.ai). From the analysis, a machine learning algorithm is used to determine students' comments, attitudes, and progress on assignments. Facial recognition and facial expression analysis are performed to measure each student's level of understanding and concentration. In addition, text analysis is performed on the number of times a student's hand is raised and the content of their comments, and participation is reflected as part of the evaluation. The analysis results are converted into a score based on each student's evaluation indicators (e.g., participation, understanding, and cooperation).

[0532] The generated evaluation results are sent to the teacher's device. The teacher can then view the evaluation results in a dedicated application on the device and check each student's score and analysis results. If necessary, the teacher can manually correct the evaluation and add comments. This result is saved as the final version and provided to each student as feedback.

[0533] Teachers can plan individual consultations and supplementary lessons based on students' levels of understanding and participation. They can also communicate the results of the assessment to students through a dedicated application and provide specific advice based on the results.

[0534] In this way, the present invention allows teachers to improve the efficiency of lesson preparation and assessment work, and to focus on communication with students. Furthermore, by improving the consistency and objectivity of assessment, students' learning progress can be accurately grasped.

[0535] For example, here is a prompt for a teacher to "introduce linear equations" to an eighth grade math class:

[0536] "Generate a lesson plan and test questions for 'Introduction to Linear Equations' based on an eighth grade mathematics textbook. The lesson plan should include the lesson progression, materials to be used, practice problems, and supporting visual materials. The test questions should include both basic and applied problems."

[0537] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0538] Step 1:

[0539] The server receives data on lesson content from the educational institution's database. For example, data on a junior high school second-year mathematics textbook is retrieved from a MySQL database. Based on the received data, the server uses a text analysis tool (e.g., SpaCy) to extract the necessary information and remove unnecessary parts. This preprocessing converts the data into a format (JSON) that is easy for the generative AI to handle.

[0540] Input: Course content data stored in the institution's database

[0541] Output: Preprocessed data in JSON format

[0542] Step 2:

[0543] The server runs a generative AI model (e.g., GPT-4) based on the preprocessed data. It generates a lesson plan and test questions based on a prompt ("Generate a lesson plan and test questions for the introduction of linear equations based on a second-year junior high school mathematics textbook."). The generated results include the lesson plan, materials to be used, practice questions, and supporting visual materials.

[0544] Input: Preprocessed JSON format data, prompt statement

[0545] Output: Auto-generated lesson plans and test questions

[0546] Step 3:

[0547] The server then sends the generated lesson plans and test questions to the educational institution's terminals, which can be electronic devices such as teachers' laptops or tablets. After sending, teachers can use a dedicated application to review these materials and make any necessary corrections. For example, they can change the layout of slides or modify the content of test questions.

[0548] Input: Auto-generated lesson plans and test questions

[0549] Output: Lesson plans and test questions sent to the teacher's device

[0550] Step 4:

[0551] When a lesson starts, the server automatically activates the cameras installed in the classrooms. These cameras record the lesson video in real time and stream it to the server, where the video data is temporarily stored.

[0552] Input: Real-time classroom video

[0553] Output: Video data of lessons stored on the server

[0554] Step 5:

[0555] The server analyzes the collected video data using a generative AI model. Machine learning algorithms (e.g., Viso.ai) are used to analyze the video data and determine student speech, behavior, and task performance. This analysis generates a score based on facial recognition and facial expression analysis to assess student engagement and comprehension.

[0556] Input: Video data of lessons stored on the server

[0557] Output: Student evaluation scores

[0558] Step 6:

[0559] The evaluation results generated by the server are then sent back to the teacher's device. The teacher can then check the results using a dedicated application on the device and manually correct them as necessary. The corrected evaluation results are saved as the final version and provided as feedback to the students.

[0560] Input: Generated student assessment scores

[0561] Output: Evaluation results sent to the teacher's terminal, evaluation results corrected by the teacher

[0562] Step 7:

[0563] The user (teacher) provides feedback on the evaluation results to students through a dedicated application. Based on the evaluation results, the user can plan individual interviews and supplementary lessons and provide specific advice to students.

[0564] Input: Evaluation results corrected by the teacher

[0565] Output: Individual feedback and advice to students

[0566] (Application example 1)

[0567] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0568] In conventional educational systems, teachers expend a great deal of effort preparing lessons and evaluating students. Furthermore, even in online classes, there are few means for evaluating students' understanding and participation in real time and providing feedback. This increases the burden on teachers and leads to issues such as a lack of communication with students. This invention aims to solve these issues and improve the efficiency of teachers' work and the quality of lessons.

[0569] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0570] In this invention, the server includes means for receiving and preprocessing data on lesson content, means for operating a generative AI model for automatically generating lesson plans and test questions based on the preprocessed data, means for transmitting the generated lesson plans and test questions to a terminal of the educational institution, means for collecting lesson video using a camera installed in the classroom, means for analyzing the collected video data and automatically generating objective evaluations of students, means for transmitting the generated evaluation results to a terminal of the educational institution, means for providing feedback to students according to their learning progress using the generative AI model, means for distributing online lesson content based on the automatically generated lesson plans and test questions, and means for recording and evaluating lessons in real time via an application installed on a smartphone or tablet. This reduces the workload of teachers and enables them to grasp students' levels of understanding and participation in real time and provide feedback.

[0571] "Course content data" refers to a set of information related to teaching materials and courses provided by educational institutions.

[0572] "Preprocessing" refers to a series of operations that prepare lesson content data into a format that is easy for the generative AI model to handle.

[0573] "Generative AI models" refer to algorithms or software that use artificial intelligence technology to automatically generate lesson plans and test questions.

[0574] A "lesson plan" is a detailed plan that includes the lesson content, materials to be used, exercises, supporting visual materials, etc.

[0575] "Test questions" refer to questions created by generative AI models to assess comprehension and proficiency of lesson content.

[0576] "Educational institution device" refers to an electronic device, such as a computer or tablet, owned by an educational institution.

[0577] "Camera" refers to video recording equipment installed in the classroom to collect footage during lessons.

[0578] "Video data" refers to video files and related information captured by a camera during class.

[0579] "Objective evaluation" refers to a quantitative evaluation calculated by a generative AI model based on students' comments, attitudes, and progress on assignments.

[0580] "Students" refers to students who participate in classes.

[0581] "Learning progress" refers to the degree to which a participant has progressed in the knowledge and skills they have acquired through classes and learning materials.

[0582] "Feedback" refers to advice and suggestions provided by a generative AI model based on a student's learning progress and level of understanding.

[0583] The present invention is a system for automating lesson preparation and evaluation tasks, and detailed embodiments thereof are described below. The system includes, as its main components, a server, terminals, users (teachers), data on lesson content, a camera for collecting lesson footage, and a generative AI model.

[0584] First, the server receives data on lesson content from the educational institution's database. For example, data from a mathematics textbook for second-year junior high school students. Next, the received data is preprocessed and converted into a format that is easy for the generative AI model to process. This ensures that the teaching material data is properly structured.

[0585] The server runs a generative AI model based on the preprocessed data to automatically generate lesson plans and test questions. For example, let's take a lesson plan for "Introduction to Linear Equations." This lesson plan includes the lesson progress, materials to be used, practice questions, and supporting visual materials. The generative AI model also generates test questions, including "basic and applied problems involving linear equations."

[0586] The generated lesson plans and test questions are sent to the educational institution's terminals, where teachers can review them and make corrections as necessary.

[0587] During lessons, cameras installed in the classroom are turned on and capture video footage of the lesson in real time. The server temporarily stores the collected video data and then analyzes it using a generative AI model. Specifically, it assesses students' comments, attitudes, and progress on assignments, and automatically generates an objective evaluation. Evaluation criteria include participation, understanding, and cooperation.

[0588] The generated evaluation results are sent to the teacher's device. The teacher reviews the results and makes corrections as necessary. The corrected evaluation results are saved as the final version and feedback is provided to the students.

[0589] Additionally, feedback is automatically provided using a generative AI model according to the student's learning progress, allowing appropriate learning support to be provided to each individual student.

[0590] The system also has a function for distributing online lesson content. It automatically generates lesson plans and test questions, enabling effective online lessons for students. It also allows lessons to be recorded and evaluated in real time via an application installed on a smartphone or tablet.

[0591] For example, if you are teaching second-year junior high school maths classes about "Introduction to Linear Equations," you might input the following prompts into the generative AI model:

[0592] Example prompt sentence:

[0593] "Generate lesson plans and test questions for an introduction to linear equations based on eighth grade mathematics materials. Include concrete examples, visual aids, and practice problems in your lesson plans."

[0594] This system will enable teachers to significantly reduce the time they spend preparing lessons and assessing them, allowing them to spend more time communicating with students, resulting in improved quality of education and more efficient teacher work.

[0595] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0596] Step 1:

[0597] The server receives data on lesson content from the educational institution's database. For example, it retrieves data on a mathematics textbook for second-year junior high school students. The input here is the educational institution's database, and the output is the received teaching material data. Specifically, it accesses the database through an API and downloads the necessary teaching material data.

[0598] Step 2:

[0599] The server preprocesses the received teaching material data. Preprocessing involves cleaning and tokenizing the text data, converting it into a format that is easy for the generative AI model to process. The input here is the received teaching material data, and the output is the preprocessed teaching material data. Specifically, it converts unstructured data into an appropriate format and removes unnecessary characters and errors.

[0600] Step 3:

[0601] The server runs a generative AI model based on the preprocessed data to automatically generate lesson plans and test questions. The input is the preprocessed teaching material data, and the output is the generated lesson plans and test questions. Specifically, a natural language processing algorithm is used to generate prompt sentences based on the specified theme.

[0602] Step 4:

[0603] The server sends the generated lesson plans and test questions to the educational institution's terminals. The input here is the generated lesson plans and test questions, and the output is the data displayed on the teacher's terminal. Specifically, the data is securely sent via the network so that the teacher can access it.

[0604] Step 5:

[0605] The camera installed in the classroom is activated and collects the lesson video in real time. The input here is the video of the lesson scene, and the output is the collected video data. Specifically, the camera device is activated remotely and recording begins.

[0606] Step 6:

[0607] The server temporarily stores the collected video data and then analyzes it using a generative AI model. The input here is the collected video data, and the output is the analysis results and an objective evaluation of the student. Specifically, video recognition technology is used to analyze the student's comments and behavior and calculate evaluation indicators.

[0608] Step 7:

[0609] The server sends the generated evaluation results to the educational institution's terminal. The input here is the analysis results and the student's objective evaluation, and the output is the evaluation results displayed on the teacher's terminal. Specifically, the evaluation results are sent via the network so that the teacher can check and correct them.

[0610] Step 8:

[0611] Teachers check the evaluation results through their terminals and make corrections as necessary. The input here is the generated evaluation results, and the output is the final evaluation results after corrections. Specifically, teachers check the evaluation results using the terminal interface and make corrections using a dedicated editing tool.

[0612] Step 9:

[0613] The server uses a generative AI model to provide feedback based on the student's learning progress. The input is the student's learning progress data, and the output is the feedback content. Specifically, it generates individual advice and supplementary learning materials based on test results and participation status.

[0614] Step 10:

[0615] The server distributes online lesson content based on the automatically generated lesson plans and test questions. The input here is the generated lesson plans and test questions, and the output is the content of the online lesson. Specifically, the content of the lesson is provided in a live streaming or video-on-demand format, allowing students to access it.

[0616] Step 11:

[0617] Lessons are recorded in real time and evaluated through an application installed on a smartphone or tablet. The input is real-time lesson footage, and the output is the recorded data and its evaluation results. Specifically, lessons are recorded using the mobile device's camera and analyzed using a generative AI model.

[0618] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0619] This invention relates to a system that aims to improve the efficiency of teachers' work by automating lesson preparation and evaluation tasks by utilizing generative AI and an emotion engine, and further improving the quality of evaluations through student emotion recognition.

[0620] System configuration overview

[0621] The system includes the following major components:

[0622] server

[0623] Terminal

[0624] User (teacher)

[0625] Class content data

[0626] Cameras that collect video footage from lessons

[0627] Generative AI Models

[0628] Emotion Engine

[0629] A natural language explanation of the program's processing

[0630] 1. Receiving and preprocessing the teaching material data

[0631] Server: Receives educational material data (e.g., a mathematics textbook for eighth-grade students) from the educational institution's database. The received data contains information in various formats, such as text, charts, and images. Preprocessing is performed to unify this information into a consistent format. Specifically, this includes text analysis, data cleansing, and format conversion.

[0632] 2. Automatic generation of lesson plans and test questions

[0633] Server: The generative AI model is run based on the preprocessed data. The lesson plan generation module is activated to generate a lesson plan for "Introduction to Linear Equations." This lesson plan automatically summarizes the lesson content, materials to be used, practice questions, and supporting visual materials. In addition, the test question generation module is activated to generate test questions including "Basic and applied problems for linear equations."

[0634] 3. Notification to instructors and corrections

[0635] Device: The generated lesson plans and test questions are sent to the teacher's device, where the teacher can review the plans and questions and make any necessary corrections.

[0636] 4. Recording lessons and collecting data

[0637] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The video data is temporarily saved in local storage and then uploaded to the cloud server.

[0638] 5. Video Data Analysis and Emotion Recognition

[0639] Server: A generative AI model runs to analyze the collected video data. The analysis module uses voice recognition to record what students say and analyzes video to determine their attitudes and progress on assignments. Furthermore, an emotion engine runs, analyzing students' facial expressions and tone of voice to recognize emotions, such as joy, sadness, anger, and surprise.

[0640] 6. Generating evaluation results and feedback

[0641] Server: Generates objective evaluations based on each student's evaluation indicators (e.g., participation, understanding, cooperation) and emotion recognition results. The generated evaluation results are organized for each individual student and presented in a visually easy-to-understand format.

[0642] Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can review them and make corrections as necessary. Because the evaluation also includes emotion recognition information, the teacher can assess the student's mental state while still understanding it.

[0643] Users: Teachers can provide feedback on revised assessment results to students and communicate their grades through individual communication, allowing students to accurately understand their strengths and weaknesses and use this information to guide their next learning steps.

[0644] Specific examples

[0645] For example, consider a situation in which a teacher introduces linear equations to second-year junior high school students in a mathematics class.

[0646] 1. Server: Receives the chapters and sections on linear equations from a second-year junior high school mathematics textbook and performs preprocessing.

[0647] 2. Server: Automatically generates lesson plans and test questions based on the preprocessed data and sends them to the teacher's device.

[0648] 3. Terminal: Teachers review lesson plans and test questions and make adjustments as needed.

[0649] 4. Server: During the lesson, the camera collects video footage and stores it on the server.

[0650] 5. Server: Analyzes video data using generative AI and an emotion engine to generate objective evaluations based on students' emotions, comments, and attitudes.

[0651] 6. Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can check and correct them.

[0652] 7. User: Teachers communicate assessment results through feedback to students and engage in individual communication.

[0653] The above is a specific "Mode for carrying out the invention." This system allows teachers to significantly reduce the time they spend preparing lessons and assessing them, allowing them to spend more time communicating with students. By combining emotion recognition, more comprehensive assessments are possible, improving students' learning experiences.

[0654] The processing flow will be explained below.

[0655] Step 1:

[0656] Server: Receives educational material data (e.g., a mathematics textbook for second-year junior high school students) from the educational institution's database. The received data contains information in a variety of formats, including text, charts, and images. Preprocessing is performed to unify this information into a consistent format. Specifically, text analysis, various format conversions, and data cleansing are performed.

[0657] Step 2:

[0658] Server: The generative AI model is run based on the preprocessed data. The lesson plan generation module is activated to generate a lesson plan for "Introduction to Linear Equations." This lesson plan automatically summarizes the lesson content, materials to be used, practice questions, and supporting visual materials. In addition, the test question generation module is activated to generate test questions including "Basic and applied problems for linear equations."

[0659] Step 3:

[0660] Server: Generated lesson plans and test questions are sent to the educational institution's terminals. The data is categorized and tagged for easy access by teachers.

[0661] Step 4:

[0662] Devices: Lesson plans and test questions are sent to teachers' devices for review. Teachers can review the content and make adjustments as needed. Specifically, they can fine-tune the lesson content, add materials, increase or decrease the number of practice questions, etc.

[0663] Step 5:

[0664] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The video data is saved in local storage and then uploaded to a cloud server.

[0665] Step 6:

[0666] Server: The collected video data is analyzed using a generative AI model and emotion engine. The generative AI model records students' comments using voice recognition and determines their attitudes and progress on assignments through video analysis. The emotion engine analyzes students' facial expressions and tone of voice to recognize their emotions. For example, emotions such as joy, sadness, anger, and surprise can be recognized.

[0667] Step 7:

[0668] Server: Generates objective evaluations based on each student's evaluation indicators (e.g., participation, understanding, cooperation) and emotion recognition results. The generated evaluation results are organized for each individual student and presented in a visually easy-to-understand format.

[0669] Step 8:

[0670] Terminal: The evaluation results are sent to the teacher's terminal. The teacher can review them and make corrections as necessary. Emotion recognition information is also included, so the teacher can evaluate the student while understanding their mental state.

[0671] Step 9:

[0672] User: Teachers provide feedback to students on the revised assessment results. Students are informed of their grades through individual communication, allowing them to accurately understand their strengths and weaknesses. This feedback helps students take the next step in their studies.

[0673] These are the specific processing steps of the program, which will enable teachers to significantly reduce the time required for lesson preparation and assessment, while also enabling comprehensive assessment that takes students' emotions into account.

[0674] Example 2

[0675] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0676] Teachers at educational institutions spend a great deal of time and effort preparing lessons and conducting assessments. It is particularly difficult to objectively assess students' progress and emotions while understanding their learning progress. They also need to provide students with prompt and accurate feedback. Furthermore, if video data from lessons could be used to assess students' learning status, more efficient educational support would be possible. However, current educational systems lack such functionality. Therefore, a new system is needed to improve teacher efficiency and comprehensively assess students.

[0677] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0678] In this invention, the server includes means for receiving and preprocessing lesson content data, means for operating a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data, means for transmitting the generated lesson plans and test questions to an educational institution's terminal, means for reviewing and correcting the generated lesson plans and test questions on the educational institution's terminal, means for collecting lesson video using a video capture device installed in the classroom, means for analyzing the collected video data and automatically generating objective evaluations of students using the generative AI model and an emotion recognition algorithm, means for transmitting the generated evaluation results to the educational institution's terminal and reviewing and correcting the evaluation results on the educational institution's terminal, and means for providing feedback to students from the educational institution's terminal. This reduces teachers' lesson preparation time and enables accurate evaluation of students' learning progress and emotions. It also allows teachers to quickly review and correct the evaluation results and provide quick and accurate feedback to students.

[0679] "Course content data" means information related to a course offered by an educational institution, including textbooks, slides, visual materials, audio files, video files, and other formats.

[0680] "Preprocessing" refers to a series of steps taken to convert received data into a consistent format, including parsing text, removing unnecessary information, and standardizing formats.

[0681] A "generative AI model" is a model that uses artificial intelligence algorithms to automatically generate lesson plans and test questions from basic data. Specifically, machine learning and neural networks are often applied.

[0682] A "lesson plan" is a written plan prepared by a teacher to conduct a lesson, and includes the content of the lesson, teaching materials to be used, practice questions, supplementary visual materials, etc.

[0683] "Test questions" are questions that are asked to assess a student's level of understanding and learning progress, and include multiple choice questions, essay questions, applied questions, etc.

[0684] "Educational institution device" refers to an electronic device, such as a computer, tablet, or smartphone, owned by an educational institution and used by faculty to review, modify, or access lesson plans and test questions.

[0685] "Filming equipment" refers to devices such as cameras and recording devices installed in classrooms to collect footage of lessons.

[0686] "Generative AI model and emotion recognition algorithm" refers to an algorithm that uses AI technology to analyze students' facial expressions and tone of voice to recognize their emotions, which can then be used to evaluate their learning attitudes and emotions.

[0687] "Objective assessment of students" means evaluating students' learning progress, level of understanding, participation, etc. based on data and algorithms, without relying on the teacher's subjective opinion.

[0688] "Feedback" refers to the act of teachers giving students grades and advice based on the results of their assessments, allowing students to understand their own strengths and weaknesses and use this information to guide their next steps in learning.

[0689] MODE FOR CARRYING OUT THE INVENTION

[0690] This invention aims to improve the work efficiency of teachers in educational institutions by automating lesson preparation and evaluation tasks. Furthermore, it aims to improve the quality of evaluations through student emotion recognition. This system mainly consists of three elements: a server, a terminal, and a user (teacher).

[0691] Receiving and preprocessing teaching material data

[0692] Server: Receives course content data from the educational institution's database. The received data contains a variety of formats, including text, charts, and images, and undergoes preprocessing to unify them into a consistent format. Preprocessing includes text analysis, data cleansing, and format conversion.

[0693] Automatic generation of lesson plans and test questions

[0694] Server: Based on the preprocessed data, a generative AI model is run to automatically generate lesson plans and test questions. The generative AI model uses machine learning algorithms, and algorithms optimized for each subject area are implemented. The generated lesson plans include the lesson progress, materials to be used, practice questions, and supporting visual materials. In addition, a test question generation module automatically generates multiple-choice and essay questions.

[0695] Instructor notification and corrections

[0696] Devices: The generated lesson plans and test questions are sent to the educational institution's devices. Teachers can review them on their devices and make corrections as needed. Corrections can be easily made through the device's editing interface.

[0697] Recording lessons and collecting data

[0698] Server: During lessons, the camera installed in the classroom is turned on and captures video of the lesson in real time. The collected video data is temporarily saved in local storage and then uploaded to the cloud server.

[0699] Video data analysis and emotion recognition

[0700] Server: Runs generative AI models and emotion recognition algorithms to analyze the collected video data. The analysis module uses voice recognition technology to record students' speech and video analysis technology to evaluate their attitudes and progress on assignments. It also uses emotion recognition algorithms to analyze students' facial expressions and tone of voice to recognize their emotions.

[0701] Generation of evaluation results and feedback

[0702] Server: Generates objective evaluations based on evaluation indicators such as student participation, understanding, and cooperation, as well as emotion recognition results. The evaluation results are organized in a visually easy-to-understand format.

[0703] Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can review them and make corrections as necessary. Emotion recognition information is also included, so the teacher can evaluate the student while understanding their mental state.

[0704] Users: Teachers can provide feedback on revised assessment results to students and communicate their grades through individual communication, allowing students to accurately identify their strengths and weaknesses and use them to guide their next learning steps.

[0705] Specific examples

[0706] For example, consider a lesson on "Introduction to Linear Equations" in mathematics for second-year junior high school students. First, the server extracts the chapters and sections on linear equations from the second-year junior high school mathematics textbook and performs preprocessing. Next, a generative AI model is run based on the preprocessed data, automatically generating lesson plans and test questions. The generated plans and questions are sent to the teacher's device, who reviews and modifies them. After that, cameras installed during the lesson collect footage of the lesson, and the server analyzes the data. The analysis results are objectively evaluated based on the students' emotions and comments, and the evaluation results are sent to the teacher's device. The teacher reviews and modifies these evaluation results and provides feedback to the students.

[0707] Prompt Sentence Examples

[0708] "Generate a lesson plan for introducing linear equations.

[0709] Lessons should include basic conceptual explanations, examples, exercises, and supporting visual materials.

[0710] The system of this invention can significantly reduce the time required for teachers to prepare and evaluate lessons, and improve the learning experience for students. By combining it with emotion recognition, more comprehensive and objective evaluation can be achieved, supporting the growth of each student.

[0711] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0712] Program processing flow

[0713] Step 1: Receiving and preprocessing the teaching material data

[0714] server:

[0715] Input: Course content data obtained from an educational institution's database (e.g., an XML file of a second-year junior high school mathematics textbook)

[0716] Processing: Parse the data and convert it into a consistent format. This involves text analysis, removal of unnecessary tags, and standardization of formats.

[0717] Output: Preprocessed data (JSON formatted lesson content data)

[0718] Specific behavior:

[0719] The server accesses the educational institution's database and retrieves the data for the specified course content. If the data is in XML format, it first parses it to extract the textbook content into text format. It then removes unnecessary tags and noise and converts it into a consistent JSON format.

[0720] Step 2: Automatically generate lesson plans and test questions

[0721] server:

[0722] Input: Preprocessed lesson content data (JSON format)

[0723] Processing: Run generative AI models to generate lesson plans and test questions, using machine learning algorithms and domain-specific optimized models.

[0724] Output: Auto-generated lesson plans and test questions (JSON or text format)

[0725] Specific behavior:

[0726] The server inputs the preprocessed lesson content data into a generative AI model. Using advanced machine learning algorithms such as the GPT-4 model, it automatically generates a lesson plan for "Introduction to Linear Equations." This plan includes teaching materials, a progress schedule, practice questions, and supplementary materials for teachers to use. A test question generation module also runs simultaneously, generating multiple-choice and essay questions to assess comprehension.

[0727] Step 3: Notify instructor and make corrections

[0728] Device:

[0729] Input: Generated lesson plans and test questions (JSON or text format)

[0730] Processing: The teacher will review these on the system and make corrections as necessary.

[0731] Output: Modified lesson plans and test questions (JSON or text format)

[0732] Specific behavior:

[0733] The generated lesson plans and test questions are sent to the teacher's device, where the teacher can review the content on the device's display screen and make any necessary changes using the editing interface, such as changing the format of a particular question or adjusting the lesson schedule.

[0734] Step 4: Recording lessons and collecting data

[0735] server:

[0736] Input: Video data of lessons collected in real time

[0737] Processing: During lessons, video footage is collected using video recording equipment installed in the classroom. The video data is temporarily stored in local storage and later uploaded to a cloud server.

[0738] Output: Lesson video data stored in cloud storage

[0739] Specific behavior:

[0740] The camera automatically starts recording as soon as the lesson begins, capturing the entire lesson. The recorded video is temporarily saved in local storage. After the lesson ends, the video data is uploaded to cloud storage by the server.

[0741] Step 5: Video data analysis and emotion recognition

[0742] server:

[0743] Input: Video data from lessons stored in cloud storage

[0744] Processing: Using generative AI models and emotion recognition algorithms, the video data is analyzed to assess student speech, attitudes, and emotions.

[0745] Output: Student objective evaluation indicators (evaluation data in JSON format)

[0746] Specific behavior:

[0747] The server retrieves the video data from cloud storage and launches the analysis module. It uses a speech recognition API to transcribe what students say, applies a posture analysis model to evaluate their attitude and progress on assignments, and uses a facial expression analysis API to recognize emotions from students' facial expressions and tone of voice.

[0748] Step 6: Generating evaluation results and feedback

[0749] server:

[0750] Input: Student objective evaluation indicators (evaluation data in JSON format)

[0751] Processing: Generate objective evaluations based on each student's participation, understanding, cooperation, and other evaluation indicators, as well as emotion recognition results. Organize the results in a visually easy-to-understand format.

[0752] Output: Evaluation results (visualized evaluation report)

[0753] Specific behavior:

[0754] The system uses k-means clustering to classify student assessment data and visualize it in a dashboard format. Assessment reports are organized for each student and are created in a visually easy-to-understand format, including supporting materials.

[0755] Device:

[0756] Input: Evaluation results (visualized evaluation report)

[0757] Processing: The teacher reviews the evaluation results and makes corrections if necessary.

[0758] Output: Corrected evaluation results (final evaluation report)

[0759] Specific behavior:

[0760] Teachers can open the evaluation report on their devices, make any necessary corrections, review the results, and add specific comments about students' strengths and weaknesses.

[0761] User:

[0762] Input: Corrected evaluation results (final evaluation report)

[0763] Processing: Teachers provide feedback to students and communicate the results of the assessment.

[0764] Output: Assessment feedback that students can understand

[0765] Specific behavior:

[0766] Teachers meet with students individually to explain the results of their assessments in detail, allowing students to understand their strengths and weaknesses and use the feedback to plan their future learning.

[0767] This system not only significantly reduces the time required for teachers to prepare and evaluate lessons, but also enables more comprehensive and objective evaluation, improving students' learning experiences and supporting the growth of each individual student.

[0768] (Application example 2)

[0769] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0770] In traditional education systems, teachers have to expend a great deal of effort preparing lessons and evaluating students, making efficient lesson management difficult and potentially resulting in a decline in the quality of education. Furthermore, in brick-and-mortar stores, it is difficult for employees to immediately grasp and respond to customer emotions, hindering customer satisfaction. To solve these issues, it is necessary to utilize generative AI models and emotion engines to automate lesson preparation and evaluation, and further improve the quality of customer service in brick-and-mortar stores.

[0771] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0772] In this invention, the server includes means for receiving and preprocessing lesson content data, means for running a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data, means for transmitting the generated lesson plans and test questions to a terminal at the educational institution, means for collecting lesson video from a camera installed in the classroom, means for analyzing the collected video data and automatically generating objective evaluations of students, means for transmitting the generated evaluation results to a terminal at the educational institution, means for application to a customer service robot that recognizes customer emotions in real time and automatically adjusts responses, means for automatically adjusting customer service scripts and generating messages based on the emotion recognition results, and means for accumulating emotion data and utilizing it for future customer service. This enables the efficiency of teachers' work, improvement of the quality of education, and improvement of the quality of customer service in physical stores.

[0773] "Class content data" refers to text data, image data, video data, and other related information that constitute the content of classes provided by educational institutions.

[0774] "Preprocessing" refers to the process of data cleansing and format conversion to analyze received lesson content data and convert it into a consistent format.

[0775] A "generative AI model" is an algorithm and its execution environment that uses artificial intelligence to automatically generate new data based on specified purposes or themes.

[0776] A "lesson plan" is a written plan that summarizes the content of the lesson, the materials to be used, practice questions, supplementary visual materials, etc.

[0777] A "test" is a set of questions or tasks designed to assess a student's understanding or achievement.

[0778] "Educational institution terminals" refer to electronic devices such as computers and tablets used by faculty and administrators affiliated with educational institutions.

[0779] "Camera installed in the classroom" refers to a recording device placed to collect video and audio from the classroom in real time.

[0780] "Video data" refers to video files collected by a camera and any accompanying audio and metadata.

[0781] "Objective evaluation" refers to the results of an objective analysis of a student's behavior, comments, emotions, etc.

[0782] "Means for recognizing customer emotions in real time" refers to technology that uses cameras and voice recognition devices to analyze a customer's facial expressions and tone of voice to instantly determine their emotional state.

[0783] "Application to customer service robots that automatically adjust responses" refers to an applied technology in which the robot instantly changes its customer service scripts and behavior based on the results of emotion recognition, providing the optimal response for each individual customer.

[0784] A "customer service script" refers to a series of lines and action steps that a customer service robot speaks to customers.

[0785] "Emotion data" refers to the inferred emotions obtained through facial expression and voice analysis of customers, as well as related metadata.

[0786] "Storage" is the process of continually collecting certain data and storing it for later use.

[0787] "Methods to utilize this information in future customer service" refers to analyzing accumulated emotional data and providing optimal customer service based on the customer's past reactions.

[0788] This invention is an integrated system for improving the efficiency of teachers' work and the quality of customer service in brick-and-mortar stores. It mainly consists of a server, terminals, and users (teachers or store staff).

[0789] Main components of the system

[0790] 1. Server

[0791] The server performs the following main functions:

[0792] Receiving and preprocessing data for lesson content:

[0793] The server receives data on lesson content from the educational institution's database. The received data includes a variety of formats, such as text data, image data, and video data. To unify this data into a consistent format, preprocessing such as text analysis, data cleansing, and format conversion is performed.

[0794] Automatic generation of lesson plans and test questions:

[0795] Based on the preprocessed data, the generative AI model is run. The lesson plan generation module automatically generates a plan for a specific lesson content, including the lesson progress, materials to be used, practice questions, visual materials, etc. Furthermore, the test question generation module runs and generates test questions corresponding to the lesson content.

[0796] Send generated data:

[0797] The generated lesson plans and test questions are sent to the educational institution's terminals, where teachers can review them and make corrections as needed.

[0798] Video data collection and analysis:

[0799] Cameras installed in classrooms are used to record lessons in progress. The collected video data is first stored in local storage and then uploaded to a cloud server. An analysis module analyzes this video data and automatically evaluates students' behavior and comments. An emotion engine is also used to recognize students' emotions and improve the quality of the evaluation.

[0800] Application in physical stores

[0801] Customer Emotion Recognition:

[0802] Using cameras and a voice recognition system installed in the store, the system analyzes customers' facial expressions and tone of voice in real time to recognize their emotions. The emotion engine then recognizes their emotions, such as joy, sadness, and surprise.

[0803] Automatic adjustment of customer service scripts:

[0804] The robot's script is adjusted in real time based on the results of emotion recognition. For example, if a customer has a troubled expression, the robot will respond with, "Are you looking for something?"

[0805] Accumulating customer data and utilizing it for the next customer visit:

[0806] The collected emotional data is stored on a server, and this data is used in future visits to provide the optimal response based on the customer's past reactions.

[0807] Hardware and Software Used

[0808] Hardware: Camera, robot (Raspberry Pi or NVIDIA Jetson)

[0809] Software: OpenCV, DeepFace, GPT-3 (emotion recognition and sentence generation)

[0810] Specific examples

[0811] For example, if a customer walks into a brick-and-mortar store with a tired expression, the camera and voice recognition system will recognize that emotion, and the customer service robot will say, "We're happy to help you if you need anything."

[0812] Example prompt sentence:

[0813] Input prompt: "Generate a response message when the customer has a confused look on their face."

[0814] Example generated: "Looking for something? Let us help."

[0815] This will enable teachers to improve their work efficiency and the quality of their education, and store staff to increase customer satisfaction.

[0816] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0817] Step 1:

[0818] Receiving and preprocessing lesson content data

[0819] The server receives lesson content data from the educational institution's database. This includes text data, image data, video data, etc. The server performs preprocessing such as text analysis, data cleansing, and format conversion to unify the received data into a consistent format. The input is lesson content data from the educational institution's database, and the output is preprocessed data in a unified format.

[0820] Step 2:

[0821] Automatic generation of lesson plans and test questions

[0822] The server runs a generative AI model based on the preprocessed data. The lesson plan generation module generates lesson plans for specific lesson content, and the test question generation module generates corresponding test questions. The input is the preprocessed data, and the output is the generated lesson plans and test questions. This includes data analysis and model generation.

[0823] Step 3:

[0824] Sending generated data

[0825] The server transmits the generated lesson plans and test questions to the educational institution's terminal. The input is the generated lesson plans and test questions, and the output is the data transmitted to the teacher's terminal. A data transmission protocol is used.

[0826] Step 4:

[0827] Collecting video data from within the classroom

[0828] The server collects video footage from classrooms via cameras installed in the classrooms. This video data is stored in local storage and then uploaded to a cloud server. The input is real-time video footage from classroom cameras, and the output is video data stored on the cloud server. This includes video collection and uploading.

[0829] Step 5:

[0830] Video data analysis and emotion recognition

[0831] The server runs a generative AI model to analyze video data stored on a cloud server. It records student speech using voice recognition, determines student attitudes and progress on assignments using video analysis, and analyzes student emotions (e.g., joy, sadness, anger, surprise) using an emotion engine. The input is the collected video data, and the output is the analysis results and emotion recognition results. It includes data analysis and emotion evaluation.

[0832] Step 6:

[0833] Generating and sending evaluation results

[0834] The server generates an objective evaluation based on each student's evaluation indicators (level of participation, understanding, cooperation, etc.) and the emotion recognition results, and sends it to the educational institution's terminal. The input is the analysis results and emotion recognition results, and the output is the generated evaluation results. The evaluation results are provided in a visually easy-to-understand format.

[0835] Step 7:

[0836] Customer Emotion Recognition

[0837] The device uses cameras and voice recognition systems installed in physical stores to analyze customers' facial expressions and tone of voice in real time, and then activates an emotion engine to determine their emotions. The input is data from the in-store cameras and voice recognition system, and the output is analyzed emotion data. Emotion data analysis is included.

[0838] Step 8:

[0839] Automatic adjustment of customer service scripts

[0840] The terminal adjusts the customer service robot's script in real time based on the emotion recognition results and generates an appropriate message. For example, it responds to a customer with a troubled expression with "Are you looking for something?" The input is the emotion recognition result, and the output is the generated customer service message. This process includes script generation and adjustment.

[0841] Step 9:

[0842] Accumulation of customer data and next use

[0843] The server stores the collected emotion data and uses it for future customer service. The input is each customer's emotion data, and the output is the stored customer data. This includes saving the data and preparing it for the next use.

[0844] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0845] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0846] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0847] [Third embodiment]

[0848] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0849] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0850] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0851] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0852] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0853] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0854] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0855] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0856] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0857] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0858] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0859] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0860] The present invention relates to a system that automates lesson preparation and evaluation tasks by utilizing generative AI, with the aim of improving the efficiency of teachers' work.

[0861] System configuration overview

[0862] The system includes the following major components:

[0863] server

[0864] Terminal

[0865] User (teacher)

[0866] Class content data

[0867] Cameras that collect video footage from lessons

[0868] Generative AI Models

[0869] A natural language explanation of the program's processing

[0870] 1. Receiving and preprocessing the teaching material data

[0871] Server: Receives educational material data (e.g., a mathematics textbook for second-year junior high school students) from the educational institution's database. The received data is preprocessed and converted into a format that is easy for the generative AI to process.

[0872] 2. Automatic generation of lesson plans and test questions

[0873] Server: Runs a generative AI model based on the preprocessed data to automatically generate a lesson plan for "Introduction to Linear Equations." The lesson plan includes the lesson progress, materials to be used, practice problems, and supplementary visual materials. The generative AI model also generates test questions, including "basic and applied problems involving linear equations."

[0874] 3. Notification to instructors and corrections

[0875] Device: The generated lesson plans and test questions are sent to the teacher's device, where the teacher can review the plans and questions and make any necessary corrections.

[0876] 4. Recording lessons and collecting data

[0877] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The collected video data is temporarily stored.

[0878] 5. Analysis of video data

[0879] Server: Analyzes collected video data using AI to assess students' comments, attitudes, and progress on assignments. Generates an objective evaluation based on each student's evaluation indicators (level of participation, understanding, cooperation, etc.).

[0880] 6. Sending evaluation results and feedback to instructors

[0881] Terminal: The generated evaluation results are sent to the teacher's terminal. The teacher checks the evaluation results on the terminal and makes corrections as necessary. The corrected evaluation results are saved as the final version.

[0882] Users: Teachers communicate assessment results through feedback to students and engage in individual communication.

[0883] Specific examples

[0884] For example, consider a situation in which a teacher introduces linear equations to second-year junior high school students in a mathematics class.

[0885] 1. Server: Receives the chapters and sections on linear equations from a second-year junior high school mathematics textbook and performs preprocessing.

[0886] 2. Server: Automatically generates lesson plans and test questions based on the preprocessed data and sends them to the teacher's device.

[0887] 3. On the device: The teacher reviews the generated lesson plans and test questions and makes any necessary adjustments.

[0888] 4. Server: During the lesson, the camera collects video footage and stores it on the server.

[0889] 5. Server: The AI ​​analyzes the video data and generates an objective evaluation based on the student's level of participation and understanding.

[0890] 6. Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can check and correct them and provide feedback to the students.

[0891] This system allows teachers to significantly reduce the time they spend preparing lessons and grading, and allows them to spend more time communicating with students. This system simultaneously improves the efficiency of teachers' work and the consistency and objectivity of grading.

[0892] The processing flow will be explained below.

[0893] Step 1:

[0894] Server: Receives educational material data (e.g., a mathematics textbook for eighth-grade students) from the educational institution's database. The received data contains information in various formats, such as text, charts, and images. Preprocessing is performed to unify this information into a consistent format. Specifically, this includes text analysis, data cleansing, and format conversion.

[0895] Step 2:

[0896] Server: The generative AI model is run based on the preprocessed data. The lesson plan generation module is activated to generate a lesson plan for "Introduction to Linear Equations." This lesson plan automatically summarizes the lesson content, materials to be used, practice questions, and supporting visual materials. In addition, the test question generation module is activated to generate test questions including "Basic and applied problems for linear equations."

[0897] Step 3:

[0898] Server: Sends the generated lesson plans and test questions to the educational institution's terminals. The data is categorized and tagged for easy access by teachers.

[0899] Step 4:

[0900] Device: Lesson plans and test questions are sent to the teacher's device. The teacher can review the content and make adjustments as necessary. Specifically, they can fine-tune the lesson content, add materials, increase or decrease the number of practice questions, etc.

[0901] Step 5:

[0902] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The video data is first saved to local storage and then uploaded to a cloud server.

[0903] Step 6:

[0904] Server: A generative AI model runs to analyze the collected video data. The analysis module records student speech using voice recognition and analyzes video to determine student attitudes and progress on assignments. This generates an objective evaluation based on each student's evaluation indicators (level of participation, understanding, cooperation, etc.).

[0905] Step 7:

[0906] Server: The generated evaluation results are sent to the teacher's terminal. The evaluation results are organized for each individual student and presented in a visually easy-to-understand format.

[0907] Step 8:

[0908] Terminal: The teacher reviews the evaluation results and makes corrections as necessary. When reviewing the evaluation results, the teacher makes adjustments based on their own observations to determine whether the evaluations automatically generated by the system are appropriate.

[0909] Step 9:

[0910] Users: Teachers can provide feedback on revised assessment results to students and communicate their grades through individual communication, allowing students to accurately understand their strengths and weaknesses and use this information to guide their next learning steps.

[0911] These are the specific steps in the program. These processes will improve the efficiency of teachers' work and provide appropriate feedback to students.

[0912] Example 1

[0913] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0914] Traditionally, teachers' work required a lot of time for lesson preparation and student evaluation, which could lead to inconsistencies in the quality of education. Furthermore, student evaluations were often subjective and inconsistent. This prevented teachers from concentrating on the essential aspects of education. Furthermore, systems that were inefficient in processing data on lesson content and collecting and analyzing lesson footage were problematic.

[0915] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0916] In this invention, the server includes: means for receiving and preprocessing lesson content data; means for operating a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data; means for transmitting the generated lesson plans and test questions to a terminal of the educational institution; means for collecting lesson videos using a camera installed in the classroom; means for analyzing the collected video data and automatically generating objective evaluations of students; means for transmitting the generated evaluation results to a terminal of the educational institution; means for teachers to confirm and correct the evaluation results received on the terminal; means for the generative AI model to receive teaching material data and preprocess using a text analysis tool; means for operating the generative AI model using prompt statements; means for converting the preprocessed data into JSON format; and means for analyzing the video data using a machine learning algorithm and generating scores based on each student's evaluation indicators. This improves the efficiency of lesson preparation and evaluation work, allowing teachers to focus on the essence of education and ensuring consistency and objectivity in student evaluations.

[0917] "Class content data" refers to teaching materials, textbooks, lecture notes, slides, and other educational materials used by educational institutions in classes.

[0918] "Preprocessing" refers to the process of structuring the received educational data and converting it into a format that is easy for the generative AI model to process, and includes text analysis and the removal of unnecessary data.

[0919] A "generative AI model" refers to an artificial intelligence model that uses machine learning algorithms to automatically generate lesson plans and test questions.

[0920] A "lesson plan" refers to an educational guideline that includes a rough outline of a lesson, such as the content of the lesson, the materials to be used, exercises, and supporting visual materials.

[0921] "Test Questions" refers to exam questions automatically generated by a generative AI model to assess students' understanding and knowledge.

[0922] "Educational institution devices" refers to electronic devices, such as computers and tablets, used within an educational institution to review or correct lesson plans and test questions.

[0923] "Filming equipment" refers to cameras and video recording devices installed in classrooms to collect footage of lessons.

[0924] "Video data" refers to video information recorded by a camera during class, and is the subject of analysis.

[0925] "Objective evaluation" refers to student evaluation indicators automatically generated by a generative AI model based on students' comments, attitudes, and progress on assignments.

[0926] "Evaluation results" refers to the student's evaluation scores and feedback information generated based on the analysis of video data by the generative AI model.

[0927] "Means for checking and correcting on a device" refers to the ability for teachers to check the lesson plans and evaluation results generated using the educational institution's devices and correct the content as necessary.

[0928] "Text analysis tools" refers to software and algorithms used in the preprocessing of educational data to analyze text data and extract specific information.

[0929] A "prompt sentence" is an input sentence used to give instructions to a generative AI model, and refers to a sentence that clearly indicates the specific content and purpose.

[0930] The "JSON format" is a lightweight data format used for data exchange and storage, and is suitable for describing preprocessed data because it has a structure that is easily readable by both humans and computers.

[0931] "Machine learning algorithms" refer to artificial intelligence techniques that learn patterns based on data and make predictions and classifications.

[0932] This invention relates to a system that automates lesson preparation and assessment tasks by utilizing generative AI, aiming to improve the efficiency of teachers' work. This system receives and preprocesses lesson content data, then automatically generates lesson plans and test questions using a generative AI model, and further analyzes classroom video data to objectively assess students.

[0933] The main components of the system consist of a server, terminals, users (teachers), data on lesson content, cameras that collect lesson footage, and generative AI models.

[0934] The server connects to the educational institution's database and receives data on lesson content. For example, when retrieving data from a junior high school second-year mathematics textbook, it receives the teaching material data from a MySQL database. The received data is preprocessed in XML format, and only the necessary information is extracted using a text analysis tool (e.g., SpaCy). During this process, the data is converted to JSON format, which is easy for the generative AI model to handle.

[0935] Based on the preprocessed data, the server runs a generative AI model (e.g., GPT-4). A prompt (e.g., "Based on a second-year junior high school mathematics textbook, please generate a lesson plan and test questions for the introduction of linear equations") is entered, and a lesson plan and test questions are generated. The generated lesson plan includes the lesson progress, materials to be used, practice questions, and supporting visual materials. At the same time, the generative AI model also generates a variety of test questions, from basic to applied.

[0936] The generated lesson plans and test questions are sent to the teacher's device. The teacher's device is an electronic device such as a laptop or tablet, and the teacher uses a dedicated application (e.g., a teacher support app) to review and revise the plans and questions. Using dedicated functions, the teacher can change the slide layout and revise the questions. The order of the tasks can also be changed using drag and drop.

[0937] At the start of a lesson, a camera installed in the classroom automatically starts up and records the lesson video in real time. The camera is installed in a position that can overlook every corner of the classroom, and a series of video data is streamed in real time to a server. The data is temporarily stored and later divided into specific frame intervals for analysis by a generative AI model.

[0938] The server analyzes the collected video data using a generative AI model (e.g., Viso.ai). From the analysis, a machine learning algorithm is used to determine students' comments, attitudes, and progress on assignments. Facial recognition and facial expression analysis are performed to measure each student's level of understanding and concentration. In addition, text analysis is performed on the number of times a student's hand is raised and the content of their comments, and participation is reflected as part of the evaluation. The analysis results are converted into a score based on each student's evaluation indicators (e.g., participation, understanding, and cooperation).

[0939] The generated evaluation results are sent to the teacher's device. The teacher can then view the evaluation results in a dedicated application on the device and check each student's score and analysis results. If necessary, the teacher can manually correct the evaluation and add comments. This result is saved as the final version and provided to each student as feedback.

[0940] Teachers can plan individual consultations and supplementary lessons based on students' levels of understanding and participation. They can also communicate the results of the assessment to students through a dedicated application and provide specific advice based on the results.

[0941] In this way, the present invention allows teachers to improve the efficiency of lesson preparation and assessment work, and to focus on communication with students. Furthermore, by improving the consistency and objectivity of assessment, students' learning progress can be accurately grasped.

[0942] For example, here is a prompt for a teacher to "introduce linear equations" to an eighth grade math class:

[0943] "Generate a lesson plan and test questions for 'Introduction to Linear Equations' based on an eighth grade mathematics textbook. The lesson plan should include the lesson progression, materials to be used, practice problems, and supporting visual materials. The test questions should include both basic and applied problems."

[0944] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0945] Step 1:

[0946] The server receives data on lesson content from the educational institution's database. For example, data on a junior high school second-year mathematics textbook is retrieved from a MySQL database. Based on the received data, the server uses a text analysis tool (e.g., SpaCy) to extract the necessary information and remove unnecessary parts. This preprocessing converts the data into a format (JSON) that is easy for the generative AI to handle.

[0947] Input: Course content data stored in the institution's database

[0948] Output: Preprocessed data in JSON format

[0949] Step 2:

[0950] The server runs a generative AI model (e.g., GPT-4) based on the preprocessed data. It generates a lesson plan and test questions based on a prompt ("Generate a lesson plan and test questions for the introduction of linear equations based on a second-year junior high school mathematics textbook."). The generated results include the lesson plan, materials to be used, practice questions, and supporting visual materials.

[0951] Input: Preprocessed JSON format data, prompt statement

[0952] Output: Auto-generated lesson plans and test questions

[0953] Step 3:

[0954] The server then sends the generated lesson plans and test questions to the educational institution's terminals, which can be electronic devices such as teachers' laptops or tablets. After sending, teachers can use a dedicated application to review these materials and make any necessary corrections. For example, they can change the layout of slides or modify the content of test questions.

[0955] Input: Auto-generated lesson plans and test questions

[0956] Output: Lesson plans and test questions sent to the teacher's device

[0957] Step 4:

[0958] When a lesson starts, the server automatically activates the cameras installed in the classrooms. These cameras record the lesson video in real time and stream it to the server, where the video data is temporarily stored.

[0959] Input: Real-time classroom video

[0960] Output: Video data of lessons stored on the server

[0961] Step 5:

[0962] The server analyzes the collected video data using a generative AI model. Machine learning algorithms (e.g., Viso.ai) are used to analyze the video data and determine student speech, behavior, and task performance. This analysis generates a score based on facial recognition and facial expression analysis to assess student engagement and comprehension.

[0963] Input: Video data of lessons stored on the server

[0964] Output: Student evaluation scores

[0965] Step 6:

[0966] The evaluation results generated by the server are then sent back to the teacher's device. The teacher can then check the results using a dedicated application on the device and manually correct them as necessary. The corrected evaluation results are saved as the final version and provided as feedback to the students.

[0967] Input: Generated student assessment scores

[0968] Output: Evaluation results sent to the teacher's terminal, evaluation results corrected by the teacher

[0969] Step 7:

[0970] The user (teacher) provides feedback on the evaluation results to students through a dedicated application. Based on the evaluation results, the user can plan individual interviews and supplementary lessons and provide specific advice to students.

[0971] Input: Evaluation results corrected by the teacher

[0972] Output: Individual feedback and advice to students

[0973] (Application example 1)

[0974] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0975] In conventional educational systems, teachers expend a great deal of effort preparing lessons and evaluating students. Furthermore, even in online classes, there are few means for evaluating students' understanding and participation in real time and providing feedback. This increases the burden on teachers and leads to issues such as a lack of communication with students. This invention aims to solve these issues and improve the efficiency of teachers' work and the quality of lessons.

[0976] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0977] In this invention, the server includes means for receiving and preprocessing data on lesson content, means for operating a generative AI model for automatically generating lesson plans and test questions based on the preprocessed data, means for transmitting the generated lesson plans and test questions to a terminal of the educational institution, means for collecting lesson video using a camera installed in the classroom, means for analyzing the collected video data and automatically generating objective evaluations of students, means for transmitting the generated evaluation results to a terminal of the educational institution, means for providing feedback to students according to their learning progress using the generative AI model, means for distributing online lesson content based on the automatically generated lesson plans and test questions, and means for recording and evaluating lessons in real time via an application installed on a smartphone or tablet. This reduces the workload of teachers and enables them to grasp students' levels of understanding and participation in real time and provide feedback.

[0978] "Course content data" refers to a set of information related to teaching materials and courses provided by educational institutions.

[0979] "Preprocessing" refers to a series of operations that prepare lesson content data into a format that is easy for the generative AI model to handle.

[0980] "Generative AI models" refer to algorithms or software that use artificial intelligence technology to automatically generate lesson plans and test questions.

[0981] A "lesson plan" is a detailed plan that includes the lesson content, materials to be used, exercises, supporting visual materials, etc.

[0982] "Test questions" refer to questions created by generative AI models to assess comprehension and proficiency of lesson content.

[0983] "Educational institution device" refers to an electronic device, such as a computer or tablet, owned by an educational institution.

[0984] "Camera" refers to video recording equipment installed in the classroom to collect footage during lessons.

[0985] "Video data" refers to video files and related information captured by a camera during class.

[0986] "Objective evaluation" refers to a quantitative evaluation calculated by a generative AI model based on students' comments, attitudes, and progress on assignments.

[0987] "Students" refers to students who participate in classes.

[0988] "Learning progress" refers to the degree to which a participant has progressed in the knowledge and skills they have acquired through classes and learning materials.

[0989] "Feedback" refers to advice and suggestions provided by a generative AI model based on a student's learning progress and level of understanding.

[0990] The present invention is a system for automating lesson preparation and evaluation tasks, and detailed embodiments thereof are described below. The system includes, as its main components, a server, terminals, users (teachers), data on lesson content, a camera for collecting lesson footage, and a generative AI model.

[0991] First, the server receives data on lesson content from the educational institution's database. For example, data from a mathematics textbook for second-year junior high school students. Next, the received data is preprocessed and converted into a format that is easy for the generative AI model to process. This ensures that the teaching material data is properly structured.

[0992] The server runs a generative AI model based on the preprocessed data to automatically generate lesson plans and test questions. For example, let's take a lesson plan for "Introduction to Linear Equations." This lesson plan includes the lesson progress, materials to be used, practice questions, and supporting visual materials. The generative AI model also generates test questions, including "basic and applied problems involving linear equations."

[0993] The generated lesson plans and test questions are sent to the educational institution's terminals, where teachers can review them and make corrections as necessary.

[0994] During lessons, cameras installed in the classroom are turned on and capture video footage of the lesson in real time. The server temporarily stores the collected video data and then analyzes it using a generative AI model. Specifically, it assesses students' comments, attitudes, and progress on assignments, and automatically generates an objective evaluation. Evaluation criteria include participation, understanding, and cooperation.

[0995] The generated evaluation results are sent to the teacher's device. The teacher reviews the results and makes corrections as necessary. The corrected evaluation results are saved as the final version and feedback is provided to the students.

[0996] Additionally, feedback is automatically provided using a generative AI model according to the student's learning progress, allowing appropriate learning support to be provided to each individual student.

[0997] The system also has a function for distributing online lesson content. It automatically generates lesson plans and test questions, enabling effective online lessons for students. It also allows lessons to be recorded and evaluated in real time via an application installed on a smartphone or tablet.

[0998] For example, if you are teaching second-year junior high school maths classes about "Introduction to Linear Equations," you might input the following prompts into the generative AI model:

[0999] Example prompt sentence:

[1000] "Generate lesson plans and test questions for an introduction to linear equations based on eighth grade mathematics materials. Include concrete examples, visual aids, and practice problems in your lesson plans."

[1001] This system will enable teachers to significantly reduce the time they spend preparing lessons and assessing them, allowing them to spend more time communicating with students, resulting in improved quality of education and more efficient teacher work.

[1002] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1003] Step 1:

[1004] The server receives data on lesson content from the educational institution's database. For example, it retrieves data on a mathematics textbook for second-year junior high school students. The input here is the educational institution's database, and the output is the received teaching material data. Specifically, it accesses the database through an API and downloads the necessary teaching material data.

[1005] Step 2:

[1006] The server preprocesses the received teaching material data. Preprocessing involves cleaning and tokenizing the text data, converting it into a format that is easy for the generative AI model to process. The input here is the received teaching material data, and the output is the preprocessed teaching material data. Specifically, it converts unstructured data into an appropriate format and removes unnecessary characters and errors.

[1007] Step 3:

[1008] The server runs a generative AI model based on the preprocessed data to automatically generate lesson plans and test questions. The input is the preprocessed teaching material data, and the output is the generated lesson plans and test questions. Specifically, a natural language processing algorithm is used to generate prompt sentences based on the specified theme.

[1009] Step 4:

[1010] The server sends the generated lesson plans and test questions to the educational institution's terminals. The input here is the generated lesson plans and test questions, and the output is the data displayed on the teacher's terminal. Specifically, the data is securely sent via the network so that the teacher can access it.

[1011] Step 5:

[1012] The camera installed in the classroom is activated and collects the lesson video in real time. The input here is the video of the lesson scene, and the output is the collected video data. Specifically, the camera device is activated remotely and recording begins.

[1013] Step 6:

[1014] The server temporarily stores the collected video data and then analyzes it using a generative AI model. The input here is the collected video data, and the output is the analysis results and an objective evaluation of the student. Specifically, video recognition technology is used to analyze the student's comments and behavior and calculate evaluation indicators.

[1015] Step 7:

[1016] The server sends the generated evaluation results to the educational institution's terminal. The input here is the analysis results and the student's objective evaluation, and the output is the evaluation results displayed on the teacher's terminal. Specifically, the evaluation results are sent via the network so that the teacher can check and correct them.

[1017] Step 8:

[1018] Teachers check the evaluation results through their terminals and make corrections as necessary. The input here is the generated evaluation results, and the output is the final evaluation results after corrections. Specifically, teachers check the evaluation results using the terminal interface and make corrections using a dedicated editing tool.

[1019] Step 9:

[1020] The server uses a generative AI model to provide feedback based on the student's learning progress. The input is the student's learning progress data, and the output is the feedback content. Specifically, it generates individual advice and supplementary learning materials based on test results and participation status.

[1021] Step 10:

[1022] The server distributes online lesson content based on the automatically generated lesson plans and test questions. The input here is the generated lesson plans and test questions, and the output is the content of the online lesson. Specifically, the content of the lesson is provided in a live streaming or video-on-demand format, allowing students to access it.

[1023] Step 11:

[1024] Lessons are recorded in real time and evaluated through an application installed on a smartphone or tablet. The input is real-time lesson footage, and the output is the recorded data and its evaluation results. Specifically, lessons are recorded using the mobile device's camera and analyzed using a generative AI model.

[1025] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1026] This invention relates to a system that aims to improve the efficiency of teachers' work by automating lesson preparation and evaluation tasks by utilizing generative AI and an emotion engine, and further improving the quality of evaluations through student emotion recognition.

[1027] System configuration overview

[1028] The system includes the following major components:

[1029] server

[1030] Terminal

[1031] User (teacher)

[1032] Class content data

[1033] Cameras that collect video footage from lessons

[1034] Generative AI Models

[1035] Emotion Engine

[1036] A natural language explanation of the program's processing

[1037] 1. Receiving and preprocessing the teaching material data

[1038] Server: Receives educational material data (e.g., a mathematics textbook for eighth-grade students) from the educational institution's database. The received data contains information in various formats, such as text, charts, and images. Preprocessing is performed to unify this information into a consistent format. Specifically, this includes text analysis, data cleansing, and format conversion.

[1039] 2. Automatic generation of lesson plans and test questions

[1040] Server: The generative AI model is run based on the preprocessed data. The lesson plan generation module is activated to generate a lesson plan for "Introduction to Linear Equations." This lesson plan automatically summarizes the lesson content, materials to be used, practice questions, and supporting visual materials. In addition, the test question generation module is activated to generate test questions including "Basic and applied problems for linear equations."

[1041] 3. Notification to instructors and corrections

[1042] Device: The generated lesson plans and test questions are sent to the teacher's device, where the teacher can review the plans and questions and make any necessary corrections.

[1043] 4. Recording lessons and collecting data

[1044] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The video data is temporarily saved in local storage and then uploaded to the cloud server.

[1045] 5. Video Data Analysis and Emotion Recognition

[1046] Server: A generative AI model runs to analyze the collected video data. The analysis module uses voice recognition to record what students say and analyzes video to determine their attitudes and progress on assignments. Furthermore, an emotion engine runs, analyzing students' facial expressions and tone of voice to recognize emotions, such as joy, sadness, anger, and surprise.

[1047] 6. Generating evaluation results and feedback

[1048] Server: Generates objective evaluations based on each student's evaluation indicators (e.g., participation, understanding, cooperation) and emotion recognition results. The generated evaluation results are organized for each individual student and presented in a visually easy-to-understand format.

[1049] Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can review them and make corrections as necessary. Because the evaluation also includes emotion recognition information, the teacher can assess the student's mental state while still understanding it.

[1050] Users: Teachers can provide feedback on revised assessment results to students and communicate their grades through individual communication, allowing students to accurately understand their strengths and weaknesses and use this information to guide their next learning steps.

[1051] Specific examples

[1052] For example, consider a situation in which a teacher introduces linear equations to second-year junior high school students in a mathematics class.

[1053] 1. Server: Receives the chapters and sections on linear equations from a second-year junior high school mathematics textbook and performs preprocessing.

[1054] 2. Server: Automatically generates lesson plans and test questions based on the preprocessed data and sends them to the teacher's device.

[1055] 3. Terminal: Teachers review lesson plans and test questions and make adjustments as needed.

[1056] 4. Server: During the lesson, the camera collects video footage and stores it on the server.

[1057] 5. Server: Analyzes video data using generative AI and an emotion engine to generate objective evaluations based on students' emotions, comments, and attitudes.

[1058] 6. Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can check and correct them.

[1059] 7. User: Teachers communicate assessment results through feedback to students and engage in individual communication.

[1060] The above is a specific "Mode for carrying out the invention." This system allows teachers to significantly reduce the time they spend preparing lessons and assessing them, allowing them to spend more time communicating with students. By combining emotion recognition, more comprehensive assessments are possible, improving students' learning experiences.

[1061] The processing flow will be explained below.

[1062] Step 1:

[1063] Server: Receives educational material data (e.g., a mathematics textbook for second-year junior high school students) from the educational institution's database. The received data contains information in a variety of formats, including text, charts, and images. Preprocessing is performed to unify this information into a consistent format. Specifically, text analysis, various format conversions, and data cleansing are performed.

[1064] Step 2:

[1065] Server: The generative AI model is run based on the preprocessed data. The lesson plan generation module is activated to generate a lesson plan for "Introduction to Linear Equations." This lesson plan automatically summarizes the lesson content, materials to be used, practice questions, and supporting visual materials. In addition, the test question generation module is activated to generate test questions including "Basic and applied problems for linear equations."

[1066] Step 3:

[1067] Server: Generated lesson plans and test questions are sent to the educational institution's terminals. The data is categorized and tagged for easy access by teachers.

[1068] Step 4:

[1069] Devices: Lesson plans and test questions are sent to teachers' devices for review. Teachers can review the content and make adjustments as needed. Specifically, they can fine-tune the lesson content, add materials, increase or decrease the number of practice questions, etc.

[1070] Step 5:

[1071] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The video data is saved in local storage and then uploaded to a cloud server.

[1072] Step 6:

[1073] Server: The collected video data is analyzed using a generative AI model and emotion engine. The generative AI model records students' comments using voice recognition and determines their attitudes and progress on assignments through video analysis. The emotion engine analyzes students' facial expressions and tone of voice to recognize their emotions. For example, emotions such as joy, sadness, anger, and surprise can be recognized.

[1074] Step 7:

[1075] Server: Generates objective evaluations based on each student's evaluation indicators (e.g., participation, understanding, cooperation) and emotion recognition results. The generated evaluation results are organized for each individual student and presented in a visually easy-to-understand format.

[1076] Step 8:

[1077] Terminal: The evaluation results are sent to the teacher's terminal. The teacher can review them and make corrections as necessary. Emotion recognition information is also included, so the teacher can evaluate the student while understanding their mental state.

[1078] Step 9:

[1079] User: Teachers provide feedback to students on the revised assessment results. Students are informed of their grades through individual communication, allowing them to accurately understand their strengths and weaknesses. This feedback helps students take the next step in their studies.

[1080] These are the specific processing steps of the program, which will enable teachers to significantly reduce the time required for lesson preparation and assessment, while also enabling comprehensive assessment that takes students' emotions into account.

[1081] Example 2

[1082] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1083] Teachers at educational institutions spend a great deal of time and effort preparing lessons and conducting assessments. It is particularly difficult to objectively assess students' progress and emotions while understanding their learning progress. They also need to provide students with prompt and accurate feedback. Furthermore, if video data from lessons could be used to assess students' learning status, more efficient educational support would be possible. However, current educational systems lack such functionality. Therefore, a new system is needed to improve teacher efficiency and comprehensively assess students.

[1084] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1085] In this invention, the server includes means for receiving and preprocessing lesson content data, means for operating a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data, means for transmitting the generated lesson plans and test questions to an educational institution's terminal, means for reviewing and correcting the generated lesson plans and test questions on the educational institution's terminal, means for collecting lesson video using a video capture device installed in the classroom, means for analyzing the collected video data and automatically generating objective evaluations of students using the generative AI model and an emotion recognition algorithm, means for transmitting the generated evaluation results to the educational institution's terminal and reviewing and correcting the evaluation results on the educational institution's terminal, and means for providing feedback to students from the educational institution's terminal. This reduces teachers' lesson preparation time and enables accurate evaluation of students' learning progress and emotions. It also allows teachers to quickly review and correct the evaluation results and provide quick and accurate feedback to students.

[1086] "Course content data" means information related to a course offered by an educational institution, including textbooks, slides, visual materials, audio files, video files, and other formats.

[1087] "Preprocessing" refers to a series of steps taken to convert received data into a consistent format, including parsing text, removing unnecessary information, and standardizing formats.

[1088] A "generative AI model" is a model that uses artificial intelligence algorithms to automatically generate lesson plans and test questions from basic data. Specifically, machine learning and neural networks are often applied.

[1089] A "lesson plan" is a written plan prepared by a teacher to conduct a lesson, and includes the content of the lesson, teaching materials to be used, practice questions, supplementary visual materials, etc.

[1090] "Test questions" are questions that are asked to assess a student's level of understanding and learning progress, and include multiple choice questions, essay questions, applied questions, etc.

[1091] "Educational institution device" refers to an electronic device, such as a computer, tablet, or smartphone, owned by an educational institution and used by faculty to review, modify, or access lesson plans and test questions.

[1092] "Filming equipment" refers to devices such as cameras and recording devices installed in classrooms to collect footage of lessons.

[1093] "Generative AI model and emotion recognition algorithm" refers to an algorithm that uses AI technology to analyze students' facial expressions and tone of voice to recognize their emotions, which can then be used to evaluate their learning attitudes and emotions.

[1094] "Objective assessment of students" means evaluating students' learning progress, level of understanding, participation, etc. based on data and algorithms, without relying on the teacher's subjective opinion.

[1095] "Feedback" refers to the act of teachers giving students grades and advice based on the results of their assessments, allowing students to understand their own strengths and weaknesses and use this information to guide their next steps in learning.

[1096] MODE FOR CARRYING OUT THE INVENTION

[1097] This invention aims to improve the work efficiency of teachers in educational institutions by automating lesson preparation and evaluation tasks. Furthermore, it aims to improve the quality of evaluations through student emotion recognition. This system mainly consists of three elements: a server, a terminal, and a user (teacher).

[1098] Receiving and preprocessing teaching material data

[1099] Server: Receives course content data from the educational institution's database. The received data contains a variety of formats, including text, charts, and images, and undergoes preprocessing to unify them into a consistent format. Preprocessing includes text analysis, data cleansing, and format conversion.

[1100] Automatic generation of lesson plans and test questions

[1101] Server: Based on the preprocessed data, a generative AI model is run to automatically generate lesson plans and test questions. The generative AI model uses machine learning algorithms, and algorithms optimized for each subject area are implemented. The generated lesson plans include the lesson progress, materials to be used, practice questions, and supporting visual materials. In addition, a test question generation module automatically generates multiple-choice and essay questions.

[1102] Instructor notification and corrections

[1103] Devices: The generated lesson plans and test questions are sent to the educational institution's devices. Teachers can review them on their devices and make corrections as needed. Corrections can be easily made through the device's editing interface.

[1104] Recording lessons and collecting data

[1105] Server: During lessons, the camera installed in the classroom is turned on and captures video of the lesson in real time. The collected video data is temporarily saved in local storage and then uploaded to the cloud server.

[1106] Video data analysis and emotion recognition

[1107] Server: Runs generative AI models and emotion recognition algorithms to analyze the collected video data. The analysis module uses voice recognition technology to record students' speech and video analysis technology to evaluate their attitudes and progress on assignments. It also uses emotion recognition algorithms to analyze students' facial expressions and tone of voice to recognize their emotions.

[1108] Generation of evaluation results and feedback

[1109] Server: Generates objective evaluations based on evaluation indicators such as student participation, understanding, and cooperation, as well as emotion recognition results. The evaluation results are organized in a visually easy-to-understand format.

[1110] Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can review them and make corrections as necessary. Emotion recognition information is also included, so the teacher can evaluate the student while understanding their mental state.

[1111] Users: Teachers can provide feedback on revised assessment results to students and communicate their grades through individual communication, allowing students to accurately identify their strengths and weaknesses and use them to guide their next learning steps.

[1112] Specific examples

[1113] For example, consider a lesson on "Introduction to Linear Equations" in mathematics for second-year junior high school students. First, the server extracts the chapters and sections on linear equations from the second-year junior high school mathematics textbook and performs preprocessing. Next, a generative AI model is run based on the preprocessed data, automatically generating lesson plans and test questions. The generated plans and questions are sent to the teacher's device, who reviews and modifies them. After that, cameras installed during the lesson collect footage of the lesson, and the server analyzes the data. The analysis results are objectively evaluated based on the students' emotions and comments, and the evaluation results are sent to the teacher's device. The teacher reviews and modifies these evaluation results and provides feedback to the students.

[1114] Prompt Sentence Examples

[1115] "Generate a lesson plan for introducing linear equations.

[1116] Lessons should include basic conceptual explanations, examples, exercises, and supporting visual materials.

[1117] The system of this invention can significantly reduce the time required for teachers to prepare and evaluate lessons, and improve the learning experience for students. By combining it with emotion recognition, more comprehensive and objective evaluation can be achieved, supporting the growth of each student.

[1118] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1119] Program processing flow

[1120] Step 1: Receiving and preprocessing the teaching material data

[1121] server:

[1122] Input: Course content data obtained from an educational institution's database (e.g., an XML file of a second-year junior high school mathematics textbook)

[1123] Processing: Parse the data and convert it into a consistent format. This involves text analysis, removal of unnecessary tags, and standardization of formats.

[1124] Output: Preprocessed data (JSON formatted lesson content data)

[1125] Specific behavior:

[1126] The server accesses the educational institution's database and retrieves the data for the specified course content. If the data is in XML format, it first parses it to extract the textbook content into text format. It then removes unnecessary tags and noise and converts it into a consistent JSON format.

[1127] Step 2: Automatically generate lesson plans and test questions

[1128] server:

[1129] Input: Preprocessed lesson content data (JSON format)

[1130] Processing: Run generative AI models to generate lesson plans and test questions, using machine learning algorithms and domain-specific optimized models.

[1131] Output: Auto-generated lesson plans and test questions (JSON or text format)

[1132] Specific behavior:

[1133] The server inputs the preprocessed lesson content data into a generative AI model. Using advanced machine learning algorithms such as the GPT-4 model, it automatically generates a lesson plan for "Introduction to Linear Equations." This plan includes teaching materials, a progress schedule, practice questions, and supplementary materials for teachers to use. A test question generation module also runs simultaneously, generating multiple-choice and essay questions to assess comprehension.

[1134] Step 3: Notify instructor and make corrections

[1135] Device:

[1136] Input: Generated lesson plans and test questions (JSON or text format)

[1137] Processing: The teacher will review these on the system and make corrections as necessary.

[1138] Output: Modified lesson plans and test questions (JSON or text format)

[1139] Specific behavior:

[1140] The generated lesson plans and test questions are sent to the teacher's device, where the teacher can review the content on the device's display screen and make any necessary changes using the editing interface, such as changing the format of a particular question or adjusting the lesson schedule.

[1141] Step 4: Recording lessons and collecting data

[1142] server:

[1143] Input: Video data of lessons collected in real time

[1144] Processing: During lessons, video footage is collected using video recording equipment installed in the classroom. The video data is temporarily stored in local storage and later uploaded to a cloud server.

[1145] Output: Lesson video data stored in cloud storage

[1146] Specific behavior:

[1147] The camera automatically starts recording as soon as the lesson begins, capturing the entire lesson. The recorded video is temporarily saved in local storage. After the lesson ends, the video data is uploaded to cloud storage by the server.

[1148] Step 5: Video data analysis and emotion recognition

[1149] server:

[1150] Input: Video data from lessons stored in cloud storage

[1151] Processing: Using generative AI models and emotion recognition algorithms, the video data is analyzed to assess student speech, attitudes, and emotions.

[1152] Output: Student objective evaluation indicators (evaluation data in JSON format)

[1153] Specific behavior:

[1154] The server retrieves the video data from cloud storage and launches the analysis module. It uses a speech recognition API to transcribe what students say, applies a posture analysis model to evaluate their attitude and progress on assignments, and uses a facial expression analysis API to recognize emotions from students' facial expressions and tone of voice.

[1155] Step 6: Generating evaluation results and feedback

[1156] server:

[1157] Input: Student objective evaluation indicators (evaluation data in JSON format)

[1158] Processing: Generate objective evaluations based on each student's participation, understanding, cooperation, and other evaluation indicators, as well as emotion recognition results. Organize the results in a visually easy-to-understand format.

[1159] Output: Evaluation results (visualized evaluation report)

[1160] Specific behavior:

[1161] The system uses k-means clustering to classify student assessment data and visualize it in a dashboard format. Assessment reports are organized for each student and are created in a visually easy-to-understand format, including supporting materials.

[1162] Device:

[1163] Input: Evaluation results (visualized evaluation report)

[1164] Processing: The teacher reviews the evaluation results and makes corrections if necessary.

[1165] Output: Corrected evaluation results (final evaluation report)

[1166] Specific behavior:

[1167] Teachers can open the evaluation report on their devices, make any necessary corrections, review the results, and add specific comments about students' strengths and weaknesses.

[1168] User:

[1169] Input: Corrected evaluation results (final evaluation report)

[1170] Processing: Teachers provide feedback to students and communicate the results of the assessment.

[1171] Output: Assessment feedback that students can understand

[1172] Specific behavior:

[1173] Teachers meet with students individually to explain the results of their assessments in detail, allowing students to understand their strengths and weaknesses and use the feedback to plan their future learning.

[1174] This system not only significantly reduces the time required for teachers to prepare and evaluate lessons, but also enables more comprehensive and objective evaluation, improving students' learning experiences and supporting the growth of each individual student.

[1175] (Application example 2)

[1176] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1177] In traditional education systems, teachers have to expend a great deal of effort preparing lessons and evaluating students, making efficient lesson management difficult and potentially resulting in a decline in the quality of education. Furthermore, in brick-and-mortar stores, it is difficult for employees to immediately grasp and respond to customer emotions, hindering customer satisfaction. To solve these issues, it is necessary to utilize generative AI models and emotion engines to automate lesson preparation and evaluation, and further improve the quality of customer service in brick-and-mortar stores.

[1178] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1179] In this invention, the server includes means for receiving and preprocessing lesson content data, means for running a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data, means for transmitting the generated lesson plans and test questions to a terminal at the educational institution, means for collecting lesson video from a camera installed in the classroom, means for analyzing the collected video data and automatically generating objective evaluations of students, means for transmitting the generated evaluation results to a terminal at the educational institution, means for application to a customer service robot that recognizes customer emotions in real time and automatically adjusts responses, means for automatically adjusting customer service scripts and generating messages based on the emotion recognition results, and means for accumulating emotion data and utilizing it for future customer service. This enables the efficiency of teachers' work, improvement of the quality of education, and improvement of the quality of customer service in physical stores.

[1180] "Class content data" refers to text data, image data, video data, and other related information that constitute the content of classes provided by educational institutions.

[1181] "Preprocessing" refers to the process of data cleansing and format conversion to analyze received lesson content data and convert it into a consistent format.

[1182] A "generative AI model" is an algorithm and its execution environment that uses artificial intelligence to automatically generate new data based on specified purposes or themes.

[1183] A "lesson plan" is a written plan that summarizes the content of the lesson, the materials to be used, practice questions, supplementary visual materials, etc.

[1184] A "test" is a set of questions or tasks designed to assess a student's understanding or achievement.

[1185] "Educational institution terminals" refer to electronic devices such as computers and tablets used by faculty and administrators affiliated with educational institutions.

[1186] "Camera installed in the classroom" refers to a recording device placed to collect video and audio from the classroom in real time.

[1187] "Video data" refers to video files collected by a camera and any accompanying audio and metadata.

[1188] "Objective evaluation" refers to the results of an objective analysis of a student's behavior, comments, emotions, etc.

[1189] "Means for recognizing customer emotions in real time" refers to technology that uses cameras and voice recognition devices to analyze a customer's facial expressions and tone of voice to instantly determine their emotional state.

[1190] "Application to customer service robots that automatically adjust responses" refers to an applied technology in which the robot instantly changes its customer service scripts and behavior based on the results of emotion recognition, providing the optimal response for each individual customer.

[1191] A "customer service script" refers to a series of lines and action steps that a customer service robot speaks to customers.

[1192] "Emotion data" refers to the inferred emotions obtained through facial expression and voice analysis of customers, as well as related metadata.

[1193] "Storage" is the process of continually collecting certain data and storing it for later use.

[1194] "Methods to utilize this information in future customer service" refers to analyzing accumulated emotional data and providing optimal customer service based on the customer's past reactions.

[1195] This invention is an integrated system for improving the efficiency of teachers' work and the quality of customer service in brick-and-mortar stores. It mainly consists of a server, terminals, and users (teachers or store staff).

[1196] Main components of the system

[1197] 1. Server

[1198] The server performs the following main functions:

[1199] Receiving and preprocessing data for lesson content:

[1200] The server receives data on lesson content from the educational institution's database. The received data includes a variety of formats, such as text data, image data, and video data. To unify this data into a consistent format, preprocessing such as text analysis, data cleansing, and format conversion is performed.

[1201] Automatic generation of lesson plans and test questions:

[1202] Based on the preprocessed data, the generative AI model is run. The lesson plan generation module automatically generates a plan for a specific lesson content, including the lesson progress, materials to be used, practice questions, visual materials, etc. Furthermore, the test question generation module runs and generates test questions corresponding to the lesson content.

[1203] Send generated data:

[1204] The generated lesson plans and test questions are sent to the educational institution's terminals, where teachers can review them and make corrections as needed.

[1205] Video data collection and analysis:

[1206] Cameras installed in classrooms are used to record lessons in progress. The collected video data is first stored in local storage and then uploaded to a cloud server. An analysis module analyzes this video data and automatically evaluates students' behavior and comments. An emotion engine is also used to recognize students' emotions and improve the quality of the evaluation.

[1207] Application in physical stores

[1208] Customer Emotion Recognition:

[1209] Using cameras and a voice recognition system installed in the store, the system analyzes customers' facial expressions and tone of voice in real time to recognize their emotions. The emotion engine then recognizes their emotions, such as joy, sadness, and surprise.

[1210] Automatic adjustment of customer service scripts:

[1211] The robot's script is adjusted in real time based on the results of emotion recognition. For example, if a customer has a troubled expression, the robot will respond with, "Are you looking for something?"

[1212] Accumulating customer data and utilizing it for the next customer visit:

[1213] The collected emotional data is stored on a server, and this data is used in future visits to provide the optimal response based on the customer's past reactions.

[1214] Hardware and Software Used

[1215] Hardware: Camera, robot (Raspberry Pi or NVIDIA Jetson)

[1216] Software: OpenCV, DeepFace, GPT-3 (emotion recognition and sentence generation)

[1217] Specific examples

[1218] For example, if a customer walks into a brick-and-mortar store with a tired expression, the camera and voice recognition system will recognize that emotion, and the customer service robot will say, "We're happy to help you if you need anything."

[1219] Example prompt sentence:

[1220] Input prompt: "Generate a response message when the customer has a confused look on their face."

[1221] Example generated: "Looking for something? Let us help."

[1222] This will enable teachers to improve their work efficiency and the quality of their education, and store staff to increase customer satisfaction.

[1223] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1224] Step 1:

[1225] Receiving and preprocessing lesson content data

[1226] The server receives lesson content data from the educational institution's database. This includes text data, image data, video data, etc. The server performs preprocessing such as text analysis, data cleansing, and format conversion to unify the received data into a consistent format. The input is lesson content data from the educational institution's database, and the output is preprocessed data in a unified format.

[1227] Step 2:

[1228] Automatic generation of lesson plans and test questions

[1229] The server runs a generative AI model based on the preprocessed data. The lesson plan generation module generates lesson plans for specific lesson content, and the test question generation module generates corresponding test questions. The input is the preprocessed data, and the output is the generated lesson plans and test questions. This includes data analysis and model generation.

[1230] Step 3:

[1231] Sending generated data

[1232] The server transmits the generated lesson plans and test questions to the educational institution's terminal. The input is the generated lesson plans and test questions, and the output is the data transmitted to the teacher's terminal. A data transmission protocol is used.

[1233] Step 4:

[1234] Collecting video data from within the classroom

[1235] The server collects video footage from classrooms via cameras installed in the classrooms. This video data is stored in local storage and then uploaded to a cloud server. The input is real-time video footage from classroom cameras, and the output is video data stored on the cloud server. This includes video collection and uploading.

[1236] Step 5:

[1237] Video data analysis and emotion recognition

[1238] The server runs a generative AI model to analyze video data stored on a cloud server. It records student speech using voice recognition, determines student attitudes and progress on assignments using video analysis, and analyzes student emotions (e.g., joy, sadness, anger, surprise) using an emotion engine. The input is the collected video data, and the output is the analysis results and emotion recognition results. It includes data analysis and emotion evaluation.

[1239] Step 6:

[1240] Generating and sending evaluation results

[1241] The server generates an objective evaluation based on each student's evaluation indicators (level of participation, understanding, cooperation, etc.) and the emotion recognition results, and sends it to the educational institution's terminal. The input is the analysis results and emotion recognition results, and the output is the generated evaluation results. The evaluation results are provided in a visually easy-to-understand format.

[1242] Step 7:

[1243] Customer Emotion Recognition

[1244] The device uses cameras and voice recognition systems installed in physical stores to analyze customers' facial expressions and tone of voice in real time, and then activates an emotion engine to determine their emotions. The input is data from the in-store cameras and voice recognition system, and the output is analyzed emotion data. Emotion data analysis is included.

[1245] Step 8:

[1246] Automatic adjustment of customer service scripts

[1247] The terminal adjusts the customer service robot's script in real time based on the emotion recognition results and generates an appropriate message. For example, it responds to a customer with a troubled expression with "Are you looking for something?" The input is the emotion recognition result, and the output is the generated customer service message. This process includes script generation and adjustment.

[1248] Step 9:

[1249] Accumulation of customer data and next use

[1250] The server stores the collected emotion data and uses it for future customer service. The input is each customer's emotion data, and the output is the stored customer data. This includes saving the data and preparing it for the next use.

[1251] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1252] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1253] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1254] [Fourth embodiment]

[1255] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1256] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1257] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1258] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1259] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1260] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1261] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1262] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1263] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1264] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1265] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1266] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1267] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1268] The present invention relates to a system that automates lesson preparation and evaluation tasks by utilizing generative AI, with the aim of improving the efficiency of teachers' work.

[1269] System configuration overview

[1270] The system includes the following major components:

[1271] server

[1272] Terminal

[1273] User (teacher)

[1274] Class content data

[1275] Cameras that collect video footage from lessons

[1276] Generative AI Models

[1277] A natural language explanation of the program's processing

[1278] 1. Receiving and preprocessing the teaching material data

[1279] Server: Receives educational material data (e.g., a mathematics textbook for second-year junior high school students) from the educational institution's database. The received data is preprocessed and converted into a format that is easy for the generative AI to process.

[1280] 2. Automatic generation of lesson plans and test questions

[1281] Server: Runs a generative AI model based on the preprocessed data to automatically generate a lesson plan for "Introduction to Linear Equations." The lesson plan includes the lesson progress, materials to be used, practice problems, and supplementary visual materials. The generative AI model also generates test questions, including "basic and applied problems involving linear equations."

[1282] 3. Notification to instructors and corrections

[1283] Device: The generated lesson plans and test questions are sent to the teacher's device, where the teacher can review the plans and questions and make any necessary corrections.

[1284] 4. Recording lessons and collecting data

[1285] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The collected video data is temporarily stored.

[1286] 5. Analysis of video data

[1287] Server: Analyzes collected video data using AI to assess students' comments, attitudes, and progress on assignments. Generates an objective evaluation based on each student's evaluation indicators (level of participation, understanding, cooperation, etc.).

[1288] 6. Sending evaluation results and feedback to instructors

[1289] Terminal: The generated evaluation results are sent to the teacher's terminal. The teacher checks the evaluation results on the terminal and makes corrections as necessary. The corrected evaluation results are saved as the final version.

[1290] Users: Teachers communicate assessment results through feedback to students and engage in individual communication.

[1291] Specific examples

[1292] For example, consider a situation in which a teacher introduces linear equations to second-year junior high school students in a mathematics class.

[1293] 1. Server: Receives the chapters and sections on linear equations from a second-year junior high school mathematics textbook and performs preprocessing.

[1294] 2. Server: Automatically generates lesson plans and test questions based on the preprocessed data and sends them to the teacher's device.

[1295] 3. On the device: The teacher reviews the generated lesson plans and test questions and makes any necessary adjustments.

[1296] 4. Server: During the lesson, the camera collects video footage and stores it on the server.

[1297] 5. Server: The AI ​​analyzes the video data and generates an objective evaluation based on the student's level of participation and understanding.

[1298] 6. Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can check and correct them and provide feedback to the students.

[1299] This system allows teachers to significantly reduce the time they spend preparing lessons and grading, and allows them to spend more time communicating with students. This system simultaneously improves the efficiency of teachers' work and the consistency and objectivity of grading.

[1300] The processing flow will be explained below.

[1301] Step 1:

[1302] Server: Receives educational material data (e.g., a mathematics textbook for eighth-grade students) from the educational institution's database. The received data contains information in various formats, such as text, charts, and images. Preprocessing is performed to unify this information into a consistent format. Specifically, this includes text analysis, data cleansing, and format conversion.

[1303] Step 2:

[1304] Server: The generative AI model is run based on the preprocessed data. The lesson plan generation module is activated to generate a lesson plan for "Introduction to Linear Equations." This lesson plan automatically summarizes the lesson content, materials to be used, practice questions, and supporting visual materials. In addition, the test question generation module is activated to generate test questions including "Basic and applied problems for linear equations."

[1305] Step 3:

[1306] Server: Sends the generated lesson plans and test questions to the educational institution's terminals. The data is categorized and tagged for easy access by teachers.

[1307] Step 4:

[1308] Device: Lesson plans and test questions are sent to the teacher's device. The teacher can review the content and make adjustments as necessary. Specifically, they can fine-tune the lesson content, add materials, increase or decrease the number of practice questions, etc.

[1309] Step 5:

[1310] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The video data is first saved to local storage and then uploaded to a cloud server.

[1311] Step 6:

[1312] Server: A generative AI model runs to analyze the collected video data. The analysis module records student speech using voice recognition and analyzes video to determine student attitudes and progress on assignments. This generates an objective evaluation based on each student's evaluation indicators (level of participation, understanding, cooperation, etc.).

[1313] Step 7:

[1314] Server: The generated evaluation results are sent to the teacher's terminal. The evaluation results are organized for each individual student and presented in a visually easy-to-understand format.

[1315] Step 8:

[1316] Terminal: The teacher reviews the evaluation results and makes corrections as necessary. When reviewing the evaluation results, the teacher makes adjustments based on their own observations to determine whether the evaluations automatically generated by the system are appropriate.

[1317] Step 9:

[1318] Users: Teachers can provide feedback on revised assessment results to students and communicate their grades through individual communication, allowing students to accurately understand their strengths and weaknesses and use this information to guide their next learning steps.

[1319] These are the specific steps in the program. These processes will improve the efficiency of teachers' work and provide appropriate feedback to students.

[1320] Example 1

[1321] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1322] Traditionally, teachers' work required a lot of time for lesson preparation and student evaluation, which could lead to inconsistencies in the quality of education. Furthermore, student evaluations were often subjective and inconsistent. This prevented teachers from concentrating on the essential aspects of education. Furthermore, systems that were inefficient in processing data on lesson content and collecting and analyzing lesson footage were problematic.

[1323] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1324] In this invention, the server includes: means for receiving and preprocessing lesson content data; means for operating a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data; means for transmitting the generated lesson plans and test questions to a terminal of the educational institution; means for collecting lesson videos using a camera installed in the classroom; means for analyzing the collected video data and automatically generating objective evaluations of students; means for transmitting the generated evaluation results to a terminal of the educational institution; means for teachers to confirm and correct the evaluation results received on the terminal; means for the generative AI model to receive teaching material data and preprocess using a text analysis tool; means for operating the generative AI model using prompt statements; means for converting the preprocessed data into JSON format; and means for analyzing the video data using a machine learning algorithm and generating scores based on each student's evaluation indicators. This improves the efficiency of lesson preparation and evaluation work, allowing teachers to focus on the essence of education and ensuring consistency and objectivity in student evaluations.

[1325] "Class content data" refers to teaching materials, textbooks, lecture notes, slides, and other educational materials used by educational institutions in classes.

[1326] "Preprocessing" refers to the process of structuring the received educational data and converting it into a format that is easy for the generative AI model to process, and includes text analysis and the removal of unnecessary data.

[1327] A "generative AI model" refers to an artificial intelligence model that uses machine learning algorithms to automatically generate lesson plans and test questions.

[1328] A "lesson plan" refers to an educational guideline that includes a rough outline of a lesson, such as the content of the lesson, the materials to be used, exercises, and supporting visual materials.

[1329] "Test Questions" refers to exam questions automatically generated by a generative AI model to assess students' understanding and knowledge.

[1330] "Educational institution devices" refers to electronic devices, such as computers and tablets, used within an educational institution to review or correct lesson plans and test questions.

[1331] "Filming equipment" refers to cameras and video recording devices installed in classrooms to collect footage of lessons.

[1332] "Video data" refers to video information recorded by a camera during class, and is the subject of analysis.

[1333] "Objective evaluation" refers to student evaluation indicators automatically generated by a generative AI model based on students' comments, attitudes, and progress on assignments.

[1334] "Evaluation results" refers to the student's evaluation scores and feedback information generated based on the analysis of video data by the generative AI model.

[1335] "Means for checking and correcting on a device" refers to the ability for teachers to check the lesson plans and evaluation results generated using the educational institution's devices and correct the content as necessary.

[1336] "Text analysis tools" refers to software and algorithms used in the preprocessing of educational data to analyze text data and extract specific information.

[1337] A "prompt sentence" is an input sentence used to give instructions to a generative AI model, and refers to a sentence that clearly indicates the specific content and purpose.

[1338] The "JSON format" is a lightweight data format used for data exchange and storage, and is suitable for describing preprocessed data because it has a structure that is easily readable by both humans and computers.

[1339] "Machine learning algorithms" refer to artificial intelligence techniques that learn patterns based on data and make predictions and classifications.

[1340] This invention relates to a system that automates lesson preparation and assessment tasks by utilizing generative AI, aiming to improve the efficiency of teachers' work. This system receives and preprocesses lesson content data, then automatically generates lesson plans and test questions using a generative AI model, and further analyzes classroom video data to objectively assess students.

[1341] The main components of the system consist of a server, terminals, users (teachers), data on lesson content, cameras that collect lesson footage, and generative AI models.

[1342] The server connects to the educational institution's database and receives data on lesson content. For example, when retrieving data from a junior high school second-year mathematics textbook, it receives the teaching material data from a MySQL database. The received data is preprocessed in XML format, and only the necessary information is extracted using a text analysis tool (e.g., SpaCy). During this process, the data is converted to JSON format, which is easy for the generative AI model to handle.

[1343] Based on the preprocessed data, the server runs a generative AI model (e.g., GPT-4). A prompt (e.g., "Based on a second-year junior high school mathematics textbook, please generate a lesson plan and test questions for the introduction of linear equations") is entered, and a lesson plan and test questions are generated. The generated lesson plan includes the lesson progress, materials to be used, practice questions, and supporting visual materials. At the same time, the generative AI model also generates a variety of test questions, from basic to applied.

[1344] The generated lesson plans and test questions are sent to the teacher's device. The teacher's device is an electronic device such as a laptop or tablet, and the teacher uses a dedicated application (e.g., a teacher support app) to review and revise the plans and questions. Using dedicated functions, the teacher can change the slide layout and revise the questions. The order of the tasks can also be changed using drag and drop.

[1345] At the start of a lesson, a camera installed in the classroom automatically starts up and records the lesson video in real time. The camera is installed in a position that can overlook every corner of the classroom, and a series of video data is streamed in real time to a server. The data is temporarily stored and later divided into specific frame intervals for analysis by a generative AI model.

[1346] The server analyzes the collected video data using a generative AI model (e.g., Viso.ai). From the analysis, a machine learning algorithm is used to determine students' comments, attitudes, and progress on assignments. Facial recognition and facial expression analysis are performed to measure each student's level of understanding and concentration. In addition, text analysis is performed on the number of times a student's hand is raised and the content of their comments, and participation is reflected as part of the evaluation. The analysis results are converted into a score based on each student's evaluation indicators (e.g., participation, understanding, and cooperation).

[1347] The generated evaluation results are sent to the teacher's device. The teacher can then view the evaluation results in a dedicated application on the device and check each student's score and analysis results. If necessary, the teacher can manually correct the evaluation and add comments. This result is saved as the final version and provided to each student as feedback.

[1348] Teachers can plan individual consultations and supplementary lessons based on students' levels of understanding and participation. They can also communicate the results of the assessment to students through a dedicated application and provide specific advice based on the results.

[1349] In this way, the present invention allows teachers to improve the efficiency of lesson preparation and assessment work, and to focus on communication with students. Furthermore, by improving the consistency and objectivity of assessment, students' learning progress can be accurately grasped.

[1350] For example, here is a prompt for a teacher to "introduce linear equations" to an eighth grade math class:

[1351] "Generate a lesson plan and test questions for 'Introduction to Linear Equations' based on an eighth grade mathematics textbook. The lesson plan should include the lesson progression, materials to be used, practice problems, and supporting visual materials. The test questions should include both basic and applied problems."

[1352] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1353] Step 1:

[1354] The server receives data on lesson content from the educational institution's database. For example, data on a junior high school second-year mathematics textbook is retrieved from a MySQL database. Based on the received data, the server uses a text analysis tool (e.g., SpaCy) to extract the necessary information and remove unnecessary parts. This preprocessing converts the data into a format (JSON) that is easy for the generative AI to handle.

[1355] Input: Course content data stored in the institution's database

[1356] Output: Preprocessed data in JSON format

[1357] Step 2:

[1358] The server runs a generative AI model (e.g., GPT-4) based on the preprocessed data. It generates a lesson plan and test questions based on a prompt ("Generate a lesson plan and test questions for the introduction of linear equations based on a second-year junior high school mathematics textbook."). The generated results include the lesson plan, materials to be used, practice questions, and supporting visual materials.

[1359] Input: Preprocessed JSON format data, prompt statement

[1360] Output: Auto-generated lesson plans and test questions

[1361] Step 3:

[1362] The server then sends the generated lesson plans and test questions to the educational institution's terminals, which can be electronic devices such as teachers' laptops or tablets. After sending, teachers can use a dedicated application to review these materials and make any necessary corrections. For example, they can change the layout of slides or modify the content of test questions.

[1363] Input: Auto-generated lesson plans and test questions

[1364] Output: Lesson plans and test questions sent to the teacher's device

[1365] Step 4:

[1366] When a lesson starts, the server automatically activates the cameras installed in the classrooms. These cameras record the lesson video in real time and stream it to the server, where the video data is temporarily stored.

[1367] Input: Real-time classroom video

[1368] Output: Video data of lessons stored on the server

[1369] Step 5:

[1370] The server analyzes the collected video data using a generative AI model. Machine learning algorithms (e.g., Viso.ai) are used to analyze the video data and determine student speech, behavior, and task performance. This analysis generates a score based on facial recognition and facial expression analysis to assess student engagement and comprehension.

[1371] Input: Video data of lessons stored on the server

[1372] Output: Student evaluation scores

[1373] Step 6:

[1374] The evaluation results generated by the server are then sent back to the teacher's device. The teacher can then check the results using a dedicated application on the device and manually correct them as necessary. The corrected evaluation results are saved as the final version and provided as feedback to the students.

[1375] Input: Generated student assessment scores

[1376] Output: Evaluation results sent to the teacher's terminal, evaluation results corrected by the teacher

[1377] Step 7:

[1378] The user (teacher) provides feedback on the evaluation results to students through a dedicated application. Based on the evaluation results, the user can plan individual interviews and supplementary lessons and provide specific advice to students.

[1379] Input: Evaluation results corrected by the teacher

[1380] Output: Individual feedback and advice to students

[1381] (Application example 1)

[1382] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1383] In conventional educational systems, teachers expend a great deal of effort preparing lessons and evaluating students. Furthermore, even in online classes, there are few means for evaluating students' understanding and participation in real time and providing feedback. This increases the burden on teachers and leads to issues such as a lack of communication with students. This invention aims to solve these issues and improve the efficiency of teachers' work and the quality of lessons.

[1384] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1385] In this invention, the server includes means for receiving and preprocessing data on lesson content, means for operating a generative AI model for automatically generating lesson plans and test questions based on the preprocessed data, means for transmitting the generated lesson plans and test questions to a terminal of the educational institution, means for collecting lesson video using a camera installed in the classroom, means for analyzing the collected video data and automatically generating objective evaluations of students, means for transmitting the generated evaluation results to a terminal of the educational institution, means for providing feedback to students according to their learning progress using the generative AI model, means for distributing online lesson content based on the automatically generated lesson plans and test questions, and means for recording and evaluating lessons in real time via an application installed on a smartphone or tablet. This reduces the workload of teachers and enables them to grasp students' levels of understanding and participation in real time and provide feedback.

[1386] "Course content data" refers to a set of information related to teaching materials and courses provided by educational institutions.

[1387] "Preprocessing" refers to a series of operations that prepare lesson content data into a format that is easy for the generative AI model to handle.

[1388] "Generative AI models" refer to algorithms or software that use artificial intelligence technology to automatically generate lesson plans and test questions.

[1389] A "lesson plan" is a detailed plan that includes the lesson content, materials to be used, exercises, supporting visual materials, etc.

[1390] "Test questions" refer to questions created by generative AI models to assess comprehension and proficiency of lesson content.

[1391] "Educational institution device" refers to an electronic device, such as a computer or tablet, owned by an educational institution.

[1392] "Camera" refers to video recording equipment installed in the classroom to collect footage during lessons.

[1393] "Video data" refers to video files and related information captured by a camera during class.

[1394] "Objective evaluation" refers to a quantitative evaluation calculated by a generative AI model based on students' comments, attitudes, and progress on assignments.

[1395] "Students" refers to students who participate in classes.

[1396] "Learning progress" refers to the degree to which a participant has progressed in the knowledge and skills they have acquired through classes and learning materials.

[1397] "Feedback" refers to advice and suggestions provided by a generative AI model based on a student's learning progress and level of understanding.

[1398] The present invention is a system for automating lesson preparation and evaluation tasks, and detailed embodiments thereof are described below. The system includes, as its main components, a server, terminals, users (teachers), data on lesson content, a camera for collecting lesson footage, and a generative AI model.

[1399] First, the server receives data on lesson content from the educational institution's database. For example, data from a mathematics textbook for second-year junior high school students. Next, the received data is preprocessed and converted into a format that is easy for the generative AI model to process. This ensures that the teaching material data is properly structured.

[1400] The server runs a generative AI model based on the preprocessed data to automatically generate lesson plans and test questions. For example, let's take a lesson plan for "Introduction to Linear Equations." This lesson plan includes the lesson progress, materials to be used, practice questions, and supporting visual materials. The generative AI model also generates test questions, including "basic and applied problems involving linear equations."

[1401] The generated lesson plans and test questions are sent to the educational institution's terminals, where teachers can review them and make corrections as necessary.

[1402] During lessons, cameras installed in the classroom are turned on and capture video footage of the lesson in real time. The server temporarily stores the collected video data and then analyzes it using a generative AI model. Specifically, it assesses students' comments, attitudes, and progress on assignments, and automatically generates an objective evaluation. Evaluation criteria include participation, understanding, and cooperation.

[1403] The generated evaluation results are sent to the teacher's device. The teacher reviews the results and makes corrections as necessary. The corrected evaluation results are saved as the final version and feedback is provided to the students.

[1404] Additionally, feedback is automatically provided using a generative AI model according to the student's learning progress, allowing appropriate learning support to be provided to each individual student.

[1405] The system also has a function for distributing online lesson content. It automatically generates lesson plans and test questions, enabling effective online lessons for students. It also allows lessons to be recorded and evaluated in real time via an application installed on a smartphone or tablet.

[1406] For example, if you are teaching second-year junior high school maths classes about "Introduction to Linear Equations," you might input the following prompts into the generative AI model:

[1407] Example prompt sentence:

[1408] "Generate lesson plans and test questions for an introduction to linear equations based on eighth grade mathematics materials. Include concrete examples, visual aids, and practice problems in your lesson plans."

[1409] This system will enable teachers to significantly reduce the time they spend preparing lessons and assessing them, allowing them to spend more time communicating with students, resulting in improved quality of education and more efficient teacher work.

[1410] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1411] Step 1:

[1412] The server receives data on lesson content from the educational institution's database. For example, it retrieves data on a mathematics textbook for second-year junior high school students. The input here is the educational institution's database, and the output is the received teaching material data. Specifically, it accesses the database through an API and downloads the necessary teaching material data.

[1413] Step 2:

[1414] The server preprocesses the received teaching material data. Preprocessing involves cleaning and tokenizing the text data, converting it into a format that is easy for the generative AI model to process. The input here is the received teaching material data, and the output is the preprocessed teaching material data. Specifically, it converts unstructured data into an appropriate format and removes unnecessary characters and errors.

[1415] Step 3:

[1416] The server runs a generative AI model based on the preprocessed data to automatically generate lesson plans and test questions. The input is the preprocessed teaching material data, and the output is the generated lesson plans and test questions. Specifically, a natural language processing algorithm is used to generate prompt sentences based on the specified theme.

[1417] Step 4:

[1418] The server sends the generated lesson plans and test questions to the educational institution's terminals. The input here is the generated lesson plans and test questions, and the output is the data displayed on the teacher's terminal. Specifically, the data is securely sent via the network so that the teacher can access it.

[1419] Step 5:

[1420] The camera installed in the classroom is activated and collects the lesson video in real time. The input here is the video of the lesson scene, and the output is the collected video data. Specifically, the camera device is activated remotely and recording begins.

[1421] Step 6:

[1422] The server temporarily stores the collected video data and then analyzes it using a generative AI model. The input here is the collected video data, and the output is the analysis results and an objective evaluation of the student. Specifically, video recognition technology is used to analyze the student's comments and behavior and calculate evaluation indicators.

[1423] Step 7:

[1424] The server sends the generated evaluation results to the educational institution's terminal. The input here is the analysis results and the student's objective evaluation, and the output is the evaluation results displayed on the teacher's terminal. Specifically, the evaluation results are sent via the network so that the teacher can check and correct them.

[1425] Step 8:

[1426] Teachers check the evaluation results through their terminals and make corrections as necessary. The input here is the generated evaluation results, and the output is the final evaluation results after corrections. Specifically, teachers check the evaluation results using the terminal interface and make corrections using a dedicated editing tool.

[1427] Step 9:

[1428] The server uses a generative AI model to provide feedback based on the student's learning progress. The input is the student's learning progress data, and the output is the feedback content. Specifically, it generates individual advice and supplementary learning materials based on test results and participation status.

[1429] Step 10:

[1430] The server distributes online lesson content based on the automatically generated lesson plans and test questions. The input here is the generated lesson plans and test questions, and the output is the content of the online lesson. Specifically, the content of the lesson is provided in a live streaming or video-on-demand format, allowing students to access it.

[1431] Step 11:

[1432] Lessons are recorded in real time and evaluated through an application installed on a smartphone or tablet. The input is real-time lesson footage, and the output is the recorded data and its evaluation results. Specifically, lessons are recorded using the mobile device's camera and analyzed using a generative AI model.

[1433] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1434] This invention relates to a system that aims to improve the efficiency of teachers' work by automating lesson preparation and evaluation tasks by utilizing generative AI and an emotion engine, and further improving the quality of evaluations through student emotion recognition.

[1435] System configuration overview

[1436] The system includes the following major components:

[1437] server

[1438] Terminal

[1439] User (teacher)

[1440] Class content data

[1441] Cameras that collect video footage from lessons

[1442] Generative AI Models

[1443] Emotion Engine

[1444] A natural language explanation of the program's processing

[1445] 1. Receiving and preprocessing the teaching material data

[1446] Server: Receives educational material data (e.g., a mathematics textbook for eighth-grade students) from the educational institution's database. The received data contains information in various formats, such as text, charts, and images. Preprocessing is performed to unify this information into a consistent format. Specifically, this includes text analysis, data cleansing, and format conversion.

[1447] 2. Automatic generation of lesson plans and test questions

[1448] Server: The generative AI model is run based on the preprocessed data. The lesson plan generation module is activated to generate a lesson plan for "Introduction to Linear Equations." This lesson plan automatically summarizes the lesson content, materials to be used, practice questions, and supporting visual materials. In addition, the test question generation module is activated to generate test questions including "Basic and applied problems for linear equations."

[1449] 3. Notification to instructors and corrections

[1450] Device: The generated lesson plans and test questions are sent to the teacher's device, where the teacher can review the plans and questions and make any necessary corrections.

[1451] 4. Recording lessons and collecting data

[1452] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The video data is temporarily saved in local storage and then uploaded to the cloud server.

[1453] 5. Video Data Analysis and Emotion Recognition

[1454] Server: A generative AI model runs to analyze the collected video data. The analysis module uses voice recognition to record what students say and analyzes video to determine their attitudes and progress on assignments. Furthermore, an emotion engine runs, analyzing students' facial expressions and tone of voice to recognize emotions, such as joy, sadness, anger, and surprise.

[1455] 6. Generating evaluation results and feedback

[1456] Server: Generates objective evaluations based on each student's evaluation indicators (e.g., participation, understanding, cooperation) and emotion recognition results. The generated evaluation results are organized for each individual student and presented in a visually easy-to-understand format.

[1457] Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can review them and make corrections as necessary. Because the evaluation also includes emotion recognition information, the teacher can assess the student's mental state while still understanding it.

[1458] Users: Teachers can provide feedback on revised assessment results to students and communicate their grades through individual communication, allowing students to accurately understand their strengths and weaknesses and use this information to guide their next learning steps.

[1459] Specific examples

[1460] For example, consider a situation in which a teacher introduces linear equations to second-year junior high school students in a mathematics class.

[1461] 1. Server: Receives the chapters and sections on linear equations from a second-year junior high school mathematics textbook and performs preprocessing.

[1462] 2. Server: Automatically generates lesson plans and test questions based on the preprocessed data and sends them to the teacher's device.

[1463] 3. Terminal: Teachers review lesson plans and test questions and make adjustments as needed.

[1464] 4. Server: During the lesson, the camera collects video footage and stores it on the server.

[1465] 5. Server: Analyzes video data using generative AI and an emotion engine to generate objective evaluations based on students' emotions, comments, and attitudes.

[1466] 6. Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can check and correct them.

[1467] 7. User: Teachers communicate assessment results through feedback to students and engage in individual communication.

[1468] The above is a specific "Mode for carrying out the invention." This system allows teachers to significantly reduce the time they spend preparing lessons and assessing them, allowing them to spend more time communicating with students. By combining emotion recognition, more comprehensive assessments are possible, improving students' learning experiences.

[1469] The processing flow will be explained below.

[1470] Step 1:

[1471] Server: Receives educational material data (e.g., a mathematics textbook for second-year junior high school students) from the educational institution's database. The received data contains information in a variety of formats, including text, charts, and images. Preprocessing is performed to unify this information into a consistent format. Specifically, text analysis, various format conversions, and data cleansing are performed.

[1472] Step 2:

[1473] Server: The generative AI model is run based on the preprocessed data. The lesson plan generation module is activated to generate a lesson plan for "Introduction to Linear Equations." This lesson plan automatically summarizes the lesson content, materials to be used, practice questions, and supporting visual materials. In addition, the test question generation module is activated to generate test questions including "Basic and applied problems for linear equations."

[1474] Step 3:

[1475] Server: Generated lesson plans and test questions are sent to the educational institution's terminals. The data is categorized and tagged for easy access by teachers.

[1476] Step 4:

[1477] Devices: Lesson plans and test questions are sent to teachers' devices for review. Teachers can review the content and make adjustments as needed. Specifically, they can fine-tune the lesson content, add materials, increase or decrease the number of practice questions, etc.

[1478] Step 5:

[1479] Server: During lessons, cameras in the classroom are turned on and capture video footage in real time. The video data is saved in local storage and then uploaded to a cloud server.

[1480] Step 6:

[1481] Server: The collected video data is analyzed using a generative AI model and emotion engine. The generative AI model records students' comments using voice recognition and determines their attitudes and progress on assignments through video analysis. The emotion engine analyzes students' facial expressions and tone of voice to recognize their emotions. For example, emotions such as joy, sadness, anger, and surprise can be recognized.

[1482] Step 7:

[1483] Server: Generates objective evaluations based on each student's evaluation indicators (e.g., participation, understanding, cooperation) and emotion recognition results. The generated evaluation results are organized for each individual student and presented in a visually easy-to-understand format.

[1484] Step 8:

[1485] Terminal: The evaluation results are sent to the teacher's terminal. The teacher can review them and make corrections as necessary. Emotion recognition information is also included, so the teacher can evaluate the student while understanding their mental state.

[1486] Step 9:

[1487] User: Teachers provide feedback to students on the revised assessment results. Students are informed of their grades through individual communication, allowing them to accurately understand their strengths and weaknesses. This feedback helps students take the next step in their studies.

[1488] These are the specific processing steps of the program, which will enable teachers to significantly reduce the time required for lesson preparation and assessment, while also enabling comprehensive assessment that takes students' emotions into account.

[1489] Example 2

[1490] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1491] Teachers at educational institutions spend a great deal of time and effort preparing lessons and conducting assessments. It is particularly difficult to objectively assess students' progress and emotions while understanding their learning progress. They also need to provide students with prompt and accurate feedback. Furthermore, if video data from lessons could be used to assess students' learning status, more efficient educational support would be possible. However, current educational systems lack such functionality. Therefore, a new system is needed to improve teacher efficiency and comprehensively assess students.

[1492] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1493] In this invention, the server includes means for receiving and preprocessing lesson content data, means for operating a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data, means for transmitting the generated lesson plans and test questions to an educational institution's terminal, means for reviewing and correcting the generated lesson plans and test questions on the educational institution's terminal, means for collecting lesson video using a video capture device installed in the classroom, means for analyzing the collected video data and automatically generating objective evaluations of students using the generative AI model and an emotion recognition algorithm, means for transmitting the generated evaluation results to the educational institution's terminal and reviewing and correcting the evaluation results on the educational institution's terminal, and means for providing feedback to students from the educational institution's terminal. This reduces teachers' lesson preparation time and enables accurate evaluation of students' learning progress and emotions. It also allows teachers to quickly review and correct the evaluation results and provide quick and accurate feedback to students.

[1494] "Course content data" means information related to a course offered by an educational institution, including textbooks, slides, visual materials, audio files, video files, and other formats.

[1495] "Preprocessing" refers to a series of steps taken to convert received data into a consistent format, including parsing text, removing unnecessary information, and standardizing formats.

[1496] A "generative AI model" is a model that uses artificial intelligence algorithms to automatically generate lesson plans and test questions from basic data. Specifically, machine learning and neural networks are often applied.

[1497] A "lesson plan" is a written plan prepared by a teacher to conduct a lesson, and includes the content of the lesson, teaching materials to be used, practice questions, supplementary visual materials, etc.

[1498] "Test questions" are questions that are asked to assess a student's level of understanding and learning progress, and include multiple choice questions, essay questions, applied questions, etc.

[1499] "Educational institution device" refers to an electronic device, such as a computer, tablet, or smartphone, owned by an educational institution and used by faculty to review, modify, or access lesson plans and test questions.

[1500] "Filming equipment" refers to devices such as cameras and recording devices installed in classrooms to collect footage of lessons.

[1501] "Generative AI model and emotion recognition algorithm" refers to an algorithm that uses AI technology to analyze students' facial expressions and tone of voice to recognize their emotions, which can then be used to evaluate their learning attitudes and emotions.

[1502] "Objective assessment of students" means evaluating students' learning progress, level of understanding, participation, etc. based on data and algorithms, without relying on the teacher's subjective opinion.

[1503] "Feedback" refers to the act of teachers giving students grades and advice based on the results of their assessments, allowing students to understand their own strengths and weaknesses and use this information to guide their next steps in learning.

[1504] MODE FOR CARRYING OUT THE INVENTION

[1505] This invention aims to improve the work efficiency of teachers in educational institutions by automating lesson preparation and evaluation tasks. Furthermore, it aims to improve the quality of evaluations through student emotion recognition. This system mainly consists of three elements: a server, a terminal, and a user (teacher).

[1506] Receiving and preprocessing teaching material data

[1507] Server: Receives course content data from the educational institution's database. The received data contains a variety of formats, including text, charts, and images, and undergoes preprocessing to unify them into a consistent format. Preprocessing includes text analysis, data cleansing, and format conversion.

[1508] Automatic generation of lesson plans and test questions

[1509] Server: Based on the preprocessed data, a generative AI model is run to automatically generate lesson plans and test questions. The generative AI model uses machine learning algorithms, and algorithms optimized for each subject area are implemented. The generated lesson plans include the lesson progress, materials to be used, practice questions, and supporting visual materials. In addition, a test question generation module automatically generates multiple-choice and essay questions.

[1510] Instructor notification and corrections

[1511] Devices: The generated lesson plans and test questions are sent to the educational institution's devices. Teachers can review them on their devices and make corrections as needed. Corrections can be easily made through the device's editing interface.

[1512] Recording lessons and collecting data

[1513] Server: During lessons, the camera installed in the classroom is turned on and captures video of the lesson in real time. The collected video data is temporarily saved in local storage and then uploaded to the cloud server.

[1514] Video data analysis and emotion recognition

[1515] Server: Runs generative AI models and emotion recognition algorithms to analyze the collected video data. The analysis module uses voice recognition technology to record students' speech and video analysis technology to evaluate their attitudes and progress on assignments. It also uses emotion recognition algorithms to analyze students' facial expressions and tone of voice to recognize their emotions.

[1516] Generation of evaluation results and feedback

[1517] Server: Generates objective evaluations based on evaluation indicators such as student participation, understanding, and cooperation, as well as emotion recognition results. The evaluation results are organized in a visually easy-to-understand format.

[1518] Terminal: The evaluation results are sent to the teacher's terminal, where the teacher can review them and make corrections as necessary. Emotion recognition information is also included, so the teacher can evaluate the student while understanding their mental state.

[1519] Users: Teachers can provide feedback on revised assessment results to students and communicate their grades through individual communication, allowing students to accurately identify their strengths and weaknesses and use them to guide their next learning steps.

[1520] Specific examples

[1521] For example, consider a lesson on "Introduction to Linear Equations" in mathematics for second-year junior high school students. First, the server extracts the chapters and sections on linear equations from the second-year junior high school mathematics textbook and performs preprocessing. Next, a generative AI model is run based on the preprocessed data, automatically generating lesson plans and test questions. The generated plans and questions are sent to the teacher's device, who reviews and modifies them. After that, cameras installed during the lesson collect footage of the lesson, and the server analyzes the data. The analysis results are objectively evaluated based on the students' emotions and comments, and the evaluation results are sent to the teacher's device. The teacher reviews and modifies these evaluation results and provides feedback to the students.

[1522] Prompt Sentence Examples

[1523] "Generate a lesson plan for introducing linear equations.

[1524] Lessons should include basic conceptual explanations, examples, exercises, and supporting visual materials.

[1525] The system of this invention can significantly reduce the time required for teachers to prepare and evaluate lessons, and improve the learning experience for students. By combining it with emotion recognition, more comprehensive and objective evaluation can be achieved, supporting the growth of each student.

[1526] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1527] Program processing flow

[1528] Step 1: Receiving and preprocessing the teaching material data

[1529] server:

[1530] Input: Course content data obtained from an educational institution's database (e.g., an XML file of a second-year junior high school mathematics textbook)

[1531] Processing: Parse the data and convert it into a consistent format. This involves text analysis, removal of unnecessary tags, and standardization of formats.

[1532] Output: Preprocessed data (JSON formatted lesson content data)

[1533] Specific behavior:

[1534] The server accesses the educational institution's database and retrieves the data for the specified course content. If the data is in XML format, it first parses it to extract the textbook content into text format. It then removes unnecessary tags and noise and converts it into a consistent JSON format.

[1535] Step 2: Automatically generate lesson plans and test questions

[1536] server:

[1537] Input: Preprocessed lesson content data (JSON format)

[1538] Processing: Run generative AI models to generate lesson plans and test questions, using machine learning algorithms and domain-specific optimized models.

[1539] Output: Auto-generated lesson plans and test questions (JSON or text format)

[1540] Specific behavior:

[1541] The server inputs the preprocessed lesson content data into a generative AI model. Using advanced machine learning algorithms such as the GPT-4 model, it automatically generates a lesson plan for "Introduction to Linear Equations." This plan includes teaching materials, a progress schedule, practice questions, and supplementary materials for teachers to use. A test question generation module also runs simultaneously, generating multiple-choice and essay questions to assess comprehension.

[1542] Step 3: Notify instructor and make corrections

[1543] Device:

[1544] Input: Generated lesson plans and test questions (JSON or text format)

[1545] Processing: The teacher will review these on the system and make corrections as necessary.

[1546] Output: Modified lesson plans and test questions (JSON or text format)

[1547] Specific behavior:

[1548] The generated lesson plans and test questions are sent to the teacher's device, where the teacher can review the content on the device's display screen and make any necessary changes using the editing interface, such as changing the format of a particular question or adjusting the lesson schedule.

[1549] Step 4: Recording lessons and collecting data

[1550] server:

[1551] Input: Video data of lessons collected in real time

[1552] Processing: During lessons, video footage is collected using video recording equipment installed in the classroom. The video data is temporarily stored in local storage and later uploaded to a cloud server.

[1553] Output: Lesson video data stored in cloud storage

[1554] Specific behavior:

[1555] The camera automatically starts recording as soon as the lesson begins, capturing the entire lesson. The recorded video is temporarily saved in local storage. After the lesson ends, the video data is uploaded to cloud storage by the server.

[1556] Step 5: Video data analysis and emotion recognition

[1557] server:

[1558] Input: Video data from lessons stored in cloud storage

[1559] Processing: Using generative AI models and emotion recognition algorithms, the video data is analyzed to assess student speech, attitudes, and emotions.

[1560] Output: Student objective evaluation indicators (evaluation data in JSON format)

[1561] Specific behavior:

[1562] The server retrieves the video data from cloud storage and launches the analysis module. It uses a speech recognition API to transcribe what students say, applies a posture analysis model to evaluate their attitude and progress on assignments, and uses a facial expression analysis API to recognize emotions from students' facial expressions and tone of voice.

[1563] Step 6: Generating evaluation results and feedback

[1564] server:

[1565] Input: Student objective evaluation indicators (evaluation data in JSON format)

[1566] Processing: Generate objective evaluations based on each student's participation, understanding, cooperation, and other evaluation indicators, as well as emotion recognition results. Organize the results in a visually easy-to-understand format.

[1567] Output: Evaluation results (visualized evaluation report)

[1568] Specific behavior:

[1569] The system uses k-means clustering to classify student assessment data and visualize it in a dashboard format. Assessment reports are organized for each student and are created in a visually easy-to-understand format, including supporting materials.

[1570] Device:

[1571] Input: Evaluation results (visualized evaluation report)

[1572] Processing: The teacher reviews the evaluation results and makes corrections if necessary.

[1573] Output: Corrected evaluation results (final evaluation report)

[1574] Specific behavior:

[1575] Teachers can open the evaluation report on their devices, make any necessary corrections, review the results, and add specific comments about students' strengths and weaknesses.

[1576] User:

[1577] Input: Corrected evaluation results (final evaluation report)

[1578] Processing: Teachers provide feedback to students and communicate the results of the assessment.

[1579] Output: Assessment feedback that students can understand

[1580] Specific behavior:

[1581] Teachers meet with students individually to explain the results of their assessments in detail, allowing students to understand their strengths and weaknesses and use the feedback to plan their future learning.

[1582] This system not only significantly reduces the time required for teachers to prepare and evaluate lessons, but also enables more comprehensive and objective evaluation, improving students' learning experiences and supporting the growth of each individual student.

[1583] (Application example 2)

[1584] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1585] In traditional education systems, teachers have to expend a great deal of effort preparing lessons and evaluating students, making efficient lesson management difficult and potentially resulting in a decline in the quality of education. Furthermore, in brick-and-mortar stores, it is difficult for employees to immediately grasp and respond to customer emotions, hindering customer satisfaction. To solve these issues, it is necessary to utilize generative AI models and emotion engines to automate lesson preparation and evaluation, and further improve the quality of customer service in brick-and-mortar stores.

[1586] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1587] In this invention, the server includes means for receiving and preprocessing lesson content data, means for running a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data, means for transmitting the generated lesson plans and test questions to a terminal at the educational institution, means for collecting lesson video from a camera installed in the classroom, means for analyzing the collected video data and automatically generating objective evaluations of students, means for transmitting the generated evaluation results to a terminal at the educational institution, means for application to a customer service robot that recognizes customer emotions in real time and automatically adjusts responses, means for automatically adjusting customer service scripts and generating messages based on the emotion recognition results, and means for accumulating emotion data and utilizing it for future customer service. This enables the efficiency of ...

Claims

1. A means for receiving and preprocessing lesson content data; means for operating a generative AI model to automatically generate lesson plans and test questions based on the preprocessed data; and means for transmitting the generated lesson plans and test questions to a terminal of the educational institution; A means of collecting lesson footage through cameras installed in classrooms, A means to analyze the collected video data and automatically generate objective evaluations of students; The system includes means for transmitting the generated assessment results to a terminal at the educational institution.

2. 2. The system according to claim 1, further comprising means for enabling the educational institution to check and correct the generated lesson plans and test questions after they have been sent to the educational institution's terminal.

3. The system of claim 1, further comprising means for the generative AI model that generates lesson plans and test questions based on the preprocessed data to use an algorithm optimized for each subject.

4. The system according to claim 1, further comprising means for analyzing the student's comments, attitude, and task performance from the collected video data.

5. 2. The system according to claim 1, further comprising means for enabling the educational institution to confirm and correct the evaluation results of the students.

Images