system

Abstract

Provide a system. [Solution] a means for accepting questions entered by students; A means for analyzing the audio and video data of recorded lecture videos; A means for understanding the question content based on the analyzed lecture content and generating an appropriate answer; a means for transmitting the generated answers to the student; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The declining birthrate and the resulting rise in education costs have become a social issue. While education costs continue to rise, the declining birthrate has created a need to provide high-quality education while reducing costs. Furthermore, with on-demand recorded lessons, students are unable to ask questions on the spot, limiting opportunities for resolving their doubts. Furthermore, if there are a large number of questions, the burden on the instructor increases, potentially affecting the quality of the education. Therefore, in order to improve the quality of recorded lessons, a system is needed that allows students to easily ask questions at any time and provides prompt answers. [Means for solving the problem]

[0005] This invention relates to a system that includes a means for accepting questions entered by students, a means for analyzing audio and video data from recorded lecture videos, a means for understanding the question based on the analyzed lecture content and generating an appropriate answer, and a means for sending the generated answer to the student. The system also includes a means for authenticating student access and a means for structuring data acquired from the lecture videos and storing it in a database. This allows students to ask questions at any time, and AI can quickly and accurately generate answers to instantly resolve their questions. This reduces the burden on instructors and reduces education costs while maintaining the quality of education.

[0006] A "student" is someone who watches and learns from a recorded lecture video.

[0007] "Means for accepting questions" refers to the interface or function for recognizing questions entered by students and incorporating them into the system.

[0008] "Lecture video" refers to video and audio data of recorded educational content.

[0009] "Means for analyzing audio and video data" refers to the technologies and mechanisms for converting the audio data of lecture videos into text and extracting useful information from the video data.

[0010] "Lecture content" refers to the entire body of knowledge and information conveyed by the instructor in class.

[0011] "Means for understanding the content of the question" refers to the process and techniques for analyzing the question entered by the student and grasping its intent and main point.

[0012] "Means for generating appropriate answers" refers to algorithms and technologies for constructing the most appropriate answer to the question.

[0013] "Means for sending answers to students" refers to communication means or interfaces for promptly transmitting generated answers to students.

[0014] "Means for access authentication" refers to the mechanism used to verify and authenticate a student's identity when logging into the system.

[0015] A "database" is a place or system where data such as video lectures, analyzed lecture content, and questions and answers from students are stored in a centralized manner.

[0016] "Structuring" refers to the process of organizing and formatting data so that it can be easily processed by a system.

[0017] "Educational costs" refers to all expenses required for education, including instructor salaries, facility fees, and costs for creating teaching materials. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention relates to a system that uses AI technology to instantly answer questions from students about recorded lecture videos. The system includes the following specific elements and processes:

[0040] System configuration

[0041] 1. Server: A central management system that stores data, analyzes data, and handles question and answering.

[0042] 2. Device: Provides an interface for students to watch recorded lessons and enter questions on their devices (PCs, tablets, smartphones, etc.).

[0043] 3. Users: Students who watch recorded lessons and ask questions.

[0044] Overview of system program processing

[0045] The program processing of this system is explained in natural language below.

[0046] 1. Student Access Authentication

[0047] User: Enter your user ID and password on the login screen.

[0048] Terminal: Sends the entered authentication information to the server.

[0049] Server: Authenticates the user against the database, and if successful, returns a list of lecture videos to the user.

[0050] 2. Start streaming recorded lecture videos

[0051] User: Select the lecture video they want to watch.

[0052] Device: Sends a request for the selected video to the server.

[0053] Server: Retrieves the relevant video from the video database and starts streaming.

[0054] 3. Analysis of lesson content

[0055] Server: Passes the audio and video portions of the lesson video to the multimodal AI.

[0056] AI model (inside the server): Converts audio into text (voice recognition) and analyzes blackboard notes and slides in the video to convert them into text (image analysis).

[0057] Server: Structures the parsed data and stores it in a database.

[0058] 4. Questions

[0059] User: If a question arises during the lesson being viewed, enter the question in the text box.

[0060] Terminal: Sends the entered question to the server.

[0061] 5. Question Answering

[0062] Server: Receives questions from users and analyzes the content of the questions.

[0063] AI model (inside the server): Compares the analyzed questions with pre-structured lecture content and generates appropriate answers.

[0064] Server: Sends the generated answer to the user's device.

[0065] User: Check the AI's answer sent back from the device.

[0066] To give a specific example, if a student doesn't understand how to calculate consumption tax and asks, "How do you calculate consumption tax?", the server receives the question and passes it to the AI ​​model. The AI ​​model searches for relevant parts of the lesson content and generates an answer: "Consumption tax is the purchase price multiplied by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen item, the consumption tax will be 100 yen." The server then sends this to the user's device. The user can instantly check the appropriate answer and resolve their question.

[0067] This system allows students to resolve their questions on the spot, significantly improving learning efficiency. In addition, since instructors do not need to respond directly to students' questions, the burden on them is reduced, making it possible to reduce education costs while maintaining the quality of education.

[0068] The processing flow will be explained below.

[0069] Step 1: Student Access Authorization

[0070] User: Enter your user ID and password on the login screen.

[0071] Terminal: Sends the entered authentication information to the server.

[0072] Server: Authenticates the user against the information stored in the database, and if authentication is successful, returns access rights and a list of available lecture videos to the user. If authentication fails, displays an error message.

[0073] Step 2: Select the recorded lesson video and start streaming

[0074] User: Select the video they want to watch from the list of lecture videos provided.

[0075] Terminal: Sends a request for the selected lecture video to the server.

[0076] Server: Retrieves the selected lecture video from the database and sends it to the user's device in streaming format. The server also sends the video playback position information.

[0077] Step 3: Analysis of lesson content

[0078] Server: The recorded lesson video is sent to the multimodal AI. The audio portion of the video is processed through speech recognition and converted into text (speech-to-text).

[0079] Server: The contents of the slides and whiteboards are subjected to image analysis processing from the video footage to extract important information (video analysis).

[0080] Server: Integrates data obtained from audio-to-text and video analysis, and stores the lecture content in a database as structured data.

[0081] Step 4: Ask a question

[0082] Users: If they have any questions while watching a recorded lesson, they can enter their questions in the dedicated text box.

[0083] Terminal: Sends the entered question to the server.

[0084] Step 5: Parsing the question and generating an answer

[0085] Server: Receives questions sent by students and analyzes the content of the questions using natural language processing (NLP) technology.

[0086] AI model (on the server): Using the analyzed question content, the model searches a pre-structured lecture content database to obtain relevant information, and then generates the most appropriate answer from the obtained relevant information.

[0087] Server: Sends the generated answers to the students' devices.

[0088] Step 6: View and review your answers

[0089] User: Checks the submitted answers on the device, resolves any doubts, and enters additional questions as necessary.

[0090] Step 7: Feedback and Data Storage (Optional)

[0091] Server: Logs all questions and answers into a database for later review by the instructor.

[0092] Instructor: Analyze the topics that students focus on asking questions about and the AI's answers, and prepare additional teaching materials and supplementary lectures as necessary.

[0093] The above processing steps allow students to ask questions immediately while watching recorded lessons, and AI can provide quick answers, improving learning efficiency. This also reduces the burden on instructors and contributes to maintaining the quality of education.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] In conventional systems that provide recorded lecture videos, it is difficult for students to get immediate answers to questions that arise while watching the video. This reduces learning efficiency and places a burden on the instructor. Therefore, there is a need for a system that allows students to solve their questions in real time while watching the recorded video.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes a means for accepting questions entered by students, a means for analyzing audio and video data of recorded lecture videos, a means for understanding the question content based on the analyzed lecture content and using a generative AI model to generate an appropriate answer, and a means for transmitting the generated answer to the student, thereby enabling the student to solve their question in real time while watching the recorded video.

[0099] "Student" refers to a person who watches and learns from lecture videos provided online or offline.

[0100] "Means for accepting questions" refers to the interface or protocol that receives questions entered by students and sends them to the server.

[0101] "Lecture video" refers to audio and video data containing recorded lecture content.

[0102] "Means for analyzing audio and video data" refers to technology that converts the audio of lecture videos into text and extracts and analyzes visual information from the video.

[0103] A "generative AI model" refers to an artificial intelligence model that generates appropriate answers based on input questions and data.

[0104] "Means for generating answers" refers to technology that generates appropriate answers to students' questions based on the lecture content analyzed using a generative AI model.

[0105] "Means for sending answers" refers to the communication technology or protocol used to send the generated answers to the student's device.

[0106] "Access Authentication Measures" means the system or method that verifies a student's credentials and verifies access rights.

[0107] "Encryption methods" refers to the technologies and protocols that encrypt information to protect data transmissions.

[0108] "Structured data" refers to data that has been systematically organized from analyzed lecture video information and stored in a database.

[0109] "Speech recognition means" refers to technology that automatically converts voice data into text.

[0110] "Image analysis means" refers to technology that extracts specific visual information from video data and converts it into text.

[0111] This invention relates to a system that uses AI technology to instantly answer questions posed by students about recorded lecture videos. The system consists of three main components: a server that functions as a central management system, terminals where students can watch lecture videos and enter questions, and users who watch the recorded lectures.

[0112] System configuration

[0113] 1. Server:

[0114] The server stores data, analyzes it, and handles question and answering. Specifically, it uses the following hardware and software:

[0115] Database: Used to store authentication information and lecture video data.

[0116] Generative AI model: For example, GPT-4 (registered trademark) is used to generate appropriate answers based on the question.

[0117] Speech recognition technology: The audio portion of the lecture video is converted into text using the Google (registered trademark) Speech-to-Text API.

[0118] Image analysis technology: The video portion of the lecture video is analyzed using Google Cloud Vision API and converted into text.

[0119] 2. Terminal:

[0120] The device used by students (PC, tablet, smartphone, etc.) provides an interface for viewing recorded lessons and entering questions. The device has the following functions:

[0121] Login interface: An interface for entering a user ID and password.

[0122] Video playback function: Lecture videos retrieved from the server are played in streaming format.

[0123] Question input interface: An interface for students to enter their questions into a text box.

[0124] 3. User:

[0125] Students watch the recorded lectures and ask questions.

[0126] Specific actions

[0127] Below, we will explain how this system works with a concrete example.

[0128] 1. User access authentication

[0129] The user enters their user ID and password on the login screen.

[0130] The terminal encrypts the entered authentication information and sends it to the server.

[0131] The server checks the authentication information stored in the database and, if authentication is successful, returns a list of lecture videos to the user.

[0132] 2. Start streaming recorded lecture videos

[0133] The user selects the lecture video they wish to watch from the list.

[0134] The terminal sends a request for the selected video to the server.

[0135] The server retrieves the relevant video from the video database and streams it.

[0136] 3. Analysis of lesson content

[0137] The server passes the audio and video portions of the lesson video to the multimodal AI. The audio portion is converted to text using the Google Speech-to-Text API, and the video portion is analyzed and converted to text using the Google Cloud Vision API.

[0138] The server structures the parsed data and stores it in a database.

[0139] 4. Questions

[0140] If a user has a question about a lesson they are watching, they can enter it in the text box.

[0141] The terminal transmits the entered question to the server.

[0142] 5. Question Answering

[0143] The server receives a question from a user and analyzes the question.

[0144] The generative AI model matches the analyzed questions with pre-structured lecture content and generates appropriate answers.

[0145] The server sends the generated answer to the user's terminal.

[0146] The user checks the AI's answers sent back from the device.

[0147] Specific examples

[0148] For example, if a student does not understand how to calculate consumption tax, he or she can input the question, "How do you calculate consumption tax?" In response to this question, the system operates as follows:

[0149] User: Enters the question "How do you calculate sales tax?" into the text box.

[0150] Terminal: Sends entered text to the server.

[0151] Server: Receives the question and sends the prompt "How do you calculate sales tax?" to the generative AI model.

[0152] Generative AI model: Searches the lecture content and generates the answer, "Consumption tax is the purchase amount multiplied by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen product, the consumption tax will be 100 yen."

[0153] Server: Sends the generated answer to the user's device.

[0154] Terminal: Receives the answer and displays it to the user.

[0155] User: Review the answers provided and understand how sales tax is calculated.

[0156] This system allows students to get answers to their questions in real time, improving learning efficiency and reducing the burden on instructors.

[0157] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0158] Step 1:

[0159] Authenticating users for access

[0160] 1. Input: The user enters their user ID and password on the login screen.

[0161] 2. Operation: The device encrypts the entered authentication information and sends it to the server using a protocol such as SSL / TLS.

[0162] 3. Input: The server checks the received authentication information against the information stored in its database.

[0163] 4. Data processing: The server authenticates the user based on the matching result. If authentication is successful, the user information is saved in the session.

[0164] 5. Output: The server generates a lecture video list for the user and sends it to the terminal.

[0165] 6. Operation: The terminal displays a list of lecture videos.

[0166] Step 2:

[0167] Streaming of recorded lecture videos begins

[0168] 1. Input: The user selects the lecture video they want to watch from the list.

[0169] 2. Action: The device sends a request for the selected video to the server.

[0170] 3. Input: The server retrieves the relevant video data from the video database.

[0171] 4. Data processing: The server converts the video data into a streaming format, often using tools such as FFmpeg.

[0172] 5. Output: The server sends the converted streaming data to the terminal.

[0173] 6. Operation: The device plays the received streaming data.

[0174] Step 3:

[0175] Analysis of lesson content

[0176] 1. Input: The server extracts the audio and video parts of the lesson video.

[0177] 2. How it works: The server passes the audio portion to the Google Speech-to-Text API for speech recognition.

[0178] 3. Input: The server receives the text data obtained by speech recognition.

[0179] 4. Data processing: The audio data is saved as text along with time information.

[0180] 5. Operation: The server passes the video portion to the Google Cloud Vision API for image analysis.

[0181] 6. Input: The server receives the text data obtained from image analysis.

[0182] 7. Data processing: The information on the board and slides in the video is saved as text along with time information.

[0183] 8. Output: The server structures the acquired text data and stores it in a database.

[0184] Step 4:

[0185] Questions accepted

[0186] 1. Input: When a user has a question about the lesson they are watching, they can enter it in the text box.

[0187] 2. Operation: The terminal sends the entered question data to the server.

[0188] 3. Input: The server adds the received query data to a queue for analysis.

[0189] 4. Output: The question data added to the queue is passed to the next question-answering process.

[0190] Step 5:

[0191] Question and Answering

[0192] 1. Input: The server retrieves the question data in the queue.

[0193] 2. Operation: The server analyzes the question using NLP (Natural Language Processing) technology.

[0194] 3. Input: The server sends the parsed question content to the generative AI model as a prompt sentence.

[0195] 4. Output: Based on a specific prompt, such as "How do you calculate sales tax?", the generative AI model generates an answer.

[0196] 5. Input: Receive the answer from the generative AI model.

[0197] 6. Data processing: The server structures the answer and sends it to the user's device.

[0198] 7. Operation: The terminal receives the generated response and displays it to the user.

[0199] 8. Output: The user checks the answer displayed on the terminal and resolves their doubt.

[0200] (Application example 1)

[0201] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0202] With conventional video lecture systems, students had no way to ask questions and get answers immediately when they had doubts. This resulted in problems such as unresolved questions and reduced learning efficiency. Furthermore, it was a significant burden on instructors to deal with questions afterward, making it difficult to provide an efficient education.

[0203] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0204] In this invention, the server includes a means for accepting questions entered by students, a means for analyzing audio and video data from recorded lecture videos, and a means for understanding the questions based on the analyzed lecture content and generating appropriate answers. This allows the generated answers to be sent immediately to students. The server also includes a means for using a mobile information terminal that enables viewing of recorded lectures and question entry, a means for performing speech recognition and question answering using a generative AI model, and a means for using a central management system that receives, processes, and transmits various data. This allows students to instantly resolve their questions using mobile information terminals such as smartphones and tablets, improving their learning efficiency.

[0205] "Students" are learners who watch recorded lecture videos and enter questions.

[0206] "Means for accepting questions" refers to an interface for accepting questions entered by students and sending them into the system.

[0207] A "lecture video" is a file containing video and audio data of recorded educational content.

[0208] "Means for analyzing audio and video data" refers to the technical methods for extracting audio and video information from recorded lecture videos, converting it into text, and performing semantic analysis.

[0209] "Means of understanding the question and generating an appropriate answer" refers to a function that analyzes questions sent by students, compares them with related lecture content, and automatically creates the most appropriate answer.

[0210] "Means for sending generated answers" refers to a mechanism for sending answers generated by AI to students' devices in real time.

[0211] "Mobile information terminal" refers to a portable information processing device such as a smartphone or tablet.

[0212] "Generative AI model" refers to a learning model that uses artificial intelligence to perform speech recognition, natural language processing, and question answering.

[0213] "Central management system" refers to the server and related infrastructure for centrally managing the reception and processing of questions from students, the analysis of lecture videos, and the generation and transmission of answers.

[0214] This invention relates to a system that uses a generative AI model to instantly answer questions posed by students about recorded lecture videos. This system is realized by the following specific components and the mechanisms by which these components work together.

[0215] System configuration

[0216] 1. Server: As a central management system, it accepts questions from students, analyzes recorded lecture videos, generates questions and answers, and sends the generated answers.

[0217] 2. Devices: Mobile information devices such as smartphones and tablets used by students provide an interface for watching lecture videos and entering questions.

[0218] 3. User: A student who watches the recorded lecture and enters questions.

[0219] Program processing description

[0220] Login Authentication

[0221] The server receives the user ID and password sent by the student from their device and checks them against a database. If authentication is successful, it returns a list of lecture videos to the device. For security reasons, this process uses encrypted communication such as SSL / TLS.

[0222] Streaming lecture videos

[0223] When a user selects a lecture video to watch on their device, the server retrieves the corresponding video data from the video database and starts streaming it to the device. The video data is saved in standard formats such as MP4 and WebM.

[0224] Analysis of lesson content

[0225] The server passes the audio and video portions of the recorded lecture video to a generative AI model, which performs speech recognition to convert the audio into text and analyzes the blackboard notes and slides in the video to generate text. The analyzed data is structured and stored in a database. This process uses natural language processing libraries (e.g., HuggingFace Transformers) and image analysis software.

[0226] Question reception and answering

[0227] When a user has a question about a lesson they are watching, they send it in text format from their device to the server. The server then passes the received question to a generative AI model, which analyzes the question and generates an appropriate answer. The generated answer is then immediately sent back to the device. This allows students to receive answers to their questions in real time.

[0228] Specific examples of programs

[0229] Authentication Example

[0230] When a user enters "User ID: student1" and "Password: password1" on the login screen on their smartphone and sends it to the server, the server checks the database and, if the authentication information is correct, returns a list of lecture videos.

[0231] Example questions and answers

[0232] When a user inputs the question, "How do you calculate economic growth rate?", the generative AI model searches for relevant lecture content and generates the answer, "Economic growth rate is an indicator that shows how much a country's economy has grown, and specifically, it is calculated as the annual rate of change in gross domestic product (GDP)." The server then sends this answer to the user's device.

[0233] Example prompt sentence:

[0234] How do you calculate economic growth rate?

[0235] In this way, this invention allows students to resolve questions in real time while watching recorded lessons, improving learning efficiency and reducing the burden on instructors. Furthermore, this system is highly convenient because it uses mobile information terminals such as smartphones and tablets.

[0236] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0237] Step 1:

[0238] The server accepts the authentication information of the user ID and password entered by the student using the terminal and compares it with the database. The input is the user ID and password, and the output is the authentication result. If the authentication is successful, the server returns a list of lecture videos to the terminal. Specifically, the authentication process is carried out securely using SSL / TLS encrypted communication.

[0239] Step 2:

[0240] The server receives the lecture video ID requested by the student from the terminal and retrieves the video data corresponding to that ID from the video database. The input is the video request information (video ID), and the output is the streaming URL of the corresponding lecture video. Specifically, the server queries the video database, generates a video URL, and sends it to the terminal.

[0241] Step 3:

[0242] The server passes the audio and video data of the lecture video to the generative AI model for analysis. The input is the audio and video data of the lecture video, and the output is the text of the lecture content. Specifically, the audio data is converted into text using voice recognition software, and at the same time, the text and slides in the video are converted into text using image analysis software. The analyzed data is then structured and stored in a database.

[0243] Step 4:

[0244] When a student has a question during a class they are watching, they use their device to enter the question into a text box and send it to the server. The input is the question text entered by the student, and the output is a confirmation by the server that the question has been received. Specifically, the question is entered into the device interface and sent to the server.

[0245] Step 5:

[0246] The server passes the questions sent by the students to a generative AI model, which analyzes the content of the questions and generates appropriate answers. The input is the question text from the students, and the output is the generated answer text. Specifically, the server uses a generative AI model (e.g., HuggingFace Transformers) to generate answers from lecture data related to the question content.

[0247] Step 6:

[0248] The server sends the generated answer text to the student's terminal. The input is the generated answer text, and the output is the answer information displayed on the student's terminal. In concrete terms, the generated answer is sent to the student's terminal and displayed in real time.

[0249] This allows students to instantly resolve any questions they may have while watching the lecture videos, significantly improving their learning efficiency.

[0250] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0251] This invention relates to a system that combines AI technology and emotion recognition to instantly answer questions from students about recorded lecture videos. The system includes the following specific elements and processes:

[0252] System configuration

[0253] 1. Server: The central management system stores data, analyzes data, handles question-answering, and recognizes emotions.

[0254] 2. Device: Provides an input interface for viewing recorded lessons, entering questions, and analyzing sentiment on devices used by students (PCs, tablets, smartphones, etc.).

[0255] 3. Users: Students who watch recorded lessons and ask questions.

[0256] Overview of system program processing

[0257] The program processing of this system is explained in natural language below.

[0258] 1. Student Access Authentication

[0259] User: Enter your user ID and password on the login screen.

[0260] Terminal: Sends the entered authentication information to the server.

[0261] Server: Authenticates the user against the information stored in the database, and if authentication is successful, returns access rights and a list of available lecture videos to the user. If authentication fails, displays an error message.

[0262] 2. Select the recorded lesson video and start streaming

[0263] User: Select the video they want to watch from the list of lecture videos provided.

[0264] Terminal: Sends a request for the selected lecture video to the server.

[0265] Server: Retrieves the selected lecture video from the database and sends it to the user's device in streaming format. The server also sends the video playback position information.

[0266] 3. Analysis of lesson content

[0267] Server: The recorded lesson video is sent to the multimodal AI. The audio portion of the video is processed through speech recognition and converted into text (speech-to-text).

[0268] Server: The contents of the slides and whiteboards are subjected to image analysis processing from the video footage to extract important information (video analysis).

[0269] Server: Integrates data obtained from audio-to-text and video analysis, and stores the lecture content in a database as structured data.

[0270] 4. Emotion Recognition by Emotion Engine

[0271] Device: Uses a camera and microphone to capture students' facial expressions and tone of voice in real time.

[0272] Terminal: Sends captured data to the server.

[0273] Server: Runs an emotion engine to recognize students' emotional states from their facial expressions and tone of voice, and generates emotion data.

[0274] 5. Accepting questions

[0275] Users: If they have any questions while watching a recorded lesson, they can enter their questions in the dedicated text box.

[0276] Terminal: Sends the entered question and emotion data to the server.

[0277] 6. Question Analysis and Answer Generation

[0278] Server: Receives questions sent by students and analyzes the content of the questions using natural language processing (NLP) technology.

[0279] AI model (on the server): Using the analyzed question content, the model searches a pre-structured lecture content database to obtain relevant information, and then generates the most appropriate answer from the obtained relevant information.

[0280] Server: Adjusts the content and tone of responses appropriately based on the student's emotional state as recognized by the emotion engine.

[0281] 7. Submitting and Viewing Your Answers

[0282] Server: Adjusts the generated answer and sends it to the user's device.

[0283] User: Checks the submitted answers on the device, resolves any doubts, and enters additional questions as necessary.

[0284] To give a specific example, if a student does not understand how to calculate consumption tax and looks confused, they can input the question, "How do you calculate consumption tax?" The server receives the question and analyzes the emotional data using an emotion engine. The AI ​​model searches for relevant parts of the lesson content and responds with, "Consumption tax is calculated by multiplying the purchase price by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen item, the consumption tax will be 100 yen." Because the emotion engine recognizes the student's confusion, it softens the tone of the response and provides additional explanation. For example, it outputs a response such as, "If you have difficulty understanding, I'll explain how to calculate it again." The user immediately confirms the appropriate answer and the problem is resolved.

[0285] This system not only allows students to resolve their questions immediately, but also improves the quality and efficiency of learning by enabling appropriate responses through emotion recognition. It also reduces the burden on instructors, reducing education costs while maintaining the quality of education.

[0286] The processing flow will be explained below.

[0287] Step 1: Student Access Authorization

[0288] User: Enter your user ID and password on the login screen.

[0289] Terminal: Sends the entered authentication information to the server.

[0290] Server: Authenticates the user against the information stored in the database, and if successful, returns access rights and a list of available lecture videos to the user. If unsuccessful, displays an error message.

[0291] Step 2: Select the recorded lesson video and start streaming

[0292] User: Select the video they want to watch from the list of lecture videos provided.

[0293] Terminal: Sends a request for the selected lecture video to the server.

[0294] Server: Retrieves the selected lecture video from the database and sends it to the user's device in streaming format, along with playback position information.

[0295] Step 3: Analysis of lesson content

[0296] Server: The recorded lesson video is sent to the multimodal AI. The audio portion of the video is processed through speech recognition and converted into text (speech-to-text).

[0297] Server: The contents of the slides and whiteboards are subjected to image analysis processing from the video footage to extract important information (video analysis).

[0298] Server: Integrates data obtained from audio-to-text and video analysis, and stores the lecture content in a database as structured data.

[0299] Step 4: Emotion Recognition with the Emotion Engine

[0300] Device: Uses a camera and microphone to capture students' facial expressions and tone of voice in real time.

[0301] Terminal: Sends captured facial expression and voice data to the server.

[0302] Server: The transmitted data is analyzed using an emotion engine, which recognizes the student's emotional state and generates emotion data.

[0303] Step 5: Ask a question

[0304] Users: If they have any questions while watching a recorded lesson, they can enter their questions in the dedicated text box.

[0305] Terminal: Sends the entered question and emotion data to the server.

[0306] Step 6: Parsing the question and generating an answer

[0307] Server: Receives questions sent by students and analyzes the content of the questions using natural language processing (NLP) technology.

[0308] AI model (inside the server): Using the analyzed question content, it searches a pre-structured database of lecture content to obtain relevant information.

[0309] Server: Generates the most appropriate answer from the relevant information obtained.

[0310] Server: Adjusts the content and tone of the generated answers accordingly based on the student's emotional state as recognized by the emotion engine.

[0311] Step 7: Submit and view your responses

[0312] Server: Sends the adjusted answer to the user's device.

[0313] User: Checks the submitted answers on the device, resolves any doubts, and enters additional questions as necessary.

[0314] To give a specific example, if a student doesn't understand how to calculate consumption tax and asks, "How do you calculate consumption tax?", the server receives the question and analyzes the emotion data using the emotion engine. The AI ​​model searches for relevant parts of the lesson content and generates an answer: "Consumption tax is the purchase price multiplied by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen item, the consumption tax will be 100 yen." The emotion engine recognizes the student's confusion and softens the tone of the answer, providing additional explanation such as, "If you have difficulty understanding, I'll explain how to calculate it again." The user can instantly confirm the appropriate answer and resolve their doubt.

[0315] Example 2

[0316] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0317] In conventional learning systems, if a student has a question while watching a recorded lesson, it is difficult to get an appropriate answer immediately on the spot. Furthermore, the system does not respond to the student's emotional state (e.g., confusion, lack of understanding, etc.), which can lead to a decrease in the quality and efficiency of learning. In addition, the instructor is often asked many questions, which increases the burden on the instructor and can lead to a decrease in the quality of education.

[0318] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for accepting questions entered by students, means for analyzing audio and video data of recorded lecture videos, means for understanding the question content based on the analyzed lecture content and generating an appropriate answer, means for capturing the students' facial expressions and tone of voice and recognizing their emotional state, means for appropriately adjusting the generated answer based on the students' emotional state, means for sending the generated answer to the students, means for authenticating the students' access, and means for structuring and storing data acquired from the lecture videos in a database. This allows students to immediately receive appropriate answers if they have questions while watching recorded lectures. Furthermore, emotion recognition technology enables responses based on the students' emotional state, improving the quality and efficiency of learning. Furthermore, it reduces the burden on instructors, reducing education costs while maintaining education quality.

[0319] A "student" is a student or learner who attends an educational institution or online course and takes lectures or classes.

[0320] "Means for accepting questions" refers to the interface and functions for accepting and processing text questions entered by users and managing them on the system.

[0321] "Lecture video" refers to audio and video data that records the contents of a class or lecture.

[0322] "Means for analyzing audio and video data" refers to processing devices and software that use voice recognition technology to convert the audio portion of recorded lecture videos into text and image recognition technology to analyze the video portion and extract information.

[0323] "Means of understanding the question and generating an appropriate answer" refers to an AI model or algorithm that uses natural language processing technology to analyze the question text entered by the user and generate an appropriate answer.

[0324] "Means for capturing facial expressions and tone of voice and recognizing emotional states" refers to a system that uses a camera and microphone to collect facial expression data and voice data of a user and analyzes their emotional state based on emotion recognition technology.

[0325] "Means for appropriately adjusting the generated response based on the emotional state of the student" refers to a processing device or algorithm that modifies or adjusts the content and tone of the generated response based on the data obtained through emotion recognition.

[0326] "Means for transmitting answers to students" refers to a system that includes a communication interface or protocol for transmitting the generated and adjusted answer data to the student's terminal.

[0327] "Means for access authentication" refers to the database and authentication server that verify the user ID and password when a student logs in to the system and authenticates access rights.

[0328] "Means for structuring data and storing it in a database" refers to a system or software that structures the analyzed lecture content data into a certain format (e.g., JSON or XML) and stores it in a database (e.g., an SQL database).

[0329] This invention relates to a system that combines AI technology and emotion recognition technology to instantly answer questions posed by students about recorded lecture videos. The system includes the following specific elements and processes:

[0330] System configuration

[0331] 1. Server: The central management system stores data, analyzes data, handles question-answering, and recognizes emotions.

[0332] 2. Terminal: A device used by students (personal computer, tablet, smartphone, etc.) that provides an input interface for watching recorded lessons, entering questions, and sentiment analysis.

[0333] 3. Users: Students who watch recorded lessons and ask questions.

[0334] Program processing explanation

[0335] 1. Student Access Authentication

[0336] User: Enter your user ID and password on the login screen.

[0337] Terminal: The entered authentication information is encrypted and sent to the server using HTTPS as the communication protocol.

[0338] Server: Compares the received authentication information with a database (e.g., MySQL (registered trademark)) and returns the authentication result to the terminal. If authentication is successful, it generates access rights and a list of available lecture videos.

[0339] 2. Select the recorded lesson video and start streaming

[0340] User: Select the video they want to watch from the lecture video list.

[0341] Terminal: Sends a request for the selected lecture video to the server. HTTPS is used as the communication protocol.

[0342] Server: Retrieves the specified video file from a video database (for example, AWS (registered trademark) S3) and streams it to the user's terminal using the HLS (HTTP Live Streaming) protocol.

[0343] 3. Analysis of lesson content

[0344] Server: Sends recorded lesson videos and analyzes them using multimodal AI (e.g., Google Cloud AI).

[0345] Server: The audio portion is converted into text using speech recognition software (e.g., Google Speech-to-Text), and the video portion is analyzed using image recognition software (e.g., Amazon Rekognition).

[0346] Server: Integrates speech-to-text data and video analysis data and stores them as structured data in a database (e.g., PostgreSQL).

[0347] 4. Emotion Recognition by Emotion Engine

[0348] Device: Uses a camera and microphone to capture students' facial expressions and tone of voice in real time.

[0349] Terminal: Sends captured facial expression and audio data to the server. It is recommended to use the WebSocket protocol.

[0350] Server: Analyzes students' emotional states using emotion recognition software (e.g., Microsoft® Azure® Emotion API) and generates emotion data.

[0351] 5. Accepting questions

[0352] Users: If they have any questions while watching a recorded lesson, they can type their questions in the text box.

[0353] Terminal: The entered question text and emotion data are sent to the server using HTTPS.

[0354] 6. Question Analysis and Answer Generation

[0355] Server: Analyzes questions sent by students using natural language processing (NLP) technology (e.g., OpenAI (registered trademark) GPT-4).

[0356] AI model: Analyzes the question, searches the lecture content database, and generates an appropriate answer.

[0357] Server: The emotion engine analyzes the student's emotional state and adjusts the response accordingly.

[0358] 7. Submitting and Viewing Your Answers

[0359] Server: Sends the adjusted answer to the user's device. HTTPS is used as the communication protocol.

[0360] User: Checks the answers on the device, resolves any doubts, and can enter additional questions as needed.

[0361] Specific examples

[0362] If a student is confused about how to calculate sales tax, they can type "How do you calculate sales tax?" into the text box.

[0363] The server receives this question and further analyzes the student's emotional data. The AI ​​model searches for relevant lecture content data and generates an answer such as, "Consumption tax is the purchase amount multiplied by the consumption tax rate. For example, if a product costs 1,000 yen and the consumption tax rate is 10%, the consumption tax will be 100 yen."

[0364] Using emotion recognition, the server detects confusion in the student, softening the tone of its responses and adding additional explanations such as, "If you have difficulty understanding, I'll explain how to do the calculation again."

[0365] The user can check the answer on the device to resolve their doubts, and can also enter additional questions if necessary.

[0366] Example prompt sentence:

[0367] "Please provide a professional explanation of the calculation of sales tax. Also, please show how to respond to students who are confused by the following question: How do you calculate sales tax?"

[0368] As described above, this system allows students to resolve any questions they may have while watching recorded lessons in real time, and responds appropriately through emotion recognition. This is expected to improve the quality and efficiency of learning, reduce the burden on instructors, and reduce education costs while maintaining the quality of education.

[0369] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0370] Step 1: Student Access Authorization

[0371] User: Enter your user ID and password. The input screen has a text box where you can enter the information.

[0372] Terminal: The entered authentication information is sent to the server via HTTPS. The input is in plain text and is encrypted before being sent.

[0373] Server: Compares the received authentication information with a database (e.g., MySQL), and if authentication is successful, generates access privileges and a list of lecture videos and sends them to the terminal. If authentication fails, generates an error message and sends it to the terminal. The input is encrypted user information, and the output is the authentication result and a list of videos or an error message.

[0374] Step 2: Select the recorded lesson video and start streaming

[0375] User: Select the video they want to watch from the lecture video list. For example, click on the video title on the list.

[0376] Terminal: Sends a request for the selected lecture video to the server via HTTPS protocol. The input is the video ID and video request information, and the output is the request data to the server.

[0377] Server: Retrieves the specified video file from the video database (e.g., AWS S3) and streams it to the device in HLS format. The input is the video ID and the output is the streaming data.

[0378] Step 3: Analysis of lesson content

[0379] Server: Sends recorded lecture videos to a multimodal AI (e.g., Google Cloud AI). The input is the lecture video data, and the output is an analysis request to the AI.

[0380] Server: The audio portion of the video is converted into text using speech recognition software (e.g., Google Speech-to-Text). The input is audio data, and the output is text data.

[0381] Server: Analyzes the video portion of the video using image recognition software (e.g., Amazon Rekognition) and extracts information from slides and whiteboards. The input is video data, and the output is the text data of the analysis results.

[0382] Server: Integrates speech-to-text data and video analysis data and stores them as structured data in a database (e.g., PostgreSQL). The input is text data and video analysis data, and the output is structured data.

[0383] Step 4: Emotion Recognition with the Emotion Engine

[0384] Device: Uses a camera and microphone to capture the student's facial expressions and tone of voice in real time. The input is real-time video and audio data.

[0385] Terminal: Sends captured data to the server. It is recommended to use the WebSocket protocol. The input is the captured data and the output is the data sent to the server.

[0386] Server: Analyzes students' emotional states using emotion recognition software (e.g., Microsoft Azure Emotion API) and generates emotion data. The input is the captured data, and the output is emotion data.

[0387] Step 5: Ask a question

[0388] Users: If they have questions while watching a recorded lesson, they can type their question in the text box. For example, they can type "I don't know how to use this formula."

[0389] Terminal: The input question and emotion data are sent to the server via HTTPS. The input is the question text and emotion data, and the output is the data sent to the server.

[0390] Step 6: Parsing the question and generating an answer

[0391] Server: Analyzes questions sent by students using natural language processing (NLP) technology (e.g., OpenAI GPT-4). The input is the question text, and the output is the analysis result.

[0392] AI model: Using the analyzed question content, it searches the lecture content database, obtains relevant information, and generates an answer. The input is the analysis result, and the output is the initial answer.

[0393] Server: Based on the emotional state of the student recognized by the emotion engine, the generated answer is adjusted accordingly. The input is the initial answer and emotional data, and the output is the adjusted answer.

[0394] Step 7: Submit and view your responses

[0395] Server: Sends the adjusted answers to the student's device. HTTPS protocol is used. The input is the adjusted answers, and the output is the data sent to the device.

[0396] User: Checks the answer on the device and resolves any doubts. User can also enter additional questions if necessary. Input is the displayed answer.

[0397] (Application example 2)

[0398] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0399] Previously, it was difficult to respond immediately to customer questions with product introduction videos in virtual stores. In particular, when customers asked specific questions about a product, it was sometimes difficult to read the customer's emotional state from their facial expressions or tone of voice, which resulted in delayed responses. This could lead to lower customer satisfaction and lost sales opportunities.

[0400] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0401] In this invention, the server includes means for accepting questions entered by students, means for analyzing audio and video data of recorded educational videos, means for understanding the content of the questions based on the analyzed educational content and generating appropriate answers, means for sending the generated answers to students, means for capturing facial expressions and tone of voice in real time and recognizing emotional states, and means for adjusting the content and tone of answers based on the recognized emotional states. This enables customer service in the virtual store to be quick and accurate, thereby improving customer satisfaction.

[0402] "Participants" are users who watch recorded educational videos and enter questions.

[0403] A "means for accepting questions" is an interface that accepts text input from a user and sends it to the system.

[0404] A "recorded educational video" is a video containing pre-recorded educational content.

[0405] "Means for analyzing audio and video data" refers to the technology and functions that convert the audio data of recorded educational videos into text and extract important information from the video data.

[0406] "Means of understanding the question and generating an appropriate answer" refers to the technology and functions that use natural language processing technology to analyze the question, retrieve relevant information from a database, and generate an answer.

[0407] The "means for transmitting the generated answers to the students" refers to an interface for presenting the generated answers to the users in real time.

[0408] "Means of capturing facial expressions and tone of voice in real time and recognizing emotional states" refers to technologies and functions that capture a user's facial expressions and tone of voice using a camera or microphone and analyze their emotions in real time.

[0409] "Means for adjusting the content and tone of responses based on a recognized emotional state" refers to technologies and functions that adjust the content and tone of generated responses based on the results of emotional analysis, making them easier for users to understand and more reassuring.

[0410] In this invention, the system is realized by the server, terminal, and user performing the following processes.

[0411] server

[0412] The server analyzes and stores various data, processes questions and answers, and recognizes emotions. Specifically, it uses the following hardware and software:

[0413] Hardware: high-performance processor, storage device, large memory

[0414] Software: Natural language processing engines (e.g., OpenAI GPT-4), emotion recognition engines, video processing software, database management systems

[0415] The server process includes:

[0416] 1. Audio and video data analysis: Converting audio data from recorded educational videos into text and extracting important information from video data. This process uses voice recognition and image analysis technologies.

[0417] 2. Question analysis and answer generation: The system uses natural language processing technology to analyze questions from students, retrieves relevant information from a database, and generates answers that correspond to the questions. The generated answers incorporate emotion recognition results as needed, adjusting the tone of the answers to reflect the student's emotional state.

[0418] 3. Emotional state recognition and adjustment: Analyzes the participant's facial expressions and tone of voice to recognize their emotional state in real time, and adjusts the content and tone of their responses accordingly.

[0419] Terminal

[0420] Devices are devices used by students, including PCs, tablets, smartphones, smart glasses, and head-mounted displays. They use the following hardware and software:

[0421] Hardware: Camera, microphone, display, storage, processor

[0422] Software: User Interface (UI) applications, video streaming applications

[0423] Terminal processing includes:

[0424] 1. Login and authentication: The student enters their user ID and password and sends the authentication information to the server.

[0425] 2. Video viewing and playback: Receives and plays recorded lecture videos provided by the server, and provides an interface for students to input questions while watching.

[0426] 3. Capture and transmit emotional data: Use a camera and microphone to capture the student's facial expressions and tone of voice and transmit them to the server.

[0427] User

[0428] A user is a student who uses this system and performs the following operations:

[0429] 1. Login: Enter your user ID and password on each terminal to log in to the system.

[0430] 2. Watch Video: Select a video from the list provided and watch the video.

[0431] 3. Enter a question: If you have any questions while watching the video, enter your question in the dedicated text box.

[0432] Examples of concrete examples and prompts

[0433] For example, if a student is watching a recorded lecture on their smartphone and types a question like, "Please tell me about the camera performance of this smartphone." If the student's facial expression sounds confused and their tone of voice sounds anxious, the server will analyze the question and generate an answer like this:

[0434] The smartphone's camera is equipped with a 12MP wide-angle lens, allowing you to take clear photos even at night. Want to hear more details? You can also call our support team if you need help.

[0435] An example of this prompt statement:

[0436] "User Question: What are the specific features of this product?

[0437] Expression: Confused.

[0438] Voice tone: Anxious.

[0439] This is a situation that requires action."

[0440] As a result, customer service at the virtual store can be provided quickly and accurately, improving customer satisfaction.

[0441] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0442] Step 1:

[0443] Login and Authentication

[0444] Terminal: The user launches the application on the terminal and enters their user ID and password. The entered authentication information is sent to the server.

[0445] Server: The server compares the received authentication information with a database and performs authentication. If authentication is successful, it returns access rights and a list of available content to the terminal. If authentication fails, it sends an error message to the terminal. The input is the user ID and password, and the output is the authentication result.

[0446] Step 2:

[0447] Select content and start streaming

[0448] Device: The user selects the video they want to watch from a content list provided by the application on the device. A request for the selected video is sent to the server.

[0449] Server: The server retrieves the selected video from the database and sends it to the device in streaming format. Video playback position information is also sent at the same time. The input is the video selection request, and the output is streaming data.

[0450] Step 3:

[0451] Video content analysis

[0452] Server: The audio data from the video is analyzed using speech recognition technology (e.g., Google Speech-to-Text) and converted into text. The video data is analyzed using image analysis technology (e.g., OpenCV) to extract important information. This data is integrated and stored in a database as structured data. The input is the audio and video data from the video, and the output is structured data.

[0453] Step 4:

[0454] Capturing and transmitting emotional data

[0455] Terminal: The camera and microphone capture the user's facial expressions and tone of voice in real time and send the data to the server. The input is real-time data from the camera and microphone, and the output is data sent to the server.

[0456] Step 5:

[0457] Recognition of emotional states

[0458] Server: Using an emotion recognition engine (e.g., Microsoft Azure Emotion API), the server analyzes the user's emotional state from the facial expression and tone of voice data sent. The analysis results are stored as internal data and used when answering questions. The input is facial expression and tone of voice data, and the output is the analysis result of the emotional state.

[0459] Step 6:

[0460] Accepting and analyzing questions

[0461] Terminal: If a user has a question while watching a video, they can enter it in the text box, which is then sent to the server.

[0462] Server: The server uses a natural language processing engine (e.g., OpenAI GPT-4) to analyze the question and retrieve relevant information from a database. The input is a text question, and the output is the analysis result based on the question.

[0463] Step 7:

[0464] Answer generation and adjustment

[0465] Server: Generates answers to questions based on information retrieved from a database. Furthermore, it adjusts the content and tone of the answers based on the results of emotional state analysis. The input is the results of question analysis and emotional state analysis, and the output is the adjusted answer.

[0466] Step 8:

[0467] Submitting and viewing answers

[0468] Server: Sends the adjusted answer to the device.

[0469] Terminal: The terminal displays the received answers to the user, who reviews the answers and asks follow-up questions if necessary. The input is the adjusted answers, and the output is the answers displayed to the user.

[0470] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0471] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0472] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0473] [Second embodiment]

[0474] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0475] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0476] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0477] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0478] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0479] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0480] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0481] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0482] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0483] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0484] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0485] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0486] This invention relates to a system that uses AI technology to instantly answer questions from students about recorded lecture videos. The system includes the following specific elements and processes:

[0487] System configuration

[0488] 1. Server: A central management system that stores data, analyzes data, and handles question and answering.

[0489] 2. Device: Provides an interface for students to watch recorded lessons and enter questions on their devices (PCs, tablets, smartphones, etc.).

[0490] 3. Users: Students who watch recorded lessons and ask questions.

[0491] Overview of system program processing

[0492] The program processing of this system is explained in natural language below.

[0493] 1. Student Access Authentication

[0494] User: Enter your user ID and password on the login screen.

[0495] Terminal: Sends the entered authentication information to the server.

[0496] Server: Authenticates the user against the database, and if successful, returns a list of lecture videos to the user.

[0497] 2. Start streaming recorded lecture videos

[0498] User: Select the lecture video they want to watch.

[0499] Device: Sends a request for the selected video to the server.

[0500] Server: Retrieves the relevant video from the video database and starts streaming.

[0501] 3. Analysis of lesson content

[0502] Server: Passes the audio and video portions of the lesson video to the multimodal AI.

[0503] AI model (inside the server): Converts audio into text (voice recognition) and analyzes blackboard notes and slides in the video to convert them into text (image analysis).

[0504] Server: Structures the parsed data and stores it in a database.

[0505] 4. Questions

[0506] User: If a question arises during the lesson being viewed, enter the question in the text box.

[0507] Terminal: Sends the entered question to the server.

[0508] 5. Question Answering

[0509] Server: Receives questions from users and analyzes the content of the questions.

[0510] AI model (inside the server): Compares the analyzed questions with pre-structured lecture content and generates appropriate answers.

[0511] Server: Sends the generated answer to the user's device.

[0512] User: Check the AI's answer sent back from the device.

[0513] To give a specific example, if a student doesn't understand how to calculate consumption tax and asks, "How do you calculate consumption tax?", the server receives the question and passes it to the AI ​​model. The AI ​​model searches for relevant parts of the lesson content and generates an answer: "Consumption tax is the purchase price multiplied by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen item, the consumption tax will be 100 yen." The server then sends this to the user's device. The user can instantly check the appropriate answer and resolve their question.

[0514] This system allows students to resolve their questions on the spot, significantly improving learning efficiency. In addition, since instructors do not need to respond directly to students' questions, the burden on them is reduced, making it possible to reduce education costs while maintaining the quality of education.

[0515] The processing flow will be explained below.

[0516] Step 1: Student Access Authorization

[0517] User: Enter your user ID and password on the login screen.

[0518] Terminal: Sends the entered authentication information to the server.

[0519] Server: Authenticates the user against the information stored in the database, and if authentication is successful, returns access rights and a list of available lecture videos to the user. If authentication fails, displays an error message.

[0520] Step 2: Select the recorded lesson video and start streaming

[0521] User: Select the video they want to watch from the list of lecture videos provided.

[0522] Terminal: Sends a request for the selected lecture video to the server.

[0523] Server: Retrieves the selected lecture video from the database and sends it to the user's device in streaming format. The server also sends the video playback position information.

[0524] Step 3: Analysis of lesson content

[0525] Server: The recorded lesson video is sent to the multimodal AI. The audio portion of the video is processed through speech recognition and converted into text (speech-to-text).

[0526] Server: The contents of the slides and whiteboards are subjected to image analysis processing from the video footage to extract important information (video analysis).

[0527] Server: Integrates data obtained from audio-to-text and video analysis, and stores the lecture content in a database as structured data.

[0528] Step 4: Ask a question

[0529] Users: If they have any questions while watching a recorded lesson, they can enter their questions in the dedicated text box.

[0530] Terminal: Sends the entered question to the server.

[0531] Step 5: Parsing the question and generating an answer

[0532] Server: Receives questions sent by students and analyzes the content of the questions using natural language processing (NLP) technology.

[0533] AI model (on the server): Using the analyzed question content, the model searches a pre-structured lecture content database to obtain relevant information, and then generates the most appropriate answer from the obtained relevant information.

[0534] Server: Sends the generated answers to the students' devices.

[0535] Step 6: View and review your answers

[0536] User: Checks the submitted answers on the device, resolves any doubts, and enters additional questions as necessary.

[0537] Step 7: Feedback and Data Storage (Optional)

[0538] Server: Logs all questions and answers into a database for later review by the instructor.

[0539] Instructor: Analyze the topics that students focus on asking questions about and the AI's answers, and prepare additional teaching materials and supplementary lectures as necessary.

[0540] The above processing steps allow students to ask questions immediately while watching recorded lessons, and AI can provide quick answers, improving learning efficiency. This also reduces the burden on instructors and contributes to maintaining the quality of education.

[0541] Example 1

[0542] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0543] In conventional systems that provide recorded lecture videos, it is difficult for students to get immediate answers to questions that arise while watching the video. This reduces learning efficiency and places a burden on the instructor. Therefore, there is a need for a system that allows students to solve their questions in real time while watching the recorded video.

[0544] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0545] In this invention, the server includes a means for accepting questions entered by students, a means for analyzing audio and video data of recorded lecture videos, a means for understanding the question content based on the analyzed lecture content and using a generative AI model to generate an appropriate answer, and a means for transmitting the generated answer to the student, thereby enabling the student to solve their question in real time while watching the recorded video.

[0546] "Student" refers to a person who watches and learns from lecture videos provided online or offline.

[0547] "Means for accepting questions" refers to the interface or protocol that receives questions entered by students and sends them to the server.

[0548] "Lecture video" refers to audio and video data containing recorded lecture content.

[0549] "Means for analyzing audio and video data" refers to technology that converts the audio of lecture videos into text and extracts and analyzes visual information from the video.

[0550] A "generative AI model" refers to an artificial intelligence model that generates appropriate answers based on input questions and data.

[0551] "Means for generating answers" refers to technology that generates appropriate answers to students' questions based on the lecture content analyzed using a generative AI model.

[0552] "Means for sending answers" refers to the communication technology or protocol used to send the generated answers to the student's device.

[0553] "Access Authentication Measures" means the system or method that verifies a student's credentials and verifies access rights.

[0554] "Encryption methods" refers to the technologies and protocols that encrypt information to protect data transmissions.

[0555] "Structured data" refers to data that has been systematically organized from analyzed lecture video information and stored in a database.

[0556] "Speech recognition means" refers to technology that automatically converts voice data into text.

[0557] "Image analysis means" refers to technology that extracts specific visual information from video data and converts it into text.

[0558] This invention relates to a system that uses AI technology to instantly answer questions posed by students about recorded lecture videos. The system consists of three main components: a server that functions as a central management system, terminals where students can watch lecture videos and enter questions, and users who watch the recorded lectures.

[0559] System configuration

[0560] 1. Server:

[0561] The server stores data, analyzes it, and handles question and answering. Specifically, it uses the following hardware and software:

[0562] Database: Used to store authentication information and lecture video data.

[0563] Generative AI model: For example, GPT-4 is used to generate appropriate answers based on the question.

[0564] Speech recognition technology: The audio portion of the lecture video is converted into text using the Google Speech-to-Text API.

[0565] Image analysis technology: The video portion of the lecture video is analyzed using Google Cloud Vision API and converted into text.

[0566] 2. Terminal:

[0567] The device used by students (PC, tablet, smartphone, etc.) provides an interface for viewing recorded lessons and entering questions. The device has the following functions:

[0568] Login interface: An interface for entering a user ID and password.

[0569] Video playback function: Lecture videos retrieved from the server are played in streaming format.

[0570] Question input interface: An interface for students to enter their questions into a text box.

[0571] 3. User:

[0572] Students watch the recorded lectures and ask questions.

[0573] Specific actions

[0574] Below, we will explain how this system works with a concrete example.

[0575] 1. User access authentication

[0576] The user enters their user ID and password on the login screen.

[0577] The terminal encrypts the entered authentication information and sends it to the server.

[0578] The server checks the authentication information stored in the database and, if authentication is successful, returns a list of lecture videos to the user.

[0579] 2. Start streaming recorded lecture videos

[0580] The user selects the lecture video they wish to watch from the list.

[0581] The terminal sends a request for the selected video to the server.

[0582] The server retrieves the relevant video from the video database and streams it.

[0583] 3. Analysis of lesson content

[0584] The server passes the audio and video portions of the lesson video to the multimodal AI. The audio portion is converted to text using the Google Speech-to-Text API, and the video portion is analyzed and converted to text using the Google Cloud Vision API.

[0585] The server structures the parsed data and stores it in a database.

[0586] 4. Questions

[0587] If a user has a question about a lesson they are watching, they can enter it in the text box.

[0588] The terminal transmits the entered question to the server.

[0589] 5. Question Answering

[0590] The server receives a question from a user and analyzes the question.

[0591] The generative AI model matches the analyzed questions with pre-structured lecture content and generates appropriate answers.

[0592] The server sends the generated answer to the user's terminal.

[0593] The user checks the AI's answers sent back from the device.

[0594] Specific examples

[0595] For example, if a student does not understand how to calculate consumption tax, he or she can input the question, "How do you calculate consumption tax?" In response to this question, the system operates as follows:

[0596] User: Enters the question "How do you calculate sales tax?" into the text box.

[0597] Terminal: Sends entered text to the server.

[0598] Server: Receives the question and sends the prompt "How do you calculate sales tax?" to the generative AI model.

[0599] Generative AI model: Searches the lecture content and generates the answer, "Consumption tax is the purchase amount multiplied by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen product, the consumption tax will be 100 yen."

[0600] Server: Sends the generated answer to the user's device.

[0601] Terminal: Receives the answer and displays it to the user.

[0602] User: Review the answers provided and understand how sales tax is calculated.

[0603] This system allows students to get answers to their questions in real time, improving learning efficiency and reducing the burden on instructors.

[0604] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0605] Step 1:

[0606] Authenticating users for access

[0607] 1. Input: The user enters their user ID and password on the login screen.

[0608] 2. Operation: The device encrypts the entered authentication information and sends it to the server using a protocol such as SSL / TLS.

[0609] 3. Input: The server checks the received authentication information against the information stored in its database.

[0610] 4. Data processing: The server authenticates the user based on the matching result. If authentication is successful, the user information is saved in the session.

[0611] 5. Output: The server generates a lecture video list for the user and sends it to the terminal.

[0612] 6. Operation: The terminal displays a list of lecture videos.

[0613] Step 2:

[0614] Streaming of recorded lecture videos begins

[0615] 1. Input: The user selects the lecture video they want to watch from the list.

[0616] 2. Action: The device sends a request for the selected video to the server.

[0617] 3. Input: The server retrieves the relevant video data from the video database.

[0618] 4. Data processing: The server converts the video data into a streaming format, often using tools such as FFmpeg.

[0619] 5. Output: The server sends the converted streaming data to the terminal.

[0620] 6. Operation: The device plays the received streaming data.

[0621] Step 3:

[0622] Analysis of lesson content

[0623] 1. Input: The server extracts the audio and video parts of the lesson video.

[0624] 2. How it works: The server passes the audio portion to the Google Speech-to-Text API for speech recognition.

[0625] 3. Input: The server receives the text data obtained by speech recognition.

[0626] 4. Data processing: The audio data is saved as text along with time information.

[0627] 5. Operation: The server passes the video portion to the Google Cloud Vision API for image analysis.

[0628] 6. Input: The server receives the text data obtained from image analysis.

[0629] 7. Data processing: The information on the board and slides in the video is saved as text along with time information.

[0630] 8. Output: The server structures the acquired text data and stores it in a database.

[0631] Step 4:

[0632] Questions accepted

[0633] 1. Input: When a user has a question about the lesson they are watching, they can enter it in the text box.

[0634] 2. Operation: The terminal sends the entered question data to the server.

[0635] 3. Input: The server adds the received query data to a queue for analysis.

[0636] 4. Output: The question data added to the queue is passed to the next question-answering process.

[0637] Step 5:

[0638] Question and Answering

[0639] 1. Input: The server retrieves the question data in the queue.

[0640] 2. Operation: The server analyzes the question using NLP (Natural Language Processing) technology.

[0641] 3. Input: The server sends the parsed question content to the generative AI model as a prompt sentence.

[0642] 4. Output: Based on a specific prompt, such as "How do you calculate sales tax?", the generative AI model generates an answer.

[0643] 5. Input: Receive the answer from the generative AI model.

[0644] 6. Data processing: The server structures the answer and sends it to the user's device.

[0645] 7. Operation: The terminal receives the generated response and displays it to the user.

[0646] 8. Output: The user checks the answer displayed on the terminal and resolves their doubt.

[0647] (Application example 1)

[0648] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0649] With conventional video lecture systems, students had no way to ask questions and get answers immediately when they had doubts. This resulted in problems such as unresolved questions and reduced learning efficiency. Furthermore, it was a significant burden on instructors to deal with questions afterward, making it difficult to provide an efficient education.

[0650] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0651] In this invention, the server includes a means for accepting questions entered by students, a means for analyzing audio and video data from recorded lecture videos, and a means for understanding the questions based on the analyzed lecture content and generating appropriate answers. This allows the generated answers to be sent immediately to students. The server also includes a means for using a mobile information terminal that enables viewing of recorded lectures and question entry, a means for performing speech recognition and question answering using a generative AI model, and a means for using a central management system that receives, processes, and transmits various data. This allows students to instantly resolve their questions using mobile information terminals such as smartphones and tablets, improving their learning efficiency.

[0652] "Students" are learners who watch recorded lecture videos and enter questions.

[0653] "Means for accepting questions" refers to an interface for accepting questions entered by students and sending them into the system.

[0654] A "lecture video" is a file containing video and audio data of recorded educational content.

[0655] "Means for analyzing audio and video data" refers to the technical methods for extracting audio and video information from recorded lecture videos, converting it into text, and performing semantic analysis.

[0656] "Means of understanding the question and generating an appropriate answer" refers to a function that analyzes questions sent by students, compares them with related lecture content, and automatically creates the most appropriate answer.

[0657] "Means for sending generated answers" refers to a mechanism for sending answers generated by AI to students' devices in real time.

[0658] "Mobile information terminal" refers to a portable information processing device such as a smartphone or tablet.

[0659] "Generative AI model" refers to a learning model that uses artificial intelligence to perform speech recognition, natural language processing, and question answering.

[0660] "Central management system" refers to the server and related infrastructure for centrally managing the reception and processing of questions from students, the analysis of lecture videos, and the generation and transmission of answers.

[0661] This invention relates to a system that uses a generative AI model to instantly answer questions posed by students about recorded lecture videos. This system is realized by the following specific components and the mechanisms by which these components work together.

[0662] System configuration

[0663] 1. Server: As a central management system, it accepts questions from students, analyzes recorded lecture videos, generates questions and answers, and sends the generated answers.

[0664] 2. Devices: Mobile information devices such as smartphones and tablets used by students provide an interface for watching lecture videos and entering questions.

[0665] 3. User: A student who watches the recorded lecture and enters questions.

[0666] Program processing description

[0667] Login Authentication

[0668] The server receives the user ID and password sent by the student from their device and checks them against a database. If authentication is successful, it returns a list of lecture videos to the device. For security reasons, this process uses encrypted communication such as SSL / TLS.

[0669] Streaming lecture videos

[0670] When a user selects a lecture video to watch on their device, the server retrieves the corresponding video data from the video database and starts streaming it to the device. The video data is saved in standard formats such as MP4 and WebM.

[0671] Analysis of lesson content

[0672] The server passes the audio and video portions of the recorded lecture video to a generative AI model, which performs speech recognition to convert the audio into text and analyzes the blackboard notes and slides in the video to generate text. The analyzed data is structured and stored in a database. This process uses natural language processing libraries (e.g., HuggingFace Transformers) and image analysis software.

[0673] Question reception and answering

[0674] When a user has a question about a lesson they are watching, they send it in text format from their device to the server. The server then passes the received question to a generative AI model, which analyzes the question and generates an appropriate answer. The generated answer is then immediately sent back to the device. This allows students to receive answers to their questions in real time.

[0675] Specific examples of programs

[0676] Authentication Example

[0677] When a user enters "User ID: student1" and "Password: password1" on the login screen on their smartphone and sends it to the server, the server checks the database and, if the authentication information is correct, returns a list of lecture videos.

[0678] Example questions and answers

[0679] When a user inputs the question, "How do you calculate economic growth rate?", the generative AI model searches for relevant lecture content and generates the answer, "Economic growth rate is an indicator that shows how much a country's economy has grown, and specifically, it is calculated as the annual rate of change in gross domestic product (GDP)." The server then sends this answer to the user's device.

[0680] Example prompt sentence:

[0681] How do you calculate economic growth rate?

[0682] In this way, this invention allows students to resolve questions in real time while watching recorded lessons, improving learning efficiency and reducing the burden on instructors. Furthermore, this system is highly convenient because it uses mobile information terminals such as smartphones and tablets.

[0683] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0684] Step 1:

[0685] The server accepts the authentication information of the user ID and password entered by the student using the terminal and compares it with the database. The input is the user ID and password, and the output is the authentication result. If the authentication is successful, the server returns a list of lecture videos to the terminal. Specifically, the authentication process is carried out securely using SSL / TLS encrypted communication.

[0686] Step 2:

[0687] The server receives the lecture video ID requested by the student from the terminal and retrieves the video data corresponding to that ID from the video database. The input is the video request information (video ID), and the output is the streaming URL of the corresponding lecture video. Specifically, the server queries the video database, generates a video URL, and sends it to the terminal.

[0688] Step 3:

[0689] The server passes the audio and video data of the lecture video to the generative AI model for analysis. The input is the audio and video data of the lecture video, and the output is the text of the lecture content. Specifically, the audio data is converted into text using voice recognition software, and at the same time, the text and slides in the video are converted into text using image analysis software. The analyzed data is then structured and stored in a database.

[0690] Step 4:

[0691] When a student has a question during a class they are watching, they use their device to enter the question into a text box and send it to the server. The input is the question text entered by the student, and the output is a confirmation by the server that the question has been received. Specifically, the question is entered into the device interface and sent to the server.

[0692] Step 5:

[0693] The server passes the questions sent by the students to a generative AI model, which analyzes the content of the questions and generates appropriate answers. The input is the question text from the students, and the output is the generated answer text. Specifically, the server uses a generative AI model (e.g., HuggingFace Transformers) to generate answers from lecture data related to the question content.

[0694] Step 6:

[0695] The server sends the generated answer text to the student's terminal. The input is the generated answer text, and the output is the answer information displayed on the student's terminal. In concrete terms, the generated answer is sent to the student's terminal and displayed in real time.

[0696] This allows students to instantly resolve any questions they may have while watching the lecture videos, significantly improving their learning efficiency.

[0697] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0698] This invention relates to a system that combines AI technology and emotion recognition to instantly answer questions from students about recorded lecture videos. The system includes the following specific elements and processes:

[0699] System configuration

[0700] 1. Server: The central management system stores data, analyzes data, handles question-answering, and recognizes emotions.

[0701] 2. Device: Provides an input interface for viewing recorded lessons, entering questions, and analyzing sentiment on devices used by students (PCs, tablets, smartphones, etc.).

[0702] 3. Users: Students who watch recorded lessons and ask questions.

[0703] Overview of system program processing

[0704] The program processing of this system is explained in natural language below.

[0705] 1. Student Access Authentication

[0706] User: Enter your user ID and password on the login screen.

[0707] Terminal: Sends the entered authentication information to the server.

[0708] Server: Authenticates the user against the information stored in the database, and if authentication is successful, returns access rights and a list of available lecture videos to the user. If authentication fails, displays an error message.

[0709] 2. Select the recorded lesson video and start streaming

[0710] User: Select the video they want to watch from the list of lecture videos provided.

[0711] Terminal: Sends a request for the selected lecture video to the server.

[0712] Server: Retrieves the selected lecture video from the database and sends it to the user's device in streaming format. The server also sends the video playback position information.

[0713] 3. Analysis of lesson content

[0714] Server: The recorded lesson video is sent to the multimodal AI. The audio portion of the video is processed through speech recognition and converted into text (speech-to-text).

[0715] Server: The contents of the slides and whiteboards are subjected to image analysis processing from the video footage to extract important information (video analysis).

[0716] Server: Integrates data obtained from audio-to-text and video analysis, and stores the lecture content in a database as structured data.

[0717] 4. Emotion Recognition by Emotion Engine

[0718] Device: Uses a camera and microphone to capture students' facial expressions and tone of voice in real time.

[0719] Terminal: Sends captured data to the server.

[0720] Server: Runs an emotion engine to recognize students' emotional states from their facial expressions and tone of voice, and generates emotion data.

[0721] 5. Accepting questions

[0722] Users: If they have any questions while watching a recorded lesson, they can enter their questions in the dedicated text box.

[0723] Terminal: Sends the entered question and emotion data to the server.

[0724] 6. Question Analysis and Answer Generation

[0725] Server: Receives questions sent by students and analyzes the content of the questions using natural language processing (NLP) technology.

[0726] AI model (on the server): Using the analyzed question content, the model searches a pre-structured lecture content database to obtain relevant information, and then generates the most appropriate answer from the obtained relevant information.

[0727] Server: Adjusts the content and tone of responses appropriately based on the student's emotional state as recognized by the emotion engine.

[0728] 7. Submitting and Viewing Your Answers

[0729] Server: Adjusts the generated answer and sends it to the user's device.

[0730] User: Checks the submitted answers on the device, resolves any doubts, and enters additional questions as necessary.

[0731] To give a specific example, if a student does not understand how to calculate consumption tax and looks confused, they can input the question, "How do you calculate consumption tax?" The server receives the question and analyzes the emotional data using an emotion engine. The AI ​​model searches for relevant parts of the lesson content and responds with, "Consumption tax is calculated by multiplying the purchase price by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen item, the consumption tax will be 100 yen." Because the emotion engine recognizes the student's confusion, it softens the tone of the response and provides additional explanation. For example, it outputs a response such as, "If you have difficulty understanding, I'll explain how to calculate it again." The user immediately confirms the appropriate answer and the problem is resolved.

[0732] This system not only allows students to resolve their questions immediately, but also improves the quality and efficiency of learning by enabling appropriate responses through emotion recognition. It also reduces the burden on instructors, reducing education costs while maintaining the quality of education.

[0733] The processing flow will be explained below.

[0734] Step 1: Student Access Authorization

[0735] User: Enter your user ID and password on the login screen.

[0736] Terminal: Sends the entered authentication information to the server.

[0737] Server: Authenticates the user against the information stored in the database, and if successful, returns access rights and a list of available lecture videos to the user. If unsuccessful, displays an error message.

[0738] Step 2: Select the recorded lesson video and start streaming

[0739] User: Select the video they want to watch from the list of lecture videos provided.

[0740] Terminal: Sends a request for the selected lecture video to the server.

[0741] Server: Retrieves the selected lecture video from the database and sends it to the user's device in streaming format, along with playback position information.

[0742] Step 3: Analysis of lesson content

[0743] Server: The recorded lesson video is sent to the multimodal AI. The audio portion of the video is processed through speech recognition and converted into text (speech-to-text).

[0744] Server: The contents of the slides and whiteboards are subjected to image analysis processing from the video footage to extract important information (video analysis).

[0745] Server: Integrates data obtained from audio-to-text and video analysis, and stores the lecture content in a database as structured data.

[0746] Step 4: Emotion Recognition with the Emotion Engine

[0747] Device: Uses a camera and microphone to capture students' facial expressions and tone of voice in real time.

[0748] Terminal: Sends captured facial expression and voice data to the server.

[0749] Server: The transmitted data is analyzed using an emotion engine, which recognizes the student's emotional state and generates emotion data.

[0750] Step 5: Ask a question

[0751] Users: If they have any questions while watching a recorded lesson, they can enter their questions in the dedicated text box.

[0752] Terminal: Sends the entered question and emotion data to the server.

[0753] Step 6: Parsing the question and generating an answer

[0754] Server: Receives questions sent by students and analyzes the content of the questions using natural language processing (NLP) technology.

[0755] AI model (inside the server): Using the analyzed question content, it searches a pre-structured database of lecture content to obtain relevant information.

[0756] Server: Generates the most appropriate answer from the relevant information obtained.

[0757] Server: Adjusts the content and tone of the generated answers accordingly based on the student's emotional state as recognized by the emotion engine.

[0758] Step 7: Submit and view your responses

[0759] Server: Sends the adjusted answer to the user's device.

[0760] User: Checks the submitted answers on the device, resolves any doubts, and enters additional questions as necessary.

[0761] To give a specific example, if a student doesn't understand how to calculate consumption tax and asks, "How do you calculate consumption tax?", the server receives the question and analyzes the emotion data using the emotion engine. The AI ​​model searches for relevant parts of the lesson content and generates an answer: "Consumption tax is the purchase price multiplied by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen item, the consumption tax will be 100 yen." The emotion engine recognizes the student's confusion and softens the tone of the answer, providing additional explanation such as, "If you have difficulty understanding, I'll explain how to calculate it again." The user can instantly confirm the appropriate answer and resolve their doubt.

[0762] Example 2

[0763] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0764] In conventional learning systems, if a student has a question while watching a recorded lesson, it is difficult to get an appropriate answer immediately on the spot. Furthermore, the system does not respond to the student's emotional state (e.g., confusion, lack of understanding, etc.), which can lead to a decrease in the quality and efficiency of learning. In addition, the instructor is often asked many questions, which increases the burden on the instructor and can lead to a decrease in the quality of education.

[0765] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for accepting questions entered by students, means for analyzing audio and video data of recorded lecture videos, means for understanding the question content based on the analyzed lecture content and generating an appropriate answer, means for capturing the students' facial expressions and tone of voice and recognizing their emotional state, means for appropriately adjusting the generated answer based on the students' emotional state, means for sending the generated answer to the students, means for authenticating the students' access, and means for structuring and storing data acquired from the lecture videos in a database. This allows students to immediately receive appropriate answers if they have questions while watching recorded lectures. Furthermore, emotion recognition technology enables responses based on the students' emotional state, improving the quality and efficiency of learning. Furthermore, it reduces the burden on instructors, reducing education costs while maintaining education quality.

[0766] A "student" is a student or learner who attends an educational institution or online course and takes lectures or classes.

[0767] "Means for accepting questions" refers to the interface and functions for accepting and processing text questions entered by users and managing them on the system.

[0768] "Lecture video" refers to audio and video data that records the contents of a class or lecture.

[0769] "Means for analyzing audio and video data" refers to processing devices and software that use voice recognition technology to convert the audio portion of recorded lecture videos into text and image recognition technology to analyze the video portion and extract information.

[0770] "Means of understanding the question and generating an appropriate answer" refers to an AI model or algorithm that uses natural language processing technology to analyze the question text entered by the user and generate an appropriate answer.

[0771] "Means for capturing facial expressions and tone of voice and recognizing emotional states" refers to a system that uses a camera and microphone to collect facial expression data and voice data of a user and analyzes their emotional state based on emotion recognition technology.

[0772] "Means for appropriately adjusting the generated response based on the emotional state of the student" refers to a processing device or algorithm that modifies or adjusts the content and tone of the generated response based on the data obtained through emotion recognition.

[0773] "Means for transmitting answers to students" refers to a system that includes a communication interface or protocol for transmitting the generated and adjusted answer data to the student's terminal.

[0774] "Means for access authentication" refers to the database and authentication server that verify the user ID and password when a student logs in to the system and authenticates access rights.

[0775] "Means for structuring data and storing it in a database" refers to a system or software that structures the analyzed lecture content data into a certain format (e.g., JSON or XML) and stores it in a database (e.g., an SQL database).

[0776] This invention relates to a system that combines AI technology and emotion recognition technology to instantly answer questions posed by students about recorded lecture videos. The system includes the following specific elements and processes:

[0777] System configuration

[0778] 1. Server: The central management system stores data, analyzes data, handles question-answering, and recognizes emotions.

[0779] 2. Terminal: A device used by students (personal computer, tablet, smartphone, etc.) that provides an input interface for watching recorded lessons, entering questions, and sentiment analysis.

[0780] 3. Users: Students who watch recorded lessons and ask questions.

[0781] Program processing explanation

[0782] 1. Student Access Authentication

[0783] User: Enter your user ID and password on the login screen.

[0784] Terminal: The entered authentication information is encrypted and sent to the server using HTTPS as the communication protocol.

[0785] Server: Compares the received authentication information with a database (e.g., MySQL) and returns the authentication result to the terminal. If authentication is successful, it generates access rights and a list of available lecture videos.

[0786] 2. Select the recorded lesson video and start streaming

[0787] User: Select the video they want to watch from the lecture video list.

[0788] Terminal: Sends a request for the selected lecture video to the server. HTTPS is used as the communication protocol.

[0789] Server: Retrieves the specified video file from a video database (e.g., AWS S3) and streams it to the user's device using the HLS (HTTP Live Streaming) protocol.

[0790] 3. Analysis of lesson content

[0791] Server: Sends recorded lesson videos and analyzes them using multimodal AI (e.g., Google Cloud AI).

[0792] Server: The audio portion is converted into text using speech recognition software (e.g., Google Speech-to-Text), and the video portion is analyzed using image recognition software (e.g., Amazon Rekognition).

[0793] Server: Integrates speech-to-text data and video analysis data and stores them as structured data in a database (e.g., PostgreSQL).

[0794] 4. Emotion Recognition by Emotion Engine

[0795] Device: Uses a camera and microphone to capture students' facial expressions and tone of voice in real time.

[0796] Terminal: Sends captured facial expression and audio data to the server. It is recommended to use the WebSocket protocol.

[0797] Server: Analyzes students' emotional states using emotion recognition software (e.g., Microsoft Azure Emotion API) and generates emotion data.

[0798] 5. Accepting questions

[0799] Users: If they have any questions while watching a recorded lesson, they can type their questions in the text box.

[0800] Terminal: The entered question text and emotion data are sent to the server using HTTPS.

[0801] 6. Question Analysis and Answer Generation

[0802] Server: Analyzes questions sent by students using natural language processing (NLP) technology (e.g., OpenAI GPT-4).

[0803] AI model: Analyzes the question, searches the lecture content database, and generates an appropriate answer.

[0804] Server: The emotion engine analyzes the student's emotional state and adjusts the response accordingly.

[0805] 7. Submitting and Viewing Your Answers

[0806] Server: Sends the adjusted answer to the user's device. HTTPS is used as the communication protocol.

[0807] User: Checks the answers on the device, resolves any doubts, and can enter additional questions as needed.

[0808] Specific examples

[0809] If a student is confused about how to calculate sales tax, they can type "How do you calculate sales tax?" into the text box.

[0810] The server receives this question and further analyzes the student's emotional data. The AI ​​model searches for relevant lecture content data and generates an answer such as, "Consumption tax is the purchase amount multiplied by the consumption tax rate. For example, if a product costs 1,000 yen and the consumption tax rate is 10%, the consumption tax will be 100 yen."

[0811] Using emotion recognition, the server detects confusion in the student, softening the tone of its responses and adding additional explanations such as, "If you have difficulty understanding, I'll explain how to do the calculation again."

[0812] The user can check the answer on the device to resolve their doubts, and can also enter additional questions if necessary.

[0813] Example prompt sentence:

[0814] "Please provide a professional explanation of the calculation of sales tax. Also, please show how to respond to students who are confused by the following question: How do you calculate sales tax?"

[0815] As described above, this system allows students to resolve any questions they may have while watching recorded lessons in real time, and responds appropriately through emotion recognition. This is expected to improve the quality and efficiency of learning, reduce the burden on instructors, and reduce education costs while maintaining the quality of education.

[0816] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0817] Step 1: Student Access Authorization

[0818] User: Enter your user ID and password. The input screen has a text box where you can enter the information.

[0819] Terminal: The entered authentication information is sent to the server via HTTPS. The input is in plain text and is encrypted before being sent.

[0820] Server: Compares the received authentication information with a database (e.g., MySQL), and if authentication is successful, generates access privileges and a list of lecture videos and sends them to the terminal. If authentication fails, generates an error message and sends it to the terminal. The input is encrypted user information, and the output is the authentication result and a list of videos or an error message.

[0821] Step 2: Select the recorded lesson video and start streaming

[0822] User: Select the video they want to watch from the lecture video list. For example, click on the video title on the list.

[0823] Terminal: Sends a request for the selected lecture video to the server via HTTPS protocol. The input is the video ID and video request information, and the output is the request data to the server.

[0824] Server: Retrieves the specified video file from the video database (e.g., AWS S3) and streams it to the device in HLS format. The input is the video ID and the output is the streaming data.

[0825] Step 3: Analysis of lesson content

[0826] Server: Sends recorded lecture videos to a multimodal AI (e.g., Google Cloud AI). The input is the lecture video data, and the output is an analysis request to the AI.

[0827] Server: The audio portion of the video is converted into text using speech recognition software (e.g., Google Speech-to-Text). The input is audio data, and the output is text data.

[0828] Server: Analyzes the video portion of the video using image recognition software (e.g., Amazon Rekognition) and extracts information from slides and whiteboards. The input is video data, and the output is the text data of the analysis results.

[0829] Server: Integrates speech-to-text data and video analysis data and stores them as structured data in a database (e.g., PostgreSQL). The input is text data and video analysis data, and the output is structured data.

[0830] Step 4: Emotion Recognition with the Emotion Engine

[0831] Device: Uses a camera and microphone to capture the student's facial expressions and tone of voice in real time. The input is real-time video and audio data.

[0832] Terminal: Sends captured data to the server. It is recommended to use the WebSocket protocol. The input is the captured data and the output is the data sent to the server.

[0833] Server: Analyzes students' emotional states using emotion recognition software (e.g., Microsoft Azure Emotion API) and generates emotion data. The input is the captured data, and the output is emotion data.

[0834] Step 5: Ask a question

[0835] Users: If they have questions while watching a recorded lesson, they can type their question in the text box. For example, they can type "I don't know how to use this formula."

[0836] Terminal: The input question and emotion data are sent to the server via HTTPS. The input is the question text and emotion data, and the output is the data sent to the server.

[0837] Step 6: Parsing the question and generating an answer

[0838] Server: Analyzes questions sent by students using natural language processing (NLP) technology (e.g., OpenAI GPT-4). The input is the question text, and the output is the analysis result.

[0839] AI model: Using the analyzed question content, it searches the lecture content database, obtains relevant information, and generates an answer. The input is the analysis result, and the output is the initial answer.

[0840] Server: Based on the emotional state of the student recognized by the emotion engine, the generated answer is adjusted accordingly. The input is the initial answer and emotional data, and the output is the adjusted answer.

[0841] Step 7: Submit and view your responses

[0842] Server: Sends the adjusted answers to the student's device. HTTPS protocol is used. The input is the adjusted answers, and the output is the data sent to the device.

[0843] User: Checks the answer on the device and resolves any doubts. User can also enter additional questions if necessary. Input is the displayed answer.

[0844] (Application example 2)

[0845] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0846] Previously, it was difficult to respond immediately to customer questions with product introduction videos in virtual stores. In particular, when customers asked specific questions about a product, it was sometimes difficult to read the customer's emotional state from their facial expressions or tone of voice, which resulted in delayed responses. This could lead to lower customer satisfaction and lost sales opportunities.

[0847] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0848] In this invention, the server includes means for accepting questions entered by students, means for analyzing audio and video data of recorded educational videos, means for understanding the content of the questions based on the analyzed educational content and generating appropriate answers, means for sending the generated answers to students, means for capturing facial expressions and tone of voice in real time and recognizing emotional states, and means for adjusting the content and tone of answers based on the recognized emotional states. This enables customer service in the virtual store to be quick and accurate, thereby improving customer satisfaction.

[0849] "Participants" are users who watch recorded educational videos and enter questions.

[0850] A "means for accepting questions" is an interface that accepts text input from a user and sends it to the system.

[0851] A "recorded educational video" is a video containing pre-recorded educational content.

[0852] "Means for analyzing audio and video data" refers to the technology and functions that convert the audio data of recorded educational videos into text and extract important information from the video data.

[0853] "Means of understanding the question and generating an appropriate answer" refers to the technology and functions that use natural language processing technology to analyze the question, retrieve relevant information from a database, and generate an answer.

[0854] The "means for transmitting the generated answers to the students" refers to an interface for presenting the generated answers to the users in real time.

[0855] "Means of capturing facial expressions and tone of voice in real time and recognizing emotional states" refers to technologies and functions that capture a user's facial expressions and tone of voice using a camera or microphone and analyze their emotions in real time.

[0856] "Means for adjusting the content and tone of responses based on a recognized emotional state" refers to technologies and functions that adjust the content and tone of generated responses based on the results of emotional analysis, making them easier for users to understand and more reassuring.

[0857] In this invention, the system is realized by the server, terminal, and user performing the following processes.

[0858] server

[0859] The server analyzes and stores various data, processes questions and answers, and recognizes emotions. Specifically, it uses the following hardware and software:

[0860] Hardware: high-performance processor, storage device, large memory

[0861] Software: Natural language processing engines (e.g., OpenAI GPT-4), emotion recognition engines, video processing software, database management systems

[0862] The server process includes:

[0863] 1. Audio and video data analysis: Converting audio data from recorded educational videos into text and extracting important information from video data. This process uses voice recognition and image analysis technologies.

[0864] 2. Question analysis and answer generation: The system uses natural language processing technology to analyze questions from students, retrieves relevant information from a database, and generates answers that correspond to the questions. The generated answers incorporate emotion recognition results as needed, adjusting the tone of the answers to reflect the student's emotional state.

[0865] 3. Emotional state recognition and adjustment: Analyzes the participant's facial expressions and tone of voice to recognize their emotional state in real time, and adjusts the content and tone of their responses accordingly.

[0866] Terminal

[0867] Devices are devices used by students, including PCs, tablets, smartphones, smart glasses, and head-mounted displays. They use the following hardware and software:

[0868] Hardware: Camera, microphone, display, storage, processor

[0869] Software: User Interface (UI) applications, video streaming applications

[0870] Terminal processing includes:

[0871] 1. Login and authentication: The student enters their user ID and password and sends the authentication information to the server.

[0872] 2. Video viewing and playback: Receives and plays recorded lecture videos provided by the server, and provides an interface for students to input questions while watching.

[0873] 3. Capture and transmit emotional data: Use a camera and microphone to capture the student's facial expressions and tone of voice and transmit them to the server.

[0874] User

[0875] A user is a student who uses this system and performs the following operations:

[0876] 1. Login: Enter your user ID and password on each terminal to log in to the system.

[0877] 2. Watch Video: Select a video from the list provided and watch the video.

[0878] 3. Enter a question: If you have any questions while watching the video, enter your question in the dedicated text box.

[0879] Examples of concrete examples and prompts

[0880] For example, if a student is watching a recorded lecture on their smartphone and types a question like, "Please tell me about the camera performance of this smartphone." If the student's facial expression sounds confused and their tone of voice sounds anxious, the server will analyze the question and generate an answer like this:

[0881] The smartphone's camera is equipped with a 12MP wide-angle lens, allowing you to take clear photos even at night. Want to hear more details? You can also call our support team if you need help.

[0882] An example of this prompt statement:

[0883] "User Question: What are the specific features of this product?

[0884] Expression: Confused.

[0885] Voice tone: Anxious.

[0886] This is a situation that requires action."

[0887] As a result, customer service at the virtual store can be provided quickly and accurately, improving customer satisfaction.

[0888] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0889] Step 1:

[0890] Login and Authentication

[0891] Terminal: The user launches the application on the terminal and enters their user ID and password. The entered authentication information is sent to the server.

[0892] Server: The server compares the received authentication information with a database and performs authentication. If authentication is successful, it returns access rights and a list of available content to the terminal. If authentication fails, it sends an error message to the terminal. The input is the user ID and password, and the output is the authentication result.

[0893] Step 2:

[0894] Select content and start streaming

[0895] Device: The user selects the video they want to watch from a content list provided by the application on the device. A request for the selected video is sent to the server.

[0896] Server: The server retrieves the selected video from the database and sends it to the device in streaming format. Video playback position information is also sent at the same time. The input is the video selection request, and the output is streaming data.

[0897] Step 3:

[0898] Video content analysis

[0899] Server: The audio data from the video is analyzed using speech recognition technology (e.g., Google Speech-to-Text) and converted into text. The video data is analyzed using image analysis technology (e.g., OpenCV) to extract important information. This data is integrated and stored in a database as structured data. The input is the audio and video data from the video, and the output is structured data.

[0900] Step 4:

[0901] Capturing and transmitting emotional data

[0902] Terminal: The camera and microphone capture the user's facial expressions and tone of voice in real time and send the data to the server. The input is real-time data from the camera and microphone, and the output is data sent to the server.

[0903] Step 5:

[0904] Recognition of emotional states

[0905] Server: Using an emotion recognition engine (e.g., Microsoft Azure Emotion API), the server analyzes the user's emotional state from the facial expression and tone of voice data sent. The analysis results are stored as internal data and used when answering questions. The input is facial expression and tone of voice data, and the output is the analysis result of the emotional state.

[0906] Step 6:

[0907] Accepting and analyzing questions

[0908] Terminal: If a user has a question while watching a video, they can enter it in the text box, which is then sent to the server.

[0909] Server: The server uses a natural language processing engine (e.g., OpenAI GPT-4) to analyze the question and retrieve relevant information from a database. The input is a text question, and the output is the analysis result based on the question.

[0910] Step 7:

[0911] Answer generation and adjustment

[0912] Server: Generates answers to questions based on information retrieved from a database. Furthermore, it adjusts the content and tone of the answers based on the results of emotional state analysis. The input is the results of question analysis and emotional state analysis, and the output is the adjusted answer.

[0913] Step 8:

[0914] Submitting and viewing answers

[0915] Server: Sends the adjusted answer to the device.

[0916] Terminal: The terminal displays the received answers to the user, who reviews the answers and asks follow-up questions if necessary. The input is the adjusted answers, and the output is the answers displayed to the user.

[0917] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0918] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0919] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0920] [Third embodiment]

[0921] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0922] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0923] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0924] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0925] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0926] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0927] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0928] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0929] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0930] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0931] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0932] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0933] This invention relates to a system that uses AI technology to instantly answer questions from students about recorded lecture videos. The system includes the following specific elements and processes:

[0934] System configuration

[0935] 1. Server: A central management system that stores data, analyzes data, and handles question and answering.

[0936] 2. Device: Provides an interface for students to watch recorded lessons and enter questions on their devices (PCs, tablets, smartphones, etc.).

[0937] 3. Users: Students who watch recorded lessons and ask questions.

[0938] Overview of system program processing

[0939] The program processing of this system is explained in natural language below.

[0940] 1. Student Access Authentication

[0941] User: Enter your user ID and password on the login screen.

[0942] Terminal: Sends the entered authentication information to the server.

[0943] Server: Authenticates the user against the database, and if successful, returns a list of lecture videos to the user.

[0944] 2. Start streaming recorded lecture videos

[0945] User: Select the lecture video they want to watch.

[0946] Device: Sends a request for the selected video to the server.

[0947] Server: Retrieves the relevant video from the video database and starts streaming.

[0948] 3. Analysis of lesson content

[0949] Server: Passes the audio and video portions of the lesson video to the multimodal AI.

[0950] AI model (inside the server): Converts audio into text (voice recognition) and analyzes blackboard notes and slides in the video to convert them into text (image analysis).

[0951] Server: Structures the parsed data and stores it in a database.

[0952] 4. Questions

[0953] User: If a question arises during the lesson being viewed, enter the question in the text box.

[0954] Terminal: Sends the entered question to the server.

[0955] 5. Question Answering

[0956] Server: Receives questions from users and analyzes the content of the questions.

[0957] AI model (inside the server): Compares the analyzed questions with pre-structured lecture content and generates appropriate answers.

[0958] Server: Sends the generated answer to the user's device.

[0959] User: Check the AI's answer sent back from the device.

[0960] To give a specific example, if a student doesn't understand how to calculate consumption tax and asks, "How do you calculate consumption tax?", the server receives the question and passes it to the AI ​​model. The AI ​​model searches for relevant parts of the lesson content and generates an answer: "Consumption tax is the purchase price multiplied by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen item, the consumption tax will be 100 yen." The server then sends this to the user's device. The user can instantly check the appropriate answer and resolve their question.

[0961] This system allows students to resolve their questions on the spot, significantly improving learning efficiency. In addition, since instructors do not need to respond directly to students' questions, the burden on them is reduced, making it possible to reduce education costs while maintaining the quality of education.

[0962] The processing flow will be explained below.

[0963] Step 1: Student Access Authorization

[0964] User: Enter your user ID and password on the login screen.

[0965] Terminal: Sends the entered authentication information to the server.

[0966] Server: Authenticates the user against the information stored in the database, and if authentication is successful, returns access rights and a list of available lecture videos to the user. If authentication fails, displays an error message.

[0967] Step 2: Select the recorded lesson video and start streaming

[0968] User: Select the video they want to watch from the list of lecture videos provided.

[0969] Terminal: Sends a request for the selected lecture video to the server.

[0970] Server: Retrieves the selected lecture video from the database and sends it to the user's device in streaming format. The server also sends the video playback position information.

[0971] Step 3: Analysis of lesson content

[0972] Server: The recorded lesson video is sent to the multimodal AI. The audio portion of the video is processed through speech recognition and converted into text (speech-to-text).

[0973] Server: The contents of the slides and whiteboards are subjected to image analysis processing from the video footage to extract important information (video analysis).

[0974] Server: Integrates data obtained from audio-to-text and video analysis, and stores the lecture content in a database as structured data.

[0975] Step 4: Ask a question

[0976] Users: If they have any questions while watching a recorded lesson, they can enter their questions in the dedicated text box.

[0977] Terminal: Sends the entered question to the server.

[0978] Step 5: Parsing the question and generating an answer

[0979] Server: Receives questions sent by students and analyzes the content of the questions using natural language processing (NLP) technology.

[0980] AI model (on the server): Using the analyzed question content, the model searches a pre-structured lecture content database to obtain relevant information, and then generates the most appropriate answer from the obtained relevant information.

[0981] Server: Sends the generated answers to the students' devices.

[0982] Step 6: View and review your answers

[0983] User: Checks the submitted answers on the device, resolves any doubts, and enters additional questions as necessary.

[0984] Step 7: Feedback and Data Storage (Optional)

[0985] Server: Logs all questions and answers into a database for later review by the instructor.

[0986] Instructor: Analyze the topics that students focus on asking questions about and the AI's answers, and prepare additional teaching materials and supplementary lectures as necessary.

[0987] The above processing steps allow students to ask questions immediately while watching recorded lessons, and AI can provide quick answers, improving learning efficiency. This also reduces the burden on instructors and contributes to maintaining the quality of education.

[0988] Example 1

[0989] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0990] In conventional systems that provide recorded lecture videos, it is difficult for students to get immediate answers to questions that arise while watching the video. This reduces learning efficiency and places a burden on the instructor. Therefore, there is a need for a system that allows students to solve their questions in real time while watching the recorded video.

[0991] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0992] In this invention, the server includes a means for accepting questions entered by students, a means for analyzing audio and video data of recorded lecture videos, a means for understanding the question content based on the analyzed lecture content and using a generative AI model to generate an appropriate answer, and a means for transmitting the generated answer to the student, thereby enabling the student to solve their question in real time while watching the recorded video.

[0993] "Student" refers to a person who watches and learns from lecture videos provided online or offline.

[0994] "Means for accepting questions" refers to the interface or protocol that receives questions entered by students and sends them to the server.

[0995] "Lecture video" refers to audio and video data containing recorded lecture content.

[0996] "Means for analyzing audio and video data" refers to technology that converts the audio of lecture videos into text and extracts and analyzes visual information from the video.

[0997] A "generative AI model" refers to an artificial intelligence model that generates appropriate answers based on input questions and data.

[0998] "Means for generating answers" refers to technology that generates appropriate answers to students' questions based on the lecture content analyzed using a generative AI model.

[0999] "Means for sending answers" refers to the communication technology or protocol used to send the generated answers to the student's device.

[1000] "Access Authentication Measures" means the system or method that verifies a student's credentials and verifies access rights.

[1001] "Encryption methods" refers to the technologies and protocols that encrypt information to protect data transmissions.

[1002] "Structured data" refers to data that has been systematically organized from analyzed lecture video information and stored in a database.

[1003] "Speech recognition means" refers to technology that automatically converts voice data into text.

[1004] "Image analysis means" refers to technology that extracts specific visual information from video data and converts it into text.

[1005] This invention relates to a system that uses AI technology to instantly answer questions posed by students about recorded lecture videos. The system consists of three main components: a server that functions as a central management system, terminals where students can watch lecture videos and enter questions, and users who watch the recorded lectures.

[1006] System configuration

[1007] 1. Server:

[1008] The server stores data, analyzes it, and handles question and answering. Specifically, it uses the following hardware and software:

[1009] Database: Used to store authentication information and lecture video data.

[1010] Generative AI model: For example, GPT-4 is used to generate appropriate answers based on the question.

[1011] Speech recognition technology: The audio portion of the lecture video is converted into text using the Google Speech-to-Text API.

[1012] Image analysis technology: The video portion of the lecture video is analyzed using Google Cloud Vision API and converted into text.

[1013] 2. Terminal:

[1014] The device used by students (PC, tablet, smartphone, etc.) provides an interface for viewing recorded lessons and entering questions. The device has the following functions:

[1015] Login interface: An interface for entering a user ID and password.

[1016] Video playback function: Lecture videos retrieved from the server are played in streaming format.

[1017] Question input interface: An interface for students to enter their questions into a text box.

[1018] 3. User:

[1019] Students watch the recorded lectures and ask questions.

[1020] Specific actions

[1021] Below, we will explain how this system works with a concrete example.

[1022] 1. User access authentication

[1023] The user enters their user ID and password on the login screen.

[1024] The terminal encrypts the entered authentication information and sends it to the server.

[1025] The server checks the authentication information stored in the database and, if authentication is successful, returns a list of lecture videos to the user.

[1026] 2. Start streaming recorded lecture videos

[1027] The user selects the lecture video they wish to watch from the list.

[1028] The terminal sends a request for the selected video to the server.

[1029] The server retrieves the relevant video from the video database and streams it.

[1030] 3. Analysis of lesson content

[1031] The server passes the audio and video portions of the lesson video to the multimodal AI. The audio portion is converted to text using the Google Speech-to-Text API, and the video portion is analyzed and converted to text using the Google Cloud Vision API.

[1032] The server structures the parsed data and stores it in a database.

[1033] 4. Questions

[1034] If a user has a question about a lesson they are watching, they can enter it in the text box.

[1035] The terminal transmits the entered question to the server.

[1036] 5. Question Answering

[1037] The server receives a question from a user and analyzes the question.

[1038] The generative AI model matches the analyzed questions with pre-structured lecture content and generates appropriate answers.

[1039] The server sends the generated answer to the user's terminal.

[1040] The user checks the AI's answers sent back from the device.

[1041] Specific examples

[1042] For example, if a student does not understand how to calculate consumption tax, he or she can input the question, "How do you calculate consumption tax?" In response to this question, the system operates as follows:

[1043] User: Enters the question "How do you calculate sales tax?" into the text box.

[1044] Terminal: Sends entered text to the server.

[1045] Server: Receives the question and sends the prompt "How do you calculate sales tax?" to the generative AI model.

[1046] Generative AI model: Searches the lecture content and generates the answer, "Consumption tax is the purchase amount multiplied by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen product, the consumption tax will be 100 yen."

[1047] Server: Sends the generated answer to the user's device.

[1048] Terminal: Receives the answer and displays it to the user.

[1049] User: Review the answers provided and understand how sales tax is calculated.

[1050] This system allows students to get answers to their questions in real time, improving learning efficiency and reducing the burden on instructors.

[1051] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1052] Step 1:

[1053] Authenticating users for access

[1054] 1. Input: The user enters their user ID and password on the login screen.

[1055] 2. Operation: The device encrypts the entered authentication information and sends it to the server using a protocol such as SSL / TLS.

[1056] 3. Input: The server checks the received authentication information against the information stored in its database.

[1057] 4. Data processing: The server authenticates the user based on the matching result. If authentication is successful, the user information is saved in the session.

[1058] 5. Output: The server generates a lecture video list for the user and sends it to the terminal.

[1059] 6. Operation: The terminal displays a list of lecture videos.

[1060] Step 2:

[1061] Streaming of recorded lecture videos begins

[1062] 1. Input: The user selects the lecture video they want to watch from the list.

[1063] 2. Action: The device sends a request for the selected video to the server.

[1064] 3. Input: The server retrieves the relevant video data from the video database.

[1065] 4. Data processing: The server converts the video data into a streaming format, often using tools such as FFmpeg.

[1066] 5. Output: The server sends the converted streaming data to the terminal.

[1067] 6. Operation: The device plays the received streaming data.

[1068] Step 3:

[1069] Analysis of lesson content

[1070] 1. Input: The server extracts the audio and video parts of the lesson video.

[1071] 2. How it works: The server passes the audio portion to the Google Speech-to-Text API for speech recognition.

[1072] 3. Input: The server receives the text data obtained by speech recognition.

[1073] 4. Data processing: The audio data is saved as text along with time information.

[1074] 5. Operation: The server passes the video portion to the Google Cloud Vision API for image analysis.

[1075] 6. Input: The server receives the text data obtained from image analysis.

[1076] 7. Data processing: The information on the board and slides in the video is saved as text along with time information.

[1077] 8. Output: The server structures the acquired text data and stores it in a database.

[1078] Step 4:

[1079] Questions accepted

[1080] 1. Input: When a user has a question about the lesson they are watching, they can enter it in the text box.

[1081] 2. Operation: The terminal sends the entered question data to the server.

[1082] 3. Input: The server adds the received query data to a queue for analysis.

[1083] 4. Output: The question data added to the queue is passed to the next question-answering process.

[1084] Step 5:

[1085] Question and Answering

[1086] 1. Input: The server retrieves the question data in the queue.

[1087] 2. Operation: The server analyzes the question using NLP (Natural Language Processing) technology.

[1088] 3. Input: The server sends the parsed question content to the generative AI model as a prompt sentence.

[1089] 4. Output: Based on a specific prompt, such as "How do you calculate sales tax?", the generative AI model generates an answer.

[1090] 5. Input: Receive the answer from the generative AI model.

[1091] 6. Data processing: The server structures the answer and sends it to the user's device.

[1092] 7. Operation: The terminal receives the generated response and displays it to the user.

[1093] 8. Output: The user checks the answer displayed on the terminal and resolves their doubt.

[1094] (Application example 1)

[1095] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1096] With conventional video lecture systems, students had no way to ask questions and get answers immediately when they had doubts. This resulted in problems such as unresolved questions and reduced learning efficiency. Furthermore, it was a significant burden on instructors to deal with questions afterward, making it difficult to provide an efficient education.

[1097] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1098] In this invention, the server includes a means for accepting questions entered by students, a means for analyzing audio and video data from recorded lecture videos, and a means for understanding the questions based on the analyzed lecture content and generating appropriate answers. This allows the generated answers to be sent immediately to students. The server also includes a means for using a mobile information terminal that enables viewing of recorded lectures and question entry, a means for performing speech recognition and question answering using a generative AI model, and a means for using a central management system that receives, processes, and transmits various data. This allows students to instantly resolve their questions using mobile information terminals such as smartphones and tablets, improving their learning efficiency.

[1099] "Students" are learners who watch recorded lecture videos and enter questions.

[1100] "Means for accepting questions" refers to an interface for accepting questions entered by students and sending them into the system.

[1101] A "lecture video" is a file containing video and audio data of recorded educational content.

[1102] "Means for analyzing audio and video data" refers to the technical methods for extracting audio and video information from recorded lecture videos, converting it into text, and performing semantic analysis.

[1103] "Means of understanding the question and generating an appropriate answer" refers to a function that analyzes questions sent by students, compares them with related lecture content, and automatically creates the most appropriate answer.

[1104] "Means for sending generated answers" refers to a mechanism for sending answers generated by AI to students' devices in real time.

[1105] "Mobile information terminal" refers to a portable information processing device such as a smartphone or tablet.

[1106] "Generative AI model" refers to a learning model that uses artificial intelligence to perform speech recognition, natural language processing, and question answering.

[1107] "Central management system" refers to the server and related infrastructure for centrally managing the reception and processing of questions from students, the analysis of lecture videos, and the generation and transmission of answers.

[1108] This invention relates to a system that uses a generative AI model to instantly answer questions posed by students about recorded lecture videos. This system is realized by the following specific components and the mechanisms by which these components work together.

[1109] System configuration

[1110] 1. Server: As a central management system, it accepts questions from students, analyzes recorded lecture videos, generates questions and answers, and sends the generated answers.

[1111] 2. Devices: Mobile information devices such as smartphones and tablets used by students provide an interface for watching lecture videos and entering questions.

[1112] 3. User: A student who watches the recorded lecture and enters questions.

[1113] Program processing description

[1114] Login Authentication

[1115] The server receives the user ID and password sent by the student from their device and checks them against a database. If authentication is successful, it returns a list of lecture videos to the device. For security reasons, this process uses encrypted communication such as SSL / TLS.

[1116] Streaming lecture videos

[1117] When a user selects a lecture video to watch on their device, the server retrieves the corresponding video data from the video database and starts streaming it to the device. The video data is saved in standard formats such as MP4 and WebM.

[1118] Analysis of lesson content

[1119] The server passes the audio and video portions of the recorded lecture video to a generative AI model, which performs speech recognition to convert the audio into text and analyzes the blackboard notes and slides in the video to generate text. The analyzed data is structured and stored in a database. This process uses natural language processing libraries (e.g., HuggingFace Transformers) and image analysis software.

[1120] Question reception and answering

[1121] When a user has a question about a lesson they are watching, they send it in text format from their device to the server. The server then passes the received question to a generative AI model, which analyzes the question and generates an appropriate answer. The generated answer is then immediately sent back to the device. This allows students to receive answers to their questions in real time.

[1122] Specific examples of programs

[1123] Authentication Example

[1124] When a user enters "User ID: student1" and "Password: password1" on the login screen on their smartphone and sends it to the server, the server checks the database and, if the authentication information is correct, returns a list of lecture videos.

[1125] Example questions and answers

[1126] When a user inputs the question, "How do you calculate economic growth rate?", the generative AI model searches for relevant lecture content and generates the answer, "Economic growth rate is an indicator that shows how much a country's economy has grown, and specifically, it is calculated as the annual rate of change in gross domestic product (GDP)." The server then sends this answer to the user's device.

[1127] Example prompt sentence:

[1128] How do you calculate economic growth rate?

[1129] In this way, this invention allows students to resolve questions in real time while watching recorded lessons, improving learning efficiency and reducing the burden on instructors. Furthermore, this system is highly convenient because it uses mobile information terminals such as smartphones and tablets.

[1130] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1131] Step 1:

[1132] The server accepts the authentication information of the user ID and password entered by the student using the terminal and compares it with the database. The input is the user ID and password, and the output is the authentication result. If the authentication is successful, the server returns a list of lecture videos to the terminal. Specifically, the authentication process is carried out securely using SSL / TLS encrypted communication.

[1133] Step 2:

[1134] The server receives the lecture video ID requested by the student from the terminal and retrieves the video data corresponding to that ID from the video database. The input is the video request information (video ID), and the output is the streaming URL of the corresponding lecture video. Specifically, the server queries the video database, generates a video URL, and sends it to the terminal.

[1135] Step 3:

[1136] The server passes the audio and video data of the lecture video to the generative AI model for analysis. The input is the audio and video data of the lecture video, and the output is the text of the lecture content. Specifically, the audio data is converted into text using voice recognition software, and at the same time, the text and slides in the video are converted into text using image analysis software. The analyzed data is then structured and stored in a database.

[1137] Step 4:

[1138] When a student has a question during a class they are watching, they use their device to enter the question into a text box and send it to the server. The input is the question text entered by the student, and the output is a confirmation by the server that the question has been received. Specifically, the question is entered into the device interface and sent to the server.

[1139] Step 5:

[1140] The server passes the questions sent by the students to a generative AI model, which analyzes the content of the questions and generates appropriate answers. The input is the question text from the students, and the output is the generated answer text. Specifically, the server uses a generative AI model (e.g., HuggingFace Transformers) to generate answers from lecture data related to the question content.

[1141] Step 6:

[1142] The server sends the generated answer text to the student's terminal. The input is the generated answer text, and the output is the answer information displayed on the student's terminal. In concrete terms, the generated answer is sent to the student's terminal and displayed in real time.

[1143] This allows students to instantly resolve any questions they may have while watching the lecture videos, significantly improving their learning efficiency.

[1144] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1145] This invention relates to a system that combines AI technology and emotion recognition to instantly answer questions from students about recorded lecture videos. The system includes the following specific elements and processes:

[1146] System configuration

[1147] 1. Server: The central management system stores data, analyzes data, handles question-answering, and recognizes emotions.

[1148] 2. Device: Provides an input interface for viewing recorded lessons, entering questions, and analyzing sentiment on devices used by students (PCs, tablets, smartphones, etc.).

[1149] 3. Users: Students who watch recorded lessons and ask questions.

[1150] Overview of system program processing

[1151] The program processing of this system is explained in natural language below.

[1152] 1. Student Access Authentication

[1153] User: Enter your user ID and password on the login screen.

[1154] Terminal: Sends the entered authentication information to the server.

[1155] Server: Authenticates the user against the information stored in the database, and if authentication is successful, returns access rights and a list of available lecture videos to the user. If authentication fails, displays an error message.

[1156] 2. Select the recorded lesson video and start streaming

[1157] User: Select the video they want to watch from the list of lecture videos provided.

[1158] Terminal: Sends a request for the selected lecture video to the server.

[1159] Server: Retrieves the selected lecture video from the database and sends it to the user's device in streaming format. The server also sends the video playback position information.

[1160] 3. Analysis of lesson content

[1161] Server: The recorded lesson video is sent to the multimodal AI. The audio portion of the video is processed through speech recognition and converted into text (speech-to-text).

[1162] Server: The contents of the slides and whiteboards are subjected to image analysis processing from the video footage to extract important information (video analysis).

[1163] Server: Integrates data obtained from audio-to-text and video analysis, and stores the lecture content in a database as structured data.

[1164] 4. Emotion Recognition by Emotion Engine

[1165] Device: Uses a camera and microphone to capture students' facial expressions and tone of voice in real time.

[1166] Terminal: Sends captured data to the server.

[1167] Server: Runs an emotion engine to recognize students' emotional states from their facial expressions and tone of voice, and generates emotion data.

[1168] 5. Accepting questions

[1169] Users: If they have any questions while watching a recorded lesson, they can enter their questions in the dedicated text box.

[1170] Terminal: Sends the entered question and emotion data to the server.

[1171] 6. Question Analysis and Answer Generation

[1172] Server: Receives questions sent by students and analyzes the content of the questions using natural language processing (NLP) technology.

[1173] AI model (on the server): Using the analyzed question content, the model searches a pre-structured lecture content database to obtain relevant information, and then generates the most appropriate answer from the obtained relevant information.

[1174] Server: Adjusts the content and tone of responses appropriately based on the student's emotional state as recognized by the emotion engine.

[1175] 7. Submitting and Viewing Your Answers

[1176] Server: Adjusts the generated answer and sends it to the user's device.

[1177] User: Checks the submitted answers on the device, resolves any doubts, and enters additional questions as necessary.

[1178] To give a specific example, if a student does not understand how to calculate consumption tax and looks confused, they can input the question, "How do you calculate consumption tax?" The server receives the question and analyzes the emotional data using an emotion engine. The AI ​​model searches for relevant parts of the lesson content and responds with, "Consumption tax is calculated by multiplying the purchase price by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen item, the consumption tax will be 100 yen." Because the emotion engine recognizes the student's confusion, it softens the tone of the response and provides additional explanation. For example, it outputs a response such as, "If you have difficulty understanding, I'll explain how to calculate it again." The user immediately confirms the appropriate answer and the problem is resolved.

[1179] This system not only allows students to resolve their questions immediately, but also improves the quality and efficiency of learning by enabling appropriate responses through emotion recognition. It also reduces the burden on instructors, reducing education costs while maintaining the quality of education.

[1180] The processing flow will be explained below.

[1181] Step 1: Student Access Authorization

[1182] User: Enter your user ID and password on the login screen.

[1183] Terminal: Sends the entered authentication information to the server.

[1184] Server: Authenticates the user against the information stored in the database, and if successful, returns access rights and a list of available lecture videos to the user. If unsuccessful, displays an error message.

[1185] Step 2: Select the recorded lesson video and start streaming

[1186] User: Select the video they want to watch from the list of lecture videos provided.

[1187] Terminal: Sends a request for the selected lecture video to the server.

[1188] Server: Retrieves the selected lecture video from the database and sends it to the user's device in streaming format, along with playback position information.

[1189] Step 3: Analysis of lesson content

[1190] Server: The recorded lesson video is sent to the multimodal AI. The audio portion of the video is processed through speech recognition and converted into text (speech-to-text).

[1191] Server: The contents of the slides and whiteboards are subjected to image analysis processing from the video footage to extract important information (video analysis).

[1192] Server: Integrates data obtained from audio-to-text and video analysis, and stores the lecture content in a database as structured data.

[1193] Step 4: Emotion Recognition with the Emotion Engine

[1194] Device: Uses a camera and microphone to capture students' facial expressions and tone of voice in real time.

[1195] Terminal: Sends captured facial expression and voice data to the server.

[1196] Server: The transmitted data is analyzed using an emotion engine, which recognizes the student's emotional state and generates emotion data.

[1197] Step 5: Ask a question

[1198] Users: If they have any questions while watching a recorded lesson, they can enter their questions in the dedicated text box.

[1199] Terminal: Sends the entered question and emotion data to the server.

[1200] Step 6: Parsing the question and generating an answer

[1201] Server: Receives questions sent by students and analyzes the content of the questions using natural language processing (NLP) technology.

[1202] AI model (inside the server): Using the analyzed question content, it searches a pre-structured database of lecture content to obtain relevant information.

[1203] Server: Generates the most appropriate answer from the relevant information obtained.

[1204] Server: Adjusts the content and tone of the generated answers accordingly based on the student's emotional state as recognized by the emotion engine.

[1205] Step 7: Submit and view your responses

[1206] Server: Sends the adjusted answer to the user's device.

[1207] User: Checks the submitted answers on the device, resolves any doubts, and enters additional questions as necessary.

[1208] To give a specific example, if a student doesn't understand how to calculate consumption tax and asks, "How do you calculate consumption tax?", the server receives the question and analyzes the emotion data using the emotion engine. The AI ​​model searches for relevant parts of the lesson content and generates an answer: "Consumption tax is the purchase price multiplied by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen item, the consumption tax will be 100 yen." The emotion engine recognizes the student's confusion and softens the tone of the answer, providing additional explanation such as, "If you have difficulty understanding, I'll explain how to calculate it again." The user can instantly confirm the appropriate answer and resolve their doubt.

[1209] Example 2

[1210] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1211] In conventional learning systems, if a student has a question while watching a recorded lesson, it is difficult to get an appropriate answer immediately on the spot. Furthermore, the system does not respond to the student's emotional state (e.g., confusion, lack of understanding, etc.), which can lead to a decrease in the quality and efficiency of learning. In addition, the instructor is often asked many questions, which increases the burden on the instructor and can lead to a decrease in the quality of education.

[1212] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for accepting questions entered by students, means for analyzing audio and video data of recorded lecture videos, means for understanding the question content based on the analyzed lecture content and generating an appropriate answer, means for capturing the students' facial expressions and tone of voice and recognizing their emotional state, means for appropriately adjusting the generated answer based on the students' emotional state, means for sending the generated answer to the students, means for authenticating the students' access, and means for structuring and storing data acquired from the lecture videos in a database. This allows students to immediately receive appropriate answers if they have questions while watching recorded lectures. Furthermore, emotion recognition technology enables responses based on the students' emotional state, improving the quality and efficiency of learning. Furthermore, it reduces the burden on instructors, reducing education costs while maintaining education quality.

[1213] A "student" is a student or learner who attends an educational institution or online course and takes lectures or classes.

[1214] "Means for accepting questions" refers to the interface and functions for accepting and processing text questions entered by users and managing them on the system.

[1215] "Lecture video" refers to audio and video data that records the contents of a class or lecture.

[1216] "Means for analyzing audio and video data" refers to processing devices and software that use voice recognition technology to convert the audio portion of recorded lecture videos into text and image recognition technology to analyze the video portion and extract information.

[1217] "Means of understanding the question and generating an appropriate answer" refers to an AI model or algorithm that uses natural language processing technology to analyze the question text entered by the user and generate an appropriate answer.

[1218] "Means for capturing facial expressions and tone of voice and recognizing emotional states" refers to a system that uses a camera and microphone to collect facial expression data and voice data of a user and analyzes their emotional state based on emotion recognition technology.

[1219] "Means for appropriately adjusting the generated response based on the emotional state of the student" refers to a processing device or algorithm that modifies or adjusts the content and tone of the generated response based on the data obtained through emotion recognition.

[1220] "Means for transmitting answers to students" refers to a system that includes a communication interface or protocol for transmitting the generated and adjusted answer data to the student's terminal.

[1221] "Means for access authentication" refers to the database and authentication server that verify the user ID and password when a student logs in to the system and authenticates access rights.

[1222] "Means for structuring data and storing it in a database" refers to a system or software that structures the analyzed lecture content data into a certain format (e.g., JSON or XML) and stores it in a database (e.g., an SQL database).

[1223] This invention relates to a system that combines AI technology and emotion recognition technology to instantly answer questions posed by students about recorded lecture videos. The system includes the following specific elements and processes:

[1224] System configuration

[1225] 1. Server: The central management system stores data, analyzes data, handles question-answering, and recognizes emotions.

[1226] 2. Terminal: A device used by students (personal computer, tablet, smartphone, etc.) that provides an input interface for watching recorded lessons, entering questions, and sentiment analysis.

[1227] 3. Users: Students who watch recorded lessons and ask questions.

[1228] Program processing explanation

[1229] 1. Student Access Authentication

[1230] User: Enter your user ID and password on the login screen.

[1231] Terminal: The entered authentication information is encrypted and sent to the server using HTTPS as the communication protocol.

[1232] Server: Compares the received authentication information with a database (e.g., MySQL) and returns the authentication result to the terminal. If authentication is successful, it generates access rights and a list of available lecture videos.

[1233] 2. Select the recorded lesson video and start streaming

[1234] User: Select the video they want to watch from the lecture video list.

[1235] Terminal: Sends a request for the selected lecture video to the server. HTTPS is used as the communication protocol.

[1236] Server: Retrieves the specified video file from a video database (e.g., AWS S3) and streams it to the user's device using the HLS (HTTP Live Streaming) protocol.

[1237] 3. Analysis of lesson content

[1238] Server: Sends recorded lesson videos and analyzes them using multimodal AI (e.g., Google Cloud AI).

[1239] Server: The audio portion is converted into text using speech recognition software (e.g., Google Speech-to-Text), and the video portion is analyzed using image recognition software (e.g., Amazon Rekognition).

[1240] Server: Integrates speech-to-text data and video analysis data and stores them as structured data in a database (e.g., PostgreSQL).

[1241] 4. Emotion Recognition by Emotion Engine

[1242] Device: Uses a camera and microphone to capture students' facial expressions and tone of voice in real time.

[1243] Terminal: Sends captured facial expression and audio data to the server. It is recommended to use the WebSocket protocol.

[1244] Server: Analyzes students' emotional states using emotion recognition software (e.g., Microsoft Azure Emotion API) and generates emotion data.

[1245] 5. Accepting questions

[1246] Users: If they have any questions while watching a recorded lesson, they can type their questions in the text box.

[1247] Terminal: The entered question text and emotion data are sent to the server using HTTPS.

[1248] 6. Question Analysis and Answer Generation

[1249] Server: Analyzes questions sent by students using natural language processing (NLP) technology (e.g., OpenAI GPT-4).

[1250] AI model: Analyzes the question, searches the lecture content database, and generates an appropriate answer.

[1251] Server: The emotion engine analyzes the student's emotional state and adjusts the response accordingly.

[1252] 7. Submitting and Viewing Your Answers

[1253] Server: Sends the adjusted answer to the user's device. HTTPS is used as the communication protocol.

[1254] User: Checks the answers on the device, resolves any doubts, and can enter additional questions as needed.

[1255] Specific examples

[1256] If a student is confused about how to calculate sales tax, they can type "How do you calculate sales tax?" into the text box.

[1257] The server receives this question and further analyzes the student's emotional data. The AI ​​model searches for relevant lecture content data and generates an answer such as, "Consumption tax is the purchase amount multiplied by the consumption tax rate. For example, if a product costs 1,000 yen and the consumption tax rate is 10%, the consumption tax will be 100 yen."

[1258] Using emotion recognition, the server detects confusion in the student, softening the tone of its responses and adding additional explanations such as, "If you have difficulty understanding, I'll explain how to do the calculation again."

[1259] The user can check the answer on the device to resolve their doubts, and can also enter additional questions if necessary.

[1260] Example prompt sentence:

[1261] "Please provide a professional explanation of the calculation of sales tax. Also, please show how to respond to students who are confused by the following question: How do you calculate sales tax?"

[1262] As described above, this system allows students to resolve any questions they may have while watching recorded lessons in real time, and responds appropriately through emotion recognition. This is expected to improve the quality and efficiency of learning, reduce the burden on instructors, and reduce education costs while maintaining the quality of education.

[1263] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1264] Step 1: Student Access Authorization

[1265] User: Enter your user ID and password. The input screen has a text box where you can enter the information.

[1266] Terminal: The entered authentication information is sent to the server via HTTPS. The input is in plain text and is encrypted before being sent.

[1267] Server: Compares the received authentication information with a database (e.g., MySQL), and if authentication is successful, generates access privileges and a list of lecture videos and sends them to the terminal. If authentication fails, generates an error message and sends it to the terminal. The input is encrypted user information, and the output is the authentication result and a list of videos or an error message.

[1268] Step 2: Select the recorded lesson video and start streaming

[1269] User: Select the video they want to watch from the lecture video list. For example, click on the video title on the list.

[1270] Terminal: Sends a request for the selected lecture video to the server via HTTPS protocol. The input is the video ID and video request information, and the output is the request data to the server.

[1271] Server: Retrieves the specified video file from the video database (e.g., AWS S3) and streams it to the device in HLS format. The input is the video ID and the output is the streaming data.

[1272] Step 3: Analysis of lesson content

[1273] Server: Sends recorded lecture videos to a multimodal AI (e.g., Google Cloud AI). The input is the lecture video data, and the output is an analysis request to the AI.

[1274] Server: The audio portion of the video is converted into text using speech recognition software (e.g., Google Speech-to-Text). The input is audio data, and the output is text data.

[1275] Server: Analyzes the video portion of the video using image recognition software (e.g., Amazon Rekognition) and extracts information from slides and whiteboards. The input is video data, and the output is the text data of the analysis results.

[1276] Server: Integrates speech-to-text data and video analysis data and stores them as structured data in a database (e.g., PostgreSQL). The input is text data and video analysis data, and the output is structured data.

[1277] Step 4: Emotion Recognition with the Emotion Engine

[1278] Device: Uses a camera and microphone to capture the student's facial expressions and tone of voice in real time. The input is real-time video and audio data.

[1279] Terminal: Sends captured data to the server. It is recommended to use the WebSocket protocol. The input is the captured data and the output is the data sent to the server.

[1280] Server: Analyzes students' emotional states using emotion recognition software (e.g., Microsoft Azure Emotion API) and generates emotion data. The input is the captured data, and the output is emotion data.

[1281] Step 5: Ask a question

[1282] Users: If they have questions while watching a recorded lesson, they can type their question in the text box. For example, they can type "I don't know how to use this formula."

[1283] Terminal: The input question and emotion data are sent to the server via HTTPS. The input is the question text and emotion data, and the output is the data sent to the server.

[1284] Step 6: Parsing the question and generating an answer

[1285] Server: Analyzes questions sent by students using natural language processing (NLP) technology (e.g., OpenAI GPT-4). The input is the question text, and the output is the analysis result.

[1286] AI model: Using the analyzed question content, it searches the lecture content database, obtains relevant information, and generates an answer. The input is the analysis result, and the output is the initial answer.

[1287] Server: Based on the emotional state of the student recognized by the emotion engine, the generated answer is adjusted accordingly. The input is the initial answer and emotional data, and the output is the adjusted answer.

[1288] Step 7: Submit and view your responses

[1289] Server: Sends the adjusted answers to the student's device. HTTPS protocol is used. The input is the adjusted answers, and the output is the data sent to the device.

[1290] User: Checks the answer on the device and resolves any doubts. User can also enter additional questions if necessary. Input is the displayed answer.

[1291] (Application example 2)

[1292] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1293] Previously, it was difficult to respond immediately to customer questions with product introduction videos in virtual stores. In particular, when customers asked specific questions about a product, it was sometimes difficult to read the customer's emotional state from their facial expressions or tone of voice, which resulted in delayed responses. This could lead to lower customer satisfaction and lost sales opportunities.

[1294] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1295] In this invention, the server includes means for accepting questions entered by students, means for analyzing audio and video data of recorded educational videos, means for understanding the content of the questions based on the analyzed educational content and generating appropriate answers, means for sending the generated answers to students, means for capturing facial expressions and tone of voice in real time and recognizing emotional states, and means for adjusting the content and tone of answers based on the recognized emotional states. This enables customer service in the virtual store to be quick and accurate, thereby improving customer satisfaction.

[1296] "Participants" are users who watch recorded educational videos and enter questions.

[1297] A "means for accepting questions" is an interface that accepts text input from a user and sends it to the system.

[1298] A "recorded educational video" is a video containing pre-recorded educational content.

[1299] "Means for analyzing audio and video data" refers to the technology and functions that convert the audio data of recorded educational videos into text and extract important information from the video data.

[1300] "Means of understanding the question and generating an appropriate answer" refers to the technology and functions that use natural language processing technology to analyze the question, retrieve relevant information from a database, and generate an answer.

[1301] The "means for transmitting the generated answers to the students" refers to an interface for presenting the generated answers to the users in real time.

[1302] "Means of capturing facial expressions and tone of voice in real time and recognizing emotional states" refers to technologies and functions that capture a user's facial expressions and tone of voice using a camera or microphone and analyze their emotions in real time.

[1303] "Means for adjusting the content and tone of responses based on a recognized emotional state" refers to technologies and functions that adjust the content and tone of generated responses based on the results of emotional analysis, making them easier for users to understand and more reassuring.

[1304] In this invention, the system is realized by the server, terminal, and user performing the following processes.

[1305] server

[1306] The server analyzes and stores various data, processes questions and answers, and recognizes emotions. Specifically, it uses the following hardware and software:

[1307] Hardware: high-performance processor, storage device, large memory

[1308] Software: Natural language processing engines (e.g., OpenAI GPT-4), emotion recognition engines, video processing software, database management systems

[1309] The server process includes:

[1310] 1. Audio and video data analysis: Converting audio data from recorded educational videos into text and extracting important information from video data. This process uses voice recognition and image analysis technologies.

[1311] 2. Question analysis and answer generation: The system uses natural language processing technology to analyze questions from students, retrieves relevant information from a database, and generates answers that correspond to the questions. The generated answers incorporate emotion recognition results as needed, adjusting the tone of the answers to reflect the student's emotional state.

[1312] 3. Emotional state recognition and adjustment: Analyzes the participant's facial expressions and tone of voice to recognize their emotional state in real time, and adjusts the content and tone of their responses accordingly.

[1313] Terminal

[1314] Devices are devices used by students, including PCs, tablets, smartphones, smart glasses, and head-mounted displays. They use the following hardware and software:

[1315] Hardware: Camera, microphone, display, storage, processor

[1316] Software: User Interface (UI) applications, video streaming applications

[1317] Terminal processing includes:

[1318] 1. Login and authentication: The student enters their user ID and password and sends the authentication information to the server.

[1319] 2. Video viewing and playback: Receives and plays recorded lecture videos provided by the server, and provides an interface for students to input questions while watching.

[1320] 3. Capture and transmit emotional data: Use a camera and microphone to capture the student's facial expressions and tone of voice and transmit them to the server.

[1321] User

[1322] A user is a student who uses this system and performs the following operations:

[1323] 1. Login: Enter your user ID and password on each terminal to log in to the system.

[1324] 2. Watch Video: Select a video from the list provided and watch the video.

[1325] 3. Enter a question: If you have any questions while watching the video, enter your question in the dedicated text box.

[1326] Examples of concrete examples and prompts

[1327] For example, if a student is watching a recorded lecture on their smartphone and types a question like, "Please tell me about the camera performance of this smartphone." If the student's facial expression sounds confused and their tone of voice sounds anxious, the server will analyze the question and generate an answer like this:

[1328] The smartphone's camera is equipped with a 12MP wide-angle lens, allowing you to take clear photos even at night. Want to hear more details? You can also call our support team if you need help.

[1329] An example of this prompt statement:

[1330] "User Question: What are the specific features of this product?

[1331] Expression: Confused.

[1332] Voice tone: Anxious.

[1333] This is a situation that requires action."

[1334] As a result, customer service at the virtual store can be provided quickly and accurately, improving customer satisfaction.

[1335] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1336] Step 1:

[1337] Login and Authentication

[1338] Terminal: The user launches the application on the terminal and enters their user ID and password. The entered authentication information is sent to the server.

[1339] Server: The server compares the received authentication information with a database and performs authentication. If authentication is successful, it returns access rights and a list of available content to the terminal. If authentication fails, it sends an error message to the terminal. The input is the user ID and password, and the output is the authentication result.

[1340] Step 2:

[1341] Select content and start streaming

[1342] Device: The user selects the video they want to watch from a content list provided by the application on the device. A request for the selected video is sent to the server.

[1343] Server: The server retrieves the selected video from the database and sends it to the device in streaming format. Video playback position information is also sent at the same time. The input is the video selection request, and the output is streaming data.

[1344] Step 3:

[1345] Video content analysis

[1346] Server: The audio data from the video is analyzed using speech recognition technology (e.g., Google Speech-to-Text) and converted into text. The video data is analyzed using image analysis technology (e.g., OpenCV) to extract important information. This data is integrated and stored in a database as structured data. The input is the audio and video data from the video, and the output is structured data.

[1347] Step 4:

[1348] Capturing and transmitting emotional data

[1349] Terminal: The camera and microphone capture the user's facial expressions and tone of voice in real time and send the data to the server. The input is real-time data from the camera and microphone, and the output is data sent to the server.

[1350] Step 5:

[1351] Recognition of emotional states

[1352] Server: Using an emotion recognition engine (e.g., Microsoft Azure Emotion API), the server analyzes the user's emotional state from the facial expression and tone of voice data sent. The analysis results are stored as internal data and used when answering questions. The input is facial expression and tone of voice data, and the output is the analysis result of the emotional state.

[1353] Step 6:

[1354] Accepting and analyzing questions

[1355] Terminal: If a user has a question while watching a video, they can enter it in the text box, which is then sent to the server.

[1356] Server: The server uses a natural language processing engine (e.g., OpenAI GPT-4) to analyze the question and retrieve relevant information from a database. The input is a text question, and the output is the analysis result based on the question.

[1357] Step 7:

[1358] Answer generation and adjustment

[1359] Server: Generates answers to questions based on information retrieved from a database. Furthermore, it adjusts the content and tone of the answers based on the results of emotional state analysis. The input is the results of question analysis and emotional state analysis, and the output is the adjusted answer.

[1360] Step 8:

[1361] Submitting and viewing answers

[1362] Server: Sends the adjusted answer to the device.

[1363] Terminal: The terminal displays the received answers to the user, who reviews the answers and asks follow-up questions if necessary. The input is the adjusted answers, and the output is the answers displayed to the user.

[1364] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1365] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1366] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1367] [Fourth embodiment]

[1368] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1369] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1370] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1371] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1372] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1373] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1374] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1375] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1376] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1377] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1378] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1379] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1380] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1381] This invention relates to a system that uses AI technology to instantly answer questions from students about recorded lecture videos. The system includes the following specific elements and processes:

[1382] System configuration

[1383] 1. Server: A central management system that stores data, analyzes data, and handles question and answering.

[1384] 2. Device: Provides an interface for students to watch recorded lessons and enter questions on their devices (PCs, tablets, smartphones, etc.).

[1385] 3. Users: Students who watch recorded lessons and ask questions.

[1386] Overview of system program processing

[1387] The program processing of this system is explained in natural language below.

[1388] 1. Student Access Authentication

[1389] User: Enter your user ID and password on the login screen.

[1390] Terminal: Sends the entered authentication information to the server.

[1391] Server: Authenticates the user against the database, and if successful, returns a list of lecture videos to the user.

[1392] 2. Start streaming recorded lecture videos

[1393] User: Select the lecture video they want to watch.

[1394] Device: Sends a request for the selected video to the server.

[1395] Server: Retrieves the relevant video from the video database and starts streaming.

[1396] 3. Analysis of lesson content

[1397] Server: Passes the audio and video portions of the lesson video to the multimodal AI.

[1398] AI model (inside the server): Converts audio into text (voice recognition) and analyzes blackboard notes and slides in the video to convert them into text (image analysis).

[1399] Server: Structures the parsed data and stores it in a database.

[1400] 4. Questions

[1401] User: If a question arises during the lesson being viewed, enter the question in the text box.

[1402] Terminal: Sends the entered question to the server.

[1403] 5. Question Answering

[1404] Server: Receives questions from users and analyzes the content of the questions.

[1405] AI model (inside the server): Compares the analyzed questions with pre-structured lecture content and generates appropriate answers.

[1406] Server: Sends the generated answer to the user's device.

[1407] User: Check the AI's answer sent back from the device.

[1408] To give a specific example, if a student doesn't understand how to calculate consumption tax and asks, "How do you calculate consumption tax?", the server receives the question and passes it to the AI ​​model. The AI ​​model searches for relevant parts of the lesson content and generates an answer: "Consumption tax is the purchase price multiplied by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen item, the consumption tax will be 100 yen." The server then sends this to the user's device. The user can instantly check the appropriate answer and resolve their question.

[1409] This system allows students to resolve their questions on the spot, significantly improving learning efficiency. In addition, since instructors do not need to respond directly to students' questions, the burden on them is reduced, making it possible to reduce education costs while maintaining the quality of education.

[1410] The processing flow will be explained below.

[1411] Step 1: Student Access Authorization

[1412] User: Enter your user ID and password on the login screen.

[1413] Terminal: Sends the entered authentication information to the server.

[1414] Server: Authenticates the user against the information stored in the database, and if authentication is successful, returns access rights and a list of available lecture videos to the user. If authentication fails, displays an error message.

[1415] Step 2: Select the recorded lesson video and start streaming

[1416] User: Select the video they want to watch from the list of lecture videos provided.

[1417] Terminal: Sends a request for the selected lecture video to the server.

[1418] Server: Retrieves the selected lecture video from the database and sends it to the user's device in streaming format. The server also sends the video playback position information.

[1419] Step 3: Analysis of lesson content

[1420] Server: The recorded lesson video is sent to the multimodal AI. The audio portion of the video is processed through speech recognition and converted into text (speech-to-text).

[1421] Server: The contents of the slides and whiteboards are subjected to image analysis processing from the video footage to extract important information (video analysis).

[1422] Server: Integrates data obtained from audio-to-text and video analysis, and stores the lecture content in a database as structured data.

[1423] Step 4: Ask a question

[1424] Users: If they have any questions while watching a recorded lesson, they can enter their questions in the dedicated text box.

[1425] Terminal: Sends the entered question to the server.

[1426] Step 5: Parsing the question and generating an answer

[1427] Server: Receives questions sent by students and analyzes the content of the questions using natural language processing (NLP) technology.

[1428] AI model (on the server): Using the analyzed question content, the model searches a pre-structured lecture content database to obtain relevant information, and then generates the most appropriate answer from the obtained relevant information.

[1429] Server: Sends the generated answers to the students' devices.

[1430] Step 6: View and review your answers

[1431] User: Checks the submitted answers on the device, resolves any doubts, and enters additional questions as necessary.

[1432] Step 7: Feedback and Data Storage (Optional)

[1433] Server: Logs all questions and answers into a database for later review by the instructor.

[1434] Instructor: Analyze the topics that students focus on asking questions about and the AI's answers, and prepare additional teaching materials and supplementary lectures as necessary.

[1435] The above processing steps allow students to ask questions immediately while watching recorded lessons, and AI can provide quick answers, improving learning efficiency. This also reduces the burden on instructors and contributes to maintaining the quality of education.

[1436] Example 1

[1437] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1438] In conventional systems that provide recorded lecture videos, it is difficult for students to get immediate answers to questions that arise while watching the video. This reduces learning efficiency and places a burden on the instructor. Therefore, there is a need for a system that allows students to solve their questions in real time while watching the recorded video.

[1439] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1440] In this invention, the server includes a means for accepting questions entered by students, a means for analyzing audio and video data of recorded lecture videos, a means for understanding the question content based on the analyzed lecture content and using a generative AI model to generate an appropriate answer, and a means for transmitting the generated answer to the student, thereby enabling the student to solve their question in real time while watching the recorded video.

[1441] "Student" refers to a person who watches and learns from lecture videos provided online or offline.

[1442] "Means for accepting questions" refers to the interface or protocol that receives questions entered by students and sends them to the server.

[1443] "Lecture video" refers to audio and video data containing recorded lecture content.

[1444] "Means for analyzing audio and video data" refers to technology that converts the audio of lecture videos into text and extracts and analyzes visual information from the video.

[1445] A "generative AI model" refers to an artificial intelligence model that generates appropriate answers based on input questions and data.

[1446] "Means for generating answers" refers to technology that generates appropriate answers to students' questions based on the lecture content analyzed using a generative AI model.

[1447] "Means for sending answers" refers to the communication technology or protocol used to send the generated answers to the student's device.

[1448] "Access Authentication Measures" means the system or method that verifies a student's credentials and verifies access rights.

[1449] "Encryption methods" refers to the technologies and protocols that encrypt information to protect data transmissions.

[1450] "Structured data" refers to data that has been systematically organized from analyzed lecture video information and stored in a database.

[1451] "Speech recognition means" refers to technology that automatically converts voice data into text.

[1452] "Image analysis means" refers to technology that extracts specific visual information from video data and converts it into text.

[1453] This invention relates to a system that uses AI technology to instantly answer questions posed by students about recorded lecture videos. The system consists of three main components: a server that functions as a central management system, terminals where students can watch lecture videos and enter questions, and users who watch the recorded lectures.

[1454] System configuration

[1455] 1. Server:

[1456] The server stores data, analyzes it, and handles question and answering. Specifically, it uses the following hardware and software:

[1457] Database: Used to store authentication information and lecture video data.

[1458] Generative AI model: For example, GPT-4 is used to generate appropriate answers based on the question.

[1459] Speech recognition technology: The audio portion of the lecture video is converted into text using the Google Speech-to-Text API.

[1460] Image analysis technology: The video portion of the lecture video is analyzed using Google Cloud Vision API and converted into text.

[1461] 2. Terminal:

[1462] The device used by students (PC, tablet, smartphone, etc.) provides an interface for viewing recorded lessons and entering questions. The device has the following functions:

[1463] Login interface: An interface for entering a user ID and password.

[1464] Video playback function: Lecture videos retrieved from the server are played in streaming format.

[1465] Question input interface: An interface for students to enter their questions into a text box.

[1466] 3. User:

[1467] Students watch the recorded lectures and ask questions.

[1468] Specific actions

[1469] Below, we will explain how this system works with a concrete example.

[1470] 1. User access authentication

[1471] The user enters their user ID and password on the login screen.

[1472] The terminal encrypts the entered authentication information and sends it to the server.

[1473] The server checks the authentication information stored in the database and, if authentication is successful, returns a list of lecture videos to the user.

[1474] 2. Start streaming recorded lecture videos

[1475] The user selects the lecture video they wish to watch from the list.

[1476] The terminal sends a request for the selected video to the server.

[1477] The server retrieves the relevant video from the video database and streams it.

[1478] 3. Analysis of lesson content

[1479] The server passes the audio and video portions of the lesson video to the multimodal AI. The audio portion is converted to text using the Google Speech-to-Text API, and the video portion is analyzed and converted to text using the Google Cloud Vision API.

[1480] The server structures the parsed data and stores it in a database.

[1481] 4. Questions

[1482] If a user has a question about a lesson they are watching, they can enter it in the text box.

[1483] The terminal transmits the entered question to the server.

[1484] 5. Question Answering

[1485] The server receives a question from a user and analyzes the question.

[1486] The generative AI model matches the analyzed questions with pre-structured lecture content and generates appropriate answers.

[1487] The server sends the generated answer to the user's terminal.

[1488] The user checks the AI's answers sent back from the device.

[1489] Specific examples

[1490] For example, if a student does not understand how to calculate consumption tax, he or she can input the question, "How do you calculate consumption tax?" In response to this question, the system operates as follows:

[1491] User: Enters the question "How do you calculate sales tax?" into the text box.

[1492] Terminal: Sends entered text to the server.

[1493] Server: Receives the question and sends the prompt "How do you calculate sales tax?" to the generative AI model.

[1494] Generative AI model: Searches the lecture content and generates the answer, "Consumption tax is the purchase amount multiplied by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen product, the consumption tax will be 100 yen."

[1495] Server: Sends the generated answer to the user's device.

[1496] Terminal: Receives the answer and displays it to the user.

[1497] User: Review the answers provided and understand how sales tax is calculated.

[1498] This system allows students to get answers to their questions in real time, improving learning efficiency and reducing the burden on instructors.

[1499] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1500] Step 1:

[1501] Authenticating users for access

[1502] 1. Input: The user enters their user ID and password on the login screen.

[1503] 2. Operation: The device encrypts the entered authentication information and sends it to the server using a protocol such as SSL / TLS.

[1504] 3. Input: The server checks the received authentication information against the information stored in its database.

[1505] 4. Data processing: The server authenticates the user based on the matching result. If authentication is successful, the user information is saved in the session.

[1506] 5. Output: The server generates a lecture video list for the user and sends it to the terminal.

[1507] 6. Operation: The terminal displays a list of lecture videos.

[1508] Step 2:

[1509] Streaming of recorded lecture videos begins

[1510] 1. Input: The user selects the lecture video they want to watch from the list.

[1511] 2. Action: The device sends a request for the selected video to the server.

[1512] 3. Input: The server retrieves the relevant video data from the video database.

[1513] 4. Data processing: The server converts the video data into a streaming format, often using tools such as FFmpeg.

[1514] 5. Output: The server sends the converted streaming data to the terminal.

[1515] 6. Operation: The device plays the received streaming data.

[1516] Step 3:

[1517] Analysis of lesson content

[1518] 1. Input: The server extracts the audio and video parts of the lesson video.

[1519] 2. How it works: The server passes the audio portion to the Google Speech-to-Text API for speech recognition.

[1520] 3. Input: The server receives the text data obtained by speech recognition.

[1521] 4. Data processing: The audio data is saved as text along with time information.

[1522] 5. Operation: The server passes the video portion to the Google Cloud Vision API for image analysis.

[1523] 6. Input: The server receives the text data obtained from image analysis.

[1524] 7. Data processing: The information on the board and slides in the video is saved as text along with time information.

[1525] 8. Output: The server structures the acquired text data and stores it in a database.

[1526] Step 4:

[1527] Questions accepted

[1528] 1. Input: When a user has a question about the lesson they are watching, they can enter it in the text box.

[1529] 2. Operation: The terminal sends the entered question data to the server.

[1530] 3. Input: The server adds the received query data to a queue for analysis.

[1531] 4. Output: The question data added to the queue is passed to the next question-answering process.

[1532] Step 5:

[1533] Question and Answering

[1534] 1. Input: The server retrieves the question data in the queue.

[1535] 2. Operation: The server analyzes the question using NLP (Natural Language Processing) technology.

[1536] 3. Input: The server sends the parsed question content to the generative AI model as a prompt sentence.

[1537] 4. Output: Based on a specific prompt, such as "How do you calculate sales tax?", the generative AI model generates an answer.

[1538] 5. Input: Receive the answer from the generative AI model.

[1539] 6. Data processing: The server structures the answer and sends it to the user's device.

[1540] 7. Operation: The terminal receives the generated response and displays it to the user.

[1541] 8. Output: The user checks the answer displayed on the terminal and resolves their doubt.

[1542] (Application example 1)

[1543] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1544] With conventional video lecture systems, students had no way to ask questions and get answers immediately when they had doubts. This resulted in problems such as unresolved questions and reduced learning efficiency. Furthermore, it was a significant burden on instructors to deal with questions afterward, making it difficult to provide an efficient education.

[1545] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1546] In this invention, the server includes a means for accepting questions entered by students, a means for analyzing audio and video data from recorded lecture videos, and a means for understanding the questions based on the analyzed lecture content and generating appropriate answers. This allows the generated answers to be sent immediately to students. The server also includes a means for using a mobile information terminal that enables viewing of recorded lectures and question entry, a means for performing speech recognition and question answering using a generative AI model, and a means for using a central management system that receives, processes, and transmits various data. This allows students to instantly resolve their questions using mobile information terminals such as smartphones and tablets, improving their learning efficiency.

[1547] "Students" are learners who watch recorded lecture videos and enter questions.

[1548] "Means for accepting questions" refers to an interface for accepting questions entered by students and sending them into the system.

[1549] A "lecture video" is a file containing video and audio data of recorded educational content.

[1550] "Means for analyzing audio and video data" refers to the technical methods for extracting audio and video information from recorded lecture videos, converting it into text, and performing semantic analysis.

[1551] "Means of understanding the question and generating an appropriate answer" refers to a function that analyzes questions sent by students, compares them with related lecture content, and automatically creates the most appropriate answer.

[1552] "Means for sending generated answers" refers to a mechanism for sending answers generated by AI to students' devices in real time.

[1553] "Mobile information terminal" refers to a portable information processing device such as a smartphone or tablet.

[1554] "Generative AI model" refers to a learning model that uses artificial intelligence to perform speech recognition, natural language processing, and question answering.

[1555] "Central management system" refers to the server and related infrastructure for centrally managing the reception and processing of questions from students, the analysis of lecture videos, and the generation and transmission of answers.

[1556] This invention relates to a system that uses a generative AI model to instantly answer questions posed by students about recorded lecture videos. This system is realized by the following specific components and the mechanisms by which these components work together.

[1557] System configuration

[1558] 1. Server: As a central management system, it accepts questions from students, analyzes recorded lecture videos, generates questions and answers, and sends the generated answers.

[1559] 2. Devices: Mobile information devices such as smartphones and tablets used by students provide an interface for watching lecture videos and entering questions.

[1560] 3. User: A student who watches the recorded lecture and enters questions.

[1561] Program processing description

[1562] Login Authentication

[1563] The server receives the user ID and password sent by the student from their device and checks them against a database. If authentication is successful, it returns a list of lecture videos to the device. For security reasons, this process uses encrypted communication such as SSL / TLS.

[1564] Streaming lecture videos

[1565] When a user selects a lecture video to watch on their device, the server retrieves the corresponding video data from the video database and starts streaming it to the device. The video data is saved in standard formats such as MP4 and WebM.

[1566] Analysis of lesson content

[1567] The server passes the audio and video portions of the recorded lecture video to a generative AI model, which performs speech recognition to convert the audio into text and analyzes the blackboard notes and slides in the video to generate text. The analyzed data is structured and stored in a database. This process uses natural language processing libraries (e.g., HuggingFace Transformers) and image analysis software.

[1568] Question reception and answering

[1569] When a user has a question about a lesson they are watching, they send it in text format from their device to the server. The server then passes the received question to a generative AI model, which analyzes the question and generates an appropriate answer. The generated answer is then immediately sent back to the device. This allows students to receive answers to their questions in real time.

[1570] Specific examples of programs

[1571] Authentication Example

[1572] When a user enters "User ID: student1" and "Password: password1" on the login screen on their smartphone and sends it to the server, the server checks the database and, if the authentication information is correct, returns a list of lecture videos.

[1573] Example questions and answers

[1574] When a user inputs the question, "How do you calculate economic growth rate?", the generative AI model searches for relevant lecture content and generates the answer, "Economic growth rate is an indicator that shows how much a country's economy has grown, and specifically, it is calculated as the annual rate of change in gross domestic product (GDP)." The server then sends this answer to the user's device.

[1575] Example prompt sentence:

[1576] How do you calculate economic growth rate?

[1577] In this way, this invention allows students to resolve questions in real time while watching recorded lessons, improving learning efficiency and reducing the burden on instructors. Furthermore, this system is highly convenient because it uses mobile information terminals such as smartphones and tablets.

[1578] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1579] Step 1:

[1580] The server accepts the authentication information of the user ID and password entered by the student using the terminal and compares it with the database. The input is the user ID and password, and the output is the authentication result. If the authentication is successful, the server returns a list of lecture videos to the terminal. Specifically, the authentication process is carried out securely using SSL / TLS encrypted communication.

[1581] Step 2:

[1582] The server receives the lecture video ID requested by the student from the terminal and retrieves the video data corresponding to that ID from the video database. The input is the video request information (video ID), and the output is the streaming URL of the corresponding lecture video. Specifically, the server queries the video database, generates a video URL, and sends it to the terminal.

[1583] Step 3:

[1584] The server passes the audio and video data of the lecture video to the generative AI model for analysis. The input is the audio and video data of the lecture video, and the output is the text of the lecture content. Specifically, the audio data is converted into text using voice recognition software, and at the same time, the text and slides in the video are converted into text using image analysis software. The analyzed data is then structured and stored in a database.

[1585] Step 4:

[1586] When a student has a question during a class they are watching, they use their device to enter the question into a text box and send it to the server. The input is the question text entered by the student, and the output is a confirmation by the server that the question has been received. Specifically, the question is entered into the device interface and sent to the server.

[1587] Step 5:

[1588] The server passes the questions sent by the students to a generative AI model, which analyzes the content of the questions and generates appropriate answers. The input is the question text from the students, and the output is the generated answer text. Specifically, the server uses a generative AI model (e.g., HuggingFace Transformers) to generate answers from lecture data related to the question content.

[1589] Step 6:

[1590] The server sends the generated answer text to the student's terminal. The input is the generated answer text, and the output is the answer information displayed on the student's terminal. In concrete terms, the generated answer is sent to the student's terminal and displayed in real time.

[1591] This allows students to instantly resolve any questions they may have while watching the lecture videos, significantly improving their learning efficiency.

[1592] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1593] This invention relates to a system that combines AI technology and emotion recognition to instantly answer questions from students about recorded lecture videos. The system includes the following specific elements and processes:

[1594] System configuration

[1595] 1. Server: The central management system stores data, analyzes data, handles question-answering, and recognizes emotions.

[1596] 2. Device: Provides an input interface for viewing recorded lessons, entering questions, and analyzing sentiment on devices used by students (PCs, tablets, smartphones, etc.).

[1597] 3. Users: Students who watch recorded lessons and ask questions.

[1598] Overview of system program processing

[1599] The program processing of this system is explained in natural language below.

[1600] 1. Student Access Authentication

[1601] User: Enter your user ID and password on the login screen.

[1602] Terminal: Sends the entered authentication information to the server.

[1603] Server: Authenticates the user against the information stored in the database, and if authentication is successful, returns access rights and a list of available lecture videos to the user. If authentication fails, displays an error message.

[1604] 2. Select the recorded lesson video and start streaming

[1605] User: Select the video they want to watch from the list of lecture videos provided.

[1606] Terminal: Sends a request for the selected lecture video to the server.

[1607] Server: Retrieves the selected lecture video from the database and sends it to the user's device in streaming format. The server also sends the video playback position information.

[1608] 3. Analysis of lesson content

[1609] Server: The recorded lesson video is sent to the multimodal AI. The audio portion of the video is processed through speech recognition and converted into text (speech-to-text).

[1610] Server: The contents of the slides and whiteboards are subjected to image analysis processing from the video footage to extract important information (video analysis).

[1611] Server: Integrates data obtained from audio-to-text and video analysis, and stores the lecture content in a database as structured data.

[1612] 4. Emotion Recognition by Emotion Engine

[1613] Device: Uses a camera and microphone to capture students' facial expressions and tone of voice in real time.

[1614] Terminal: Sends captured data to the server.

[1615] Server: Runs an emotion engine to recognize students' emotional states from their facial expressions and tone of voice, and generates emotion data.

[1616] 5. Accepting questions

[1617] Users: If they have any questions while watching a recorded lesson, they can enter their questions in the dedicated text box.

[1618] Terminal: Sends the entered question and emotion data to the server.

[1619] 6. Question Analysis and Answer Generation

[1620] Server: Receives questions sent by students and analyzes the content of the questions using natural language processing (NLP) technology.

[1621] AI model (on the server): Using the analyzed question content, the model searches a pre-structured lecture content database to obtain relevant information, and then generates the most appropriate answer from the obtained relevant information.

[1622] Server: Adjusts the content and tone of responses appropriately based on the student's emotional state as recognized by the emotion engine.

[1623] 7. Submitting and Viewing Your Answers

[1624] Server: Adjusts the generated answer and sends it to the user's device.

[1625] User: Checks the submitted answers on the device, resolves any doubts, and enters additional questions as necessary.

[1626] To give a specific example, if a student does not understand how to calculate consumption tax and looks confused, they can input the question, "How do you calculate consumption tax?" The server receives the question and analyzes the emotional data using an emotion engine. The AI ​​model searches for relevant parts of the lesson content and responds with, "Consumption tax is calculated by multiplying the purchase price by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen item, the consumption tax will be 100 yen." Because the emotion engine recognizes the student's confusion, it softens the tone of the response and provides additional explanation. For example, it outputs a response such as, "If you have difficulty understanding, I'll explain how to calculate it again." The user immediately confirms the appropriate answer and the problem is resolved.

[1627] This system not only allows students to resolve their questions immediately, but also improves the quality and efficiency of learning by enabling appropriate responses through emotion recognition. It also reduces the burden on instructors, reducing education costs while maintaining the quality of education.

[1628] The processing flow will be explained below.

[1629] Step 1: Student Access Authorization

[1630] User: Enter your user ID and password on the login screen.

[1631] Terminal: Sends the entered authentication information to the server.

[1632] Server: Authenticates the user against the information stored in the database, and if successful, returns access rights and a list of available lecture videos to the user. If unsuccessful, displays an error message.

[1633] Step 2: Select the recorded lesson video and start streaming

[1634] User: Select the video they want to watch from the list of lecture videos provided.

[1635] Terminal: Sends a request for the selected lecture video to the server.

[1636] Server: Retrieves the selected lecture video from the database and sends it to the user's device in streaming format, along with playback position information.

[1637] Step 3: Analysis of lesson content

[1638] Server: The recorded lesson video is sent to the multimodal AI. The audio portion of the video is processed through speech recognition and converted into text (speech-to-text).

[1639] Server: The contents of the slides and whiteboards are subjected to image analysis processing from the video footage to extract important information (video analysis).

[1640] Server: Integrates data obtained from audio-to-text and video analysis, and stores the lecture content in a database as structured data.

[1641] Step 4: Emotion Recognition with the Emotion Engine

[1642] Device: Uses a camera and microphone to capture students' facial expressions and tone of voice in real time.

[1643] Terminal: Sends captured facial expression and voice data to the server.

[1644] Server: The transmitted data is analyzed using an emotion engine, which recognizes the student's emotional state and generates emotion data.

[1645] Step 5: Ask a question

[1646] Users: If they have any questions while watching a recorded lesson, they can enter their questions in the dedicated text box.

[1647] Terminal: Sends the entered question and emotion data to the server.

[1648] Step 6: Parsing the question and generating an answer

[1649] Server: Receives questions sent by students and analyzes the content of the questions using natural language processing (NLP) technology.

[1650] AI model (inside the server): Using the analyzed question content, it searches a pre-structured database of lecture content to obtain relevant information.

[1651] Server: Generates the most appropriate answer from the relevant information obtained.

[1652] Server: Adjusts the content and tone of the generated answers accordingly based on the student's emotional state as recognized by the emotion engine.

[1653] Step 7: Submit and view your responses

[1654] Server: Sends the adjusted answer to the user's device.

[1655] User: Checks the submitted answers on the device, resolves any doubts, and enters additional questions as necessary.

[1656] To give a specific example, if a student doesn't understand how to calculate consumption tax and asks, "How do you calculate consumption tax?", the server receives the question and analyzes the emotion data using the emotion engine. The AI ​​model searches for relevant parts of the lesson content and generates an answer: "Consumption tax is the purchase price multiplied by the consumption tax rate. For example, if the consumption tax rate is 10% for a 1,000 yen item, the consumption tax will be 100 yen." The emotion engine recognizes the student's confusion and softens the tone of the answer, providing additional explanation such as, "If you have difficulty understanding, I'll explain how to calculate it again." The user can instantly confirm the appropriate answer and resolve their doubt.

[1657] Example 2

[1658] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1659] In conventional learning systems, if a student has a question while watching a recorded lesson, it is difficult to get an appropriate answer immediately on the spot. Furthermore, the system does not respond to the student's emotional state (e.g., confusion, lack of understanding, etc.), which can lead to a decrease in the quality and efficiency of learning. In addition, the instructor is often asked many questions, which increases the burden on the instructor and can lead to a decrease in the quality of education.

[1660] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for accepting questions entered by students, means for analyzing audio and video data of recorded lecture videos, means for understanding the question content based on the analyzed lecture content and generating an appropriate answer, means for capturing the students' facial expressions and tone of voice and recognizing their emotional state, means for appropriately adjusting the generated answer based on the students' emotional state, means for sending the generated answer to the students, means for authenticating the students' access, and means for structuring and storing data acquired from the lecture videos in a database. This allows students to immediately receive appropriate answers if they have questions while watching recorded lectures. Furthermore, emotion recognition technology enables responses based on the students' emotional state, improving the quality and efficiency of learning. Furthermore, it reduces the burden on instructors, reducing education costs while maintaining education quality.

[1661] A "student" is a student or learner who attends an educational institution or online course and takes lectures or classes.

[1662] "Means for accepting questions" refers to the interface and functions for accepting and processing text questions entered by users and managing them on the system.

[1663] "Lecture video" refers to audio and video data that records the contents of a class or lecture.

[1664] "Means for analyzing audio and video data" refers to processing devices and software that use voice recognition technology to convert the audio portion of recorded lecture videos into text and image recognition technology to analyze the video portion and extract information.

[1665] "Means of understanding the question and generating an appropriate answer" refers to an AI model or algorithm that uses natural language processing technology to analyze the question text entered by the user and generate an appropriate answer.

[1666] "Means for capturing facial expressions and tone of voice and recognizing emotional states" refers to a system that uses a camera and microphone to collect facial expression data and voice data of a user and analyzes their emotional state based on emotion recognition technology.

[1667] "Means for appropriately adjusting the generated response based on the emotional state of the student" refers to a processing device or algorithm that modifies or adjusts the content and tone of the generated response based on the data obtained through emotion recognition.

[1668] "Means for transmitting answers to students" refers to a system that includes a communication interface or protocol for transmitting the generated and adjusted answer data to the student's terminal.

[1669] "Means for access authentication" refers to the database and authentication server that verify the user ID and password when a student logs in to the system and authenticates access rights.

[1670] "Means for structuring data and storing it in a database" refers to a system or software that structures the analyzed lecture content data into a certain format (e.g., JSON or XML) and stores it in a database (e.g., an SQL database).

[1671] This invention relates to a system that combines AI technology and emotion recognition technology to instantly answer questions posed by students about recorded lecture videos. The system includes the following specific elements and processes:

[1672] System configuration

[1673] 1. Server: The central management system stores data, analyzes data, handles question-answering, and recognizes emotions.

[1674] 2. Terminal: A device used by students (personal computer, tablet, smartphone, etc.) that provides an input interface for watching recorded lessons, entering questions, and sentiment analysis.

[1675] 3. Users: Students who watch recorded lessons and ask questions.

[1676] Program processing explanation

[1677] 1. Student Access Authentication

[1678] User: Enter your user ID and password on the login screen.

[1679] Terminal: The entered authentication information is encrypted and sent to the server using HTTPS as the communication protocol.

[1680] Server: Compares the received authentication information with a database (e.g., MySQL) and returns the authentication result to the terminal. If authentication is successful, it generates access rights and a list of available lecture videos.

[1681] 2. Select the recorded lesson video and start streaming

[1682] User: Select the video they want to watch from the lecture video list.

[1683] Terminal: Sends a request for the selected lecture video to the server. HTTPS is used as the communication protocol.

[1684] Server: Retrieves the specified video file from a video database (e.g., AWS S3) and streams it to the user's device using the HLS (HTTP Live Streaming) protocol.

[1685] 3. Analysis of lesson content

[1686] Server: Sends recorded lesson videos and analyzes them using multimodal AI (e.g., Google Cloud AI).

[1687] Server: The audio portion is converted into text using speech recognition software (e.g., Google Speech-to-Text), and the video portion is analyzed using image recognition software (e.g., Amazon Rekognition).

[1688] Server: Integrates speech-to-text data and video analysis data and stores them as structured data in a database (e.g., PostgreSQL).

[1689] 4. Emotion Recognition by Emotion Engine

[1690] Device: Uses a camera and microphone to capture students' facial expressions and tone of voice in real time.

[1691] Terminal: Sends captured facial expression and audio data to the server. It is recommended to use the WebSocket protocol.

[1692] Server: Analyzes students' emotional states using emotion recognition software (e.g., Microsoft Azure Emotion API) and generates emotion data.

[1693] 5. Accepting questions

[1694] Users: If they have any questions while watching a recorded lesson, they can type their questions in the text box.

[1695] Terminal: The entered question text and emotion data are sent to the server using HTTPS.

[1696] 6. Question Analysis and Answer Generation

[1697] Server: Analyzes questions sent by students using natural language processing (NLP) technology (e.g., OpenAI GPT-4).

[1698] AI model: Analyzes the question, searches the lecture content database, and generates an appropriate answer.

[1699] Server: The emotion engine analyzes the student's emotional state and adjusts the response accordingly.

[1700] 7. Submitting and Viewing Your Answers

[1701] Server: Sends the adjusted answer to the user's device. HTTPS is used as the communication protocol.

[1702] User: Checks the answers on the device, resolves any doubts, and can enter additional questions as needed.

[1703] Specific examples

[1704] If a student is confused about how to calculate sales tax, they can type "How do you calculate sales tax?" into the text box.

[1705] The server receives this question and further analyzes the student's emotional data. The AI ​​model searches for relevant lecture content data and generates an answer such as, "Consumption tax is the purchase amount multiplied by the consumption tax rate. For example, if a product costs 1,000 yen and the consumption tax rate is 10%, the consumption tax will be 100 yen."

[1706] Using emotion recognition, the server detects confusion in the student, softening the tone of its responses and adding additional explanations such as, "If you have difficulty understanding, I'll explain how to do the calculation again."

[1707] The user can check the answer on the device to resolve their doubts, and can also enter additional questions if necessary.

[1708] Example prompt sentence:

[1709] "Please provide a professional explanation of the calculation of sales tax. Also, please show how to respond to students who are confused by the following question: How do you calculate sales tax?"

[1710] As described above, this system allows students to resolve any questions they may have while watching recorded lessons in real time, and responds appropriately through emotion recognition. This is expected to improve the quality and efficiency of learning, reduce the burden on instructors, and reduce education costs while maintaining the quality of education.

[1711] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1712] Step 1: Student Access Authorization

[1713] User: Enter your user ID and password. The input screen has a text box where you can enter the information.

[1714] Terminal: The entered authentication information is sent to the server via HTTPS. The input is in plain text and is encrypted before being sent.

[1715] Server: Compares the received authentication information with a database (e.g., MySQL), and if authentication is successful, generates access privileges and a list of lecture videos and sends them to the terminal. If authentication fails, generates an error message and sends it to the terminal. The input is encrypted user information, and the output is the authentication result and a list of videos or an error message.

[1716] Step 2: Select the recorded lesson video and start streaming

[1717] User: Select the video they want to watch from the lecture video list. For example, click on the video title on the list.

[1718] Terminal: Sends a request for the selected lecture video to the server via HTTPS protocol. The input is the video ID and video request information, and the output is the request data to the server.

[1719] Server: Retrieves the specified video file from the video database (e.g., AWS S3) and streams it to the device in HLS format. The input is the video ID and the output is the streaming data.

[1720] Step 3: Analysis of lesson content

[1721] Server: Sends recorded lecture videos to a multimodal AI (e.g., Google Cloud AI). The input is the lecture video data, and the output is an analysis request to the AI.

[1722] Server: The audio portion of the video is converted into text using speech recognition software (e.g., Google Speech-to-Text). The input is audio data, and the output is text data.

[1723] Server: Analyzes the video portion of the video using image recognition software (e.g., Amazon Rekognition) and extracts information from slides and whiteboards. The input is video data, and the output is the text data of the analysis results.

[1724] Server: Integrates speech-to-text data and video analysis data and stores them as structured data in a database (e.g., PostgreSQL). The input is text data and video analysis data, and the output is structured data.

[1725] Step 4: Emotion Recognition with the Emotion Engine

[1726] Device: Uses a camera and microphone to capture the student's facial expressions and tone of voice in real time. The input is real-time video and audio data.

[1727] Terminal: Sends captured data to the server. It is recommended to use the WebSocket protocol. The input is the captured data and the output is the data sent to the server.

[1728] Server: Analyzes students' emotional states using emotion recognition software (e.g., Microsoft Azure Emotion API) and generates emotion data. The input is the captured data, and the output is emotion data.

[1729] Step 5: Ask a question

[1730] Users: If they have questions while watching a recorded lesson, they can type their question in the text box. For example, they can type "I don't know how to use this formula."

[1731] Terminal: The input question and emotion data are sent to the server via HTTPS. The input is the question text and emotion data, and the output is the data sent to the server.

[1732] Step 6: Parsing the question and generating an answer

[1733] Server: Analyzes questions sent by students using natural language processing (NLP) technology (e.g., OpenAI GPT-4). The input is the question text, and the output is the analysis result.

[1734] AI model: Using the analyzed question content, it searches the lecture content database, obtains relevant information, and generates an answer. The input is the analysis result, and the output is the initial answer.

[1735] Server: Based on the emotional state of the student recognized by the emotion engine, the generated answer is adjusted accordingly. The input is the initial answer and emotional data, and the output is the adjusted answer.

[1736] Step 7: Submit and view your responses

[1737] Server: Sends the adjusted answers to the student's device. HTTPS protocol is used. The input is the adjusted answers, and the output is the data sent to the device.

[1738] User: Checks the answer on the device and resolves any doubts. User can also enter additional questions if necessary. Input is the displayed answer.

[1739] (Application example 2)

[1740] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1741] Previously, it was difficult to respond immediately to customer questions with product introduction videos in virtual stores. In particular, when customers asked specific questions about a product, it was sometimes difficult to read the customer's emotional state from their facial expressions or tone of voice, which resulted in delayed responses. This could lead to lower customer satisfaction and lost sales opportunities.

[1742] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1743] In this invention, the server includes means for accepting questions entered by students, means for analyzing audio and video data of recorded educational videos, means for understanding the content of the questions based on the analyzed educational content and generating appropriate answers, means for sending the generated answers to students, means for capturing facial expressions and tone of voice in real time and recognizing emotional states, and means for adjusting the content and tone of answers based on the recognized emotional states. This enables customer service in the virtual store to be quick and accurate, thereby improving customer satisfaction.

[1744] "Participants" are users who watch recorded educational videos and enter questions.

[1745] A "means for accepting questions" is an interface that accepts text input from a user and sends it to the system.

[1746] A "recorded educational video" is a video containing pre-recorded educational content.

[1747] "Means for analyzing audio and video data" refers to the technology and functions that convert the audio data of recorded educational videos into text and extract important information from the video data.

[1748] "Means of understanding the question and generating an appropriate answer" refers to the technology and functions that use natural language processing technology to analyze the question, retrieve relevant information from a database, and generate an answer.

[1749] The "means for transmitting the generated answers to the students" refers to an interface for presenting the generated answers to the users in real time.

[1750] "Means of capturing facial expressions and tone of voice in real time and recognizing emotional states" refers to technologies and functions that capture a user's facial expressions and tone of voice using a camera or microphone and analyze their emotions in real time.

[1751] "Means for adjusting the content and tone of responses based on a recognized emotional state" refers to technologies and functions that adjust the content and tone of generated responses based on the results of emotional analysis, making them easier for users to understand and more reassuring.

[1752] In this invention, the system is realized by the server, terminal, and user performing the following processes.

[1753] server

[1754] The server analyzes and stores various data, processes questions and answers, and recognizes emotions. Specifically, it uses the following hardware and software:

[1755] Hardware: high-performance processor, storage device, large memory

[1756] Software: Natural language processing engines (e.g., OpenAI GPT-4), emotion recognition engines, video processing software, database management systems

[1757] The server process includes:

[1758] 1. Audio and video data analysis: Converting audio data from recorded educational videos into text and extracting important information from video data. This process uses voice recognition and image analysis technologies.

[1759] 2. Question analysis and answer generation: The system uses natural language processing technology to analyze questions from students, retrieves relevant information from a database, and generates answers that correspond to the questions. The generated answers incorporate emotion recognition results as needed, adjusting the tone of the answers to reflect the student's emotional state.

[1760] 3. Emotional state recognition and adjustment: Analyzes the participant's facial expressions and tone of voice to recognize their emotional state in real time, and adjusts the content and tone of their responses accordingly.

[1761] Terminal

[1762] Devices are devices used by students, including PCs, tablets, smartphones, smart glasses, and head-mounted displays. They use the following hardware and software:

[1763] Hardware: Camera, microphone, display, storage, processor

[1764] Software: User Interface (UI) applications, video streaming applications

[1765] Terminal processing includes:

[1766] 1. Login and authentication: The student enters their user ID and password and sends the authentication information to the server.

[1767] 2. Video viewing and playback: Receives and plays recorded lecture videos provided by the server, and provides an interface for students to input questions while watching.

[1768] 3. Capture and transmit emotional data: Use a camera and microphone to capture the student's facial expressions and tone of voice and transmit them to the server.

[1769] User

[1770] A user is a student who uses this system and performs the following operations:

[1771] 1. Login: Enter your user ID and password on each terminal to log in to the system.

[1772] 2. Watch Video: Select a video from the list provided and watch the video.

[1773] 3. Enter a question: If you have any questions while watching the video, enter your question in the dedicated text box.

[1774] Examples of concrete examples and prompts

[1775] For example, if a student is watching a recorded lecture on their smartphone and types a question like, "Please tell me about the camera performance of this smartphone." If the student's facial expression sounds confused and their tone of voice sounds anxious, the server will analyze the question and generate an answer like this:

[1776] The smartphone's camera is equipped with a 12MP wide-angle lens, allowing you to take clear photos even at night. Want to hear more details? You can also call our support team if you need help.

[1777] An example of this prompt statement:

[1778] "User Question: What are the specific features of this product?

[1779] Expression: Confused.

[1780] Voice tone: Anxious.

[1781] This is a situation that requires action."

[1782] As a result, customer service at the virtual store can be provided quickly and accurately, improving customer satisfaction.

[1783] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1784] Step 1:

[1785] Login and Authentication

[1786] Terminal: The user launches the application on the terminal and enters their user ID and password. The entered authentication information is sent to the server.

[1787] Server: The server compares the received authentication information with a database and performs authentication. If authentication is successful, it returns access rights and a list of available content to the terminal. If authentication fails, it sends an error message to the terminal. The input is the user ID and password, and the output is the authentication result.

[1788] Step 2:

[1789] Select content and start streaming

[1790] Device: The user selects the video they want to watch from a content list provided by the application on the device. A request for the selected video is sent to the server.

[1791] Server: The server retrieves the selected video from the database and sends it to the device in streaming format. Video playback position information is also sent at the same time. The input is the video selection request, and the output is streaming data.

[1792] Step 3:

[1793] Video content analysis

[1794] Server: The audio data from the video is analyzed using speech recognition technology (e.g., Google Speech-to-Text) and converted into text. The video data is analyzed using image analysis technology (e.g., OpenCV) to extract important information. This data is integrated and stored in a database as structured data. The input is the audio and video data from the video, and the output is structured data.

[1795] Step 4:

[1796] Capturing and transmitting emotional data

[1797] Terminal: The camera and microphone capture the user's facial expressions and tone of voice in real time and send the data to the server. The input is real-time data from the camera and microphone, and the output is data sent to the server.

[1798] Step 5:

[1799] Recognition of emotional states

[1800] Server: Using an emotion recognition engine (e.g., Microsoft Azure Emotion API), the server analyzes the user's emotional state from the facial expression and tone of voice data sent. The analysis results are stored as internal data and used when answering questions. The input is facial expression and tone of voice data, and the output is the analysis result of the emotional state.

[1801] Step 6:

[1802] Accepting and analyzing questions

[1803] Terminal: If a user has a question while watching a video, they can enter it in the text box, which is then sent to the server.

[1804] Server: The server uses a natural language processing engine (e.g., OpenAI GPT-4) to analyze the question and retrieve relevant information from a database. The input is a text question, and the output is the analysis result based on the question.

[1805] Step 7:

[1806] Answer generation and adjustment

[1807] Server: Generates answers to questions based on information retrieved from a database. Furthermore, it adjusts the content and tone of the answers based on the results of emotional state analysis. The input is the results of question analysis and emotional state analysis, and the output is the adjusted answer.

[1808] Step 8:

[1809] Submitting and viewing answers

[1810] Server: Sends the adjusted answer to the device.

[1811] Terminal: The terminal displays the received answers to the user, who reviews the answers and asks follow-up questions if necessary. The input is the adjusted answers, and the output is the answers displayed to the user.

[1812] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1813] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1814] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1815] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1816] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1817] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1818] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1819] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1820] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1821] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1822] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1823] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1824] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1825] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1826] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1827] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1828] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1829] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1830] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1831] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1832] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1833] The following is further disclosed regarding the above embodiment.

[1834] (Claim 1)

[1835] a means for accepting questions entered by students;

[1836] A means for analyzing the audio and video data of recorded lecture videos;

[1837] A means for understanding the question content based on the analyzed lecture content and generating an appropriate answer;

[1838] a means for transmitting the generated answers to the student;

[1839] A system including:

[1840] (Claim 2)

[1841] a means for authenticating student access;

[1842] 10. The system of claim 1.

[1843] (Claim 3)

[1844] A means to structure the data obtained from the lesson videos and store it in a database,

[1845] 10. The system of claim 1.

[1846] "Example 1"

[1847] (Claim 1)

[1848] a means for accepting questions entered by students;

[1849] A means for analyzing the audio and video data of recorded lecture videos;

[1850] A means for using a generative AI model to understand the question based on the analyzed lecture content and generate an appropriate answer; and

[1851] a means for transmitting the generated answers to the student;

[1852] A system including:

[1853] (Claim 2)

[1854] a means for authenticating student access;

[1855] a means for encrypting data transmission and reception;

[1856] 10. The system of claim 1.

[1857] (Claim 3)

[1858] A means to structure the data obtained from the lesson videos and store it in a database,

[1859] a speech recognition means for converting the speech data into text;

[1860] Image analysis means for converting notes and slides in the video into text;

[1861] 10. The system of claim 1.

[1862] "Application Example 1"

[1863] (Claim 1)

[1864] a means for accepting questions entered by students;

[1865] A means for analyzing the audio and video data of recorded lecture videos;

[1866] A means for understanding the question content based on the analyzed lecture content and generating an appropriate answer;

[1867] a means for transmitting the generated answers to the student;

[1868] A means for using a mobile information terminal that allows viewing of recorded lessons and inputting questions;

[1869] A means for using generative AI models to perform speech recognition and question answering;

[1870] A means for utilizing a central management system for receiving, processing, and transmitting various data;

[1871] A system including:

[1872] (Claim 2)

[1873] a means for authenticating student access;

[1874] 10. The system of claim 1.

[1875] (Claim 3)

[1876] A means to structure the data obtained from the lesson videos and store it in a database,

[1877] 10. The system of claim 1.

[1878] "Example 2: Combining Emotion Engines"

[1879] (Claim 1)

[1880] a means for accepting questions entered by students;

[1881] A means for analyzing audio and video data of the recorded lecture video;

[1882] A means for understanding the question content based on the analyzed lecture content and generating an appropriate answer;

[1883] a means of capturing students' facial expressions and tone of voice to recognize their emotional state;

[1884] means for adaptively adjusting the generated answers based on the emotional state of the student;

[1885] a means for transmitting the generated answers to the student;

[1886] A system including:

[1887] (Claim 2)

[1888] a means for authenticating student access;

[1889] 10. The system of claim 1.

[1890] (Claim 3)

[1891] A means to structure the data obtained from the lesson videos and store it in a database,

[1892] 10. The system of claim 1.

[1893] "Application example 2 when combining emotion engines"

[1894] (Claim 1)

[1895] a means for accepting questions entered by the student;

[1896] A means for analyzing audio and video data of recorded educational videos;

[1897] A means for understanding the question content based on the analyzed educational content and generating an appropriate answer;

[1898] a means for transmitting the generated answers to the student;

[1899] A means of capturing facial expressions and tone of voice in real time to recognize emotional states;

[1900] a means of adjusting the content and tone of responses based on perceived emotional states;

[1901] A system including:

[1902] (Claim 2)

[1903] a means for authenticating student access;

[1904] 10. The system of claim 1.

[1905] (Claim 3)

[1906] A means of structuring the data obtained from the educational videos and storing it in a database;

[1907] 10. The system of claim 1. [Explanation of symbols]

[1908] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for accepting questions entered by students; A means of analyzing the audio and video data of recorded lecture videos; A means for understanding the question content based on the analyzed lecture content and generating an appropriate answer; a means for transmitting the generated answers to the student; A system including:

2. a means for authenticating student access; The system of claim 1 .

3. A means to structure the data obtained from the lesson videos and store it in a database, The system of claim 1 .

Images