system

Abstract

We provide the system. [Solution] Filming equipment for recording educational activities on video, A method for storing recorded video on an information storage device and managing it on a cloud platform, Means for providing educational resources to users via information processing means, A means of receiving viewing requests from users, editing the video based on the requested duration, and generating a summarized version, A means of responding to user questions and generating answers using artificial intelligence technology, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is to solve the problem of providing a flexible learning environment according to individual circumstances while minimizing the learning delay of students who have missed educational activities. In the conventional method, there has been a concern that the learning delay due to absence has an adverse effect on the student's grades and understanding. Also, since there has been a lack of means to efficiently learn the lesson content of the absent days, it has been difficult for students to learn again at their own pace.

Means for Solving the Problems

[0005] This invention provides a system that automatically records educational activities using video recording means and stores the recordings in an information storage device. Information processing means can edit the video of the lesson based on the viewing time desired by the students and provide it as a summarized version including important points. Furthermore, by using natural language processing to build a system that responds quickly and accurately to student questions, it becomes possible to deepen the understanding of individual students. In this way, absent students can continue their learning in near real-time.

[0006] "Educational activities" refer to activities conducted by teachers in schools and educational institutions, such as classes and lectures, for the purpose of transmitting knowledge and supporting learning.

[0007] "Video recording" refers to the process of saving the content of classes and lectures in video format using recording devices such as cameras.

[0008] An "information storage device" refers to a storage medium installed inside a computer or server that is capable of storing digital data for a long period of time.

[0009] "Information processing means" refers to the functions of computers and software used to perform tasks such as analyzing, organizing, converting, or editing digital information.

[0010] "Users" refer to students who use this system to view educational activity videos and utilize the question-and-answer function.

[0011] A "viewing request" refers to the act of a user requesting the system to play a video of a class or lecture on a specific day or time.

[0012] A "machine learning algorithm" refers to a set of mathematical models and computational procedures used to learn patterns and meanings from data and improve subsequent decisions and predictions.

[0013] "Natural language processing" refers to artificial intelligence technologies that enable computers to understand, interpret, and generate human language. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention provides an embodiment using an electronic system for efficiently managing the process of recording and providing educational activities. Specifically, a camera device installed in the classroom where the lesson is held captures the educational activity as video, and a server receives and records this video in real time. As a result, the entire lesson is saved in digital format.

[0036] The server automatically uploads recorded video to a cloud-based information storage device, and each video is assigned detailed metadata about the lesson. This metadata includes the subject name, teacher's name, and lesson date, and serves as an index for users to search for videos later.

[0037] Users log in through a portal or application running on a dedicated terminal, identify the classes they missed, and request to view them. At this time, users specify the viewing time as needed, and the server edits the video based on this information.

[0038] The server uses machine learning algorithms to extract important information from video data. Through extensive data analysis, it automatically generates a summarized version based on viewing requests and converts it into a format playable on the device.

[0039] Furthermore, if a user has a question while watching, they can enter it through their device. A natural language processing unit operating on the server side analyzes this question and generates the optimal response based on a digital library and AI models. The response is then displayed on the device or provided verbally.

[0040] A concrete example is a user who missed a math class due to illness and later watches a 30-minute condensed version of the 1-hour class on their device. In this case, if a question arises about the differentiation of a function, the user can immediately send the question to the system, and the server can quickly return an answer to support further understanding.

[0041] In this way, this invention improves the learning experience of absent students in educational settings and provides advanced educational services tailored to individual circumstances.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The server receives video from cameras installed in the classroom in real time and automatically starts recording at the start of the class. Recording automatically stops at the end of the class, and the obtained video is saved in digital format.

[0045] Step 2:

[0046] The server analyzes the recorded video, adds metadata (subject name, teacher name, lesson date), and then uploads the video data to a cloud-based storage device. During this process, an index is also generated to improve the video's availability.

[0047] Step 3:

[0048] Users log in to a dedicated portal or application, select the specific class day they missed, and send a viewing request from their device to the server. This request also includes the desired viewing time.

[0049] Step 4:

[0050] The server searches for stored video data based on the specified date and subject, and uses machine learning algorithms to analyze and edit the videos to match the user's desired viewing time. This process extracts key content, which is then prepared as a summarized video.

[0051] Step 5:

[0052] The edited video data is provided to the user's device in streaming format for immediate viewing. The user then begins watching this summarized video on their device.

[0053] Step 6:

[0054] If a user has a question while watching, they can enter it on their device, and this question will be sent to the server in real time.

[0055] Step 7:

[0056] The server receives a question and analyzes its content using natural language processing techniques. To generate an appropriate answer, the server consults databases and AI models as needed.

[0057] Step 8:

[0058] The generated answers are displayed to the user via their device or provided as audio responses. This allows users to resolve their questions and deepen their learning on the spot.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] In today's educational environment, a major challenge is the limited learning opportunities resulting from class absences and restricted access to educational resources. Furthermore, the lack of mechanisms for efficient review of lesson content, extraction of key information, and prompt answers to individual student questions is also a problem. This raises concerns about a decline in the quality of learning.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes a visual information acquisition device for electronically recording educational activities, means for storing the acquired visual information in a remote information storage means, and means for providing educational materials to users via a computer processing means. This makes it possible to easily replay missed lessons as visual information, enabling efficient review and deepening of learning. Furthermore, by utilizing computational learning algorithms and intelligent models, important information can be extracted and responses to user inquiries can be provided quickly, realizing high-quality individualized learning.

[0064] "Educational activities" refer to all activities related to learning, such as classes, lectures, and practical training, conducted at schools and other educational institutions.

[0065] "To record electronically" refers to saving information in digital format using a video camera or other recording device.

[0066] A "visual information acquisition device" refers to a device that captures educational activities in real time and saves them as video data.

[0067] "Remote information storage means" refers to technology that stores acquired data in a physically distant location, such as cloud storage or a remote server.

[0068] "Computer processing means" refers to software and hardware systems that automatically perform tasks such as organizing, searching, playing back, and editing information.

[0069] A "computational learning algorithm" refers to a group of analytical methods that extract patterns from large amounts of data to support decision-making.

[0070] An "intelligent model" refers to an artificial intelligence system designed to answer human questions through natural language processing and data analysis.

[0071] "Individualized learning" refers to a form of education that is tailored to the individual needs and pace of each learner.

[0072] This invention is a system for recording and providing educational activities in digital format, and aims to improve the learning experience in educational institutions.

[0073] The server uses visual information acquisition devices, i.e., cameras, installed in the classroom to capture lessons in real time and collect the video data. The collected video is temporarily stored in the server's storage and then automatically uploaded to a cloud-based remote information storage system. At this time, metadata related to the lesson, such as the subject name, teacher's name, and lesson date, is added.

[0074] Users log in using a dedicated terminal via a provided portal or application. Authenticated users can select a desired lesson from a list of previously recorded lessons and specify the viewing time. Based on the viewing request, the server analyzes the video data, uses computational learning algorithms to extract important information, and generates a summarized version edited to fit the specified time. This summarized version is then converted into a data format playable on the terminal.

[0075] Furthermore, users can input questions that arise during viewing through their device. The server analyzes these questions using an intelligent model based on natural language processing and generates quick and appropriate answers. This allows users to resolve their questions on the spot and enhance their learning effectiveness.

[0076] As a concrete example, consider a user who wants to review a math class they missed for health reasons. This user can access the system and watch a 30-minute summary of the one-hour class. If they want to learn more about the differentiation of functions while watching, they can enter a question into the system, and the server will provide an appropriate explanation through natural language processing. An example of a prompt would be, "Please explain the basic concepts of differentiation in under 5 minutes."

[0077] As described above, this system combines the latest video technology and AI technology to improve the efficiency and quality of individualized learning.

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] The server receives video data in real time from a visual information acquisition device installed in the classroom. At this time, the video data is temporarily stored in the server's storage. The input is a video stream from the camera, and the output is buffered video data.

[0081] Step 2:

[0082] The server adds relevant metadata such as subject name, teacher name, and class date to the temporarily stored video data and uploads it to a cloud-based remote information storage system. The input for this step is the video data and associated metadata, and the output is the video data stored in the cloud storage.

[0083] Step 3:

[0084] Users log in to the system using a dedicated terminal. Once authentication is complete, the terminal displays a list of classes the user can access. The input at this time is the user's authentication information, and the output is a class list tailored to the user.

[0085] Step 4:

[0086] The user selects the class they wish to view and submits a viewing request based on the specified time. The terminal sends this information to the server. The input is the user's selected class information and viewing time, and the output is the viewing request sent to the server.

[0087] Step 5:

[0088] Based on user viewing requests, the server uses a computational learning algorithm to analyze video data, extract important scenes, and create a summary. The input is unedited video data and a specified viewing time, while the output is edited summary video data.

[0089] Step 6:

[0090] The server converts the generated summary video data into a format playable on the terminal and sends it to the user's terminal. In this step, the input is the summary video data, and the output is data in a playable format.

[0091] Step 7:

[0092] If a user has a question while watching, they enter it via their device. The device then sends this question to the server. The input is the user's question, and the output is the question request sent to the server.

[0093] Step 8:

[0094] The server uses an intelligent model to analyze the user's question and generate a quick and appropriate answer. The input is the question data, and the output is the answer data. The generated answer is returned to the terminal and presented in a format that the user can see or hear.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] In educational activities, there is a problem in effectively recording and providing lesson content. Specifically, there is a lack of efficient systems for students who are absent to catch up on lesson content. Furthermore, there is a need to provide an environment where questions that arise during viewing can be answered immediately.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes means for video recording educational activities and managing them on a cloud platform, means for using machine learning algorithms to generate summarized video versions, and means for providing answers to user questions using artificial intelligence technology. This enables absent students to effectively catch up on class content and obtain answers to their questions in real time.

[0100] "Educational activities" refer to activities aimed at learning, such as classes and lectures.

[0101] "Video recording" refers to the process of saving visual information, including movement, as digital data using a camera or other recording device.

[0102] A "cloud infrastructure" is a structure of remote servers used to store and manage data via the internet.

[0103] An "information storage device" is a hardware or software system for storing data.

[0104] "Information processing" is the process of analyzing and transforming data to generate useful information.

[0105] A "viewing request" is a request from a user who wants to watch a video at a specific time or with a specific content.

[0106] A "summary version" is a video that has been reconstructed in a shortened format by extracting the important parts from the original video data.

[0107] A "machine learning algorithm" is a method that allows computers to learn patterns from data and automatically perform predictions and classifications.

[0108] "Artificial intelligence technology" is the technology that enables machines to mimic human intelligence and act accordingly.

[0109] "Natural language processing" is a technology that enables computers to understand, generate, and analyze human language.

[0110] A "question" is a question posed by a user to gain knowledge or information.

[0111] An "answer" is a response or explanation given in response to a question.

[0112] To implement this invention, the server records educational activities on video and stores the data on a cloud infrastructure. The hardware includes a camera and a cloud server that collects video data in real time and uploads it to the cloud. AWS® or Google® Cloud are suitable as the cloud infrastructure.

[0113] The server uses machine learning algorithms to extract important information from the video and generates a summarized version of the video according to the user's viewing request. Here, machine learning models are built using frameworks such as TENSORFLOW® and PyTorch.

[0114] Furthermore, questions posed by users while viewing content on their devices are analyzed using natural language processing technology. By applying generative AI models such as OpenAI's GPT-3, the system generates and provides the user with the most appropriate answers based on this analysis.

[0115] As a concrete example, there is a feature that allows users to watch a summarized video of any subject they missed in class. For instance, a user could enter a prompt such as, "I would like to watch a summarized version of the math class. Please explain the differentiation of functions." In response to this prompt, the system can display a summarized version of the relevant class and automatically provide an explanation of the differentiation of functions.

[0116] In this way, this invention aims to provide a means for efficiently and flexibly reproducing and utilizing educational activities, thereby supporting the learning of individual users.

[0117] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0118] Step 1:

[0119] The server begins receiving video data in real time from cameras in the classroom where educational activities are taking place. The input is video data from the cameras, and the output is video data stored on the cloud infrastructure. At this stage, the server efficiently acquires the data using data streaming technology.

[0120] Step 2:

[0121] The server automatically generates metadata (such as class name, teacher name, and date) related to the received video data and adds it when uploading to the cloud. The input is the received video data and class information, and the output is the video data on the cloud with the metadata attached. A database system is used to effectively manage the metadata.

[0122] Step 3:

[0123] The user sends a viewing request to the server via their device. The input is the user's viewing request (subject, viewing time, etc.), and the output is data that is processed as a request to the server. Based on this information, the server applies a machine learning algorithm to prepare to generate a summarized version of the video.

[0124] Step 4:

[0125] The server uses machine learning algorithms to extract important information and edits and generates summarized videos based on viewing requests. The input is the original video data and the user's viewing request, and the output is summarized video data. TensorFlow and PyTorch are used to build models and analyze the video data.

[0126] Step 5:

[0127] When a user enters a question while watching, the device sends that prompt to the server. The input is the user's question, and the output is the query data sent to the server. The text data is formatted in preparation for natural language processing.

[0128] Step 6:

[0129] The server uses a generative AI model to analyze user questions and generate the optimal answer. The input is the user's question text, and the output is the analyzed answer text. Natural language processing is performed using OpenAI's GPT-3, etc., to generate the answer based on relevant information.

[0130] Step 7:

[0131] The terminal displays the answer received from the server to the user. The input is the answer text sent from the server, and the output is the answer information displayed on the terminal. The displayed answer is presented clearly through the user interface.

[0132] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0133] This invention is a system for effectively experiencing classes and educational activities remotely, and is particularly intended to provide flexible learning support for absent students. This system is equipped with an emotion engine that recognizes the user's emotions in real time and has the function to dynamically adjust the learning experience.

[0134] The server receives video from cameras installed in the classroom and records the entire lesson. This video is stored in the cloud and tagged with metadata. Users can view it through an application. Machine learning algorithms are used for video editing, editing and presenting important learning points according to the viewer's preferred time.

[0135] Furthermore, the user's device is equipped with a camera and microphone, and an emotion engine is incorporated that analyzes the user's emotions in real time from their facial expressions and voice through these devices. When the user watches the video, the emotion engine can identify emotions such as joy, confusion, and anxiety. This information is sent to the server, which uses this feedback to make adjustments to optimize the viewing experience.

[0136] For example, if a user expresses confusion, the server will help them understand by repeating the problematic section or displaying additional explanations. This entire process is automated, creating an environment where users can focus on learning. Furthermore, natural language processing technology enables an interactive learning experience by generating accurate answers to user questions in real time and displaying them on the device.

[0137] A concrete example would be a scenario where, while watching a math lesson, a user becomes confused by a calculus concept; the emotion engine detects this emotion, and the server immediately provides supplementary explanations. In this way, the goal is to provide personalized education by offering adaptive learning support through emotional feedback.

[0138] The following describes the processing flow.

[0139] Step 1:

[0140] The server receives video footage from cameras installed in the classroom and automatically starts recording according to the set start time. When the class ends, it stops recording and transfers the video data to cloud storage.

[0141] Step 2:

[0142] The server adds metadata to the recorded video, storing information such as the subject name, class date, and teacher's name in a database. This allows users to easily search for specific classes.

[0143] Step 3:

[0144] Users use a dedicated application to identify classes they have missed and send requests to view them from their device to the server. They can also include requests for preferred viewing times and specific explanations.

[0145] Step 4:

[0146] The server receives viewing requests from users and edits the video using a machine learning algorithm based on the desired viewing time. Through this process, a summarized video highlighting key learning points is generated.

[0147] Step 5:

[0148] The edited, summarized video is delivered to the user's device via streaming. The device then displays the received video to the user and begins playback.

[0149] Step 6:

[0150] The user's device has a built-in camera and microphone, and the emotion engine uses this data to analyze the user's emotions in real time from their facial expressions and voice. The results are then reported to the server.

[0151] Step 7:

[0152] Based on data from the emotion engine, the server adjusts the viewing experience by providing additional explanations or clarifications, or by pausing playback and inserting simpler explanations, if the user's emotions indicate anxiety or confusion.

[0153] Step 8:

[0154] When a user has a question, they input it through their device, and this question is transmitted to the server in real time. Using natural language processing technology, the server analyzes the question, generates an accurate and immediate answer, and displays it on the device.

[0155] Step 9:

[0156] All viewing and interaction records are stored on the server and used to track user learning progress and design feedback. This allows for continuous improvement of the quality of educational services.

[0157] (Example 2)

[0158] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0159] In today's educational environment, despite the growing demand for distance learning, there is a problem in that it cannot adequately accommodate individual learning paces and styles. In particular, there is a challenge in that learning effectiveness decreases when beneficiaries cannot participate in real-time classes or when their understanding of the lesson content is insufficient. This invention aims to improve the quality of learning by providing an environment in which beneficiaries can learn efficiently and flexibly remotely.

[0160] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0161] In this invention, the server includes recording means for video recording educational activities, means for storing the recorded video in an information storage device, and means for providing educational resources to beneficiaries via information processing means. This enables beneficiaries to participate in classes in real time from remote locations and enjoy a high-quality learning experience.

[0162] "Educational activities" are a set of actions organized to impart specific knowledge or skills.

[0163] "Recording means" refers to devices and methods for storing information such as audio and video for extended periods.

[0164] "Information storage device" refers to hardware that stores digital data and makes it available for retrieval as needed.

[0165] "Information processing means" refers to the mechanisms of devices and software that analyze collected data and provide it to users as useful information.

[0166] A "beneficiary" refers to an individual or group that utilizes educational resources and gains learning opportunities through this system.

[0167] A "viewing request" refers to a request from a beneficiary who wishes to view specific video content.

[0168] "Machine learning" refers to algorithms or methods that enable computers to recognize patterns from given data and make predictions and decisions more effectively.

[0169] "Natural language processing" refers to the field of technology that enables computers to understand, analyze, and generate responses to human language.

[0170] A "generative model" refers to an artificial intelligence model that has the ability to generate new data from given data.

[0171] "Emotional analysis" refers to a technology that analyzes an individual's facial expressions and voice to recognize their emotional state at that time.

[0172] This invention is a system for effectively experiencing educational activities remotely. Its primary purpose is to provide flexible learning support for beneficiaries who miss classes. The system incorporates an emotion engine that analyzes the beneficiary's emotions in real time and dynamically adjusts the learning experience accordingly.

[0173] The server receives video from cameras installed in the classroom and records the lesson in progress. This video is stored in a cloud-based information storage device and tagged with metadata. The metadata includes lesson content, date, and instructor's name. The system uses machine learning algorithms for video editing, enabling it to extract key learning points and generate summary videos tailored to the viewer's preferred viewing time.

[0174] The device is equipped with a camera and microphone, and incorporates an emotion engine that analyzes the user's emotions through their facial expressions and voice. The analyzed data is sent to a server, and the viewing experience is adjusted based on the user's emotional feedback. For example, if the user shows confusion or anxiety, the relevant section of the video may be played repeatedly, or additional text explanations may be provided.

[0175] Users can watch recorded lesson videos through the application. If a user has a question while watching, they can ask it in real time using natural language processing technology. A generative AI model understands the question, generates an accurate answer, and displays it on the device. This enables an interactive learning experience where beneficiaries can instantly resolve questions that arise during their studies.

[0176] For example, if a beneficiary is watching a math lesson and becomes confused by a calculus concept, the device's emotion engine detects this confusion. This information is sent to a server, which then repeatedly explains key points of calculus or provides supplementary materials. This automated process allows beneficiaries to learn at their own pace.

[0177] Another example of a prompt might be, "Please explain the Fundamental Theorem of Integration in detail. Please briefly explain the terminology and include examples." This type of prompt allows the generative AI model to generate appropriate answers and provide effective learning support to the beneficiary.

[0178] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0179] Step 1:

[0180] The server receives video in real time from cameras installed in the classroom and records the progress of the lesson. The input is video data from the cameras, and the output is the recorded lesson video. The video data is immediately saved to a cloud storage device, and metadata regarding the date, time, and content of the lecture is generated and attached. The server checks the video quality and audio and saves it in the appropriate format.

[0181] Step 2:

[0182] The server processes the stored lecture videos using a machine learning algorithm. The input is the recorded lecture video and its metadata, and the output is an edited video containing summarized learning points. The machine learning algorithm transcribes the audio in the video and extracts specific keywords. Based on this information, it automatically edits the important parts and generates a shortened version according to the viewer's desired length.

[0183] Step 3:

[0184] The device uses its built-in camera and microphone to record the user's facial expressions and voice in real time. The input is the user's video and audio data, and the output is the analysis results regarding the user's emotional state. An emotion engine analyzes the data and identifies emotions such as joy and confusion in real time. Sensors detect the user's body movements and voice tone, and the analysis results are sent to the server.

[0185] Step 4:

[0186] The server optimizes the viewing experience based on emotional feedback received from the device. The input is the result of the user's emotional analysis, and the output is dynamically adjusted viewing content. For example, if the user shows confusion, the server will automatically repeat the relevant section of the video or display additional explanations in text. This feedback loop allows the user to understand the content more deeply.

[0187] Step 5:

[0188] When a user enters a question through the application interface, the server generates an answer using a generative AI model. The input is the user's text question, and the output is the generated answer. The server uses natural language processing to understand the question and feeds appropriate prompt sentences into the generative AI model. The resulting answer is then displayed on the terminal. The system accurately displays the necessary information according to the question.

[0189] (Application Example 2)

[0190] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0191] In rehabilitation and daily living training for the elderly and dementia patients, there is a challenge in providing support tailored to each individual's cognitive state and emotions. Existing rehabilitation systems offer uniform programs, making it difficult to effectively improve users' cognitive abilities and maintain their motivation. Furthermore, the lack of automatic content adjustment based on user responses limits the effectiveness of the support.

[0192] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0193] In this invention, the server includes video recording means, means for analyzing the user's cognitive state using an emotion analysis engine, means for storing the recorded video in an information storage device, and means for providing educational resources and emotional feedback-based content to the user through information processing means. This makes it possible to provide an individually optimized rehabilitation program that is tailored to the user's emotional state.

[0194] "Video recording means" refers to devices and technologies used to record educational activities and rehabilitation sessions, making them available for later analysis and viewing.

[0195] An "emotion analysis engine" refers to software or algorithms that analyze a user's facial expressions and voice through a camera or microphone to determine their emotional state in real time.

[0196] "Information storage device" refers to a storage device or cloud system that securely stores captured video and data for later access and analysis.

[0197] "Information processing means" refers to computing resources and processing methods used to generate and provide appropriate content and feedback to users based on collected data.

[0198] "Emotional feedback" refers to adjustments and adaptations to optimized content and activities provided in response to the user's emotional state.

[0199] "Content" refers to the collective term for images, audio, text, interactive elements, etc., provided to users and used for educational or supportive purposes.

[0200] "User" refers to an individual who is the subject of this invention and who participates in educational activities or rehabilitation programs.

[0201] To realize this invention, a series of systems involving a server, terminals, and users are used. First, the server stores video data acquired from terminals equipped with cameras and microphones in an information storage device. In this process, it is necessary to record the entire rehabilitation or educational session using recording means. The recorded data is stored in the information storage device and later used for analysis and feedback generation.

[0202] The device incorporates an emotion analysis engine that analyzes the user's facial expressions and voice in real time. This sends data to the server to determine the user's emotional and cognitive state. At this stage, machine learning libraries such as TensorFlow and Keras are used for facial recognition, and the Google Cloud Speech-to-Text API is utilized for voice analysis. Furthermore, IBM Watson® sentiment analysis is employed for emotion analysis.

[0203] The server processes this data and delivers content through emotional feedback. Specifically, it improves the individual experience by optimizing and presenting content such as videos, images, and music selected according to the user's emotional state.

[0204] For example, if a user shows curiosity during a rehabilitation session, the server immediately displays content related to relevant history or interesting topics. This improves the user's attention and enhances the effectiveness of the session.

[0205] An example of a prompt message is, "Based on the user's sentiment analysis, list content that is likely to be of interest and present it in the appropriate order."

[0206] This system enables flexible rehabilitation and educational support tailored to the individual needs of each user.

[0207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0208] Step 1:

[0209] The device uses a camera and microphone to collect video and audio of the user. The input is real-time video and audio data, which is recorded in high resolution. The output is raw data converted into a data format that can be processed by the emotion analysis engine. Specifically, it captures the user's face and voice as digital data, performs noise reduction, and processes the data as needed.

[0210] Step 2:

[0211] The device analyzes collected video and audio data using an emotion analysis engine. The input is the formatted data obtained in step 1, and the output is a quantitative indicator showing the user's emotional state. Specifically, it analyzes facial expressions using TensorFlow and Keras, and extracts emotions from audio using the Google Cloud Speech-to-Text API.

[0212] Step 3:

[0213] The server receives data from the sentiment analysis engine and selects appropriate content based on the user's emotional state. The input is an indicator of the emotional state obtained in step 2, and the output is a list of selected content objects. Specifically, it searches the database for relevant videos, music, and text and lists the most suitable ones.

[0214] Step 4:

[0215] The server streams the selected content to the terminal through an information processing system. The input is the content list generated in step 3, and the output is the content set on the user interface for the user to view. Specifically, it performs conversion to the content format and optimization according to bandwidth to provide the user with an uninterrupted viewing experience.

[0216] Step 5:

[0217] The device delivers the streamed content to the user and records the user's reaction again. The input is the content streamed from the server, and the output is new emotional response data. Specifically, it captures facial expressions and voice again using the camera and microphone and sends the data to the server.

[0218] Step 6:

[0219] The server re-analyzes the sentiment data and adjusts the content provided as needed. The input is the sentiment data obtained in step 5, and the output is the re-selection of content and the addition of detailed explanations as required. Specifically, a generative AI model is used to evaluate the effectiveness of the previous content and provide new content or supplementary information.

[0220] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0221] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0222] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0223] [Second Embodiment]

[0224] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0225] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0226] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0227] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0228] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0229] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0230] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0231] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0232] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0233] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0234] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0235] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0236] This invention provides an embodiment using an electronic system for efficiently managing the process of recording and providing educational activities. Specifically, a camera device installed in the classroom where the lesson is held captures the educational activity as video, and a server receives and records this video in real time. As a result, the entire lesson is saved in digital format.

[0237] The server automatically uploads recorded video to a cloud-based information storage device, and each video is assigned detailed metadata about the lesson. This metadata includes the subject name, teacher's name, and lesson date, and serves as an index for users to search for videos later.

[0238] Users log in through a portal or application running on a dedicated terminal, identify the classes they missed, and request to view them. At this time, users specify the viewing time as needed, and the server edits the video based on this information.

[0239] The server uses machine learning algorithms to extract important information from video data. Through extensive data analysis, it automatically generates a summarized version based on viewing requests and converts it into a format playable on the device.

[0240] Furthermore, if a user has a question while watching, they can enter it through their device. A natural language processing unit operating on the server side analyzes this question and generates the optimal response based on a digital library and AI models. The response is then displayed on the device or provided verbally.

[0241] A concrete example is a user who missed a math class due to illness and later watches a 30-minute condensed version of the 1-hour class on their device. In this case, if a question arises about the differentiation of a function, the user can immediately send the question to the system, and the server can quickly return an answer to support further understanding.

[0242] In this way, this invention improves the learning experience of absent students in educational settings and provides advanced educational services tailored to individual circumstances.

[0243] The following describes the processing flow.

[0244] Step 1:

[0245] The server receives video from cameras installed in the classroom in real time and automatically starts recording at the start of the class. Recording automatically stops at the end of the class, and the obtained video is saved in digital format.

[0246] Step 2:

[0247] The server analyzes the recorded video, adds metadata (subject name, teacher name, lesson date), and then uploads the video data to a cloud-based storage device. During this process, an index is also generated to improve the video's availability.

[0248] Step 3:

[0249] Users log in to a dedicated portal or application, select the specific class day they missed, and send a viewing request from their device to the server. This request also includes the desired viewing time.

[0250] Step 4:

[0251] The server searches for stored video data based on the specified date and subject, and uses machine learning algorithms to analyze and edit the videos to match the user's desired viewing time. This process extracts key content, which is then prepared as a summarized video.

[0252] Step 5:

[0253] The edited video data is provided to the user's device in streaming format for immediate viewing. The user then begins watching this summarized video on their device.

[0254] Step 6:

[0255] If a user has a question while watching, they can enter it on their device, and this question will be sent to the server in real time.

[0256] Step 7:

[0257] The server receives a question and analyzes its content using natural language processing techniques. To generate an appropriate answer, the server consults databases and AI models as needed.

[0258] Step 8:

[0259] The generated answers are displayed to the user via their device or provided as audio responses. This allows users to resolve their questions and deepen their learning on the spot.

[0260] (Example 1)

[0261] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0262] In today's educational environment, a major challenge is the limited learning opportunities resulting from class absences and restricted access to educational resources. Furthermore, the lack of mechanisms for efficient review of lesson content, extraction of key information, and prompt answers to individual student questions is also a problem. This raises concerns about a decline in the quality of learning.

[0263] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0264] In this invention, the server includes a visual information acquisition device for electronically recording educational activities, means for storing the acquired visual information in a remote information storage means, and means for providing educational materials to users via a computer processing means. This makes it possible to easily replay missed lessons as visual information, enabling efficient review and deepening of learning. Furthermore, by utilizing computational learning algorithms and intelligent models, important information can be extracted and responses to user inquiries can be provided quickly, realizing high-quality individualized learning.

[0265] "Educational activities" refer to all activities related to learning, such as classes, lectures, and practical training, conducted at schools and other educational institutions.

[0266] "To record electronically" refers to saving information in digital format using a video camera or other recording device.

[0267] A "visual information acquisition device" refers to a device that captures educational activities in real time and saves them as video data.

[0268] "Remote information storage means" refers to technology that stores acquired data in a physically distant location, such as cloud storage or a remote server.

[0269] "Computer processing means" refers to software and hardware systems that automatically perform tasks such as organizing, searching, playing back, and editing information.

[0270] A "computational learning algorithm" refers to a group of analytical methods that extract patterns from large amounts of data to support decision-making.

[0271] An "intelligent model" refers to an artificial intelligence system designed to answer human questions through natural language processing and data analysis.

[0272] "Individualized learning" refers to a form of education that is tailored to the individual needs and pace of each learner.

[0273] This invention is a system for recording and providing educational activities in digital format, and aims to improve the learning experience in educational institutions.

[0274] The server uses visual information acquisition devices, i.e., cameras, installed in the classroom to capture lessons in real time and collect the video data. The collected video is temporarily stored in the server's storage and then automatically uploaded to a cloud-based remote information storage system. At this time, metadata related to the lesson, such as the subject name, teacher's name, and lesson date, is added.

[0275] Users log in using a dedicated terminal via a provided portal or application. Authenticated users can select a desired lesson from a list of previously recorded lessons and specify the viewing time. Based on the viewing request, the server analyzes the video data, uses computational learning algorithms to extract important information, and generates a summarized version edited to fit the specified time. This summarized version is then converted into a data format playable on the terminal.

[0276] Furthermore, users can input questions that arise during viewing through their device. The server analyzes these questions using an intelligent model based on natural language processing and generates quick and appropriate answers. This allows users to resolve their questions on the spot and enhance their learning effectiveness.

[0277] As a concrete example, consider a user who wants to review a math class they missed for health reasons. This user can access the system and watch a 30-minute summary of the one-hour class. If they want to learn more about the differentiation of functions while watching, they can enter a question into the system, and the server will provide an appropriate explanation through natural language processing. An example of a prompt would be, "Please explain the basic concepts of differentiation in under 5 minutes."

[0278] As described above, this system combines the latest video technology and AI technology to improve the efficiency and quality of individualized learning.

[0279] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0280] Step 1:

[0281] The server receives video data in real time from a visual information acquisition device installed in the classroom. At this time, the video data is temporarily stored in the server's storage. The input is a video stream from the camera, and the output is buffered video data.

[0282] Step 2:

[0283] The server assigns relevant metadata such as the course name, teacher name, class date, etc. to the temporarily saved video data, and uploads this to cloud-based remote information storage means. The input for this step is the video data and the accompanying metadata, and the output is the video data stored in cloud storage.

[0284] Step 3:

[0285] The user logs in to the system using a dedicated terminal. The terminal displays a list of classes that the user can access after authentication is completed. The input at this time is the user's authentication information, and the output is a class list corresponding to the user.

[0286] Step 4:

[0287] The user selects the class they want to watch and makes a viewing request based on the specified time. The terminal sends this information to the server. The input is the class information selected by the user and the viewing time, and the output is a viewing request to the server.

[0288] Step 5:

[0289] The server analyzes the video data using a computational learning algorithm based on the user's viewing request, extracts important scenes, and creates a summary version. The input is the unedited video data and the viewing time specification, and the output is the edited summary version of the video data.

[0290] Step 6:

[0291] The server converts the generated summary version of the video data into a format that can be played on the terminal and sends it to the user's terminal. The input in this step is the summary version of the video data, and the output is data in a playable format.

[0292] Step 7:

[0293] If a user has a question while watching, they enter it via their device. The device then sends this question to the server. The input is the user's question, and the output is the question request sent to the server.

[0294] Step 8:

[0295] The server uses an intelligent model to analyze the user's question and generate a quick and appropriate answer. The input is the question data, and the output is the answer data. The generated answer is returned to the terminal and presented in a format that the user can see or hear.

[0296] (Application Example 1)

[0297] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0298] In educational activities, there is a problem in effectively recording and providing lesson content. Specifically, there is a lack of efficient systems for students who are absent to catch up on lesson content. Furthermore, there is a need to provide an environment where questions that arise during viewing can be answered immediately.

[0299] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0300] In this invention, the server includes means for video recording educational activities and managing them on a cloud platform, means for using machine learning algorithms to generate summarized video versions, and means for providing answers to user questions using artificial intelligence technology. This enables absent students to effectively catch up on class content and obtain answers to their questions in real time.

[0301] "Educational activities" refer to activities aimed at learning, such as classes and lectures.

[0302] "Video recording" refers to the act of storing visual information including movement as digital data using a photographing device such as a camera.

[0303] "Cloud infrastructure" refers to the structure of remote servers for storing and managing data via the Internet.

[0304] "Information storage device" refers to a hardware or software system for storing data.

[0305] "Information processing" refers to the process of analyzing and converting data to generate useful information.

[0306] "Viewing request" refers to a request by a user to view a video at a specific time or of specific content.

[0307] "Summary version" refers to a video that extracts important parts from the original video data and reconstructs them in a shortened form.

[0308] "Machine learning algorithm" refers to a method for a computer to learn patterns from data and perform automatic prediction and classification.

[0309] "Artificial intelligence technology" refers to technology for a machine to act by imitating human intelligence.

[0310] "Natural language processing" refers to technology for a computer to understand, generate, and analyze human language.

[0311] "Question" refers to an inquiry made by a user to obtain knowledge or information.

[0312] "Answer" refers to a response or explanation given to a question.

[0313] To implement this invention, the server records educational activities on video and stores the data on a cloud infrastructure. The hardware includes a camera and a cloud server that collects video data in real time and uploads it to the cloud. AWS or Google Cloud are suitable as the cloud infrastructure.

[0314] The server uses machine learning algorithms to extract important information from the video and generates a summarized version of the video according to the user's viewing request. Here, machine learning models are built using frameworks such as TensorFlow and PyTorch.

[0315] Furthermore, questions posed by users while viewing content on their devices are analyzed using natural language processing technology. By applying generative AI models such as OpenAI's GPT-3, the system generates and provides the user with the most appropriate answers based on this analysis.

[0316] As a concrete example, there is a feature that allows users to watch a summarized video of any subject they missed in class. For instance, a user could enter a prompt such as, "I would like to watch a summarized version of the math class. Please explain the differentiation of functions." In response to this prompt, the system can display a summarized version of the relevant class and automatically provide an explanation of the differentiation of functions.

[0317] In this way, this invention aims to provide a means for efficiently and flexibly reproducing and utilizing educational activities, thereby supporting the learning of individual users.

[0318] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0319] Step 1:

[0320] The server begins receiving video data in real time from cameras in the classroom where educational activities are taking place. The input is video data from the cameras, and the output is video data stored on the cloud infrastructure. At this stage, the server efficiently acquires the data using data streaming technology.

[0321] Step 2:

[0322] The server automatically generates metadata (such as class name, teacher name, and date) related to the received video data and adds it when uploading to the cloud. The input is the received video data and class information, and the output is the video data on the cloud with the metadata attached. A database system is used to effectively manage the metadata.

[0323] Step 3:

[0324] The user sends a viewing request to the server via their device. The input is the user's viewing request (subject, viewing time, etc.), and the output is data that is processed as a request to the server. Based on this information, the server applies a machine learning algorithm to prepare to generate a summarized version of the video.

[0325] Step 4:

[0326] The server uses machine learning algorithms to extract important information and edits and generates summarized videos based on viewing requests. The input is the original video data and the user's viewing request, and the output is summarized video data. TensorFlow and PyTorch are used to build models and analyze the video data.

[0327] Step 5:

[0328] When a user enters a question while watching, the device sends that prompt to the server. The input is the user's question, and the output is the query data sent to the server. The text data is formatted in preparation for natural language processing.

[0329] Step 6:

[0330] The server uses a generative AI model to analyze user questions and generate the optimal answer. The input is the user's question text, and the output is the analyzed answer text. Natural language processing is performed using OpenAI's GPT-3, etc., to generate the answer based on relevant information.

[0331] Step 7:

[0332] The terminal displays the answer received from the server to the user. The input is the answer text sent from the server, and the output is the answer information displayed on the terminal. The displayed answer is presented clearly through the user interface.

[0333] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0334] This invention is a system for effectively experiencing classes and educational activities remotely, and is particularly intended to provide flexible learning support for absent students. This system is equipped with an emotion engine that recognizes the user's emotions in real time and has the function to dynamically adjust the learning experience.

[0335] The server receives video from cameras installed in the classroom and records the entire lesson. This video is stored in the cloud and tagged with metadata. Users can view it through an application. Machine learning algorithms are used for video editing, editing and presenting important learning points according to the viewer's preferred time.

[0336] Furthermore, the user's device is equipped with a camera and microphone, and an emotion engine is incorporated that analyzes the user's emotions in real time from their facial expressions and voice through these devices. When the user watches the video, the emotion engine can identify emotions such as joy, confusion, and anxiety. This information is sent to the server, which uses this feedback to make adjustments to optimize the viewing experience.

[0337] For example, if a user expresses confusion, the server will help them understand by repeating the problematic section or displaying additional explanations. This entire process is automated, creating an environment where users can focus on learning. Furthermore, natural language processing technology enables an interactive learning experience by generating accurate answers to user questions in real time and displaying them on the device.

[0338] A concrete example would be a scenario where, while watching a math lesson, a user becomes confused by a calculus concept; the emotion engine detects this emotion, and the server immediately provides supplementary explanations. In this way, the goal is to provide personalized education by offering adaptive learning support through emotional feedback.

[0339] The following describes the processing flow.

[0340] Step 1:

[0341] The server receives video footage from cameras installed in the classroom and automatically starts recording according to the set start time. When the class ends, it stops recording and transfers the video data to cloud storage.

[0342] Step 2:

[0343] The server adds metadata to the recorded video, storing information such as the subject name, class date, and teacher's name in a database. This allows users to easily search for specific classes.

[0344] Step 3:

[0345] Users use a dedicated application to identify classes they have missed and send requests to view them from their device to the server. They can also include requests for preferred viewing times and specific explanations.

[0346] Step 4:

[0347] The server receives viewing requests from users and edits the video using a machine learning algorithm based on the desired viewing time. Through this process, a summarized video highlighting key learning points is generated.

[0348] Step 5:

[0349] The edited, summarized video is delivered to the user's device via streaming. The device then displays the received video to the user and begins playback.

[0350] Step 6:

[0351] The user's device has a built-in camera and microphone, and the emotion engine uses this data to analyze the user's emotions in real time from their facial expressions and voice. The results are then reported to the server.

[0352] Step 7:

[0353] Based on data from the emotion engine, the server adjusts the viewing experience by providing additional explanations or clarifications, or by pausing playback and inserting simpler explanations, if the user's emotions indicate anxiety or confusion.

[0354] Step 8:

[0355] When a user has a question, they input it through their device, and this question is transmitted to the server in real time. Using natural language processing technology, the server analyzes the question, generates an accurate and immediate answer, and displays it on the device.

[0356] Step 9:

[0357] All viewing and interaction records are stored on the server and used to track user learning progress and design feedback. This allows for continuous improvement of the quality of educational services.

[0358] (Example 2)

[0359] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0360] In today's educational environment, despite the growing demand for distance learning, there is a problem in that it cannot adequately accommodate individual learning paces and styles. In particular, there is a challenge in that learning effectiveness decreases when beneficiaries cannot participate in real-time classes or when their understanding of the lesson content is insufficient. This invention aims to improve the quality of learning by providing an environment in which beneficiaries can learn efficiently and flexibly remotely.

[0361] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0362] In this invention, the server includes recording means for video recording educational activities, means for storing the recorded video in an information storage device, and means for providing educational resources to beneficiaries via information processing means. This enables beneficiaries to participate in classes in real time from remote locations and enjoy a high-quality learning experience.

[0363] "Educational activities" are a set of actions organized to impart specific knowledge or skills.

[0364] "Recording means" refers to devices and methods for storing information such as audio and video for extended periods.

[0365] "Information storage device" refers to hardware that stores digital data and makes it available for retrieval as needed.

[0366] "Information processing means" refers to the mechanisms of devices and software that analyze collected data and provide it to users as useful information.

[0367] A "beneficiary" refers to an individual or group that utilizes educational resources and gains learning opportunities through this system.

[0368] A "viewing request" refers to a request from a beneficiary who wishes to view specific video content.

[0369] "Machine learning" refers to algorithms or methods that enable computers to recognize patterns from given data and make predictions and decisions more effectively.

[0370] "Natural language processing" refers to the field of technology that enables computers to understand, analyze, and generate responses to human language.

[0371] A "generative model" refers to an artificial intelligence model that has the ability to generate new data from given data.

[0372] "Emotional analysis" refers to a technology that analyzes an individual's facial expressions and voice to recognize their emotional state at that time.

[0373] This invention is a system for effectively experiencing educational activities remotely. Its primary purpose is to provide flexible learning support for beneficiaries who miss classes. The system incorporates an emotion engine that analyzes the beneficiary's emotions in real time and dynamically adjusts the learning experience accordingly.

[0374] The server receives video from cameras installed in the classroom and records the lesson in progress. This video is stored in a cloud-based information storage device and tagged with metadata. The metadata includes lesson content, date, and instructor's name. The system uses machine learning algorithms for video editing, enabling it to extract key learning points and generate summary videos tailored to the viewer's preferred viewing time.

[0375] The device is equipped with a camera and microphone, and incorporates an emotion engine that analyzes the user's emotions through their facial expressions and voice. The analyzed data is sent to a server, and the viewing experience is adjusted based on the user's emotional feedback. For example, if the user shows confusion or anxiety, the relevant section of the video may be played repeatedly, or additional text explanations may be provided.

[0376] Users can watch recorded lesson videos through the application. If a user has a question while watching, they can ask it in real time using natural language processing technology. A generative AI model understands the question, generates an accurate answer, and displays it on the device. This enables an interactive learning experience where beneficiaries can instantly resolve questions that arise during their studies.

[0377] For example, if a beneficiary is watching a math lesson and becomes confused by a calculus concept, the device's emotion engine detects this confusion. This information is sent to a server, which then repeatedly explains key points of calculus or provides supplementary materials. This automated process allows beneficiaries to learn at their own pace.

[0378] Another example of a prompt might be, "Please explain the Fundamental Theorem of Integration in detail. Please briefly explain the terminology and include examples." This type of prompt allows the generative AI model to generate appropriate answers and provide effective learning support to the beneficiary.

[0379] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0380] Step 1:

[0381] The server receives video in real time from cameras installed in the classroom and records the progress of the lesson. The input is video data from the cameras, and the output is the recorded lesson video. The video data is immediately saved to a cloud storage device, and metadata regarding the date, time, and content of the lecture is generated and attached. The server checks the video quality and audio and saves it in the appropriate format.

[0382] Step 2:

[0383] The server processes the stored lecture videos using a machine learning algorithm. The input is the recorded lecture video and its metadata, and the output is an edited video containing summarized learning points. The machine learning algorithm transcribes the audio in the video and extracts specific keywords. Based on this information, it automatically edits the important parts and generates a shortened version according to the viewer's desired length.

[0384] Step 3:

[0385] The device uses its built-in camera and microphone to record the user's facial expressions and voice in real time. The input is the user's video and audio data, and the output is the analysis results regarding the user's emotional state. An emotion engine analyzes the data and identifies emotions such as joy and confusion in real time. Sensors detect the user's body movements and voice tone, and the analysis results are sent to the server.

[0386] Step 4:

[0387] The server optimizes the viewing experience based on emotional feedback received from the device. The input is the result of the user's emotional analysis, and the output is dynamically adjusted viewing content. For example, if the user shows confusion, the server will automatically repeat the relevant section of the video or display additional explanations in text. This feedback loop allows the user to understand the content more deeply.

[0388] Step 5:

[0389] When a user enters a question through the application interface, the server generates an answer using a generative AI model. The input is the user's text question, and the output is the generated answer. The server uses natural language processing to understand the question and feeds appropriate prompt sentences into the generative AI model. The resulting answer is then displayed on the terminal. The system accurately displays the necessary information according to the question.

[0390] (Application Example 2)

[0391] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0392] In rehabilitation and daily living training for the elderly and dementia patients, there is a challenge in providing support tailored to each individual's cognitive state and emotions. Existing rehabilitation systems offer uniform programs, making it difficult to effectively improve users' cognitive abilities and maintain their motivation. Furthermore, the lack of automatic content adjustment based on user responses limits the effectiveness of the support.

[0393] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0394] In this invention, the server includes video recording means, means for analyzing the user's cognitive state using an emotion analysis engine, means for storing the recorded video in an information storage device, and means for providing educational resources and emotional feedback-based content to the user through information processing means. This makes it possible to provide an individually optimized rehabilitation program that is tailored to the user's emotional state.

[0395] "Video recording means" refers to devices and technologies used to record educational activities and rehabilitation sessions, making them available for later analysis and viewing.

[0396] An "emotion analysis engine" refers to software or algorithms that analyze a user's facial expressions and voice through a camera or microphone to determine their emotional state in real time.

[0397] "Information storage device" refers to a storage device or cloud system that securely stores captured video and data for later access and analysis.

[0398] "Information processing means" refers to computing resources and processing methods used to generate and provide appropriate content and feedback to users based on collected data.

[0399] "Emotional feedback" refers to adjustments and adaptations to optimized content and activities provided in response to the user's emotional state.

[0400] "Content" refers to the collective term for images, audio, text, interactive elements, etc., provided to users and used for educational or supportive purposes.

[0401] "User" refers to an individual who is the subject of this invention and who participates in educational activities or rehabilitation programs.

[0402] To realize this invention, a series of systems involving a server, terminals, and users are used. First, the server stores video data acquired from terminals equipped with cameras and microphones in an information storage device. In this process, it is necessary to record the entire rehabilitation or educational session using recording means. The recorded data is stored in the information storage device and later used for analysis and feedback generation.

[0403] The device incorporates an emotion analysis engine that analyzes the user's facial expressions and voice in real time. This data is then sent to a server to determine the user's emotional and cognitive state. At this stage, machine learning libraries such as TensorFlow and Keras are used for facial recognition, and the Google Cloud Speech-to-Text API is utilized for voice analysis. Furthermore, IBM Watson's emotion analysis is employed for sentiment analysis.

[0404] The server processes this data and delivers content through emotional feedback. Specifically, it improves the individual experience by optimizing and presenting content such as videos, images, and music selected according to the user's emotional state.

[0405] For example, if a user shows curiosity during a rehabilitation session, the server immediately displays content related to relevant history or interesting topics. This improves the user's attention and enhances the effectiveness of the session.

[0406] An example of a prompt message is, "Based on the user's sentiment analysis, list content that is likely to be of interest and present it in the appropriate order."

[0407] This system enables flexible rehabilitation and educational support tailored to the individual needs of each user.

[0408] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0409] Step 1:

[0410] The device uses a camera and microphone to collect video and audio of the user. The input is real-time video and audio data, which is recorded in high resolution. The output is raw data converted into a data format that can be processed by the emotion analysis engine. Specifically, it captures the user's face and voice as digital data, performs noise reduction, and processes the data as needed.

[0411] Step 2:

[0412] The device analyzes collected video and audio data using an emotion analysis engine. The input is the formatted data obtained in step 1, and the output is a quantitative indicator showing the user's emotional state. Specifically, it analyzes facial expressions using TensorFlow and Keras, and extracts emotions from audio using the Google Cloud Speech-to-Text API.

[0413] Step 3:

[0414] The server receives data from the sentiment analysis engine and selects appropriate content based on the user's emotional state. The input is an indicator of the emotional state obtained in step 2, and the output is a list of selected content objects. Specifically, it searches the database for relevant videos, music, and text and lists the most suitable ones.

[0415] Step 4:

[0416] The server streams the selected content to the terminal through an information processing system. The input is the content list generated in step 3, and the output is the content set on the user interface for the user to view. Specifically, it performs conversion to the content format and optimization according to bandwidth to provide the user with an uninterrupted viewing experience.

[0417] Step 5:

[0418] The device delivers the streamed content to the user and records the user's reaction again. The input is the content streamed from the server, and the output is new emotional response data. Specifically, it captures facial expressions and voice again using the camera and microphone and sends the data to the server.

[0419] Step 6:

[0420] The server re-analyzes the sentiment data and adjusts the content provided as needed. The input is the sentiment data obtained in step 5, and the output is the re-selection of content and the addition of detailed explanations as required. Specifically, a generative AI model is used to evaluate the effectiveness of the previous content and provide new content or supplementary information.

[0421] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0422] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0423] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0424] [Third Embodiment]

[0425] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0426] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0427] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0428] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0429] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0430] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0431] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0432] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0433] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0434] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0435] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0436] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0437] This invention provides an embodiment using an electronic system for efficiently managing the process of recording and providing educational activities. Specifically, a camera device installed in the classroom where the lesson is held captures the educational activity as video, and a server receives and records this video in real time. As a result, the entire lesson is saved in digital format.

[0438] The server automatically uploads recorded video to a cloud-based information storage device, and each video is assigned detailed metadata about the lesson. This metadata includes the subject name, teacher's name, and lesson date, and serves as an index for users to search for videos later.

[0439] Users log in through a portal or application running on a dedicated terminal, identify the classes they missed, and request to view them. At this time, users specify the viewing time as needed, and the server edits the video based on this information.

[0440] The server uses machine learning algorithms to extract important information from video data. Through extensive data analysis, it automatically generates a summarized version based on viewing requests and converts it into a format playable on the device.

[0441] Furthermore, if a user has a question while watching, they can enter it through their device. A natural language processing unit operating on the server side analyzes this question and generates the optimal response based on a digital library and AI models. The response is then displayed on the device or provided verbally.

[0442] A concrete example is a user who missed a math class due to illness and later watches a 30-minute condensed version of the 1-hour class on their device. In this case, if a question arises about the differentiation of a function, the user can immediately send the question to the system, and the server can quickly return an answer to support further understanding.

[0443] In this way, this invention improves the learning experience of absent students in educational settings and provides advanced educational services tailored to individual circumstances.

[0444] The following describes the processing flow.

[0445] Step 1:

[0446] The server receives video from cameras installed in the classroom in real time and automatically starts recording at the start of the class. Recording automatically stops at the end of the class, and the obtained video is saved in digital format.

[0447] Step 2:

[0448] The server analyzes the recorded video, adds metadata (subject name, teacher name, lesson date), and then uploads the video data to a cloud-based storage device. During this process, an index is also generated to improve the video's availability.

[0449] Step 3:

[0450] Users log in to a dedicated portal or application, select the specific class day they missed, and send a viewing request from their device to the server. This request also includes the desired viewing time.

[0451] Step 4:

[0452] The server searches for stored video data based on the specified date and subject, and uses machine learning algorithms to analyze and edit the videos to match the user's desired viewing time. This process extracts key content, which is then prepared as a summarized video.

[0453] Step 5:

[0454] The edited video data is provided to the user's device in streaming format for immediate viewing. The user then begins watching this summarized video on their device.

[0455] Step 6:

[0456] If a user has a question while watching, they can enter it on their device, and this question will be sent to the server in real time.

[0457] Step 7:

[0458] The server receives a question and analyzes its content using natural language processing techniques. To generate an appropriate answer, the server consults databases and AI models as needed.

[0459] Step 8:

[0460] The generated answers are displayed to the user via their device or provided as audio responses. This allows users to resolve their questions and deepen their learning on the spot.

[0461] (Example 1)

[0462] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0463] In today's educational environment, a major challenge is the limited learning opportunities resulting from class absences and restricted access to educational resources. Furthermore, the lack of mechanisms for efficient review of lesson content, extraction of key information, and prompt answers to individual student questions is also a problem. This raises concerns about a decline in the quality of learning.

[0464] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0465] In this invention, the server includes a visual information acquisition device for electronically recording educational activities, means for storing the acquired visual information in a remote information storage means, and means for providing educational materials to users via a computer processing means. This makes it possible to easily replay missed lessons as visual information, enabling efficient review and deepening of learning. Furthermore, by utilizing computational learning algorithms and intelligent models, important information can be extracted and responses to user inquiries can be provided quickly, realizing high-quality individualized learning.

[0466] "Educational activities" refer to all activities related to learning, such as classes, lectures, and practical training, conducted at schools and other educational institutions.

[0467] "To record electronically" refers to saving information in digital format using a video camera or other recording device.

[0468] A "visual information acquisition device" refers to a device that captures educational activities in real time and saves them as video data.

[0469] "Remote information storage means" refers to technology that stores acquired data in a physically distant location, such as cloud storage or a remote server.

[0470] "Computer processing means" refers to software and hardware systems that automatically perform tasks such as organizing, searching, playing back, and editing information.

[0471] A "computational learning algorithm" refers to a group of analytical methods that extract patterns from large amounts of data to support decision-making.

[0472] An "intelligent model" refers to an artificial intelligence system designed to answer human questions through natural language processing and data analysis.

[0473] "Individualized learning" refers to a form of education that is tailored to the individual needs and pace of each learner.

[0474] This invention is a system for recording and providing educational activities in digital format, and aims to improve the learning experience in educational institutions.

[0475] The server uses visual information acquisition devices, i.e., cameras, installed in the classroom to capture lessons in real time and collect the video data. The collected video is temporarily stored in the server's storage and then automatically uploaded to a cloud-based remote information storage system. At this time, metadata related to the lesson, such as the subject name, teacher's name, and lesson date, is added.

[0476] Users log in using a dedicated terminal via a provided portal or application. Authenticated users can select a desired lesson from a list of previously recorded lessons and specify the viewing time. Based on the viewing request, the server analyzes the video data, uses computational learning algorithms to extract important information, and generates a summarized version edited to fit the specified time. This summarized version is then converted into a data format playable on the terminal.

[0477] Furthermore, users can input questions that arise during viewing through their device. The server analyzes these questions using an intelligent model based on natural language processing and generates quick and appropriate answers. This allows users to resolve their questions on the spot and enhance their learning effectiveness.

[0478] As a concrete example, consider a user who wants to review a math class they missed for health reasons. This user can access the system and watch a 30-minute summary of the one-hour class. If they want to learn more about the differentiation of functions while watching, they can enter a question into the system, and the server will provide an appropriate explanation through natural language processing. An example of a prompt would be, "Please explain the basic concepts of differentiation in under 5 minutes."

[0479] As described above, this system combines the latest video technology and AI technology to improve the efficiency and quality of individualized learning.

[0480] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0481] Step 1:

[0482] The server receives video data in real time from a visual information acquisition device installed in the classroom. At this time, the video data is temporarily stored in the server's storage. The input is a video stream from the camera, and the output is buffered video data.

[0483] Step 2:

[0484] The server adds relevant metadata such as subject name, teacher name, and class date to the temporarily stored video data and uploads it to a cloud-based remote information storage system. The input for this step is the video data and associated metadata, and the output is the video data stored in the cloud storage.

[0485] Step 3:

[0486] Users log in to the system using a dedicated terminal. Once authentication is complete, the terminal displays a list of classes the user can access. The input at this time is the user's authentication information, and the output is a class list tailored to the user.

[0487] Step 4:

[0488] The user selects the class they wish to view and submits a viewing request based on the specified time. The terminal sends this information to the server. The input is the user's selected class information and viewing time, and the output is the viewing request sent to the server.

[0489] Step 5:

[0490] Based on user viewing requests, the server uses a computational learning algorithm to analyze video data, extract important scenes, and create a summary. The input is unedited video data and a specified viewing time, while the output is edited summary video data.

[0491] Step 6:

[0492] The server converts the generated summary video data into a format playable on the terminal and sends it to the user's terminal. In this step, the input is the summary video data, and the output is data in a playable format.

[0493] Step 7:

[0494] If a user has a question while watching, they enter it via their device. The device then sends this question to the server. The input is the user's question, and the output is the question request sent to the server.

[0495] Step 8:

[0496] The server uses an intelligent model to analyze the user's question and generate a quick and appropriate answer. The input is the question data, and the output is the answer data. The generated answer is returned to the terminal and presented in a format that the user can see or hear.

[0497] (Application Example 1)

[0498] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0499] In educational activities, there is a problem in effectively recording and providing lesson content. Specifically, there is a lack of efficient systems for students who are absent to catch up on lesson content. Furthermore, there is a need to provide an environment where questions that arise during viewing can be answered immediately.

[0500] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0501] In this invention, the server includes means for video recording educational activities and managing them on a cloud platform, means for using machine learning algorithms to generate summarized video versions, and means for providing answers to user questions using artificial intelligence technology. This enables absent students to effectively catch up on class content and obtain answers to their questions in real time.

[0502] "Educational activities" refer to activities aimed at learning, such as classes and lectures.

[0503] "Video recording" refers to the process of saving visual information, including movement, as digital data using a camera or other recording device.

[0504] A "cloud infrastructure" is a structure of remote servers used to store and manage data via the internet.

[0505] An "information storage device" is a hardware or software system for storing data.

[0506] "Information processing" is the process of analyzing and transforming data to generate useful information.

[0507] A "viewing request" is a request from a user who wants to watch a video at a specific time or with a specific content.

[0508] A "summary version" is a video that has been reconstructed in a shortened format by extracting the important parts from the original video data.

[0509] A "machine learning algorithm" is a method that allows computers to learn patterns from data and automatically perform predictions and classifications.

[0510] "Artificial intelligence technology" is the technology that enables machines to mimic human intelligence and act accordingly.

[0511] "Natural language processing" is a technology that enables computers to understand, generate, and analyze human language.

[0512] A "question" is a question posed by a user to gain knowledge or information.

[0513] An "answer" is a response or explanation given in response to a question.

[0514] To implement this invention, the server records educational activities on video and stores the data on a cloud infrastructure. The hardware includes a camera and a cloud server that collects video data in real time and uploads it to the cloud. AWS or Google Cloud are suitable as the cloud infrastructure.

[0515] The server uses machine learning algorithms to extract important information from the video and generates a summarized version of the video according to the user's viewing request. Here, machine learning models are built using frameworks such as TensorFlow and PyTorch.

[0516] Furthermore, questions posed by users while viewing content on their devices are analyzed using natural language processing technology. By applying generative AI models such as OpenAI's GPT-3, the system generates and provides the user with the most appropriate answers based on this analysis.

[0517] As a concrete example, there is a feature that allows users to watch a summarized video of any subject they missed in class. For instance, a user could enter a prompt such as, "I would like to watch a summarized version of the math class. Please explain the differentiation of functions." In response to this prompt, the system can display a summarized version of the relevant class and automatically provide an explanation of the differentiation of functions.

[0518] In this way, this invention aims to provide a means for efficiently and flexibly reproducing and utilizing educational activities, thereby supporting the learning of individual users.

[0519] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0520] Step 1:

[0521] The server begins receiving video data in real time from cameras in the classroom where educational activities are taking place. The input is video data from the cameras, and the output is video data stored on the cloud infrastructure. At this stage, the server efficiently acquires the data using data streaming technology.

[0522] Step 2:

[0523] The server automatically generates metadata (such as class name, teacher name, and date) related to the received video data and adds it when uploading to the cloud. The input is the received video data and class information, and the output is the video data on the cloud with the metadata attached. A database system is used to effectively manage the metadata.

[0524] Step 3:

[0525] The user sends a viewing request to the server via their device. The input is the user's viewing request (subject, viewing time, etc.), and the output is data that is processed as a request to the server. Based on this information, the server applies a machine learning algorithm to prepare to generate a summarized version of the video.

[0526] Step 4:

[0527] The server uses machine learning algorithms to extract important information and edits and generates summarized videos based on viewing requests. The input is the original video data and the user's viewing request, and the output is summarized video data. TensorFlow and PyTorch are used to build models and analyze the video data.

[0528] Step 5:

[0529] When a user enters a question while watching, the device sends that prompt to the server. The input is the user's question, and the output is the query data sent to the server. The text data is formatted in preparation for natural language processing.

[0530] Step 6:

[0531] The server uses a generative AI model to analyze user questions and generate the optimal answer. The input is the user's question text, and the output is the analyzed answer text. Natural language processing is performed using OpenAI's GPT-3, etc., to generate the answer based on relevant information.

[0532] Step 7:

[0533] The terminal displays the answer received from the server to the user. The input is the answer text sent from the server, and the output is the answer information displayed on the terminal. The displayed answer is presented clearly through the user interface.

[0534] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0535] This invention is a system for effectively experiencing classes and educational activities remotely, and is particularly intended to provide flexible learning support for absent students. This system is equipped with an emotion engine that recognizes the user's emotions in real time and has the function to dynamically adjust the learning experience.

[0536] The server receives video from cameras installed in the classroom and records the entire lesson. This video is stored in the cloud and tagged with metadata. Users can view it through an application. Machine learning algorithms are used for video editing, editing and presenting important learning points according to the viewer's preferred time.

[0537] Furthermore, the user's device is equipped with a camera and microphone, and an emotion engine is incorporated that analyzes the user's emotions in real time from their facial expressions and voice through these devices. When the user watches the video, the emotion engine can identify emotions such as joy, confusion, and anxiety. This information is sent to the server, which uses this feedback to make adjustments to optimize the viewing experience.

[0538] For example, if a user expresses confusion, the server will help them understand by repeating the problematic section or displaying additional explanations. This entire process is automated, creating an environment where users can focus on learning. Furthermore, natural language processing technology enables an interactive learning experience by generating accurate answers to user questions in real time and displaying them on the device.

[0539] A concrete example would be a scenario where, while watching a math lesson, a user becomes confused by a calculus concept; the emotion engine detects this emotion, and the server immediately provides supplementary explanations. In this way, the goal is to provide personalized education by offering adaptive learning support through emotional feedback.

[0540] The following describes the processing flow.

[0541] Step 1:

[0542] The server receives video footage from cameras installed in the classroom and automatically starts recording according to the set start time. When the class ends, it stops recording and transfers the video data to cloud storage.

[0543] Step 2:

[0544] The server adds metadata to the recorded video, storing information such as the subject name, class date, and teacher's name in a database. This allows users to easily search for specific classes.

[0545] Step 3:

[0546] Users use a dedicated application to identify classes they have missed and send requests to view them from their device to the server. They can also include requests for preferred viewing times and specific explanations.

[0547] Step 4:

[0548] The server receives viewing requests from users and edits the video using a machine learning algorithm based on the desired viewing time. Through this process, a summarized video highlighting key learning points is generated.

[0549] Step 5:

[0550] The edited, summarized video is delivered to the user's device via streaming. The device then displays the received video to the user and begins playback.

[0551] Step 6:

[0552] The user's device has a built-in camera and microphone, and the emotion engine uses this data to analyze the user's emotions in real time from their facial expressions and voice. The results are then reported to the server.

[0553] Step 7:

[0554] Based on data from the emotion engine, the server adjusts the viewing experience by providing additional explanations or clarifications, or by pausing playback and inserting simpler explanations, if the user's emotions indicate anxiety or confusion.

[0555] Step 8:

[0556] When a user has a question, they input it through their device, and this question is transmitted to the server in real time. Using natural language processing technology, the server analyzes the question, generates an accurate and immediate answer, and displays it on the device.

[0557] Step 9:

[0558] All viewing and interaction records are stored on the server and used to track user learning progress and design feedback. This allows for continuous improvement of the quality of educational services.

[0559] (Example 2)

[0560] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0561] In today's educational environment, despite the growing demand for distance learning, there is a problem in that it cannot adequately accommodate individual learning paces and styles. In particular, there is a challenge in that learning effectiveness decreases when beneficiaries cannot participate in real-time classes or when their understanding of the lesson content is insufficient. This invention aims to improve the quality of learning by providing an environment in which beneficiaries can learn efficiently and flexibly remotely.

[0562] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0563] In this invention, the server includes recording means for video recording educational activities, means for storing the recorded video in an information storage device, and means for providing educational resources to beneficiaries via information processing means. This enables beneficiaries to participate in classes in real time from remote locations and enjoy a high-quality learning experience.

[0564] "Educational activities" are a set of actions organized to impart specific knowledge or skills.

[0565] "Recording means" refers to devices and methods for storing information such as audio and video for extended periods.

[0566] "Information storage device" refers to hardware that stores digital data and makes it available for retrieval as needed.

[0567] "Information processing means" refers to the mechanisms of devices and software that analyze collected data and provide it to users as useful information.

[0568] A "beneficiary" refers to an individual or group that utilizes educational resources and gains learning opportunities through this system.

[0569] A "viewing request" refers to a request from a beneficiary who wishes to view specific video content.

[0570] "Machine learning" refers to algorithms or methods that enable computers to recognize patterns from given data and make predictions and decisions more effectively.

[0571] "Natural language processing" refers to the field of technology that enables computers to understand, analyze, and generate responses to human language.

[0572] A "generative model" refers to an artificial intelligence model that has the ability to generate new data from given data.

[0573] "Emotional analysis" refers to a technology that analyzes an individual's facial expressions and voice to recognize their emotional state at that time.

[0574] This invention is a system for effectively experiencing educational activities remotely. Its primary purpose is to provide flexible learning support for beneficiaries who miss classes. The system incorporates an emotion engine that analyzes the beneficiary's emotions in real time and dynamically adjusts the learning experience accordingly.

[0575] The server receives video from cameras installed in the classroom and records the lesson in progress. This video is stored in a cloud-based information storage device and tagged with metadata. The metadata includes lesson content, date, and instructor's name. The system uses machine learning algorithms for video editing, enabling it to extract key learning points and generate summary videos tailored to the viewer's preferred viewing time.

[0576] The device is equipped with a camera and microphone, and incorporates an emotion engine that analyzes the user's emotions through their facial expressions and voice. The analyzed data is sent to a server, and the viewing experience is adjusted based on the user's emotional feedback. For example, if the user shows confusion or anxiety, the relevant section of the video may be played repeatedly, or additional text explanations may be provided.

[0577] Users can watch recorded lesson videos through the application. If a user has a question while watching, they can ask it in real time using natural language processing technology. A generative AI model understands the question, generates an accurate answer, and displays it on the device. This enables an interactive learning experience where beneficiaries can instantly resolve questions that arise during their studies.

[0578] For example, if a beneficiary is watching a math lesson and becomes confused by a calculus concept, the device's emotion engine detects this confusion. This information is sent to a server, which then repeatedly explains key points of calculus or provides supplementary materials. This automated process allows beneficiaries to learn at their own pace.

[0579] Another example of a prompt might be, "Please explain the Fundamental Theorem of Integration in detail. Please briefly explain the terminology and include examples." This type of prompt allows the generative AI model to generate appropriate answers and provide effective learning support to the beneficiary.

[0580] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0581] Step 1:

[0582] The server receives video in real time from cameras installed in the classroom and records the progress of the lesson. The input is video data from the cameras, and the output is the recorded lesson video. The video data is immediately saved to a cloud storage device, and metadata regarding the date, time, and content of the lecture is generated and attached. The server checks the video quality and audio and saves it in the appropriate format.

[0583] Step 2:

[0584] The server processes the stored lecture videos using a machine learning algorithm. The input is the recorded lecture video and its metadata, and the output is an edited video containing summarized learning points. The machine learning algorithm transcribes the audio in the video and extracts specific keywords. Based on this information, it automatically edits the important parts and generates a shortened version according to the viewer's desired length.

[0585] Step 3:

[0586] The device uses its built-in camera and microphone to record the user's facial expressions and voice in real time. The input is the user's video and audio data, and the output is the analysis results regarding the user's emotional state. An emotion engine analyzes the data and identifies emotions such as joy and confusion in real time. Sensors detect the user's body movements and voice tone, and the analysis results are sent to the server.

[0587] Step 4:

[0588] The server optimizes the viewing experience based on emotional feedback received from the device. The input is the result of the user's emotional analysis, and the output is dynamically adjusted viewing content. For example, if the user shows confusion, the server will automatically repeat the relevant section of the video or display additional explanations in text. This feedback loop allows the user to understand the content more deeply.

[0589] Step 5:

[0590] When a user enters a question through the application interface, the server generates an answer using a generative AI model. The input is the user's text question, and the output is the generated answer. The server uses natural language processing to understand the question and feeds appropriate prompt sentences into the generative AI model. The resulting answer is then displayed on the terminal. The system accurately displays the necessary information according to the question.

[0591] (Application Example 2)

[0592] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0593] In rehabilitation and daily living training for the elderly and dementia patients, there is a challenge in providing support tailored to each individual's cognitive state and emotions. Existing rehabilitation systems offer uniform programs, making it difficult to effectively improve users' cognitive abilities and maintain their motivation. Furthermore, the lack of automatic content adjustment based on user responses limits the effectiveness of the support.

[0594] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0595] In this invention, the server includes video recording means, means for analyzing the user's cognitive state using an emotion analysis engine, means for storing the recorded video in an information storage device, and means for providing educational resources and emotional feedback-based content to the user through information processing means. This makes it possible to provide an individually optimized rehabilitation program that is tailored to the user's emotional state.

[0596] "Video recording means" refers to devices and technologies used to record educational activities and rehabilitation sessions, making them available for later analysis and viewing.

[0597] An "emotion analysis engine" refers to software or algorithms that analyze a user's facial expressions and voice through a camera or microphone to determine their emotional state in real time.

[0598] "Information storage device" refers to a storage device or cloud system that securely stores captured video and data for later access and analysis.

[0599] "Information processing means" refers to computing resources and processing methods used to generate and provide appropriate content and feedback to users based on collected data.

[0600] "Emotional feedback" refers to adjustments and adaptations to optimized content and activities provided in response to the user's emotional state.

[0601] "Content" refers to the collective term for images, audio, text, interactive elements, etc., provided to users and used for educational or supportive purposes.

[0602] "User" refers to an individual who is the subject of this invention and who participates in educational activities or rehabilitation programs.

[0603] To realize this invention, a series of systems involving a server, terminals, and users are used. First, the server stores video data acquired from terminals equipped with cameras and microphones in an information storage device. In this process, it is necessary to record the entire rehabilitation or educational session using recording means. The recorded data is stored in the information storage device and later used for analysis and feedback generation.

[0604] The device incorporates an emotion analysis engine that analyzes the user's facial expressions and voice in real time. This data is then sent to a server to determine the user's emotional and cognitive state. At this stage, machine learning libraries such as TensorFlow and Keras are used for facial recognition, and the Google Cloud Speech-to-Text API is utilized for voice analysis. Furthermore, IBM Watson's emotion analysis is employed for sentiment analysis.

[0605] The server processes this data and delivers content through emotional feedback. Specifically, it improves the individual experience by optimizing and presenting content such as videos, images, and music selected according to the user's emotional state.

[0606] For example, if a user shows curiosity during a rehabilitation session, the server immediately displays content related to relevant history or interesting topics. This improves the user's attention and enhances the effectiveness of the session.

[0607] An example of a prompt message is, "Based on the user's sentiment analysis, list content that is likely to be of interest and present it in the appropriate order."

[0608] This system enables flexible rehabilitation and educational support tailored to the individual needs of each user.

[0609] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0610] Step 1:

[0611] The device uses a camera and microphone to collect video and audio of the user. The input is real-time video and audio data, which is recorded in high resolution. The output is raw data converted into a data format that can be processed by the emotion analysis engine. Specifically, it captures the user's face and voice as digital data, performs noise reduction, and processes the data as needed.

[0612] Step 2:

[0613] The device analyzes collected video and audio data using an emotion analysis engine. The input is the formatted data obtained in step 1, and the output is a quantitative indicator showing the user's emotional state. Specifically, it analyzes facial expressions using TensorFlow and Keras, and extracts emotions from audio using the Google Cloud Speech-to-Text API.

[0614] Step 3:

[0615] The server receives data from the sentiment analysis engine and selects appropriate content based on the user's emotional state. The input is an indicator of the emotional state obtained in step 2, and the output is a list of selected content objects. Specifically, it searches the database for relevant videos, music, and text and lists the most suitable ones.

[0616] Step 4:

[0617] The server streams the selected content to the terminal through an information processing system. The input is the content list generated in step 3, and the output is the content set on the user interface for the user to view. Specifically, it performs conversion to the content format and optimization according to bandwidth to provide the user with an uninterrupted viewing experience.

[0618] Step 5:

[0619] The device delivers the streamed content to the user and records the user's reaction again. The input is the content streamed from the server, and the output is new emotional response data. Specifically, it captures facial expressions and voice again using the camera and microphone and sends the data to the server.

[0620] Step 6:

[0621] The server re-analyzes the sentiment data and adjusts the content provided as needed. The input is the sentiment data obtained in step 5, and the output is the re-selection of content and the addition of detailed explanations as required. Specifically, a generative AI model is used to evaluate the effectiveness of the previous content and provide new content or supplementary information.

[0622] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0623] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0624] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0625] [Fourth Embodiment]

[0626] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0627] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0628] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0629] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0630] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0631] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0632] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0633] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0634] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0635] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0636] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0637] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0638] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0639] This invention provides an embodiment using an electronic system for efficiently managing the process of recording and providing educational activities. Specifically, a camera device installed in the classroom where the lesson is held captures the educational activity as video, and a server receives and records this video in real time. As a result, the entire lesson is saved in digital format.

[0640] The server automatically uploads recorded video to a cloud-based information storage device, and each video is assigned detailed metadata about the lesson. This metadata includes the subject name, teacher's name, and lesson date, and serves as an index for users to search for videos later.

[0641] Users log in through a portal or application running on a dedicated terminal, identify the classes they missed, and request to view them. At this time, users specify the viewing time as needed, and the server edits the video based on this information.

[0642] The server uses machine learning algorithms to extract important information from video data. Through extensive data analysis, it automatically generates a summarized version based on viewing requests and converts it into a format playable on the device.

[0643] Furthermore, if a user has a question while watching, they can enter it through their device. A natural language processing unit operating on the server side analyzes this question and generates the optimal response based on a digital library and AI models. The response is then displayed on the device or provided verbally.

[0644] A concrete example is a user who missed a math class due to illness and later watches a 30-minute condensed version of the 1-hour class on their device. In this case, if a question arises about the differentiation of a function, the user can immediately send the question to the system, and the server can quickly return an answer to support further understanding.

[0645] In this way, this invention improves the learning experience of absent students in educational settings and provides advanced educational services tailored to individual circumstances.

[0646] The following describes the processing flow.

[0647] Step 1:

[0648] The server receives video from cameras installed in the classroom in real time and automatically starts recording at the start of the class. Recording automatically stops at the end of the class, and the obtained video is saved in digital format.

[0649] Step 2:

[0650] The server analyzes the recorded video, adds metadata (subject name, teacher name, lesson date), and then uploads the video data to a cloud-based storage device. During this process, an index is also generated to improve the video's availability.

[0651] Step 3:

[0652] Users log in to a dedicated portal or application, select the specific class day they missed, and send a viewing request from their device to the server. This request also includes the desired viewing time.

[0653] Step 4:

[0654] The server searches for stored video data based on the specified date and subject, and uses machine learning algorithms to analyze and edit the videos to match the user's desired viewing time. This process extracts key content, which is then prepared as a summarized video.

[0655] Step 5:

[0656] The edited video data is provided to the user's device in streaming format for immediate viewing. The user then begins watching this summarized video on their device.

[0657] Step 6:

[0658] If a user has a question while watching, they can enter it on their device, and this question will be sent to the server in real time.

[0659] Step 7:

[0660] The server receives a question and analyzes its content using natural language processing techniques. To generate an appropriate answer, the server consults databases and AI models as needed.

[0661] Step 8:

[0662] The generated answers are displayed to the user via their device or provided as audio responses. This allows users to resolve their questions and deepen their learning on the spot.

[0663] (Example 1)

[0664] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0665] In today's educational environment, a major challenge is the limited learning opportunities resulting from class absences and restricted access to educational resources. Furthermore, the lack of mechanisms for efficient review of lesson content, extraction of key information, and prompt answers to individual student questions is also a problem. This raises concerns about a decline in the quality of learning.

[0666] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0667] In this invention, the server includes a visual information acquisition device for electronically recording educational activities, means for storing the acquired visual information in a remote information storage means, and means for providing educational materials to users via a computer processing means. This makes it possible to easily replay missed lessons as visual information, enabling efficient review and deepening of learning. Furthermore, by utilizing computational learning algorithms and intelligent models, important information can be extracted and responses to user inquiries can be provided quickly, realizing high-quality individualized learning.

[0668] "Educational activities" refer to all activities related to learning, such as classes, lectures, and practical training, conducted at schools and other educational institutions.

[0669] "To record electronically" refers to saving information in digital format using a video camera or other recording device.

[0670] A "visual information acquisition device" refers to a device that captures educational activities in real time and saves them as video data.

[0671] "Remote information storage means" refers to technology that stores acquired data in a physically distant location, such as cloud storage or a remote server.

[0672] "Computer processing means" refers to software and hardware systems that automatically perform tasks such as organizing, searching, playing back, and editing information.

[0673] A "computational learning algorithm" refers to a group of analytical methods that extract patterns from large amounts of data to support decision-making.

[0674] An "intelligent model" refers to an artificial intelligence system designed to answer human questions through natural language processing and data analysis.

[0675] "Individualized learning" refers to a form of education that is tailored to the individual needs and pace of each learner.

[0676] This invention is a system for recording and providing educational activities in digital format, and aims to improve the learning experience in educational institutions.

[0677] The server uses visual information acquisition devices, i.e., cameras, installed in the classroom to capture lessons in real time and collect the video data. The collected video is temporarily stored in the server's storage and then automatically uploaded to a cloud-based remote information storage system. At this time, metadata related to the lesson, such as the subject name, teacher's name, and lesson date, is added.

[0678] Users log in using a dedicated terminal via a provided portal or application. Authenticated users can select a desired lesson from a list of previously recorded lessons and specify the viewing time. Based on the viewing request, the server analyzes the video data, uses computational learning algorithms to extract important information, and generates a summarized version edited to fit the specified time. This summarized version is then converted into a data format playable on the terminal.

[0679] Furthermore, users can input questions that arise during viewing through their device. The server analyzes these questions using an intelligent model based on natural language processing and generates quick and appropriate answers. This allows users to resolve their questions on the spot and enhance their learning effectiveness.

[0680] As a concrete example, consider a user who wants to review a math class they missed for health reasons. This user can access the system and watch a 30-minute summary of the one-hour class. If they want to learn more about the differentiation of functions while watching, they can enter a question into the system, and the server will provide an appropriate explanation through natural language processing. An example of a prompt would be, "Please explain the basic concepts of differentiation in under 5 minutes."

[0681] As described above, this system combines the latest video technology and AI technology to improve the efficiency and quality of individualized learning.

[0682] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0683] Step 1:

[0684] The server receives video data in real time from a visual information acquisition device installed in the classroom. At this time, the video data is temporarily stored in the server's storage. The input is a video stream from the camera, and the output is buffered video data.

[0685] Step 2:

[0686] The server adds relevant metadata such as subject name, teacher name, and class date to the temporarily stored video data and uploads it to a cloud-based remote information storage system. The input for this step is the video data and associated metadata, and the output is the video data stored in the cloud storage.

[0687] Step 3:

[0688] Users log in to the system using a dedicated terminal. Once authentication is complete, the terminal displays a list of classes the user can access. The input at this time is the user's authentication information, and the output is a class list tailored to the user.

[0689] Step 4:

[0690] The user selects the class they wish to view and submits a viewing request based on the specified time. The terminal sends this information to the server. The input is the user's selected class information and viewing time, and the output is the viewing request sent to the server.

[0691] Step 5:

[0692] Based on user viewing requests, the server uses a computational learning algorithm to analyze video data, extract important scenes, and create a summary. The input is unedited video data and a specified viewing time, while the output is edited summary video data.

[0693] Step 6:

[0694] The server converts the generated summary video data into a format playable on the terminal and sends it to the user's terminal. In this step, the input is the summary video data, and the output is data in a playable format.

[0695] Step 7:

[0696] If a user has a question while watching, they enter it via their device. The device then sends this question to the server. The input is the user's question, and the output is the question request sent to the server.

[0697] Step 8:

[0698] The server uses an intelligent model to analyze the user's question and generate a quick and appropriate answer. The input is the question data, and the output is the answer data. The generated answer is returned to the terminal and presented in a format that the user can see or hear.

[0699] (Application Example 1)

[0700] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0701] In educational activities, there is a problem in effectively recording and providing lesson content. Specifically, there is a lack of efficient systems for students who are absent to catch up on lesson content. Furthermore, there is a need to provide an environment where questions that arise during viewing can be answered immediately.

[0702] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0703] In this invention, the server includes means for video recording educational activities and managing them on a cloud platform, means for using machine learning algorithms to generate summarized video versions, and means for providing answers to user questions using artificial intelligence technology. This enables absent students to effectively catch up on class content and obtain answers to their questions in real time.

[0704] "Educational activities" refer to activities aimed at learning, such as classes and lectures.

[0705] "Video recording" refers to the process of saving visual information, including movement, as digital data using a camera or other recording device.

[0706] A "cloud infrastructure" is a structure of remote servers used to store and manage data via the internet.

[0707] An "information storage device" is a hardware or software system for storing data.

[0708] "Information processing" is the process of analyzing and transforming data to generate useful information.

[0709] A "viewing request" is a request from a user who wants to watch a video at a specific time or with a specific content.

[0710] A "summary version" is a video that has been reconstructed in a shortened format by extracting the important parts from the original video data.

[0711] A "machine learning algorithm" is a method that allows computers to learn patterns from data and automatically perform predictions and classifications.

[0712] "Artificial intelligence technology" is the technology that enables machines to mimic human intelligence and act accordingly.

[0713] "Natural language processing" is a technology that enables computers to understand, generate, and analyze human language.

[0714] A "question" is a question posed by a user to gain knowledge or information.

[0715] An "answer" is a response or explanation given in response to a question.

[0716] To implement this invention, the server records educational activities on video and stores the data on a cloud infrastructure. The hardware includes a camera and a cloud server that collects video data in real time and uploads it to the cloud. AWS or Google Cloud are suitable as the cloud infrastructure.

[0717] The server uses machine learning algorithms to extract important information from the video and generates a summarized version of the video according to the user's viewing request. Here, machine learning models are built using frameworks such as TensorFlow and PyTorch.

[0718] Furthermore, questions posed by users while viewing content on their devices are analyzed using natural language processing technology. By applying generative AI models such as OpenAI's GPT-3, the system generates and provides the user with the most appropriate answers based on this analysis.

[0719] As a concrete example, there is a feature that allows users to watch a summarized video of any subject they missed in class. For instance, a user could enter a prompt such as, "I would like to watch a summarized version of the math class. Please explain the differentiation of functions." In response to this prompt, the system can display a summarized version of the relevant class and automatically provide an explanation of the differentiation of functions.

[0720] In this way, this invention aims to provide a means for efficiently and flexibly reproducing and utilizing educational activities, thereby supporting the learning of individual users.

[0721] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0722] Step 1:

[0723] The server begins receiving video data in real time from cameras in the classroom where educational activities are taking place. The input is video data from the cameras, and the output is video data stored on the cloud infrastructure. At this stage, the server efficiently acquires the data using data streaming technology.

[0724] Step 2:

[0725] The server automatically generates metadata (such as class name, teacher name, and date) related to the received video data and adds it when uploading to the cloud. The input is the received video data and class information, and the output is the video data on the cloud with the metadata attached. A database system is used to effectively manage the metadata.

[0726] Step 3:

[0727] The user sends a viewing request to the server via their device. The input is the user's viewing request (subject, viewing time, etc.), and the output is data that is processed as a request to the server. Based on this information, the server applies a machine learning algorithm to prepare to generate a summarized version of the video.

[0728] Step 4:

[0729] The server uses machine learning algorithms to extract important information and edits and generates summarized videos based on viewing requests. The input is the original video data and the user's viewing request, and the output is summarized video data. TensorFlow and PyTorch are used to build models and analyze the video data.

[0730] Step 5:

[0731] When a user enters a question while watching, the device sends that prompt to the server. The input is the user's question, and the output is the query data sent to the server. The text data is formatted in preparation for natural language processing.

[0732] Step 6:

[0733] The server uses a generative AI model to analyze user questions and generate the optimal answer. The input is the user's question text, and the output is the analyzed answer text. Natural language processing is performed using OpenAI's GPT-3, etc., to generate the answer based on relevant information.

[0734] Step 7:

[0735] The terminal displays the answer received from the server to the user. The input is the answer text sent from the server, and the output is the answer information displayed on the terminal. The displayed answer is presented clearly through the user interface.

[0736] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0737] This invention is a system for effectively experiencing classes and educational activities remotely, and is particularly intended to provide flexible learning support for absent students. This system is equipped with an emotion engine that recognizes the user's emotions in real time and has the function to dynamically adjust the learning experience.

[0738] The server receives video from cameras installed in the classroom and records the entire lesson. This video is stored in the cloud and tagged with metadata. Users can view it through an application. Machine learning algorithms are used for video editing, editing and presenting important learning points according to the viewer's preferred time.

[0739] Furthermore, the user's device is equipped with a camera and microphone, and an emotion engine is incorporated that analyzes the user's emotions in real time from their facial expressions and voice through these devices. When the user watches the video, the emotion engine can identify emotions such as joy, confusion, and anxiety. This information is sent to the server, which uses this feedback to make adjustments to optimize the viewing experience.

[0740] For example, if a user expresses confusion, the server will help them understand by repeating the problematic section or displaying additional explanations. This entire process is automated, creating an environment where users can focus on learning. Furthermore, natural language processing technology enables an interactive learning experience by generating accurate answers to user questions in real time and displaying them on the device.

[0741] A concrete example would be a scenario where, while watching a math lesson, a user becomes confused by a calculus concept; the emotion engine detects this emotion, and the server immediately provides supplementary explanations. In this way, the goal is to provide personalized education by offering adaptive learning support through emotional feedback.

[0742] The following describes the processing flow.

[0743] Step 1:

[0744] The server receives video footage from cameras installed in the classroom and automatically starts recording according to the set start time. When the class ends, it stops recording and transfers the video data to cloud storage.

[0745] Step 2:

[0746] The server adds metadata to the recorded video, storing information such as the subject name, class date, and teacher's name in a database. This allows users to easily search for specific classes.

[0747] Step 3:

[0748] Users use a dedicated application to identify classes they have missed and send requests to view them from their device to the server. They can also include requests for preferred viewing times and specific explanations.

[0749] Step 4:

[0750] The server receives viewing requests from users and edits the video using a machine learning algorithm based on the desired viewing time. Through this process, a summarized video highlighting key learning points is generated.

[0751] Step 5:

[0752] The edited, summarized video is delivered to the user's device via streaming. The device then displays the received video to the user and begins playback.

[0753] Step 6:

[0754] The user's device has a built-in camera and microphone, and the emotion engine uses this data to analyze the user's emotions in real time from their facial expressions and voice. The results are then reported to the server.

[0755] Step 7:

[0756] Based on data from the emotion engine, the server adjusts the viewing experience by providing additional explanations or clarifications, or by pausing playback and inserting simpler explanations, if the user's emotions indicate anxiety or confusion.

[0757] Step 8:

[0758] When a user has a question, they input it through their device, and this question is transmitted to the server in real time. Using natural language processing technology, the server analyzes the question, generates an accurate and immediate answer, and displays it on the device.

[0759] Step 9:

[0760] All viewing and interaction records are stored on the server and used to track user learning progress and design feedback. This allows for continuous improvement of the quality of educational services.

[0761] (Example 2)

[0762] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0763] In today's educational environment, despite the growing demand for distance learning, there is a problem in that it cannot adequately accommodate individual learning paces and styles. In particular, there is a challenge in that learning effectiveness decreases when beneficiaries cannot participate in real-time classes or when their understanding of the lesson content is insufficient. This invention aims to improve the quality of learning by providing an environment in which beneficiaries can learn efficiently and flexibly remotely.

[0764] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0765] In this invention, the server includes recording means for video recording educational activities, means for storing the recorded video in an information storage device, and means for providing educational resources to beneficiaries via information processing means. This enables beneficiaries to participate in classes in real time from remote locations and enjoy a high-quality learning experience.

[0766] "Educational activities" are a set of actions organized to impart specific knowledge or skills.

[0767] "Recording means" refers to devices and methods for storing information such as audio and video for extended periods.

[0768] "Information storage device" refers to hardware that stores digital data and makes it available for retrieval as needed.

[0769] "Information processing means" refers to the mechanisms of devices and software that analyze collected data and provide it to users as useful information.

[0770] A "beneficiary" refers to an individual or group that utilizes educational resources and gains learning opportunities through this system.

[0771] A "viewing request" refers to a request from a beneficiary who wishes to view specific video content.

[0772] "Machine learning" refers to algorithms or methods that enable computers to recognize patterns from given data and make predictions and decisions more effectively.

[0773] "Natural language processing" refers to the field of technology that enables computers to understand, analyze, and generate responses to human language.

[0774] A "generative model" refers to an artificial intelligence model that has the ability to generate new data from given data.

[0775] "Emotional analysis" refers to a technology that analyzes an individual's facial expressions and voice to recognize their emotional state at that time.

[0776] This invention is a system for effectively experiencing educational activities remotely. Its primary purpose is to provide flexible learning support for beneficiaries who miss classes. The system incorporates an emotion engine that analyzes the beneficiary's emotions in real time and dynamically adjusts the learning experience accordingly.

[0777] The server receives video from cameras installed in the classroom and records the lesson in progress. This video is stored in a cloud-based information storage device and tagged with metadata. The metadata includes lesson content, date, and instructor's name. The system uses machine learning algorithms for video editing, enabling it to extract key learning points and generate summary videos tailored to the viewer's preferred viewing time.

[0778] The device is equipped with a camera and microphone, and incorporates an emotion engine that analyzes the user's emotions through their facial expressions and voice. The analyzed data is sent to a server, and the viewing experience is adjusted based on the user's emotional feedback. For example, if the user shows confusion or anxiety, the relevant section of the video may be played repeatedly, or additional text explanations may be provided.

[0779] Users can watch recorded lesson videos through the application. If a user has a question while watching, they can ask it in real time using natural language processing technology. A generative AI model understands the question, generates an accurate answer, and displays it on the device. This enables an interactive learning experience where beneficiaries can instantly resolve questions that arise during their studies.

[0780] For example, if a beneficiary is watching a math lesson and becomes confused by a calculus concept, the device's emotion engine detects this confusion. This information is sent to a server, which then repeatedly explains key points of calculus or provides supplementary materials. This automated process allows beneficiaries to learn at their own pace.

[0781] Another example of a prompt might be, "Please explain the Fundamental Theorem of Integration in detail. Please briefly explain the terminology and include examples." This type of prompt allows the generative AI model to generate appropriate answers and provide effective learning support to the beneficiary.

[0782] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0783] Step 1:

[0784] The server receives video in real time from cameras installed in the classroom and records the progress of the lesson. The input is video data from the cameras, and the output is the recorded lesson video. The video data is immediately saved to a cloud storage device, and metadata regarding the date, time, and content of the lecture is generated and attached. The server checks the video quality and audio and saves it in the appropriate format.

[0785] Step 2:

[0786] The server processes the stored lecture videos using a machine learning algorithm. The input is the recorded lecture video and its metadata, and the output is an edited video containing summarized learning points. The machine learning algorithm transcribes the audio in the video and extracts specific keywords. Based on this information, it automatically edits the important parts and generates a shortened version according to the viewer's desired length.

[0787] Step 3:

[0788] The device uses its built-in camera and microphone to record the user's facial expressions and voice in real time. The input is the user's video and audio data, and the output is the analysis results regarding the user's emotional state. An emotion engine analyzes the data and identifies emotions such as joy and confusion in real time. Sensors detect the user's body movements and voice tone, and the analysis results are sent to the server.

[0789] Step 4:

[0790] The server optimizes the viewing experience based on emotional feedback received from the device. The input is the result of the user's emotional analysis, and the output is dynamically adjusted viewing content. For example, if the user shows confusion, the server will automatically repeat the relevant section of the video or display additional explanations in text. This feedback loop allows the user to understand the content more deeply.

[0791] Step 5:

[0792] When a user enters a question through the application interface, the server generates an answer using a generative AI model. The input is the user's text question, and the output is the generated answer. The server uses natural language processing to understand the question and feeds appropriate prompt sentences into the generative AI model. The resulting answer is then displayed on the terminal. The system accurately displays the necessary information according to the question.

[0793] (Application Example 2)

[0794] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0795] In rehabilitation and daily living training for the elderly and dementia patients, there is a challenge in providing support tailored to each individual's cognitive state and emotions. Existing rehabilitation systems offer uniform programs, making it difficult to effectively improve users' cognitive abilities and maintain their motivation. Furthermore, the lack of automatic content adjustment based on user responses limits the effectiveness of the support.

[0796] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0797] In this invention, the server includes video recording means, means for analyzing the user's cognitive state using an emotion analysis engine, means for storing the recorded video in an information storage device, and means for providing educational resources and emotional feedback-based content to the user through information processing means. This makes it possible to provide an individually optimized rehabilitation program that is tailored to the user's emotional state.

[0798] "Video recording means" refers to devices and technologies used to record educational activities and rehabilitation sessions, making them available for later analysis and viewing.

[0799] An "emotion analysis engine" refers to software or algorithms that analyze a user's facial expressions and voice through a camera or microphone to determine their emotional state in real time.

[0800] "Information storage device" refers to a storage device or cloud system that securely stores captured video and data for later access and analysis.

[0801] "Information processing means" refers to computing resources and processing methods used to generate and provide appropriate content and feedback to users based on collected data.

[0802] "Emotional feedback" refers to adjustments and adaptations to optimized content and activities provided in response to the user's emotional state.

[0803] "Content" refers to the collective term for images, audio, text, interactive elements, etc., provided to users and used for educational or supportive purposes.

[0804] "User" refers to an individual who is the subject of this invention and who participates in educational activities or rehabilitation programs.

[0805] To realize this invention, a series of systems involving a server, terminals, and users are used. First, the server stores video data acquired from terminals equipped with cameras and microphones in an information storage device. In this process, it is necessary to record the entire rehabilitation or educational session using recording means. The recorded data is stored in the information storage device and later used for analysis and feedback generation.

[0806] The device incorporates an emotion analysis engine that analyzes the user's facial expressions and voice in real time. This data is then sent to a server to determine the user's emotional and cognitive state. At this stage, machine learning libraries such as TensorFlow and Keras are used for facial recognition, and the Google Cloud Speech-to-Text API is utilized for voice analysis. Furthermore, IBM Watson's emotion analysis is employed for sentiment analysis.

[0807] The server processes this data and delivers content through emotional feedback. Specifically, it improves the individual experience by optimizing and presenting content such as videos, images, and music selected according to the user's emotional state.

[0808] For example, if a user shows curiosity during a rehabilitation session, the server immediately displays content related to relevant history or interesting topics. This improves the user's attention and enhances the effectiveness of the session.

[0809] An example of a prompt message is, "Based on the user's sentiment analysis, list content that is likely to be of interest and present it in the appropriate order."

[0810] This system enables flexible rehabilitation and educational support tailored to the individual needs of each user.

[0811] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0812] Step 1:

[0813] The device uses a camera and microphone to collect video and audio of the user. The input is real-time video and audio data, which is recorded in high resolution. The output is raw data converted into a data format that can be processed by the emotion analysis engine. Specifically, it captures the user's face and voice as digital data, performs noise reduction, and processes the data as needed.

[0814] Step 2:

[0815] The device analyzes collected video and audio data using an emotion analysis engine. The input is the formatted data obtained in step 1, and the output is a quantitative indicator showing the user's emotional state. Specifically, it analyzes facial expressions using TensorFlow and Keras, and extracts emotions from audio using the Google Cloud Speech-to-Text API.

[0816] Step 3:

[0817] The server receives data from the sentiment analysis engine and selects appropriate content based on the user's emotional state. The input is an indicator of the emotional state obtained in step 2, and the output is a list of selected content objects. Specifically, it searches the database for relevant videos, music, and text and lists the most suitable ones.

[0818] Step 4:

[0819] The server streams the selected content to the terminal through an information processing system. The input is the content list generated in step 3, and the output is the content set on the user interface for the user to view. Specifically, it performs conversion to the content format and optimization according to bandwidth to provide the user with an uninterrupted viewing experience.

[0820] Step 5:

[0821] The device delivers the streamed content to the user and records the user's reaction again. The input is the content streamed from the server, and the output is new emotional response data. Specifically, it captures facial expressions and voice again using the camera and microphone and sends the data to the server.

[0822] Step 6:

[0823] The server re-analyzes the sentiment data and adjusts the content provided as needed. The input is the sentiment data obtained in step 5, and the output is the re-selection of content and the addition of detailed explanations as required. Specifically, a generative AI model is used to evaluate the effectiveness of the previous content and provide new content or supplementary information.

[0824] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0825] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0826] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0827] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0828] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0829] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0830] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0831] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0832] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0833] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0834] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0835] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0836] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0837] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0838] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0839] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0840] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0841] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0842] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0843] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0844] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0845] The following is further disclosed regarding the embodiments described above.

[0846] (Claim 1)

[0847] Filming equipment for recording educational activities on video,

[0848] A means for storing recorded video in an information storage device,

[0849] Means for providing educational resources to users via information processing means,

[0850] A means of receiving viewing requests from users and editing the video based on the requested time,

[0851] A means of responding to user questions,

[0852] A system that includes this.

[0853] (Claim 2)

[0854] The system according to claim 1, wherein the means for editing video extracts important information using a machine learning algorithm.

[0855] (Claim 3)

[0856] The system according to claim 1, wherein the means for generating an answer uses an artificial intelligence model based on natural language processing to generate an answer to a user's question.

[0857] "Example 1"

[0858] (Claim 1)

[0859] A visual information acquisition device and means for electronically recording educational activities,

[0860] A means for storing acquired visual information in a remote information storage means,

[0861] A means of providing educational materials to users via computer processing means,

[0862] A means for receiving playback requests from users and adjusting visual information based on the specified playback time,

[0863] A means of generating responses to inquiries from users,

[0864] A system that includes this.

[0865] (Claim 2)

[0866] The system according to claim 1, wherein the means for adjusting visual information selects important data using a computational learning algorithm.

[0867] (Claim 3)

[0868] The system according to claim 1, wherein the means for generating a response uses an intelligent model based on language analysis to generate an answer to a user inquiry.

[0869] "Application Example 1"

[0870] (Claim 1)

[0871] Filming equipment for recording educational activities on video,

[0872] A method for storing recorded video on an information storage device and managing it on a cloud platform,

[0873] Means for providing educational resources to users via information processing means,

[0874] A means of receiving viewing requests from users, editing the video based on the requested duration, and generating a summarized version,

[0875] A means of responding to user questions and generating answers using artificial intelligence technology,

[0876] A system that includes this.

[0877] (Claim 2)

[0878] The system according to claim 1, wherein the means for editing video extracts important information using a machine learning algorithm and generates summarized content.

[0879] (Claim 3)

[0880] The system according to claim 1, wherein the means for generating an answer uses an artificial intelligence model based on natural language processing to generate an answer to a user's question.

[0881] "Example 2 of combining an emotion engine"

[0882] (Claim 1)

[0883] Recording means for video recording educational activities,

[0884] A means for storing recorded video in an information storage device,

[0885] Means for providing educational resources to beneficiaries through information processing means,

[0886] A means of receiving viewing requests from beneficiaries and editing the video based on the requested time,

[0887] A means of responding to questions from beneficiaries,

[0888] A means of analyzing the emotions of beneficiaries and dynamically adjusting the learning experience based on that information,

[0889] A system that includes this.

[0890] (Claim 2)

[0891] The system according to claim 1, wherein the means for editing video is to extract important information using machine learning.

[0892] (Claim 3)

[0893] The system according to claim 1, wherein the means for generating the answer uses a generative model based on natural language processing to generate an answer to the beneficiary's question.

[0894] "Application example 2 when combining with an emotional engine"

[0895] (Claim 1)

[0896] Using video recording equipment and an emotion analysis engine, the user's cognitive state is analyzed, and the equipment and

[0897] Means for storing recorded video in an information storage device,

[0898] Means for providing users with educational resources and emotionally responsive content through information processing tools,

[0899] A means of receiving viewing requests from users and providing edited versions of the video based on the requested time,

[0900] A means for generating responses to user questions using natural language processing,

[0901] A system that includes this.

[0902] (Claim 2)

[0903] The system according to claim 1, wherein the video editing means automatically extracts content based on important information and emotional feedback using a machine learning algorithm.

[0904] (Claim 3)

[0905] The system according to claim 1, wherein the question response generation means generates the optimal answer to the user's question using a generative model based on natural language processing. [Explanation of Symbols]

[0906] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Filming equipment for recording educational activities on video, A method for storing recorded video on an information storage device and managing it on a cloud platform, Means for providing educational resources to users via information processing means, A means of receiving viewing requests from users, editing the video based on the requested duration, and generating a summarized version, A means of responding to user questions and generating answers using artificial intelligence technology, A system that includes this.

2. The system according to claim 1, wherein the means for editing video extracts important information using a machine learning algorithm and generates summarized content.

3. The system according to claim 1, wherein the means for generating an answer uses an artificial intelligence model based on natural language processing to generate an answer to a user's question.

Images