System

Abstract

A system is provided.SOLUTION: The system includes a means for receiving authentication information of a user, collating the authentication information with a database to perform authentication, a means for analyzing a past learning data of the user and generating a customized question set, a means for analyzing an answer of the user and generating individual feedback, a means for storing an answer result and a learning history of the user and evaluating progress, and a means for proposing a new learning plan and additional teaching materials.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional education systems face the challenge of being unable to provide learning support tailored to individual students' needs and levels of understanding. As a result, some students may not gain an adequate understanding, resulting in reduced learning effectiveness. It is also difficult for teachers and parents to grasp students' progress in real time, which can result in delays in providing appropriate instruction. A method is needed to resolve these issues and provide optimal education for each student. [Means for solving the problem]

[0005] The present invention provides an educational support system that includes a means for receiving a user's authentication information and verifying it against a database, a means for analyzing the user's past learning data and generating a customized problem set, a means for analyzing the user's answers and generating individualized feedback, a means for saving the user's answers and learning history and evaluating the user's progress, and a means for proposing new learning plans and additional learning materials. This system generates questions based on the user's past answer patterns, analyzes the user's level of understanding using AI technology, and evaluates the user's progress, making it possible to provide an appropriate learning plan for each student.

[0006] "Authentication information" is information required to verify a user's identity, and typically includes a username and password.

[0007] A "database" is a repository of information for storing user authentication information and past learning data.

[0008] "Verification" is the process of comparing received authentication information with information in a database as part of verifying a user's identity.

[0009] "Study data" refers to data that includes information such as the user's past study content, answer results, and progress.

[0010] A "problem set" is a collection of questions that are presented to help a user learn.

[0011] An "answer" is a response submitted by a user to a question.

[0012] "Feedback" refers to evaluations and advice provided based on the user's answers.

[0013] "Study history" refers to a record of the learning that the user has done up to now.

[0014] "Progress" is an index that indicates the current stage of learning that the user has progressed to.

[0015] A "study plan" refers to a plan for a user to effectively study, and specifically includes the content and order of study.

[0016] "Additional Learning Materials" are additional learning resources or materials provided to the user for further learning.

[0017] "AI technology" is a technology that uses artificial intelligence to analyze data, recognize patterns, make predictions, and more. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention provides an educational support system for providing an optimal educational experience to each student. The system includes means for receiving user authentication information and authenticating the user by comparing it with a database, means for analyzing the user's past learning data and generating a customized problem set, means for analyzing the user's answers and generating individualized feedback, means for storing the user's answers and learning history and evaluating the user's progress, and means for suggesting new learning plans and additional learning materials.

[0040] System processing flow

[0041] Authenticating Users

[0042] 1. Device:

[0043] The user opens the login screen, enters their username and password, and clicks the "Login" button.

[0044] 2. Server:

[0045] Authentication information received from the terminal is verified against the database for authentication. If authentication is successful, the user's learning data is read and the user ID is saved in the session. If authentication fails, an error message is generated and sent to the terminal.

[0046] 3. Terminal:

[0047] Receives authentication results from the server, and if authentication is successful, displays the dashboard. If authentication fails, displays an error message.

[0048] Selecting learning modules and generating problem sets

[0049] 4. User:

[0050] Select the module you want to study from your dashboard.

[0051] 5. Terminal:

[0052] The user's selection information is sent to the server.

[0053] 6. Server:

[0054] Based on the received selection information, the system analyzes the user's past learning data, generates a customized question set, and transmits it to the terminal.

[0055] 7. Terminal:

[0056] The problem set received from the server is displayed to the user.

[0057] Answers and feedback

[0058] 8. Users:

[0059] Enter the answer to the question that appears.

[0060] 9. Terminal:

[0061] The user's answers are sent to the server in real time.

[0062] 10. Server:

[0063] The system analyzes the received answers and determines whether they are correct or not. It also uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[0064] 11. Terminal:

[0065] Display feedback received from the server to the user.

[0066] Evaluating progress and suggesting new learning plans

[0067] 12. Server:

[0068] It stores all of the user's answers and learning history, evaluates their progress, and generates and sends new learning plans and additional learning materials to the device.

[0069] 13. Terminal:

[0070] Display new learning plans and additional learning materials received from the server to the user.

[0071] Specific examples

[0072] Example 1: Math Problems

[0073] 1. Device:

[0074] A user selects a mathematics module and requests an entry-level problem set.

[0075] 2. Server:

[0076] A customized problem set including basic arithmetic operations is generated from past learning data and sent to the device.

[0077] 3. Terminal:

[0078] A user answers "4" to the question "2 + 2 = ?" and sends the answer to the server.

[0079] 4. Server:

[0080] The answer is analyzed, confirmed to be correct, and feedback is sent to the device saying, "That's correct. Please proceed to the next question."

[0081] 5. Terminal:

[0082] Display feedback received from the server to the user.

[0083] In this way, the system of the present invention provides an optimal learning experience in real time based on the user's individual needs and level of understanding, improving students' learning efficiency and allowing teachers and parents to easily track progress.

[0084] The processing flow will be explained below.

[0085] Step 1:

[0086] Terminal: The user opens the login screen, enters their username and password, and clicks the "Login" button.

[0087] Step 2:

[0088] Server: Authenticates the user by checking the authentication information received from the terminal against the database. If authentication is successful, reads the user's learning data and stores the user ID in the session. If authentication fails, generates an error message and sends it to the terminal.

[0089] Step 3:

[0090] Terminal: Receives authentication result from the server, and displays the dashboard if authentication is successful. If authentication fails, displays an error message.

[0091] Step 4:

[0092] Users: Select the module they want to study from the dashboard.

[0093] Step 5:

[0094] Terminal: Sends the user's selection information to the server.

[0095] Step 6:

[0096] Server: Based on the received selection information, the server analyzes the user's past learning data, generates a customized problem set, and sends it to the device.

[0097] Step 7:

[0098] Terminal: Displays the problem set received from the server to the user.

[0099] Step 8:

[0100] User: Enter the answer to the question displayed.

[0101] Step 9:

[0102] Terminal: Sends the user's answers to the server in real time.

[0103] Step 10:

[0104] Server: Analyzes the received answers and determines whether they are correct. Furthermore, it uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[0105] Step 11:

[0106] Terminal: Displays the feedback received from the server to the user.

[0107] Step 12:

[0108] Server: Stores all user answers and learning history, evaluates progress, and generates new learning plans and additional learning materials and sends them to the device.

[0109] Step 13:

[0110] Terminal: Displays new learning plans and additional learning materials received from the server to the user.

[0111] Step 14:

[0112] User: Proceed to the next learning step based on the new learning plan or additional materials.

[0113] Example 1

[0114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] Conventional educational support systems face the challenge of providing an educational experience tailored to each user's individual learning needs and level of understanding. Specifically, they lack the functionality to effectively analyze a user's past learning data to generate optimal problem sets or provide individually customized feedback in real time. This makes it difficult to improve learning efficiency and accurately evaluate progress.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0117] In this invention, the server includes means for receiving the user's authentication information and verifying it against a database, means for analyzing the user's past learning data to generate a customized problem set, and means for analyzing the user's answers to generate individual feedback, thereby providing an optimal educational experience based on the user's individual needs and level of understanding, improving learning efficiency, and enabling accurate assessment of progress.

[0118] "User authentication information" refers to information such as a username and password that a user enters when accessing a system.

[0119] A "database" refers to a storage device for systematically storing and managing user authentication information, learning data, answer results, etc.

[0120] "Matching" refers to the process of comparing received information with existing data to see if there is a match.

[0121] A "customized problem set" refers to a collection of study questions that are individually optimized based on the user's past study data and answer patterns.

[0122] "Analyzing answers" refers to the process of evaluating the answers entered by the user and determining whether they are correct or incorrect.

[0123] "Individual feedback" refers to study advice and comments for individual users that are generated based on the user's answer results and past study data.

[0124] "Storage" refers to the act of recording data on a storage device so that it can be used at a later time.

[0125] "Progress assessment" refers to the process of analyzing a user's learning history and answer results to determine their current level of understanding and progress.

[0126] "New learning plan" refers to a plan that suggests future learning content and methods based on the user's progress.

[0127] "Additional Learning Materials" refers to additional learning materials or resources provided to assist the user in their learning.

[0128] "Terminal" refers to an electronic device used by a user to access and operate the system.

[0129] "Computing device" refers to an electronic device for analyzing data and performing calculations.

[0130] "AI technology" refers to technology that uses artificial intelligence to analyze, predict, optimize, and perform other functions on data.

[0131] "Learning history" refers to a record of the user's learning activities, answer results, and feedback.

[0132] The present invention provides an educational support system for providing an optimal educational experience to each student. The system includes means for receiving user authentication information and authenticating the user by comparing it with a database, means for analyzing the user's past learning data and generating a customized problem set, means for analyzing the user's answers and generating individualized feedback, means for storing the user's answers and learning history and evaluating the user's progress, and means for suggesting new learning plans and additional learning materials.

[0133] The basic configuration of the entire system is as follows:

[0134] Receive user authentication information: When a user accesses the system, they enter their username and password on the login screen, and that information is sent to the server, where it is checked against a MySQL database.

[0135] Authentication by checking against database: The server uses a MySQL database to check the entered authentication information against the information in the database. If authentication is successful, the user ID is saved in the session and a successful authentication response is sent to the terminal.

[0136] Analyzing the user's past learning data: Based on the received selection information, the server analyzes the user's learning history using Python and SciPy libraries to generate a customized problem set.

[0137] Generate customized problem sets: Based on the module selected by the user, the server generates an optimal problem set based on the user's past learning data. This problem set is sent to the terminal and displayed to the user.

[0138] Analyzing user answers: When a user enters an answer to a question, the answer is sent to the server in real time. The server analyzes the received answer and determines whether it is correct or incorrect.

[0139] Generate personalized feedback: Based on the received answer results, the server uses AI technology (e.g., a generative AI model) to generate personalized feedback and send it to the device.

[0140] Saving user answers and learning history: All user answers and learning history are saved in cloud storage such as AWS (registered trademark) S3.

[0141] Progress evaluation: The server analyzes the stored data using Apache Spark (registered trademark) and evaluates the user's progress. Based on the evaluation results, a new learning plan and additional learning materials are generated and sent to the device.

[0142] The following flow can be considered as a specific example of how this system can be used.

[0143] 1. Terminal: The user selects the mathematics module and requests a beginner level problem set (clicks the "Math" button, sets the level to "Beginner" and presses the "Start" button).

[0144] 2. Server: Reads past training data from a MySQL database, generates a customized problem set including basic arithmetic problems using Python and SciPy, and sends it to the device.

[0145] 3. Terminal: Displays the problem set received from the server to the user (for example, displays the problem "2 + 2 = ?").

[0146] 4. User: Answers "4" to the displayed question "2 + 2 = ?" and clicks the button to submit the answer.

[0147] 5. Terminal: The user's answer "4" is sent to the server in real time.

[0148] 6. Server: Analyzes the received answer and verifies that it is correct. Using the generative AI model, it generates feedback such as "That's correct. Please proceed to the next question." and sends it to the device.

[0149] 7. Terminal: Displays the feedback received from the server to the user.

[0150] Examples of prompts for a generative AI model include:

[0151] "You are an AI teaching assistant. Create a customized set of math problems for a beginner student who has previously struggled with basic arithmetic (addition and subtraction)."

[0152] In this way, the system provides users with an optimal educational experience based on their individual needs and level of understanding, thereby improving their learning efficiency.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Step 1:

[0155] Terminal: The user opens a login screen and enters their username and password. When they click the "Login" button, the information they entered is sent to the server. The input is the username and password, and the output is the transmission of authentication information to the server.

[0156] Step 2:

[0157] Server: Receives the sent authentication information. Checks the received data against the database. Specifically, it compares the entered username and password with the user information stored in the MySQL database. If authentication is successful, it saves the user ID in the session and generates a response indicating successful authentication. If authentication fails, it generates an error message. The input is authentication information and the output is the authentication result.

[0158] Step 3:

[0159] Terminal: Receives authentication results from the server. If authentication is successful, displays the user's dashboard. If authentication fails, displays an error message. The input is the authentication result from the server, and the output is displaying the dashboard or an error message.

[0160] Step 4:

[0161] User: Selects the module they wish to learn from the list of learning modules displayed on the dashboard. The user's selection operation sends the selection information from the terminal to the server. The input is the selection of the learning module, and the output is the transmission of the selection information to the server.

[0162] Step 5:

[0163] Server: Based on the received selection information, it reads past learning data from the database. It uses Python and the SciPy library to analyze the user's learning history. Based on this analysis, it generates a customized problem set. The generated problem set is sent to the device. The input is the selection information and past learning data, and the output is a customized problem set.

[0164] Step 6:

[0165] Terminal: Displays the problem set received from the server to the user. The input is the problem set from the server, and the output is the display of the problem set.

[0166] Step 7:

[0167] User: Enters the answer to the displayed question and clicks the button to send the answer. The answer entered by the user is sent from the terminal to the server. The input is the answer to the question, and the output is the transmission of the answer to the server.

[0168] Step 8:

[0169] Server: Analyzes the received answers. First, it determines whether the answer is correct or incorrect. Next, it uses AI technology to generate individual feedback based on the user's answer and past learning data. The generated feedback is sent to the device. The input is the user's answer, and the output is the correctness determination and feedback.

[0170] Step 9:

[0171] Terminal: Receives feedback sent from the server. Displays the received feedback to the user. The input is the feedback from the server, and the output is the display of the feedback.

[0172] Step 10:

[0173] Server: All of the user's answer results and learning history are stored in cloud storage. Specifically, a storage service such as AWS S3 is used. Progress is evaluated using Apache Spark based on the stored data. Based on the evaluation results, new learning plans and additional learning materials are generated and sent to the device. The input is all answer results and learning history, and the output is the stored data and the progress evaluation results.

[0174] Step 11:

[0175] Terminal: Receives new learning plans and additional learning materials sent from the server. Displays them to the user. The input is the new learning plans and additional learning materials from the server, and the output is their display.

[0176] (Application example 1)

[0177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] Conventional training support systems have had the challenge of providing real-time feedback and progress visualization specific to improving factory workers' skills. It was particularly difficult to generate appropriate problem sets and provide individualized feedback for tasks using robots in factories. Furthermore, there was a lack of methods for providing customized learning plans based on the worker's level of skill acquisition.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0180] In this invention, the server includes means for receiving a user's authentication information and verifying it against a database for authentication, means for analyzing the user's past learning data to generate a customized work task set, and means for analyzing the user's answers to generate individual feedback. This allows factory workers to receive real-time feedback while undergoing work skill training using robots. Furthermore, the server can evaluate the user's progress and provide new learning plans and additional learning materials, thereby enabling effective skill improvement.

[0181] "User authentication information" refers to information required when a user accesses a system, and is primarily composed of a user name and password.

[0182] A "database" is a system for systematically storing and managing various information such as authentication information and learning data.

[0183] A "customized work task set" is a collection of work tasks that are individually designed based on the user's past learning data and answer patterns.

[0184] "Individual feedback" is information such as an evaluation and areas for improvement for a specific user that is generated based on the user's answers and work performance.

[0185] "Progress" refers to a state that indicates the learning results and skill acquisition level that the user has achieved so far.

[0186] A "new learning plan" is a new learning or training method suggested based on the user's current skill level and learning progress.

[0187] "Additional Learning Materials" are new educational content recommended for users to further their learning.

[0188] "Factory workers" are employees who work in factories using robots.

[0189] "Skills practice" is practical training to improve specific work skills.

[0190] A "smartphone application" is a software program that runs on a smartphone and can be operated by a user through an interface.

[0191] "Real-time" means immediate response to user operations and inputs.

[0192] "Generative AI model technology" refers to artificial intelligence technology that generates optimal answers and feedback based on large amounts of data.

[0193] "Answer patterns" refer to the tendencies and methods of answering questions that a user has used in the past.

[0194] A "robot" is a mechanical device that can automatically perform programmed tasks.

[0195] This invention is an educational support system that allows factory workers to effectively practice their skills using robots. The system is composed of a server, a smartphone, and a factory robot as its main components. The functions of each component and their specific implementation methods are described below.

[0196] Authenticating Users

[0197] The server receives the user's authentication information and performs authentication by checking it against the database. If authentication is successful, the server reads the user's learning data and stores the user ID in the session. The smartphone application receives the authentication result, and if authentication is successful, displays the dashboard.

[0198] Generate a customized set of work tasks

[0199] When a user selects a specific work task (e.g., welding or assembly) from the dashboard, the smartphone sends that information to the server. Based on the received selection, the server analyzes the user's past learning data and answer patterns to generate a customized work task set. This task set is sent to the smartphone and used as a guide for the user to start working.

[0200] Generate answers and feedback

[0201] When a user performs a task and enters the answer into their smartphone, the smartphone sends the information to the server in real time. The server analyzes the received answer and determines whether it is correct or incorrect. Furthermore, using generative AI model technology, the server generates personalized feedback based on the user's answer and past learning data and sends it to the smartphone. The feedback includes important hints and specific advice to help the user improve their work.

[0202] Evaluating progress and suggesting new learning plans

[0203] The server stores the user's answers and learning history, evaluates their progress, and generates a new learning plan and additional learning materials to guide the user to the next step in their learning, which are then sent to the user's smartphone.

[0204] Hardware and software used

[0205] Hardware

[0206] Smartphone

[0207] server

[0208] Factory robots

[0209] software

[0210] Flask (a web application framework written in Python)

[0211] scikit-learn (library for evaluation calculations)

[0212] Specific examples

[0213] Below are some specific examples of how this educational support system can be used.

[0214] Log in

[0215] The user logs in by entering "user1" and "pass123" into the smartphone app.

[0216] The server verifies the authentication information and returns the username if successful.

[0217] Selecting and generating work tasks

[0218] The user selects the "welding task" on the smartphone app.

[0219] The server generates a customized set of "welding" tasks and sends them to the smartphone.

[0220] Examples: "weld_joint_a: OK", "weld_joint_b: OK", "weld_joint_c: Fail"

[0221] Task execution and feedback

[0222] The user performs the work task and inputs the results into the smartphone.

[0223] Input examples: "OK", "OK", "Fail"

[0224] The server analyzes the answers and generates feedback.

[0225] The user sees feedback on their smartphone: "Great job! Continue to the next skill step."

[0226] Prompt Sentence Examples

[0227] Using the examples below as a reference, please describe the application's processing in natural language so that it can be explained to a human.

[0228] The user logs in by entering "user1" and "pass123" on the smartphone app. If authentication is successful on the server, the user name is displayed.

[0229] Next, the user selects the "welding" task on the smartphone app, and the server generates a customized task set and sends it to the smartphone.

[0230] Example of generated task set: "weld_joint_a: OK", "weld_joint_b: OK", "weld_joint_c: Fail"

[0231] The user performs the task and enters the results into a smartphone app.

[0232] Input examples: "OK", "OK", "Fail"

[0233] The server analyzes the results and generates feedback, which is displayed on the smartphone: "Great job! Proceed to the next skill step."

[0234] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0235] Step 1: User authentication

[0236] Input: The user attempts to log in by entering their username and password into the smartphone app.

[0237] How it works: The smartphone app sends this authentication information to the server.

[0238] Data processing: The server compares the received authentication information with the data stored in the database.

[0239] Output: If authentication is successful, read the user information and save the user ID in the session. If authentication fails, generate an error message and send it to the terminal.

[0240] Step 2: View the dashboard

[0241] Input: The authentication result from the server (success or failure).

[0242] Behavior: The smartphone app displays a dashboard if authentication is successful, or an error message if authentication is unsuccessful.

[0243] Data processing: None.

[0244] Output: Display a dashboard or display an error message.

[0245] Step 3: Select a work task

[0246] Input: The user selects a specific work task (e.g., welding or assembly) from the dashboard.

[0247] Operation: The smartphone app sends the selected work task information to the server.

[0248] Data processing: None.

[0249] Output: Work task information sent to the server.

[0250] Step 4: Generate a customized set of work tasks

[0251] Input: The server receives the user's selected task information, past learning data, and answer patterns.

[0252] How it works: The server uses a generative AI model to analyze incoming data and generate a customized set of work tasks.

[0253] Data processing: Analysis of past learning data and answer patterns, and generation of task sets using a generative AI model.

[0254] Output: The generated customized task set is sent to your smartphone.

[0255] Step 5: Performing work tasks

[0256] Input: A customized set of work tasks sent from the server.

[0257] Action: The user performs the work task displayed on the smartphone app and enters the results into the app.

[0258] Data processing: None.

[0259] Output: User's answer result.

[0260] Step 6: Analyze answers and generate feedback

[0261] Input: User answer results sent from the smartphone app.

[0262] How it works: The server receives and analyzes the answers, uses a generative AI model to determine whether the answer is correct, and generates personalized feedback.

[0263] Data processing: Analysis of answer results and generation of feedback using a generative AI model.

[0264] Output: The generated feedback is sent to a smartphone app.

[0265] Step 7: View your feedback

[0266] Input: Feedback sent by the server.

[0267] What it does: The smartphone app displays the received feedback to the user.

[0268] Data processing: None.

[0269] Output: Display feedback.

[0270] Step 8: Evaluate progress and propose a new learning plan

[0271] Input: All of the user's answers and learning history.

[0272] How it works: The server analyzes this data, assesses progress, and uses generative AI models and algorithms to generate new learning plans and additional learning materials.

[0273] Data processing: Saving answer results and learning history, evaluating progress, and generating new learning plans and additional learning materials.

[0274] Output: Send new learning plans and additional learning materials to a smartphone app.

[0275] Step 9: View your learning plan

[0276] Input: New study plans and additional materials sent from the server.

[0277] How it works: The smartphone app displays the plan in place.

[0278] Data processing: None.

[0279] Output: Display of new learning plans and additional learning materials.

[0280] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0281] The present invention provides an educational support system for providing an optimal educational experience to each student. The system includes a means for receiving user authentication information and authenticating it against a database, a means for analyzing the user's past learning data and generating a customized problem set, a means for analyzing the user's answers and generating individualized feedback, a means for saving the user's answers and learning history and evaluating the user's progress, and a means for suggesting new learning plans and additional learning materials. The system further includes an emotion engine that recognizes the user's emotions and generates feedback based on the user's emotional state to improve learning efficiency.

[0282] System processing flow

[0283] Authenticating Users

[0284] 1. Device:

[0285] The user opens the login screen, enters their username and password, and clicks the "Login" button.

[0286] 2. Server:

[0287] Authentication information received from the terminal is verified against the database for authentication. If authentication is successful, the user's learning data is read and the user ID is saved in the session. If authentication fails, an error message is generated and sent to the terminal.

[0288] 3. Terminal:

[0289] Receives authentication results from the server, and if authentication is successful, displays the dashboard. If authentication fails, displays an error message.

[0290] Selecting learning modules and generating problem sets

[0291] 4. User:

[0292] Select the module you want to study from your dashboard.

[0293] 5. Terminal:

[0294] The user's selection information is sent to the server.

[0295] 6. Server:

[0296] Based on the received selection information, the system analyzes the user's past learning data, generates a customized question set, and transmits it to the terminal.

[0297] 7. Terminal:

[0298] The problem set received from the server is displayed to the user.

[0299] Answers and feedback

[0300] 8. Users:

[0301] Enter the answer to the question that appears.

[0302] 9. Terminal:

[0303] The user's answers are sent to the server in real time.

[0304] 10. Server:

[0305] The system analyzes the received answers and determines whether they are correct or not. It also uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[0306] 11. Terminal:

[0307] Display feedback received from the server to the user.

[0308] Evaluating progress and suggesting new learning plans

[0309] 12. Server:

[0310] It stores all of the user's answers and learning history, evaluates their progress, and generates and sends new learning plans and additional learning materials to the device.

[0311] 13. Terminal:

[0312] Display new learning plans and additional learning materials received from the server to the user.

[0313] Introducing the Emotion Engine

[0314] The emotion engine has the ability to recognize the user's emotional state by analyzing the user's facial expressions, tone of voice, and other biometric information. Using this information, the following process is carried out to improve learning efficiency.

[0315] Emotion recognition and feedback with emotion engine

[0316] 1. Device:

[0317] Emotional information such as the user's facial expressions and tone of voice is collected through a camera and microphone.

[0318] 2. Server:

[0319] The emotion engine analyzes this emotion information and identifies the user's emotional state (e.g., excitement, fatigue, confusion, etc.).

[0320] 3. Server:

[0321] It generates feedback based on the emotional state obtained from the emotion engine. For example, if the user is tired, it generates feedback such as "Take a short break."

[0322] 4. Terminal:

[0323] Feedback based on the emotional state received from the server is displayed to the user.

[0324] Responding to emotional states

[0325] The emotional engine can adjust the difficulty and format of questions based on the user's emotional state. For example, if the user is confused, it can simplify the questions or provide hints. The emotional state obtained from the emotional engine is also added to the training data and reflected in the assessment of progress and comprehension.

[0326] Specific examples

[0327] Example 1: Math problems and emotional feedback

[0328] 1. Device:

[0329] A user selects a mathematics module and requests an entry-level problem set.

[0330] 2. Server:

[0331] A customized problem set including basic arithmetic operations is generated from past learning data and sent to the device.

[0332] 3. Terminal:

[0333] A user answers "4" to the question "2 + 2 = ?" and sends the answer to the server.

[0334] 4. Server:

[0335] The answer is analyzed, confirmed to be correct, and feedback is generated saying, "That's correct. Please proceed to the next question."

[0336] 5. Terminal:

[0337] Display feedback received from the server to the user.

[0338] 6. Terminal (Emotion Engine):

[0339] The camera analyzes the user's facial expressions and detects when the user is concentrating.

[0340] 7. Server (Emotion Engine):

[0341] It checks the user's concentration and generates feedback to encourage them to continue learning.

[0342] 8. Terminal:

[0343] Display feedback received from the server to the user.

[0344] In this way, the system of the present invention provides an optimal learning experience in real time based on the user's individual needs and level of understanding. Furthermore, the introduction of an emotion engine enables customization based on the user's emotional state, further improving learning efficiency. This results in improved learning outcomes for students, and allows teachers and parents to easily track progress.

[0345] The processing flow will be explained below.

[0346] System processing flow

[0347] Authenticating Users

[0348] Step 1:

[0349] Terminal: The user opens the login screen, enters their username and password, and clicks the "Login" button.

[0350] Step 2:

[0351] Server: Authenticates the user by checking the authentication information received from the terminal against the database. If authentication is successful, reads the user's learning data and stores the user ID in the session. If authentication fails, generates an error message and sends it to the terminal.

[0352] Step 3:

[0353] Terminal: Receives authentication result from the server, and displays the dashboard if authentication is successful. If authentication fails, displays an error message.

[0354] Selecting learning modules and generating problem sets

[0355] Step 4:

[0356] Users: Select the module they want to study from the dashboard.

[0357] Step 5:

[0358] Terminal: Sends the user's selection information to the server.

[0359] Step 6:

[0360] Server: Based on the received selection information, the server analyzes the user's past learning data, generates a customized problem set, and sends it to the device.

[0361] Step 7:

[0362] Terminal: Displays the problem set received from the server to the user.

[0363] Answers and feedback

[0364] Step 8:

[0365] User: Enter the answer to the question displayed.

[0366] Step 9:

[0367] Terminal: Sends the user's answers to the server in real time.

[0368] Step 10:

[0369] Server: Analyzes the received answers and determines whether they are correct. Furthermore, it uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[0370] Step 11:

[0371] Terminal: Displays the feedback received from the server to the user.

[0372] Evaluating progress and suggesting new learning plans

[0373] Step 12:

[0374] Server: Stores all user answers and learning history, evaluates progress, and generates new learning plans and additional learning materials and sends them to the device.

[0375] Step 13:

[0376] Terminal: Displays new learning plans and additional learning materials received from the server to the user.

[0377] Introducing the Emotion Engine

[0378] Emotion recognition and feedback with emotion engine

[0379] Step 14:

[0380] Device: Collects emotional information such as the user's facial expressions and tone of voice through a camera and microphone.

[0381] Step 15:

[0382] Server: The emotion engine analyzes this emotion information and identifies the user's emotional state (e.g., excitement, fatigue, confusion, etc.).

[0383] Step 16:

[0384] Server: Generates feedback based on the emotional state obtained from the emotion engine. For example, if the user is tired, it generates feedback such as "Take a short break."

[0385] Step 17:

[0386] Terminal: Displays feedback to the user based on the emotional state received from the server.

[0387] Responding to emotional states

[0388] Step 18:

[0389] Server: Using the emotion engine, adjust the difficulty and format of questions according to the user's emotional state. For example, if the user is confused, simplify the question or provide hints.

[0390] Step 19:

[0391] Server: The emotional state obtained from the emotion engine is added to the training data and reflected in the assessment of progress and understanding.

[0392] Specific examples

[0393] Example 1: Math problems and emotional feedback

[0394] Step 1:

[0395] Terminal: The user selects a mathematics module and requests an entry-level problem set.

[0396] Step 2:

[0397] Server: Generates a customized problem set, including basic arithmetic operations, from past learning data and sends it to the device.

[0398] Step 3:

[0399] Terminal: The user answers "4" to the question "2 + 2 = ?" and sends the answer to the server.

[0400] Step 4:

[0401] Server: Analyzes the answer, verifies it is correct, and generates feedback such as "That's correct. Please proceed to the next question."

[0402] Step 5:

[0403] Terminal: Displays the feedback received from the server to the user.

[0404] Step 6:

[0405] Terminal (emotion engine): The camera analyzes the user's facial expressions and detects when the user is concentrating.

[0406] Step 7:

[0407] Server (emotion engine): Checks the user's concentration and generates feedback to encourage them to continue learning.

[0408] Step 8:

[0409] Terminal: Displays the feedback received from the server to the user.

[0410] In this way, the system of the present invention provides an optimal learning experience in real time based on the user's individual needs and level of understanding. Furthermore, the introduction of an emotion engine enables customization based on the user's emotional state, further improving learning efficiency. This results in improved learning outcomes for students, and allows teachers and parents to easily track progress.

[0411] Example 2

[0412] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0413] Conventional educational support systems have struggled to provide an optimized educational experience for each individual user. In particular, it has been difficult to grasp a user's learning situation in real time and provide feedback based on that understanding, and customization to accommodate emotional states has been insufficient. As a result, learners are unable to properly understand their own learning situation, resulting in a decline in learning effectiveness.

[0414] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving user authentication information and verifying it against a database for authentication, means for analyzing the user's past learning data and generating a customized problem set, means for analyzing the user's answers and generating individual feedback, means for saving the user's answers and learning history and evaluating the user's progress, means for suggesting new learning plans and additional learning materials, means for collecting and analyzing the user's emotional information and generating feedback based on the user's emotional state, means for transmitting the user's answers in real time, and means for displaying the generated feedback to the user. This makes it possible to provide an optimal educational experience in real time based on the user's individual learning situation and emotional state.

[0415] "Means for receiving user authentication information and verifying it against a database" refers to the process of receiving user authentication information (such as username and password) sent from the terminal and comparing that information with the registered information in a database to verify the user's legitimacy.

[0416] "Means for analyzing a user's past learning data and generating a customized problem set" refers to the process of analyzing a user's learning history and achievements and generating individually optimized learning problems based on that.

[0417] The "means for analyzing the user's answers and generating personalized feedback" is a process for analyzing the answers submitted by the user and generating feedback that provides specific guidance based on the results.

[0418] "Means for saving the user's answers and learning history and evaluating the progress" refers to the process of recording the correctness of the user's answers and learning history, and evaluating the user's learning progress and level of understanding based on this.

[0419] The "means for suggesting new study plans and additional learning materials" is a process for providing the user with the next study content and supplementary learning materials based on the user's evaluation results.

[0420] "Means for generating feedback based on a user's emotional state by collecting and analyzing the user's emotional information" refers to a process of using a camera or microphone to collect the user's facial expressions and tone of voice, analyzing the user's emotional state based on that information, and providing appropriate feedback.

[0421] "Means for transmitting user answer results in real time" refers to a process in which the results are immediately transmitted to the server each time a user inputs an answer.

[0422] The "means for displaying the generated feedback to the user" is a process for displaying the feedback sent from the server on the screen of the terminal and providing it to the user.

[0423] The present invention is an educational support system for providing an optimal educational experience to each student. The system includes means for receiving user authentication information, authenticating the user by checking it against a database, analyzing the user's past learning data to generate a customized problem set, and analyzing the user's answers to generate personalized feedback. The system also stores the user's answers and learning history, evaluates the user's progress, and suggests new learning plans and additional learning materials. The system also incorporates an emotion engine for collecting and analyzing the user's emotional information and generating feedback based on the user's emotional state.

[0424] The entire system functions through the interaction of three parties: the server, the terminal, and the user. The specific configuration and operation will be explained below.

[0425] First, the user opens a login screen using the terminal and enters their username and password. This login information is sent to the server using JavaScript (registered trademark). The server uses the Python Flask framework to check the authentication information against a database (e.g., MySQL). If authentication is successful, the server loads the user's learning data and saves the user ID in the session. If authentication fails, a response including an error message is sent to the terminal. Based on the response from the server, the terminal displays a dashboard if the authentication was successful, or an error message if the authentication failed.

[0426] Next, the user selects the module they wish to learn from the dashboard. This selection information is sent to the server via JavaScript. Based on the received selection information, the server analyzes the user's past learning data using machine learning libraries such as scikit-learn. As a result, a problem set optimized for the user is generated and sent to the device. The device then dynamically generates the received problem set using HTML and JavaScript and displays it to the user.

[0427] When a user enters an answer to a question and clicks the submit button, the device sends the answer information to the server in real time. The server uses Python to analyze the answer and determine whether it is correct. It also uses AI technologies such as TENSORFLOW (registered trademark) and PyTorch to generate individual feedback based on the analysis results and past learning data and send it to the device. The device then displays the received feedback to the user.

[0428] The server stores all of the user's answers and learning history in a database. The server evaluates the user's progress based on this information. Based on the progress and evaluation results, it generates new learning plans and additional learning materials and sends them in JSON format to the device. The device then displays the received plans and learning materials to the user.

[0429] Furthermore, a means for recognizing the user's emotional state is incorporated using an emotion engine. The device collects the user's facial expressions and tone of voice through a camera and microphone, and analyzes them using an emotion engine (e.g., OpenCV or the Facial Emotion Recognition (FER) library). Based on the analysis results, the server generates feedback according to the user's emotional state (e.g., "Let's take a short break") and sends it to the device. The device then displays this feedback to the user.

[0430] Examples of concrete examples and prompts

[0431] As a concrete example, let's consider an operation in which a user selects a mathematics module and requests a set of beginner-level arithmetic problems. The server generates problems such as "2 + 2 = ?" based on past learning data and sends them to the device. If the user answers "4," the server determines that this is the correct answer and generates feedback saying, "That's correct. Please proceed to the next problem." The device displays this feedback, and also recognizes the user's state of concentration via the emotion engine, generating and displaying feedback encouraging the user to continue learning.

[0432] Examples of prompts might include "Design a system that returns analysis results in real time after the user enters their answer" or "Explain the process of generating a customized problem set and displaying it to the user."

[0433] In this way, the educational support system according to the present invention can provide an optimal educational experience in real time based on the user's learning situation and emotional state.

[0434] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0435] Step 1: Authenticate the user

[0436] 1.1 Terminal:

[0437] You open a login screen, enter your username and password, and click the "Login" button. The computer program sends this information to the server using JavaScript.

[0438] Input: Username, Password

[0439] Output: Authentication information sent to the server

[0440] 1.2 Server:

[0441] The authentication information received from the device is checked against a database (e.g., MySQL). The Python Flask framework is used to compare the authentication information with the database, and if a match is found, authentication is successful. The user ID is stored in the session and training data is loaded. If authentication fails, an error message is generated.

[0442] Input: Authentication information (user name, password)

[0443] Output: Authentication result (success / failure), user data (if authentication is successful)

[0444] 1.3 Terminal:

[0445] Receives the authentication result from the server and displays the dashboard if successful, or an error message if unsuccessful.

[0446] Input: Authentication result (success / failure)

[0447] Output: Dashboard (if successful), Error message (if unsuccessful)

[0448] Step 2: Selecting a learning module and generating a problem set

[0449] 2.1 User:

[0450] From the dashboard, select the module you want to study. Your selection is collected using JavaScript.

[0451] Input: Module selection information

[0452] Output: Selections sent to the server

[0453] 2.2 Terminal:

[0454] Send the selection to the server. The selection is sent asynchronously using an AJAX request.

[0455] Input: Selection information

[0456] Output: Selections sent to the server

[0457] 2.3 Server:

[0458] Receives the selection information and analyzes the user's past training data. Analyzes the training data using Python and scikit-learn to generate a customized problem set. The generated problem set is sent to the device in JSON format.

[0459] Input: Selection information, past learning data

[0460] Output: Problem set in JSON format

[0461] 2.4 Terminal:

[0462] The problem set received from the server is displayed to the user. Problems are dynamically generated using JavaScript and HTML and displayed on the screen.

[0463] Input: Problem set (JSON format)

[0464] Output: The displayed problem set

[0465] Step 3: Answers and feedback

[0466] 3.1 User:

[0467] Enter your answer to the question and click the "Submit" button. The answer will be entered into the form and collected by JavaScript.

[0468] Input: Answer

[0469] Output: Answer sent to the server

[0470] 3.2 Terminal:

[0471] Send the user's answers to the server in real time using an AJAX request.

[0472] Input: Answer

[0473] Output: Answer sent to the server

[0474] 3.3 Server:

[0475] The received answers are analyzed and judged to be correct (using Python), and then individual feedback is generated using AI technologies such as TensorFlow and PyTorch and sent to the device.

[0476] Input: Answers, past learning data

[0477] Output: Individual feedback

[0478] 3.4 Terminal:

[0479] Display the feedback received from the server to the user. Use JavaScript to display the feedback message on the screen.

[0480] Input: Feedback

[0481] Output: Displayed feedback

[0482] Step 4: Evaluate progress and propose a new learning plan

[0483] 4.1 Server:

[0484] All user answers and learning history are stored in a database, and progress is evaluated. Based on the evaluation results, new learning plans and additional learning materials are generated and sent to the device in JSON format.

[0485] Input: Answer results, learning history

[0486] Output: New learning plan, additional materials (JSON format)

[0487] 4.2 Terminal:

[0488] Display new learning plans and additional learning materials received from the server to the user. Plans and learning materials are dynamically generated using JavaScript and HTML and displayed on the screen.

[0489] Input: New learning plan, additional materials (JSON format)

[0490] Output: New learning plan displayed, additional materials

[0491] Step 5: Emotion recognition and feedback by the emotion engine

[0492] 5.1 Terminal:

[0493] Emotional information such as facial expressions and tone of voice of the user is collected through a camera and microphone. OpenCV and Python libraries are used for the emotion engine.

[0494] Input: facial expression data, voice data

[0495] Output: Emotion information sent to the server

[0496] 5.2 Server:

[0497] The emotion engine analyzes the emotion information and identifies the user's emotional state. Based on the analysis results, it generates appropriate feedback and sends it to the device.

[0498] Input: Emotion information

[0499] Output: Feedback (based on emotional state)

[0500] 5.3 Terminal:

[0501] Display feedback to the user based on the emotional state received from the server. Use JavaScript to display the feedback message on the screen.

[0502] Input: Feedback (based on emotional state)

[0503] Output: Displayed feedback

[0504] (Application example 2)

[0505] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0506] Conventional education and training support systems have difficulty adapting flexibly to the learning progress and emotional state of individual users and workers, and have often provided uniform learning plans and training procedures. As a result, optimal feedback and training support tailored to each individual's level of understanding and emotional state has not been provided, leading to a decline in learning efficiency and work efficiency.

[0507] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving a user's authentication information and verifying it against a database for authentication; means for analyzing the user's past learning data and generating a customized problem set; means for analyzing the user's answers and generating individual feedback; means for saving the user's answers and learning history and evaluating progress; means for suggesting new learning plans and additional learning materials; means for analyzing the worker's past training data and presenting customized work procedures; means including an emotion engine for recognizing emotional states and generating feedback; and means for evaluating progress and suggesting the next training or work procedures. This makes it possible to provide each user or worker with an optimal learning and training experience tailored to their learning situation and emotional state.

[0508] "User authentication information" refers to information used to identify a person accessing a system and verify their identity.

[0509] A "database" is a collection of information used to store and manage user authentication information, learning data, and training data.

[0510] A "customized problem set" is a collection of problems optimized for each individual user, generated by analyzing the user's past learning data and answer patterns.

[0511] "Individual feedback" is advice and evaluation specific to a user that is generated based on the user's answers and progress.

[0512] "Progress" is information that indicates the progress and results of a user's learning or training.

[0513] "New study plans and additional learning materials" are new study plans and supplementary learning materials that are suggested based on the user's progress and level of understanding.

[0514] A "work procedure" is a series of steps or methods for accomplishing a particular task.

[0515] An "emotion engine" is a system that analyzes emotional information such as a user's facial expression and tone of voice to recognize the user's emotional state.

[0516] "Feedback" refers to evaluations and advice provided in response to a user's actions or input.

[0517] "Training data" refers to data that includes records of training that workers have previously undergone and the learning outcomes based on that training.

[0518] The present invention is a real-time training support system for factory workers, which uses smart glasses to provide real-time assistance to workers while they are working. Detailed description of the embodiments of the present invention will be given below.

[0519] First, the system includes the following hardware and software components:

[0520] Hardware: Smart glasses (camera, display)

[0521] software:

[0522] OpenCV: Camera image capture and display

[0523] Dlib: A face detection and recognition model

[0524] EmotionRecognizer: A model for recognizing emotional states

[0525] Data Management Module: Acquire and suggest customized training data

[0526] Server communication module: Sending and receiving feedback

[0527] System processing flow

[0528] 1. User authentication:

[0529] A worker puts on the smart glasses and logs in to the system. The user's authentication information (username and password) is sent from the smart glasses device to the server and checked against the database. If authentication is successful, the worker's training data is loaded and the session begins.

[0530] 2. Providing training procedures:

[0531] Based on the worker's past training data, customized work instructions are generated and displayed on the smart glasses' display, and the instructions are adjusted based on each worker's skill level and past performance.

[0532] 3. Recognition of emotional states:

[0533] A camera mounted on the smart glasses captures the worker's facial expressions, which are then analyzed by EmotionRecognizer, which identifies the worker's emotional state, such as whether they are tired, focused, or confused.

[0534] 4. Generate personalized feedback:

[0535] Based on the recognized emotional state, the server generates appropriate feedback in real time, such as "Take a break" if the worker is tired, or "Revisit the procedure" if the worker is confused.

[0536] 5. Progress evaluation and training plan suggestions:

[0537] The server stores all the worker's training data and evaluates their progress, generating new training plans and additional work instructions to prepare them for the next job.

[0538] Specific examples

[0539] For example, when a worker is tightening a nut, smart glasses will display the work instructions and analyze the worker's facial expressions using a facial recognition camera. If the smart glasses detect that the worker is tired, they will provide feedback such as "Take a short break." If the glasses detect that the worker is confused by the work instructions, they will again provide specific instructions such as "Turn the screw clockwise."

[0540] Example prompts for generative AI models

[0541] "Generate optimal feedback when workers feel fatigued during work."

[0542] "Propose procedural support for workers if they are confused."

[0543] In this way, the system of the present invention can improve work efficiency and safety by providing optimal support in real time according to the condition of each individual worker.

[0544] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0545] Step 1:

[0546] The user puts on the smart glasses and enters their username and password on the login screen. The input information is sent from the device to the server. The server compares the received authentication information with a database and performs authentication. If authentication is successful, the worker's training data is read and the session begins. If authentication fails, an error message is sent to the device and displayed.

[0547] Step 2:

[0548] After the worker is authenticated, he or she selects a training module. The terminal sends the selection information to the server. The server analyzes the worker's past training data based on the received selection information. Based on the analysis results, the server generates a customized work procedure and sends it to the terminal. The terminal displays the generated work procedure on the display of the smart glasses.

[0549] Step 3:

[0550] A camera mounted on the smart glasses captures the worker's facial expressions. The captured video is sent from the device to a server. The server then uses an emotion engine (EmotionRecognizer) to analyze the video data and recognize the worker's emotional state (fatigue, concentration, confusion, etc.). The recognized emotional state is then used to evaluate progress.

[0551] Step 4:

[0552] The server generates appropriate feedback in real time based on the recognized emotional state. For example, if fatigue is recognized, feedback such as "Take a short break" is generated. The generated feedback is sent to the device and displayed on the smart glasses display.

[0553] Step 5:

[0554] The server stores all of the worker's training data and evaluates their progress. The evaluation results are used to generate the next training or additional work procedures. The server generates new training plans or additional work procedures and sends them to the device. The device displays the new training plans or additional work procedures on the smart glasses display.

[0555] Step 6:

[0556] While the worker is performing additional work steps, the camera in the smart glasses again captures their facial expressions and sends them to the server, which then uses the emotion engine to analyze their emotional state again and updates the feedback as needed, ensuring optimal support is always provided based on the worker's condition.

[0557] This allows users to receive optimal feedback and work procedures in real time based on their individual learning situation and emotional state.

[0558] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0559] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0560] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0561] [Second embodiment]

[0562] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0563] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0564] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0565] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0566] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0567] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0568] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0569] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0570] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0571] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0572] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0573] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0574] The present invention provides an educational support system for providing an optimal educational experience to each student. The system includes means for receiving user authentication information and authenticating the user by comparing it with a database, means for analyzing the user's past learning data and generating a customized problem set, means for analyzing the user's answers and generating individualized feedback, means for storing the user's answers and learning history and evaluating the user's progress, and means for suggesting new learning plans and additional learning materials.

[0575] System processing flow

[0576] Authenticating Users

[0577] 1. Device:

[0578] The user opens the login screen, enters their username and password, and clicks the "Login" button.

[0579] 2. Server:

[0580] Authentication information received from the terminal is verified against the database for authentication. If authentication is successful, the user's learning data is read and the user ID is saved in the session. If authentication fails, an error message is generated and sent to the terminal.

[0581] 3. Terminal:

[0582] Receives authentication results from the server, and if authentication is successful, displays the dashboard. If authentication fails, displays an error message.

[0583] Selecting learning modules and generating problem sets

[0584] 4. User:

[0585] Select the module you want to study from your dashboard.

[0586] 5. Terminal:

[0587] The user's selection information is sent to the server.

[0588] 6. Server:

[0589] Based on the received selection information, the system analyzes the user's past learning data, generates a customized question set, and transmits it to the terminal.

[0590] 7. Terminal:

[0591] The problem set received from the server is displayed to the user.

[0592] Answers and feedback

[0593] 8. Users:

[0594] Enter the answer to the question that appears.

[0595] 9. Terminal:

[0596] The user's answers are sent to the server in real time.

[0597] 10. Server:

[0598] The system analyzes the received answers and determines whether they are correct or not. It also uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[0599] 11. Terminal:

[0600] Display feedback received from the server to the user.

[0601] Evaluating progress and suggesting new learning plans

[0602] 12. Server:

[0603] It stores all of the user's answers and learning history, evaluates their progress, and generates and sends new learning plans and additional learning materials to the device.

[0604] 13. Terminal:

[0605] Display new learning plans and additional learning materials received from the server to the user.

[0606] Specific examples

[0607] Example 1: Math Problems

[0608] 1. Device:

[0609] A user selects a mathematics module and requests an entry-level problem set.

[0610] 2. Server:

[0611] A customized problem set including basic arithmetic operations is generated from past learning data and sent to the device.

[0612] 3. Terminal:

[0613] A user answers "4" to the question "2 + 2 = ?" and sends the answer to the server.

[0614] 4. Server:

[0615] The answer is analyzed, confirmed to be correct, and feedback is sent to the device saying, "That's correct. Please proceed to the next question."

[0616] 5. Terminal:

[0617] Display feedback received from the server to the user.

[0618] In this way, the system of the present invention provides an optimal learning experience in real time based on the user's individual needs and level of understanding, improving students' learning efficiency and allowing teachers and parents to easily track progress.

[0619] The processing flow will be explained below.

[0620] Step 1:

[0621] Terminal: The user opens the login screen, enters their username and password, and clicks the "Login" button.

[0622] Step 2:

[0623] Server: Authenticates the user by checking the authentication information received from the terminal against the database. If authentication is successful, reads the user's learning data and stores the user ID in the session. If authentication fails, generates an error message and sends it to the terminal.

[0624] Step 3:

[0625] Terminal: Receives authentication result from the server, and displays the dashboard if authentication is successful. If authentication fails, displays an error message.

[0626] Step 4:

[0627] Users: Select the module they want to study from the dashboard.

[0628] Step 5:

[0629] Terminal: Sends the user's selection information to the server.

[0630] Step 6:

[0631] Server: Based on the received selection information, the server analyzes the user's past learning data, generates a customized problem set, and sends it to the device.

[0632] Step 7:

[0633] Terminal: Displays the problem set received from the server to the user.

[0634] Step 8:

[0635] User: Enter the answer to the question displayed.

[0636] Step 9:

[0637] Terminal: Sends the user's answers to the server in real time.

[0638] Step 10:

[0639] Server: Analyzes the received answers and determines whether they are correct. Furthermore, it uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[0640] Step 11:

[0641] Terminal: Displays the feedback received from the server to the user.

[0642] Step 12:

[0643] Server: Stores all user answers and learning history, evaluates progress, and generates new learning plans and additional learning materials and sends them to the device.

[0644] Step 13:

[0645] Terminal: Displays new learning plans and additional learning materials received from the server to the user.

[0646] Step 14:

[0647] User: Proceed to the next learning step based on the new learning plan or additional materials.

[0648] Example 1

[0649] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0650] Conventional educational support systems face the challenge of providing an educational experience tailored to each user's individual learning needs and level of understanding. Specifically, they lack the functionality to effectively analyze a user's past learning data to generate optimal problem sets or provide individually customized feedback in real time. This makes it difficult to improve learning efficiency and accurately evaluate progress.

[0651] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0652] In this invention, the server includes means for receiving the user's authentication information and verifying it against a database, means for analyzing the user's past learning data to generate a customized problem set, and means for analyzing the user's answers to generate individual feedback, thereby providing an optimal educational experience based on the user's individual needs and level of understanding, improving learning efficiency, and enabling accurate assessment of progress.

[0653] "User authentication information" refers to information such as a username and password that a user enters when accessing a system.

[0654] A "database" refers to a storage device for systematically storing and managing user authentication information, learning data, answer results, etc.

[0655] "Matching" refers to the process of comparing received information with existing data to see if there is a match.

[0656] A "customized problem set" refers to a collection of study questions that are individually optimized based on the user's past study data and answer patterns.

[0657] "Analyzing answers" refers to the process of evaluating the answers entered by the user and determining whether they are correct or incorrect.

[0658] "Individual feedback" refers to study advice and comments for individual users that are generated based on the user's answer results and past study data.

[0659] "Storage" refers to the act of recording data on a storage device so that it can be used at a later time.

[0660] "Progress assessment" refers to the process of analyzing a user's learning history and answer results to determine their current level of understanding and progress.

[0661] "New learning plan" refers to a plan that suggests future learning content and methods based on the user's progress.

[0662] "Additional Learning Materials" refers to additional learning materials or resources provided to assist the user in their learning.

[0663] "Terminal" refers to an electronic device used by a user to access and operate the system.

[0664] "Computing device" refers to an electronic device for analyzing data and performing calculations.

[0665] "AI technology" refers to technology that uses artificial intelligence to analyze, predict, optimize, and perform other functions on data.

[0666] "Learning history" refers to a record of the user's learning activities, answer results, and feedback.

[0667] The present invention provides an educational support system for providing an optimal educational experience to each student. The system includes means for receiving user authentication information and authenticating the user by comparing it with a database, means for analyzing the user's past learning data and generating a customized problem set, means for analyzing the user's answers and generating individualized feedback, means for storing the user's answers and learning history and evaluating the user's progress, and means for suggesting new learning plans and additional learning materials.

[0668] The basic configuration of the entire system is as follows:

[0669] Receive user credentials: When a user accesses the system, they enter their username and password on a login screen, which is sent to the server and checked against a MySQL database.

[0670] Authentication by checking against database: The server uses a MySQL database to check the entered authentication information against the information in the database. If authentication is successful, the user ID is saved in the session and a successful authentication response is sent to the terminal.

[0671] Analyzing the user's past learning data: Based on the received selection information, the server analyzes the user's learning history using Python and SciPy libraries to generate a customized problem set.

[0672] Generate customized problem sets: Based on the module selected by the user, the server generates an optimal problem set based on the user's past learning data. This problem set is sent to the terminal and displayed to the user.

[0673] Analyzing user answers: When a user enters an answer to a question, the answer is sent to the server in real time. The server analyzes the received answer and determines whether it is correct or incorrect.

[0674] Generate personalized feedback: Based on the received answer results, the server uses AI technology (e.g., a generative AI model) to generate personalized feedback and send it to the device.

[0675] Save user's answer results and learning history: All user's answer results and learning history are saved in cloud storage such as AWS S3.

[0676] Progress evaluation: The server analyzes the stored data using Apache Spark and evaluates the user's progress. Based on the evaluation results, a new learning plan and additional learning materials are generated and sent to the device.

[0677] The following flow can be considered as a specific example of how this system can be used.

[0678] 1. Terminal: The user selects the mathematics module and requests a beginner level problem set (clicks the "Math" button, sets the level to "Beginner" and presses the "Start" button).

[0679] 2. Server: Reads past training data from a MySQL database, generates a customized problem set including basic arithmetic problems using Python and SciPy, and sends it to the device.

[0680] 3. Terminal: Displays the problem set received from the server to the user (for example, displays the problem "2 + 2 = ?").

[0681] 4. User: Answers "4" to the displayed question "2 + 2 = ?" and clicks the button to submit the answer.

[0682] 5. Terminal: The user's answer "4" is sent to the server in real time.

[0683] 6. Server: Analyzes the received answer and verifies that it is correct. Using the generative AI model, it generates feedback such as "That's correct. Please proceed to the next question." and sends it to the device.

[0684] 7. Terminal: Displays the feedback received from the server to the user.

[0685] Examples of prompts for a generative AI model include:

[0686] "You are an AI teaching assistant. Create a customized set of math problems for a beginner student who has previously struggled with basic arithmetic (addition and subtraction)."

[0687] In this way, the system provides users with an optimal educational experience based on their individual needs and level of understanding, thereby improving their learning efficiency.

[0688] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0689] Step 1:

[0690] Terminal: The user opens a login screen and enters their username and password. When they click the "Login" button, the information they entered is sent to the server. The input is the username and password, and the output is the transmission of authentication information to the server.

[0691] Step 2:

[0692] Server: Receives the sent authentication information. Checks the received data against the database. Specifically, it compares the entered username and password with the user information stored in the MySQL database. If authentication is successful, it saves the user ID in the session and generates a response indicating successful authentication. If authentication fails, it generates an error message. The input is authentication information and the output is the authentication result.

[0693] Step 3:

[0694] Terminal: Receives authentication results from the server. If authentication is successful, displays the user's dashboard. If authentication fails, displays an error message. The input is the authentication result from the server, and the output is displaying the dashboard or an error message.

[0695] Step 4:

[0696] User: Selects the module they wish to learn from the list of learning modules displayed on the dashboard. The user's selection operation sends the selection information from the terminal to the server. The input is the selection of the learning module, and the output is the transmission of the selection information to the server.

[0697] Step 5:

[0698] Server: Based on the received selection information, it reads past learning data from the database. It uses Python and the SciPy library to analyze the user's learning history. Based on this analysis, it generates a customized problem set. The generated problem set is sent to the device. The input is the selection information and past learning data, and the output is a customized problem set.

[0699] Step 6:

[0700] Terminal: Displays the problem set received from the server to the user. The input is the problem set from the server, and the output is the display of the problem set.

[0701] Step 7:

[0702] User: Enters the answer to the displayed question and clicks the button to send the answer. The answer entered by the user is sent from the terminal to the server. The input is the answer to the question, and the output is the transmission of the answer to the server.

[0703] Step 8:

[0704] Server: Analyzes the received answers. First, it determines whether the answer is correct or incorrect. Next, it uses AI technology to generate individual feedback based on the user's answer and past learning data. The generated feedback is sent to the device. The input is the user's answer, and the output is the correctness determination and feedback.

[0705] Step 9:

[0706] Terminal: Receives feedback sent from the server. Displays the received feedback to the user. The input is the feedback from the server, and the output is the display of the feedback.

[0707] Step 10:

[0708] Server: All of the user's answer results and learning history are stored in cloud storage. Specifically, a storage service such as AWS S3 is used. Progress is evaluated using Apache Spark based on the stored data. Based on the evaluation results, new learning plans and additional learning materials are generated and sent to the device. The input is all answer results and learning history, and the output is the stored data and the progress evaluation results.

[0709] Step 11:

[0710] Terminal: Receives new learning plans and additional learning materials sent from the server. Displays them to the user. The input is the new learning plans and additional learning materials from the server, and the output is their display.

[0711] (Application example 1)

[0712] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0713] Conventional training support systems have had the challenge of providing real-time feedback and progress visualization specific to improving factory workers' skills. It was particularly difficult to generate appropriate problem sets and provide individualized feedback for tasks using robots in factories. Furthermore, there was a lack of methods for providing customized learning plans based on the worker's level of skill acquisition.

[0714] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0715] In this invention, the server includes means for receiving a user's authentication information and verifying it against a database for authentication, means for analyzing the user's past learning data to generate a customized work task set, and means for analyzing the user's answers to generate individual feedback. This allows factory workers to receive real-time feedback while undergoing work skill training using robots. Furthermore, the server can evaluate the user's progress and provide new learning plans and additional learning materials, thereby enabling effective skill improvement.

[0716] "User authentication information" refers to information required when a user accesses a system, and is primarily composed of a user name and password.

[0717] A "database" is a system for systematically storing and managing various information such as authentication information and learning data.

[0718] A "customized work task set" is a collection of work tasks that are individually designed based on the user's past learning data and answer patterns.

[0719] "Individual feedback" is information such as an evaluation and areas for improvement for a specific user that is generated based on the user's answers and work performance.

[0720] "Progress" refers to a state that indicates the learning results and skill acquisition level that the user has achieved so far.

[0721] A "new learning plan" is a new learning or training method suggested based on the user's current skill level and learning progress.

[0722] "Additional Learning Materials" are new educational content recommended for users to further their learning.

[0723] "Factory workers" are employees who work in factories using robots.

[0724] "Skills practice" is practical training to improve specific work skills.

[0725] A "smartphone application" is a software program that runs on a smartphone and can be operated by a user through an interface.

[0726] "Real-time" means immediate response to user operations and inputs.

[0727] "Generative AI model technology" refers to artificial intelligence technology that generates optimal answers and feedback based on large amounts of data.

[0728] "Answer patterns" refer to the tendencies and methods of answering questions that a user has used in the past.

[0729] A "robot" is a mechanical device that can automatically perform programmed tasks.

[0730] This invention is an educational support system that allows factory workers to effectively practice their skills using robots. The system is composed of a server, a smartphone, and a factory robot as its main components. The functions of each component and their specific implementation methods are described below.

[0731] Authenticating Users

[0732] The server receives the user's authentication information and performs authentication by checking it against the database. If authentication is successful, the server reads the user's learning data and stores the user ID in the session. The smartphone application receives the authentication result, and if authentication is successful, displays the dashboard.

[0733] Generate a customized set of work tasks

[0734] When a user selects a specific work task (e.g., welding or assembly) from the dashboard, the smartphone sends that information to the server. Based on the received selection, the server analyzes the user's past learning data and answer patterns to generate a customized work task set. This task set is sent to the smartphone and used as a guide for the user to start working.

[0735] Generate answers and feedback

[0736] When a user performs a task and enters the answer into their smartphone, the smartphone sends the information to the server in real time. The server analyzes the received answer and determines whether it is correct or incorrect. Furthermore, using generative AI model technology, the server generates personalized feedback based on the user's answer and past learning data and sends it to the smartphone. The feedback includes important hints and specific advice to help the user improve their work.

[0737] Evaluating progress and suggesting new learning plans

[0738] The server stores the user's answers and learning history, evaluates their progress, and generates a new learning plan and additional learning materials to guide the user to the next step in their learning, which are then sent to the user's smartphone.

[0739] Hardware and software used

[0740] Hardware

[0741] Smartphone

[0742] server

[0743] Factory robots

[0744] software

[0745] Flask (a web application framework written in Python)

[0746] scikit-learn (library for evaluation calculations)

[0747] Specific examples

[0748] Below are some specific examples of how this educational support system can be used.

[0749] Log in

[0750] The user logs in by entering "user1" and "pass123" into the smartphone app.

[0751] The server verifies the authentication information and returns the username if successful.

[0752] Selecting and generating work tasks

[0753] The user selects the "welding task" on the smartphone app.

[0754] The server generates a customized set of "welding" tasks and sends them to the smartphone.

[0755] Examples: "weld_joint_a: OK", "weld_joint_b: OK", "weld_joint_c: Fail"

[0756] Task execution and feedback

[0757] The user performs the work task and inputs the results into the smartphone.

[0758] Input examples: "OK", "OK", "Fail"

[0759] The server analyzes the answers and generates feedback.

[0760] The user sees feedback on their smartphone: "Great job! Continue to the next skill step."

[0761] Prompt Sentence Examples

[0762] Using the examples below as a reference, please describe the application's processing in natural language so that it can be explained to a human.

[0763] The user logs in by entering "user1" and "pass123" on the smartphone app. If authentication is successful on the server, the user name is displayed.

[0764] Next, the user selects the "welding" task on the smartphone app, and the server generates a customized task set and sends it to the smartphone.

[0765] Example of generated task set: "weld_joint_a: OK", "weld_joint_b: OK", "weld_joint_c: Fail"

[0766] The user performs the task and enters the results into a smartphone app.

[0767] Input examples: "OK", "OK", "Fail"

[0768] The server analyzes the results and generates feedback, which is displayed on the smartphone: "Great job! Proceed to the next skill step."

[0769] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0770] Step 1: User authentication

[0771] Input: The user attempts to log in by entering their username and password into the smartphone app.

[0772] How it works: The smartphone app sends this authentication information to the server.

[0773] Data processing: The server compares the received authentication information with the data stored in the database.

[0774] Output: If authentication is successful, read the user information and save the user ID in the session. If authentication fails, generate an error message and send it to the terminal.

[0775] Step 2: View the dashboard

[0776] Input: The authentication result from the server (success or failure).

[0777] Behavior: The smartphone app displays a dashboard if authentication is successful, or an error message if authentication is unsuccessful.

[0778] Data processing: None.

[0779] Output: Display a dashboard or display an error message.

[0780] Step 3: Select a work task

[0781] Input: The user selects a specific work task (e.g., welding or assembly) from the dashboard.

[0782] Operation: The smartphone app sends the selected work task information to the server.

[0783] Data processing: None.

[0784] Output: Work task information sent to the server.

[0785] Step 4: Generate a customized set of work tasks

[0786] Input: The server receives the user's selected task information, past learning data, and answer patterns.

[0787] How it works: The server uses a generative AI model to analyze incoming data and generate a customized set of work tasks.

[0788] Data processing: Analysis of past learning data and answer patterns, and generation of task sets using a generative AI model.

[0789] Output: The generated customized task set is sent to your smartphone.

[0790] Step 5: Performing work tasks

[0791] Input: A customized set of work tasks sent from the server.

[0792] Action: The user performs the work task displayed on the smartphone app and enters the results into the app.

[0793] Data processing: None.

[0794] Output: User's answer result.

[0795] Step 6: Analyze answers and generate feedback

[0796] Input: User answer results sent from the smartphone app.

[0797] How it works: The server receives and analyzes the answers, uses a generative AI model to determine whether the answer is correct, and generates personalized feedback.

[0798] Data processing: Analysis of answer results and generation of feedback using a generative AI model.

[0799] Output: The generated feedback is sent to a smartphone app.

[0800] Step 7: View your feedback

[0801] Input: Feedback sent by the server.

[0802] What it does: The smartphone app displays the received feedback to the user.

[0803] Data processing: None.

[0804] Output: Display feedback.

[0805] Step 8: Evaluate progress and propose a new learning plan

[0806] Input: All of the user's answers and learning history.

[0807] How it works: The server analyzes this data, assesses progress, and uses generative AI models and algorithms to generate new learning plans and additional learning materials.

[0808] Data processing: Saving answer results and learning history, evaluating progress, and generating new learning plans and additional learning materials.

[0809] Output: Send new learning plans and additional learning materials to a smartphone app.

[0810] Step 9: View your learning plan

[0811] Input: New study plans and additional materials sent from the server.

[0812] How it works: The smartphone app displays the plan in place.

[0813] Data processing: None.

[0814] Output: Display of new learning plans and additional learning materials.

[0815] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0816] The present invention provides an educational support system for providing an optimal educational experience to each student. The system includes a means for receiving user authentication information and authenticating it against a database, a means for analyzing the user's past learning data and generating a customized problem set, a means for analyzing the user's answers and generating individualized feedback, a means for saving the user's answers and learning history and evaluating the user's progress, and a means for suggesting new learning plans and additional learning materials. The system further includes an emotion engine that recognizes the user's emotions and generates feedback based on the user's emotional state to improve learning efficiency.

[0817] System processing flow

[0818] Authenticating Users

[0819] 1. Device:

[0820] The user opens the login screen, enters their username and password, and clicks the "Login" button.

[0821] 2. Server:

[0822] Authentication information received from the terminal is verified against the database for authentication. If authentication is successful, the user's learning data is read and the user ID is saved in the session. If authentication fails, an error message is generated and sent to the terminal.

[0823] 3. Terminal:

[0824] Receives authentication results from the server, and if authentication is successful, displays the dashboard. If authentication fails, displays an error message.

[0825] Selecting learning modules and generating problem sets

[0826] 4. User:

[0827] Select the module you want to study from your dashboard.

[0828] 5. Terminal:

[0829] The user's selection information is sent to the server.

[0830] 6. Server:

[0831] Based on the received selection information, the system analyzes the user's past learning data, generates a customized question set, and transmits it to the terminal.

[0832] 7. Terminal:

[0833] The problem set received from the server is displayed to the user.

[0834] Answers and feedback

[0835] 8. Users:

[0836] Enter the answer to the question that appears.

[0837] 9. Terminal:

[0838] The user's answers are sent to the server in real time.

[0839] 10. Server:

[0840] The system analyzes the received answers and determines whether they are correct or not. It also uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[0841] 11. Terminal:

[0842] Display feedback received from the server to the user.

[0843] Evaluating progress and suggesting new learning plans

[0844] 12. Server:

[0845] It stores all of the user's answers and learning history, evaluates their progress, and generates and sends new learning plans and additional learning materials to the device.

[0846] 13. Terminal:

[0847] Display new learning plans and additional learning materials received from the server to the user.

[0848] Introducing the Emotion Engine

[0849] The emotion engine has the ability to recognize the user's emotional state by analyzing the user's facial expressions, tone of voice, and other biometric information. Using this information, the following process is carried out to improve learning efficiency.

[0850] Emotion recognition and feedback with emotion engine

[0851] 1. Device:

[0852] Emotional information such as the user's facial expressions and tone of voice is collected through a camera and microphone.

[0853] 2. Server:

[0854] The emotion engine analyzes this emotion information and identifies the user's emotional state (e.g., excitement, fatigue, confusion, etc.).

[0855] 3. Server:

[0856] It generates feedback based on the emotional state obtained from the emotion engine. For example, if the user is tired, it generates feedback such as "Take a short break."

[0857] 4. Terminal:

[0858] Feedback based on the emotional state received from the server is displayed to the user.

[0859] Responding to emotional states

[0860] The emotional engine can adjust the difficulty and format of questions based on the user's emotional state. For example, if the user is confused, it can simplify the questions or provide hints. The emotional state obtained from the emotional engine is also added to the training data and reflected in the assessment of progress and comprehension.

[0861] Specific examples

[0862] Example 1: Math problems and emotional feedback

[0863] 1. Device:

[0864] A user selects a mathematics module and requests an entry-level problem set.

[0865] 2. Server:

[0866] A customized problem set including basic arithmetic operations is generated from past learning data and sent to the device.

[0867] 3. Terminal:

[0868] A user answers "4" to the question "2 + 2 = ?" and sends the answer to the server.

[0869] 4. Server:

[0870] The answer is analyzed, confirmed to be correct, and feedback is generated saying, "That's correct. Please proceed to the next question."

[0871] 5. Terminal:

[0872] Display feedback received from the server to the user.

[0873] 6. Terminal (Emotion Engine):

[0874] The camera analyzes the user's facial expressions and detects when the user is concentrating.

[0875] 7. Server (Emotion Engine):

[0876] It checks the user's concentration and generates feedback to encourage them to continue learning.

[0877] 8. Terminal:

[0878] Display feedback received from the server to the user.

[0879] In this way, the system of the present invention provides an optimal learning experience in real time based on the user's individual needs and level of understanding. Furthermore, the introduction of an emotion engine enables customization based on the user's emotional state, further improving learning efficiency. This results in improved learning outcomes for students, and allows teachers and parents to easily track progress.

[0880] The processing flow will be explained below.

[0881] System processing flow

[0882] Authenticating Users

[0883] Step 1:

[0884] Terminal: The user opens the login screen, enters their username and password, and clicks the "Login" button.

[0885] Step 2:

[0886] Server: Authenticates the user by checking the authentication information received from the terminal against the database. If authentication is successful, reads the user's learning data and stores the user ID in the session. If authentication fails, generates an error message and sends it to the terminal.

[0887] Step 3:

[0888] Terminal: Receives authentication result from the server, and displays the dashboard if authentication is successful. If authentication fails, displays an error message.

[0889] Selecting learning modules and generating problem sets

[0890] Step 4:

[0891] Users: Select the module they want to study from the dashboard.

[0892] Step 5:

[0893] Terminal: Sends the user's selection information to the server.

[0894] Step 6:

[0895] Server: Based on the received selection information, the server analyzes the user's past learning data, generates a customized problem set, and sends it to the device.

[0896] Step 7:

[0897] Terminal: Displays the problem set received from the server to the user.

[0898] Answers and feedback

[0899] Step 8:

[0900] User: Enter the answer to the question displayed.

[0901] Step 9:

[0902] Terminal: Sends the user's answers to the server in real time.

[0903] Step 10:

[0904] Server: Analyzes the received answers and determines whether they are correct. Furthermore, it uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[0905] Step 11:

[0906] Terminal: Displays the feedback received from the server to the user.

[0907] Evaluating progress and suggesting new learning plans

[0908] Step 12:

[0909] Server: Stores all user answers and learning history, evaluates progress, and generates new learning plans and additional learning materials and sends them to the device.

[0910] Step 13:

[0911] Terminal: Displays new learning plans and additional learning materials received from the server to the user.

[0912] Introducing the Emotion Engine

[0913] Emotion recognition and feedback with emotion engine

[0914] Step 14:

[0915] Device: Collects emotional information such as the user's facial expressions and tone of voice through a camera and microphone.

[0916] Step 15:

[0917] Server: The emotion engine analyzes this emotion information and identifies the user's emotional state (e.g., excitement, fatigue, confusion, etc.).

[0918] Step 16:

[0919] Server: Generates feedback based on the emotional state obtained from the emotion engine. For example, if the user is tired, it generates feedback such as "Take a short break."

[0920] Step 17:

[0921] Terminal: Displays feedback to the user based on the emotional state received from the server.

[0922] Responding to emotional states

[0923] Step 18:

[0924] Server: Using the emotion engine, adjust the difficulty and format of questions according to the user's emotional state. For example, if the user is confused, simplify the question or provide hints.

[0925] Step 19:

[0926] Server: The emotional state obtained from the emotion engine is added to the training data and reflected in the assessment of progress and understanding.

[0927] Specific examples

[0928] Example 1: Math problems and emotional feedback

[0929] Step 1:

[0930] Terminal: The user selects a mathematics module and requests an entry-level problem set.

[0931] Step 2:

[0932] Server: Generates a customized problem set, including basic arithmetic operations, from past learning data and sends it to the device.

[0933] Step 3:

[0934] Terminal: The user answers "4" to the question "2 + 2 = ?" and sends the answer to the server.

[0935] Step 4:

[0936] Server: Analyzes the answer, verifies it is correct, and generates feedback such as "That's correct. Please proceed to the next question."

[0937] Step 5:

[0938] Terminal: Displays the feedback received from the server to the user.

[0939] Step 6:

[0940] Terminal (emotion engine): The camera analyzes the user's facial expressions and detects when the user is concentrating.

[0941] Step 7:

[0942] Server (emotion engine): Checks the user's concentration and generates feedback to encourage them to continue learning.

[0943] Step 8:

[0944] Terminal: Displays the feedback received from the server to the user.

[0945] In this way, the system of the present invention provides an optimal learning experience in real time based on the user's individual needs and level of understanding. Furthermore, the introduction of an emotion engine enables customization based on the user's emotional state, further improving learning efficiency. This results in improved learning outcomes for students, and allows teachers and parents to easily track progress.

[0946] Example 2

[0947] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0948] Conventional educational support systems have struggled to provide an optimized educational experience for each individual user. In particular, it has been difficult to grasp a user's learning situation in real time and provide feedback based on that understanding, and customization to accommodate emotional states has been insufficient. As a result, learners are unable to properly understand their own learning situation, resulting in a decline in learning effectiveness.

[0949] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving user authentication information and verifying it against a database for authentication, means for analyzing the user's past learning data and generating a customized problem set, means for analyzing the user's answers and generating individual feedback, means for saving the user's answers and learning history and evaluating the user's progress, means for suggesting new learning plans and additional learning materials, means for collecting and analyzing the user's emotional information and generating feedback based on the user's emotional state, means for transmitting the user's answers in real time, and means for displaying the generated feedback to the user. This makes it possible to provide an optimal educational experience in real time based on the user's individual learning situation and emotional state.

[0950] "Means for receiving user authentication information and verifying it against a database" refers to the process of receiving user authentication information (such as username and password) sent from the terminal and comparing that information with the registered information in a database to verify the user's legitimacy.

[0951] "Means for analyzing a user's past learning data and generating a customized problem set" refers to the process of analyzing a user's learning history and achievements and generating individually optimized learning problems based on that.

[0952] The "means for analyzing the user's answers and generating personalized feedback" is a process for analyzing the answers submitted by the user and generating feedback that provides specific guidance based on the results.

[0953] "Means for saving the user's answers and learning history and evaluating the progress" refers to the process of recording the correctness of the user's answers and learning history, and evaluating the user's learning progress and level of understanding based on this.

[0954] The "means for suggesting new study plans and additional learning materials" is a process for providing the user with the next study content and supplementary learning materials based on the user's evaluation results.

[0955] "Means for generating feedback based on a user's emotional state by collecting and analyzing the user's emotional information" refers to a process of using a camera or microphone to collect the user's facial expressions and tone of voice, analyzing the user's emotional state based on that information, and providing appropriate feedback.

[0956] "Means for transmitting user answer results in real time" refers to a process in which the results are immediately transmitted to the server each time a user inputs an answer.

[0957] The "means for displaying the generated feedback to the user" is a process for displaying the feedback sent from the server on the screen of the terminal and providing it to the user.

[0958] The present invention is an educational support system for providing an optimal educational experience to each student. The system includes means for receiving user authentication information, authenticating the user by checking it against a database, analyzing the user's past learning data to generate a customized problem set, and analyzing the user's answers to generate personalized feedback. The system also stores the user's answers and learning history, evaluates the user's progress, and suggests new learning plans and additional learning materials. The system also incorporates an emotion engine for collecting and analyzing the user's emotional information and generating feedback based on the user's emotional state.

[0959] The entire system functions through the interaction of three parties: the server, the terminal, and the user. The specific configuration and operation will be explained below.

[0960] First, the user opens a login screen using the terminal and enters their username and password. This login information is sent to the server using JavaScript. The server uses the Python Flask framework to check the authentication information against a database (e.g., MySQL). If authentication is successful, the server loads the user's learning data and saves the user ID in the session. If authentication fails, a response including an error message is sent to the terminal. Based on the response from the server, the terminal displays a dashboard if the authentication was successful, or an error message if the authentication failed.

[0961] Next, the user selects the module they wish to learn from the dashboard. This selection information is sent to the server via JavaScript. Based on the received selection information, the server analyzes the user's past learning data using machine learning libraries such as scikit-learn. As a result, a problem set optimized for the user is generated and sent to the device. The device then dynamically generates the received problem set using HTML and JavaScript and displays it to the user.

[0962] When a user enters an answer to a question and clicks the submit button, the device sends the answer information to the server in real time. The server uses Python to analyze the answer and determine whether it is correct. It then uses AI technologies such as TensorFlow and PyTorch to generate individual feedback based on the analysis results and past learning data and sends it to the device. The device then displays the received feedback to the user.

[0963] The server stores all of the user's answers and learning history in a database. The server evaluates the user's progress based on this information. Based on the progress and evaluation results, it generates new learning plans and additional learning materials and sends them in JSON format to the device. The device then displays the received plans and learning materials to the user.

[0964] Furthermore, a means for recognizing the user's emotional state is incorporated using an emotion engine. The device collects the user's facial expressions and tone of voice through a camera and microphone, and analyzes them using an emotion engine (e.g., OpenCV or the Facial Emotion Recognition (FER) library). Based on the analysis results, the server generates feedback according to the user's emotional state (e.g., "Let's take a short break") and sends it to the device. The device then displays this feedback to the user.

[0965] Examples of concrete examples and prompts

[0966] As a concrete example, let's consider an operation in which a user selects a mathematics module and requests a set of beginner-level arithmetic problems. The server generates problems such as "2 + 2 = ?" based on past learning data and sends them to the device. If the user answers "4," the server determines that this is the correct answer and generates feedback saying, "That's correct. Please proceed to the next problem." The device displays this feedback, and also recognizes the user's state of concentration via the emotion engine, generating and displaying feedback encouraging the user to continue learning.

[0967] Examples of prompts might include "Design a system that returns analysis results in real time after the user enters their answer" or "Explain the process of generating a customized problem set and displaying it to the user."

[0968] In this way, the educational support system according to the present invention can provide an optimal educational experience in real time based on the user's learning situation and emotional state.

[0969] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0970] Step 1: Authenticate the user

[0971] 1.1 Terminal:

[0972] You open a login screen, enter your username and password, and click the "Login" button. The computer program sends this information to the server using JavaScript.

[0973] Input: Username, Password

[0974] Output: Authentication information sent to the server

[0975] 1.2 Server:

[0976] The authentication information received from the device is checked against a database (e.g., MySQL). The Python Flask framework is used to compare the authentication information with the database, and if a match is found, authentication is successful. The user ID is stored in the session and training data is loaded. If authentication fails, an error message is generated.

[0977] Input: Authentication information (user name, password)

[0978] Output: Authentication result (success / failure), user data (if authentication is successful)

[0979] 1.3 Terminal:

[0980] Receives the authentication result from the server and displays the dashboard if successful, or an error message if unsuccessful.

[0981] Input: Authentication result (success / failure)

[0982] Output: Dashboard (if successful), Error message (if unsuccessful)

[0983] Step 2: Selecting a learning module and generating a problem set

[0984] 2.1 User:

[0985] From the dashboard, select the module you want to study. Your selection is collected using JavaScript.

[0986] Input: Module selection information

[0987] Output: Selections sent to the server

[0988] 2.2 Terminal:

[0989] Send the selection to the server. The selection is sent asynchronously using an AJAX request.

[0990] Input: Selection information

[0991] Output: Selections sent to the server

[0992] 2.3 Server:

[0993] Receives the selection information and analyzes the user's past training data. Analyzes the training data using Python and scikit-learn to generate a customized problem set. The generated problem set is sent to the device in JSON format.

[0994] Input: Selection information, past learning data

[0995] Output: Problem set in JSON format

[0996] 2.4 Terminal:

[0997] The problem set received from the server is displayed to the user. Problems are dynamically generated using JavaScript and HTML and displayed on the screen.

[0998] Input: Problem set (JSON format)

[0999] Output: The displayed problem set

[1000] Step 3: Answers and feedback

[1001] 3.1 User:

[1002] Enter your answer to the question and click the "Submit" button. The answer will be entered into the form and collected by JavaScript.

[1003] Input: Answer

[1004] Output: Answer sent to the server

[1005] 3.2 Terminal:

[1006] Send the user's answers to the server in real time using an AJAX request.

[1007] Input: Answer

[1008] Output: Answer sent to the server

[1009] 3.3 Server:

[1010] The received answers are analyzed and judged to be correct (using Python), and then individual feedback is generated using AI technologies such as TensorFlow and PyTorch and sent to the device.

[1011] Input: Answers, past learning data

[1012] Output: Individual feedback

[1013] 3.4 Terminal:

[1014] Display the feedback received from the server to the user. Use JavaScript to display the feedback message on the screen.

[1015] Input: Feedback

[1016] Output: Displayed feedback

[1017] Step 4: Evaluate progress and propose a new learning plan

[1018] 4.1 Server:

[1019] All user answers and learning history are stored in a database, and progress is evaluated. Based on the evaluation results, new learning plans and additional learning materials are generated and sent to the device in JSON format.

[1020] Input: Answer results, learning history

[1021] Output: New learning plan, additional materials (JSON format)

[1022] 4.2 Terminal:

[1023] Display new learning plans and additional learning materials received from the server to the user. Plans and learning materials are dynamically generated using JavaScript and HTML and displayed on the screen.

[1024] Input: New learning plan, additional materials (JSON format)

[1025] Output: New learning plan displayed, additional materials

[1026] Step 5: Emotion recognition and feedback by the emotion engine

[1027] 5.1 Terminal:

[1028] Emotional information such as facial expressions and tone of voice of the user is collected through a camera and microphone. OpenCV and Python libraries are used for the emotion engine.

[1029] Input: facial expression data, voice data

[1030] Output: Emotion information sent to the server

[1031] 5.2 Server:

[1032] The emotion engine analyzes the emotion information and identifies the user's emotional state. Based on the analysis results, it generates appropriate feedback and sends it to the device.

[1033] Input: Emotion information

[1034] Output: Feedback (based on emotional state)

[1035] 5.3 Terminal:

[1036] Display feedback to the user based on the emotional state received from the server. Use JavaScript to display the feedback message on the screen.

[1037] Input: Feedback (based on emotional state)

[1038] Output: Displayed feedback

[1039] (Application example 2)

[1040] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1041] Conventional education and training support systems have difficulty adapting flexibly to the learning progress and emotional state of individual users and workers, and have often provided uniform learning plans and training procedures. As a result, optimal feedback and training support tailored to each individual's level of understanding and emotional state has not been provided, leading to a decline in learning efficiency and work efficiency.

[1042] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving a user's authentication information and verifying it against a database for authentication; means for analyzing the user's past learning data and generating a customized problem set; means for analyzing the user's answers and generating individual feedback; means for saving the user's answers and learning history and evaluating progress; means for suggesting new learning plans and additional learning materials; means for analyzing the worker's past training data and presenting customized work procedures; means including an emotion engine for recognizing emotional states and generating feedback; and means for evaluating progress and suggesting the next training or work procedures. This makes it possible to provide each user or worker with an optimal learning and training experience tailored to their learning situation and emotional state.

[1043] "User authentication information" refers to information used to identify a person accessing a system and verify their identity.

[1044] A "database" is a collection of information used to store and manage user authentication information, learning data, and training data.

[1045] A "customized problem set" is a collection of problems optimized for each individual user, generated by analyzing the user's past learning data and answer patterns.

[1046] "Individual feedback" is advice and evaluation specific to a user that is generated based on the user's answers and progress.

[1047] "Progress" is information that indicates the progress and results of a user's learning or training.

[1048] "New study plans and additional learning materials" are new study plans and supplementary learning materials that are suggested based on the user's progress and level of understanding.

[1049] A "work procedure" is a series of steps or methods for accomplishing a particular task.

[1050] An "emotion engine" is a system that analyzes emotional information such as a user's facial expression and tone of voice to recognize the user's emotional state.

[1051] "Feedback" refers to evaluations and advice provided in response to a user's actions or input.

[1052] "Training data" refers to data that includes records of training that workers have previously undergone and the learning outcomes based on that training.

[1053] The present invention is a real-time training support system for factory workers, which uses smart glasses to provide real-time assistance to workers while they are working. Detailed description of the embodiments of the present invention will be given below.

[1054] First, the system includes the following hardware and software components:

[1055] Hardware: Smart glasses (camera, display)

[1056] software:

[1057] OpenCV: Camera image capture and display

[1058] Dlib: A face detection and recognition model

[1059] EmotionRecognizer: A model for recognizing emotional states

[1060] Data Management Module: Acquire and suggest customized training data

[1061] Server communication module: Sending and receiving feedback

[1062] System processing flow

[1063] 1. User authentication:

[1064] A worker puts on the smart glasses and logs in to the system. The user's authentication information (username and password) is sent from the smart glasses device to the server and checked against the database. If authentication is successful, the worker's training data is loaded and the session begins.

[1065] 2. Providing training procedures:

[1066] Based on the worker's past training data, customized work instructions are generated and displayed on the smart glasses' display, and the instructions are adjusted based on each worker's skill level and past performance.

[1067] 3. Recognition of emotional states:

[1068] A camera mounted on the smart glasses captures the worker's facial expressions, which are then analyzed by EmotionRecognizer, which identifies the worker's emotional state, such as whether they are tired, focused, or confused.

[1069] 4. Generate personalized feedback:

[1070] Based on the recognized emotional state, the server generates appropriate feedback in real time, such as "Take a break" if the worker is tired, or "Revisit the procedure" if the worker is confused.

[1071] 5. Progress evaluation and training plan suggestions:

[1072] The server stores all the worker's training data and evaluates their progress, generating new training plans and additional work instructions to prepare them for the next job.

[1073] Specific examples

[1074] For example, when a worker is tightening a nut, smart glasses will display the work instructions and analyze the worker's facial expressions using a facial recognition camera. If the smart glasses detect that the worker is tired, they will provide feedback such as "Take a short break." If the glasses detect that the worker is confused by the work instructions, they will again provide specific instructions such as "Turn the screw clockwise."

[1075] Example prompts for generative AI models

[1076] "Generate optimal feedback when workers feel fatigued during work."

[1077] "Propose procedural support for workers if they are confused."

[1078] In this way, the system of the present invention can improve work efficiency and safety by providing optimal support in real time according to the condition of each individual worker.

[1079] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1080] Step 1:

[1081] The user puts on the smart glasses and enters their username and password on the login screen. The input information is sent from the device to the server. The server compares the received authentication information with a database and performs authentication. If authentication is successful, the worker's training data is read and the session begins. If authentication fails, an error message is sent to the device and displayed.

[1082] Step 2:

[1083] After the worker is authenticated, he or she selects a training module. The terminal sends the selection information to the server. The server analyzes the worker's past training data based on the received selection information. Based on the analysis results, the server generates a customized work procedure and sends it to the terminal. The terminal displays the generated work procedure on the display of the smart glasses.

[1084] Step 3:

[1085] A camera mounted on the smart glasses captures the worker's facial expressions. The captured video is sent from the device to a server. The server then uses an emotion engine (EmotionRecognizer) to analyze the video data and recognize the worker's emotional state (fatigue, concentration, confusion, etc.). The recognized emotional state is then used to evaluate progress.

[1086] Step 4:

[1087] The server generates appropriate feedback in real time based on the recognized emotional state. For example, if fatigue is recognized, feedback such as "Take a short break" is generated. The generated feedback is sent to the device and displayed on the smart glasses display.

[1088] Step 5:

[1089] The server stores all of the worker's training data and evaluates their progress. The evaluation results are used to generate the next training or additional work procedures. The server generates new training plans or additional work procedures and sends them to the device. The device displays the new training plans or additional work procedures on the smart glasses display.

[1090] Step 6:

[1091] While the worker is performing additional work steps, the camera in the smart glasses again captures their facial expressions and sends them to the server, which then uses the emotion engine to analyze their emotional state again and updates the feedback as needed, ensuring optimal support is always provided based on the worker's condition.

[1092] This allows users to receive optimal feedback and work procedures in real time based on their individual learning situation and emotional state.

[1093] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1094] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1095] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1096] [Third embodiment]

[1097] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1098] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1099] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1100] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1101] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1102] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1103] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1104] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1105] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1106] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1107] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1108] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1109] The present invention provides an educational support system for providing an optimal educational experience to each student. The system includes means for receiving user authentication information and authenticating the user by comparing it with a database, means for analyzing the user's past learning data and generating a customized problem set, means for analyzing the user's answers and generating individualized feedback, means for storing the user's answers and learning history and evaluating the user's progress, and means for suggesting new learning plans and additional learning materials.

[1110] System processing flow

[1111] Authenticating Users

[1112] 1. Device:

[1113] The user opens the login screen, enters their username and password, and clicks the "Login" button.

[1114] 2. Server:

[1115] Authentication information received from the terminal is verified against the database for authentication. If authentication is successful, the user's learning data is read and the user ID is saved in the session. If authentication fails, an error message is generated and sent to the terminal.

[1116] 3. Terminal:

[1117] Receives authentication results from the server, and if authentication is successful, displays the dashboard. If authentication fails, displays an error message.

[1118] Selecting learning modules and generating problem sets

[1119] 4. User:

[1120] Select the module you want to study from your dashboard.

[1121] 5. Terminal:

[1122] The user's selection information is sent to the server.

[1123] 6. Server:

[1124] Based on the received selection information, the system analyzes the user's past learning data, generates a customized question set, and transmits it to the terminal.

[1125] 7. Terminal:

[1126] The problem set received from the server is displayed to the user.

[1127] Answers and feedback

[1128] 8. Users:

[1129] Enter the answer to the question that appears.

[1130] 9. Terminal:

[1131] The user's answers are sent to the server in real time.

[1132] 10. Server:

[1133] The system analyzes the received answers and determines whether they are correct or not. It also uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[1134] 11. Terminal:

[1135] Display feedback received from the server to the user.

[1136] Evaluating progress and suggesting new learning plans

[1137] 12. Server:

[1138] It stores all of the user's answers and learning history, evaluates their progress, and generates and sends new learning plans and additional learning materials to the device.

[1139] 13. Terminal:

[1140] Display new learning plans and additional learning materials received from the server to the user.

[1141] Specific examples

[1142] Example 1: Math Problems

[1143] 1. Device:

[1144] A user selects a mathematics module and requests an entry-level problem set.

[1145] 2. Server:

[1146] A customized problem set including basic arithmetic operations is generated from past learning data and sent to the device.

[1147] 3. Terminal:

[1148] A user answers "4" to the question "2 + 2 = ?" and sends the answer to the server.

[1149] 4. Server:

[1150] The answer is analyzed, confirmed to be correct, and feedback is sent to the device saying, "That's correct. Please proceed to the next question."

[1151] 5. Terminal:

[1152] Display feedback received from the server to the user.

[1153] In this way, the system of the present invention provides an optimal learning experience in real time based on the user's individual needs and level of understanding, improving students' learning efficiency and allowing teachers and parents to easily track progress.

[1154] The processing flow will be explained below.

[1155] Step 1:

[1156] Terminal: The user opens the login screen, enters their username and password, and clicks the "Login" button.

[1157] Step 2:

[1158] Server: Authenticates the user by checking the authentication information received from the terminal against the database. If authentication is successful, reads the user's learning data and stores the user ID in the session. If authentication fails, generates an error message and sends it to the terminal.

[1159] Step 3:

[1160] Terminal: Receives authentication result from the server, and displays the dashboard if authentication is successful. If authentication fails, displays an error message.

[1161] Step 4:

[1162] Users: Select the module they want to study from the dashboard.

[1163] Step 5:

[1164] Terminal: Sends the user's selection information to the server.

[1165] Step 6:

[1166] Server: Based on the received selection information, the server analyzes the user's past learning data, generates a customized problem set, and sends it to the device.

[1167] Step 7:

[1168] Terminal: Displays the problem set received from the server to the user.

[1169] Step 8:

[1170] User: Enter the answer to the question displayed.

[1171] Step 9:

[1172] Terminal: Sends the user's answers to the server in real time.

[1173] Step 10:

[1174] Server: Analyzes the received answers and determines whether they are correct. Furthermore, it uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[1175] Step 11:

[1176] Terminal: Displays the feedback received from the server to the user.

[1177] Step 12:

[1178] Server: Stores all user answers and learning history, evaluates progress, and generates new learning plans and additional learning materials and sends them to the device.

[1179] Step 13:

[1180] Terminal: Displays new learning plans and additional learning materials received from the server to the user.

[1181] Step 14:

[1182] User: Proceed to the next learning step based on the new learning plan or additional materials.

[1183] Example 1

[1184] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1185] Conventional educational support systems face the challenge of providing an educational experience tailored to each user's individual learning needs and level of understanding. Specifically, they lack the functionality to effectively analyze a user's past learning data to generate optimal problem sets or provide individually customized feedback in real time. This makes it difficult to improve learning efficiency and accurately evaluate progress.

[1186] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1187] In this invention, the server includes means for receiving the user's authentication information and verifying it against a database, means for analyzing the user's past learning data to generate a customized problem set, and means for analyzing the user's answers to generate individual feedback, thereby providing an optimal educational experience based on the user's individual needs and level of understanding, improving learning efficiency, and enabling accurate assessment of progress.

[1188] "User authentication information" refers to information such as a username and password that a user enters when accessing a system.

[1189] A "database" refers to a storage device for systematically storing and managing user authentication information, learning data, answer results, etc.

[1190] "Matching" refers to the process of comparing received information with existing data to see if there is a match.

[1191] A "customized problem set" refers to a collection of study questions that are individually optimized based on the user's past study data and answer patterns.

[1192] "Analyzing answers" refers to the process of evaluating the answers entered by the user and determining whether they are correct or incorrect.

[1193] "Individual feedback" refers to study advice and comments for individual users that are generated based on the user's answer results and past study data.

[1194] "Storage" refers to the act of recording data on a storage device so that it can be used at a later time.

[1195] "Progress assessment" refers to the process of analyzing a user's learning history and answer results to determine their current level of understanding and progress.

[1196] "New learning plan" refers to a plan that suggests future learning content and methods based on the user's progress.

[1197] "Additional Learning Materials" refers to additional learning materials or resources provided to assist the user in their learning.

[1198] "Terminal" refers to an electronic device used by a user to access and operate the system.

[1199] "Computing device" refers to an electronic device for analyzing data and performing calculations.

[1200] "AI technology" refers to technology that uses artificial intelligence to analyze, predict, optimize, and perform other functions on data.

[1201] "Learning history" refers to a record of the user's learning activities, answer results, and feedback.

[1202] The present invention provides an educational support system for providing an optimal educational experience to each student. The system includes means for receiving user authentication information and authenticating the user by comparing it with a database, means for analyzing the user's past learning data and generating a customized problem set, means for analyzing the user's answers and generating individualized feedback, means for storing the user's answers and learning history and evaluating the user's progress, and means for suggesting new learning plans and additional learning materials.

[1203] The basic configuration of the entire system is as follows:

[1204] Receive user credentials: When a user accesses the system, they enter their username and password on a login screen, which is sent to the server and checked against a MySQL database.

[1205] Authentication by checking against database: The server uses a MySQL database to check the entered authentication information against the information in the database. If authentication is successful, the user ID is saved in the session and a successful authentication response is sent to the terminal.

[1206] Analyzing the user's past learning data: Based on the received selection information, the server analyzes the user's learning history using Python and SciPy libraries to generate a customized problem set.

[1207] Generate customized problem sets: Based on the module selected by the user, the server generates an optimal problem set based on the user's past learning data. This problem set is sent to the terminal and displayed to the user.

[1208] Analyzing user answers: When a user enters an answer to a question, the answer is sent to the server in real time. The server analyzes the received answer and determines whether it is correct or incorrect.

[1209] Generate personalized feedback: Based on the received answer results, the server uses AI technology (e.g., a generative AI model) to generate personalized feedback and send it to the device.

[1210] Save user's answer results and learning history: All user's answer results and learning history are saved in cloud storage such as AWS S3.

[1211] Progress evaluation: The server analyzes the stored data using Apache Spark and evaluates the user's progress. Based on the evaluation results, a new learning plan and additional learning materials are generated and sent to the device.

[1212] The following flow can be considered as a specific example of how this system can be used.

[1213] 1. Terminal: The user selects the mathematics module and requests a beginner level problem set (clicks the "Math" button, sets the level to "Beginner" and presses the "Start" button).

[1214] 2. Server: Reads past training data from a MySQL database, generates a customized problem set including basic arithmetic problems using Python and SciPy, and sends it to the device.

[1215] 3. Terminal: Displays the problem set received from the server to the user (for example, displays the problem "2 + 2 = ?").

[1216] 4. User: Answers "4" to the displayed question "2 + 2 = ?" and clicks the button to submit the answer.

[1217] 5. Terminal: The user's answer "4" is sent to the server in real time.

[1218] 6. Server: Analyzes the received answer and verifies that it is correct. Using the generative AI model, it generates feedback such as "That's correct. Please proceed to the next question." and sends it to the device.

[1219] 7. Terminal: Displays the feedback received from the server to the user.

[1220] Examples of prompts for a generative AI model include:

[1221] "You are an AI teaching assistant. Create a customized set of math problems for a beginner student who has previously struggled with basic arithmetic (addition and subtraction)."

[1222] In this way, the system provides users with an optimal educational experience based on their individual needs and level of understanding, thereby improving their learning efficiency.

[1223] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1224] Step 1:

[1225] Terminal: The user opens a login screen and enters their username and password. When they click the "Login" button, the information they entered is sent to the server. The input is the username and password, and the output is the transmission of authentication information to the server.

[1226] Step 2:

[1227] Server: Receives the sent authentication information. Checks the received data against the database. Specifically, it compares the entered username and password with the user information stored in the MySQL database. If authentication is successful, it saves the user ID in the session and generates a response indicating successful authentication. If authentication fails, it generates an error message. The input is authentication information and the output is the authentication result.

[1228] Step 3:

[1229] Terminal: Receives authentication results from the server. If authentication is successful, displays the user's dashboard. If authentication fails, displays an error message. The input is the authentication result from the server, and the output is displaying the dashboard or an error message.

[1230] Step 4:

[1231] User: Selects the module they wish to learn from the list of learning modules displayed on the dashboard. The user's selection operation sends the selection information from the terminal to the server. The input is the selection of the learning module, and the output is the transmission of the selection information to the server.

[1232] Step 5:

[1233] Server: Based on the received selection information, it reads past learning data from the database. It uses Python and the SciPy library to analyze the user's learning history. Based on this analysis, it generates a customized problem set. The generated problem set is sent to the device. The input is the selection information and past learning data, and the output is a customized problem set.

[1234] Step 6:

[1235] Terminal: Displays the problem set received from the server to the user. The input is the problem set from the server, and the output is the display of the problem set.

[1236] Step 7:

[1237] User: Enters the answer to the displayed question and clicks the button to send the answer. The answer entered by the user is sent from the terminal to the server. The input is the answer to the question, and the output is the transmission of the answer to the server.

[1238] Step 8:

[1239] Server: Analyzes the received answers. First, it determines whether the answer is correct or incorrect. Next, it uses AI technology to generate individual feedback based on the user's answer and past learning data. The generated feedback is sent to the device. The input is the user's answer, and the output is the correctness determination and feedback.

[1240] Step 9:

[1241] Terminal: Receives feedback sent from the server. Displays the received feedback to the user. The input is the feedback from the server, and the output is the display of the feedback.

[1242] Step 10:

[1243] Server: All of the user's answer results and learning history are stored in cloud storage. Specifically, a storage service such as AWS S3 is used. Progress is evaluated using Apache Spark based on the stored data. Based on the evaluation results, new learning plans and additional learning materials are generated and sent to the device. The input is all answer results and learning history, and the output is the stored data and the progress evaluation results.

[1244] Step 11:

[1245] Terminal: Receives new learning plans and additional learning materials sent from the server. Displays them to the user. The input is the new learning plans and additional learning materials from the server, and the output is their display.

[1246] (Application example 1)

[1247] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1248] Conventional training support systems have had the challenge of providing real-time feedback and progress visualization specific to improving factory workers' skills. It was particularly difficult to generate appropriate problem sets and provide individualized feedback for tasks using robots in factories. Furthermore, there was a lack of methods for providing customized learning plans based on the worker's level of skill acquisition.

[1249] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1250] In this invention, the server includes means for receiving a user's authentication information and verifying it against a database for authentication, means for analyzing the user's past learning data to generate a customized work task set, and means for analyzing the user's answers to generate individual feedback. This allows factory workers to receive real-time feedback while undergoing work skill training using robots. Furthermore, the server can evaluate the user's progress and provide new learning plans and additional learning materials, thereby enabling effective skill improvement.

[1251] "User authentication information" refers to information required when a user accesses a system, and is primarily composed of a user name and password.

[1252] A "database" is a system for systematically storing and managing various information such as authentication information and learning data.

[1253] A "customized work task set" is a collection of work tasks that are individually designed based on the user's past learning data and answer patterns.

[1254] "Individual feedback" is information such as an evaluation and areas for improvement for a specific user that is generated based on the user's answers and work performance.

[1255] "Progress" refers to a state that indicates the learning results and skill acquisition level that the user has achieved so far.

[1256] A "new learning plan" is a new learning or training method suggested based on the user's current skill level and learning progress.

[1257] "Additional Learning Materials" are new educational content recommended for users to further their learning.

[1258] "Factory workers" are employees who work in factories using robots.

[1259] "Skills practice" is practical training to improve specific work skills.

[1260] A "smartphone application" is a software program that runs on a smartphone and can be operated by a user through an interface.

[1261] "Real-time" means immediate response to user operations and inputs.

[1262] "Generative AI model technology" refers to artificial intelligence technology that generates optimal answers and feedback based on large amounts of data.

[1263] "Answer patterns" refer to the tendencies and methods of answering questions that a user has used in the past.

[1264] A "robot" is a mechanical device that can automatically perform programmed tasks.

[1265] This invention is an educational support system that allows factory workers to effectively practice their skills using robots. The system is composed of a server, a smartphone, and a factory robot as its main components. The functions of each component and their specific implementation methods are described below.

[1266] Authenticating Users

[1267] The server receives the user's authentication information and performs authentication by checking it against the database. If authentication is successful, the server reads the user's learning data and stores the user ID in the session. The smartphone application receives the authentication result, and if authentication is successful, displays the dashboard.

[1268] Generate a customized set of work tasks

[1269] When a user selects a specific work task (e.g., welding or assembly) from the dashboard, the smartphone sends that information to the server. Based on the received selection, the server analyzes the user's past learning data and answer patterns to generate a customized work task set. This task set is sent to the smartphone and used as a guide for the user to start working.

[1270] Generate answers and feedback

[1271] When a user performs a task and enters the answer into their smartphone, the smartphone sends the information to the server in real time. The server analyzes the received answer and determines whether it is correct or incorrect. Furthermore, using generative AI model technology, the server generates personalized feedback based on the user's answer and past learning data and sends it to the smartphone. The feedback includes important hints and specific advice to help the user improve their work.

[1272] Evaluating progress and suggesting new learning plans

[1273] The server stores the user's answers and learning history, evaluates their progress, and generates a new learning plan and additional learning materials to guide the user to the next step in their learning, which are then sent to the user's smartphone.

[1274] Hardware and software used

[1275] Hardware

[1276] Smartphone

[1277] server

[1278] Factory robots

[1279] software

[1280] Flask (a web application framework written in Python)

[1281] scikit-learn (library for evaluation calculations)

[1282] Specific examples

[1283] Below are some specific examples of how this educational support system can be used.

[1284] Log in

[1285] The user logs in by entering "user1" and "pass123" into the smartphone app.

[1286] The server verifies the authentication information and returns the username if successful.

[1287] Selecting and generating work tasks

[1288] The user selects the "welding task" on the smartphone app.

[1289] The server generates a customized set of "welding" tasks and sends them to the smartphone.

[1290] Examples: "weld_joint_a: OK", "weld_joint_b: OK", "weld_joint_c: Fail"

[1291] Task execution and feedback

[1292] The user performs the work task and inputs the results into the smartphone.

[1293] Input examples: "OK", "OK", "Fail"

[1294] The server analyzes the answers and generates feedback.

[1295] The user sees feedback on their smartphone: "Great job! Continue to the next skill step."

[1296] Prompt Sentence Examples

[1297] Using the examples below as a reference, please describe the application's processing in natural language so that it can be explained to a human.

[1298] The user logs in by entering "user1" and "pass123" on the smartphone app. If authentication is successful on the server, the user name is displayed.

[1299] Next, the user selects the "welding" task on the smartphone app, and the server generates a customized task set and sends it to the smartphone.

[1300] Example of generated task set: "weld_joint_a: OK", "weld_joint_b: OK", "weld_joint_c: Fail"

[1301] The user performs the task and enters the results into a smartphone app.

[1302] Input examples: "OK", "OK", "Fail"

[1303] The server analyzes the results and generates feedback, which is displayed on the smartphone: "Great job! Proceed to the next skill step."

[1304] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1305] Step 1: User authentication

[1306] Input: The user attempts to log in by entering their username and password into the smartphone app.

[1307] How it works: The smartphone app sends this authentication information to the server.

[1308] Data processing: The server compares the received authentication information with the data stored in the database.

[1309] Output: If authentication is successful, read the user information and save the user ID in the session. If authentication fails, generate an error message and send it to the terminal.

[1310] Step 2: View the dashboard

[1311] Input: The authentication result from the server (success or failure).

[1312] Behavior: The smartphone app displays a dashboard if authentication is successful, or an error message if authentication is unsuccessful.

[1313] Data processing: None.

[1314] Output: Display a dashboard or display an error message.

[1315] Step 3: Select a work task

[1316] Input: The user selects a specific work task (e.g., welding or assembly) from the dashboard.

[1317] Operation: The smartphone app sends the selected work task information to the server.

[1318] Data processing: None.

[1319] Output: Work task information sent to the server.

[1320] Step 4: Generate a customized set of work tasks

[1321] Input: The server receives the user's selected task information, past learning data, and answer patterns.

[1322] How it works: The server uses a generative AI model to analyze incoming data and generate a customized set of work tasks.

[1323] Data processing: Analysis of past learning data and answer patterns, and generation of task sets using a generative AI model.

[1324] Output: The generated customized task set is sent to your smartphone.

[1325] Step 5: Performing work tasks

[1326] Input: A customized set of work tasks sent from the server.

[1327] Action: The user performs the work task displayed on the smartphone app and enters the results into the app.

[1328] Data processing: None.

[1329] Output: User's answer result.

[1330] Step 6: Analyze answers and generate feedback

[1331] Input: User answer results sent from the smartphone app.

[1332] How it works: The server receives and analyzes the answers, uses a generative AI model to determine whether the answer is correct, and generates personalized feedback.

[1333] Data processing: Analysis of answer results and generation of feedback using a generative AI model.

[1334] Output: The generated feedback is sent to a smartphone app.

[1335] Step 7: View your feedback

[1336] Input: Feedback sent by the server.

[1337] What it does: The smartphone app displays the received feedback to the user.

[1338] Data processing: None.

[1339] Output: Display feedback.

[1340] Step 8: Evaluate progress and propose a new learning plan

[1341] Input: All of the user's answers and learning history.

[1342] How it works: The server analyzes this data, assesses progress, and uses generative AI models and algorithms to generate new learning plans and additional learning materials.

[1343] Data processing: Saving answer results and learning history, evaluating progress, and generating new learning plans and additional learning materials.

[1344] Output: Send new learning plans and additional learning materials to a smartphone app.

[1345] Step 9: View your learning plan

[1346] Input: New study plans and additional materials sent from the server.

[1347] How it works: The smartphone app displays the plan in place.

[1348] Data processing: None.

[1349] Output: Display of new learning plans and additional learning materials.

[1350] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1351] The present invention provides an educational support system for providing an optimal educational experience to each student. The system includes a means for receiving user authentication information and authenticating it against a database, a means for analyzing the user's past learning data and generating a customized problem set, a means for analyzing the user's answers and generating individualized feedback, a means for saving the user's answers and learning history and evaluating the user's progress, and a means for suggesting new learning plans and additional learning materials. The system further includes an emotion engine that recognizes the user's emotions and generates feedback based on the user's emotional state to improve learning efficiency.

[1352] System processing flow

[1353] Authenticating Users

[1354] 1. Device:

[1355] The user opens the login screen, enters their username and password, and clicks the "Login" button.

[1356] 2. Server:

[1357] Authentication information received from the terminal is verified against the database for authentication. If authentication is successful, the user's learning data is read and the user ID is saved in the session. If authentication fails, an error message is generated and sent to the terminal.

[1358] 3. Terminal:

[1359] Receives authentication results from the server, and if authentication is successful, displays the dashboard. If authentication fails, displays an error message.

[1360] Selecting learning modules and generating problem sets

[1361] 4. User:

[1362] Select the module you want to study from your dashboard.

[1363] 5. Terminal:

[1364] The user's selection information is sent to the server.

[1365] 6. Server:

[1366] Based on the received selection information, the system analyzes the user's past learning data, generates a customized question set, and transmits it to the terminal.

[1367] 7. Terminal:

[1368] The problem set received from the server is displayed to the user.

[1369] Answers and feedback

[1370] 8. Users:

[1371] Enter the answer to the question that appears.

[1372] 9. Terminal:

[1373] The user's answers are sent to the server in real time.

[1374] 10. Server:

[1375] The system analyzes the received answers and determines whether they are correct or not. It also uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[1376] 11. Terminal:

[1377] Display feedback received from the server to the user.

[1378] Evaluating progress and suggesting new learning plans

[1379] 12. Server:

[1380] It stores all of the user's answers and learning history, evaluates their progress, and generates and sends new learning plans and additional learning materials to the device.

[1381] 13. Terminal:

[1382] Display new learning plans and additional learning materials received from the server to the user.

[1383] Introducing the Emotion Engine

[1384] The emotion engine has the ability to recognize the user's emotional state by analyzing the user's facial expressions, tone of voice, and other biometric information. Using this information, the following process is carried out to improve learning efficiency.

[1385] Emotion recognition and feedback with emotion engine

[1386] 1. Device:

[1387] Emotional information such as the user's facial expressions and tone of voice is collected through a camera and microphone.

[1388] 2. Server:

[1389] The emotion engine analyzes this emotion information and identifies the user's emotional state (e.g., excitement, fatigue, confusion, etc.).

[1390] 3. Server:

[1391] It generates feedback based on the emotional state obtained from the emotion engine. For example, if the user is tired, it generates feedback such as "Take a short break."

[1392] 4. Terminal:

[1393] Feedback based on the emotional state received from the server is displayed to the user.

[1394] Responding to emotional states

[1395] The emotional engine can adjust the difficulty and format of questions based on the user's emotional state. For example, if the user is confused, it can simplify the questions or provide hints. The emotional state obtained from the emotional engine is also added to the training data and reflected in the assessment of progress and comprehension.

[1396] Specific examples

[1397] Example 1: Math problems and emotional feedback

[1398] 1. Device:

[1399] A user selects a mathematics module and requests an entry-level problem set.

[1400] 2. Server:

[1401] A customized problem set including basic arithmetic operations is generated from past learning data and sent to the device.

[1402] 3. Terminal:

[1403] A user answers "4" to the question "2 + 2 = ?" and sends the answer to the server.

[1404] 4. Server:

[1405] The answer is analyzed, confirmed to be correct, and feedback is generated saying, "That's correct. Please proceed to the next question."

[1406] 5. Terminal:

[1407] Display feedback received from the server to the user.

[1408] 6. Terminal (Emotion Engine):

[1409] The camera analyzes the user's facial expressions and detects when the user is concentrating.

[1410] 7. Server (Emotion Engine):

[1411] It checks the user's concentration and generates feedback to encourage them to continue learning.

[1412] 8. Terminal:

[1413] Display feedback received from the server to the user.

[1414] In this way, the system of the present invention provides an optimal learning experience in real time based on the user's individual needs and level of understanding. Furthermore, the introduction of an emotion engine enables customization based on the user's emotional state, further improving learning efficiency. This results in improved learning outcomes for students, and allows teachers and parents to easily track progress.

[1415] The processing flow will be explained below.

[1416] System processing flow

[1417] Authenticating Users

[1418] Step 1:

[1419] Terminal: The user opens the login screen, enters their username and password, and clicks the "Login" button.

[1420] Step 2:

[1421] Server: Authenticates the user by checking the authentication information received from the terminal against the database. If authentication is successful, reads the user's learning data and stores the user ID in the session. If authentication fails, generates an error message and sends it to the terminal.

[1422] Step 3:

[1423] Terminal: Receives authentication result from the server, and displays the dashboard if authentication is successful. If authentication fails, displays an error message.

[1424] Selecting learning modules and generating problem sets

[1425] Step 4:

[1426] Users: Select the module they want to study from the dashboard.

[1427] Step 5:

[1428] Terminal: Sends the user's selection information to the server.

[1429] Step 6:

[1430] Server: Based on the received selection information, the server analyzes the user's past learning data, generates a customized problem set, and sends it to the device.

[1431] Step 7:

[1432] Terminal: Displays the problem set received from the server to the user.

[1433] Answers and feedback

[1434] Step 8:

[1435] User: Enter the answer to the question displayed.

[1436] Step 9:

[1437] Terminal: Sends the user's answers to the server in real time.

[1438] Step 10:

[1439] Server: Analyzes the received answers and determines whether they are correct. Furthermore, it uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[1440] Step 11:

[1441] Terminal: Displays the feedback received from the server to the user.

[1442] Evaluating progress and suggesting new learning plans

[1443] Step 12:

[1444] Server: Stores all user answers and learning history, evaluates progress, and generates new learning plans and additional learning materials and sends them to the device.

[1445] Step 13:

[1446] Terminal: Displays new learning plans and additional learning materials received from the server to the user.

[1447] Introducing the Emotion Engine

[1448] Emotion recognition and feedback with emotion engine

[1449] Step 14:

[1450] Device: Collects emotional information such as the user's facial expressions and tone of voice through a camera and microphone.

[1451] Step 15:

[1452] Server: The emotion engine analyzes this emotion information and identifies the user's emotional state (e.g., excitement, fatigue, confusion, etc.).

[1453] Step 16:

[1454] Server: Generates feedback based on the emotional state obtained from the emotion engine. For example, if the user is tired, it generates feedback such as "Take a short break."

[1455] Step 17:

[1456] Terminal: Displays feedback to the user based on the emotional state received from the server.

[1457] Responding to emotional states

[1458] Step 18:

[1459] Server: Using the emotion engine, adjust the difficulty and format of questions according to the user's emotional state. For example, if the user is confused, simplify the question or provide hints.

[1460] Step 19:

[1461] Server: The emotional state obtained from the emotion engine is added to the training data and reflected in the assessment of progress and understanding.

[1462] Specific examples

[1463] Example 1: Math problems and emotional feedback

[1464] Step 1:

[1465] Terminal: The user selects a mathematics module and requests an entry-level problem set.

[1466] Step 2:

[1467] Server: Generates a customized problem set, including basic arithmetic operations, from past learning data and sends it to the device.

[1468] Step 3:

[1469] Terminal: The user answers "4" to the question "2 + 2 = ?" and sends the answer to the server.

[1470] Step 4:

[1471] Server: Analyzes the answer, verifies it is correct, and generates feedback such as "That's correct. Please proceed to the next question."

[1472] Step 5:

[1473] Terminal: Displays the feedback received from the server to the user.

[1474] Step 6:

[1475] Terminal (emotion engine): The camera analyzes the user's facial expressions and detects when the user is concentrating.

[1476] Step 7:

[1477] Server (emotion engine): Checks the user's concentration and generates feedback to encourage them to continue learning.

[1478] Step 8:

[1479] Terminal: Displays the feedback received from the server to the user.

[1480] In this way, the system of the present invention provides an optimal learning experience in real time based on the user's individual needs and level of understanding. Furthermore, the introduction of an emotion engine enables customization based on the user's emotional state, further improving learning efficiency. This results in improved learning outcomes for students, and allows teachers and parents to easily track progress.

[1481] Example 2

[1482] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1483] Conventional educational support systems have struggled to provide an optimized educational experience for each individual user. In particular, it has been difficult to grasp a user's learning situation in real time and provide feedback based on that understanding, and customization to accommodate emotional states has been insufficient. As a result, learners are unable to properly understand their own learning situation, resulting in a decline in learning effectiveness.

[1484] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving user authentication information and verifying it against a database for authentication, means for analyzing the user's past learning data and generating a customized problem set, means for analyzing the user's answers and generating individual feedback, means for saving the user's answers and learning history and evaluating the user's progress, means for suggesting new learning plans and additional learning materials, means for collecting and analyzing the user's emotional information and generating feedback based on the user's emotional state, means for transmitting the user's answers in real time, and means for displaying the generated feedback to the user. This makes it possible to provide an optimal educational experience in real time based on the user's individual learning situation and emotional state.

[1485] "Means for receiving user authentication information and verifying it against a database" refers to the process of receiving user authentication information (such as username and password) sent from the terminal and comparing that information with the registered information in a database to verify the user's legitimacy.

[1486] "Means for analyzing a user's past learning data and generating a customized problem set" refers to the process of analyzing a user's learning history and achievements and generating individually optimized learning problems based on that.

[1487] The "means for analyzing the user's answers and generating personalized feedback" is a process for analyzing the answers submitted by the user and generating feedback that provides specific guidance based on the results.

[1488] "Means for saving the user's answers and learning history and evaluating the progress" refers to the process of recording the correctness of the user's answers and learning history, and evaluating the user's learning progress and level of understanding based on this.

[1489] The "means for suggesting new study plans and additional learning materials" is a process for providing the user with the next study content and supplementary learning materials based on the user's evaluation results.

[1490] "Means for generating feedback based on a user's emotional state by collecting and analyzing the user's emotional information" refers to a process of using a camera or microphone to collect the user's facial expressions and tone of voice, analyzing the user's emotional state based on that information, and providing appropriate feedback.

[1491] "Means for transmitting user answer results in real time" refers to a process in which the results are immediately transmitted to the server each time a user inputs an answer.

[1492] The "means for displaying the generated feedback to the user" is a process for displaying the feedback sent from the server on the screen of the terminal and providing it to the user.

[1493] The present invention is an educational support system for providing an optimal educational experience to each student. The system includes means for receiving user authentication information, authenticating the user by checking it against a database, analyzing the user's past learning data to generate a customized problem set, and analyzing the user's answers to generate personalized feedback. The system also stores the user's answers and learning history, evaluates the user's progress, and suggests new learning plans and additional learning materials. The system also incorporates an emotion engine for collecting and analyzing the user's emotional information and generating feedback based on the user's emotional state.

[1494] The entire system functions through the interaction of three parties: the server, the terminal, and the user. The specific configuration and operation will be explained below.

[1495] First, the user opens a login screen using the terminal and enters their username and password. This login information is sent to the server using JavaScript. The server uses the Python Flask framework to check the authentication information against a database (e.g., MySQL). If authentication is successful, the server loads the user's learning data and saves the user ID in the session. If authentication fails, a response including an error message is sent to the terminal. Based on the response from the server, the terminal displays a dashboard if the authentication was successful, or an error message if the authentication failed.

[1496] Next, the user selects the module they wish to learn from the dashboard. This selection information is sent to the server via JavaScript. Based on the received selection information, the server analyzes the user's past learning data using machine learning libraries such as scikit-learn. As a result, a problem set optimized for the user is generated and sent to the device. The device then dynamically generates the received problem set using HTML and JavaScript and displays it to the user.

[1497] When a user enters an answer to a question and clicks the submit button, the device sends the answer information to the server in real time. The server uses Python to analyze the answer and determine whether it is correct. It then uses AI technologies such as TensorFlow and PyTorch to generate individual feedback based on the analysis results and past learning data and sends it to the device. The device then displays the received feedback to the user.

[1498] The server stores all of the user's answers and learning history in a database. The server evaluates the user's progress based on this information. Based on the progress and evaluation results, it generates new learning plans and additional learning materials and sends them in JSON format to the device. The device then displays the received plans and learning materials to the user.

[1499] Furthermore, a means for recognizing the user's emotional state is incorporated using an emotion engine. The device collects the user's facial expressions and tone of voice through a camera and microphone, and analyzes them using an emotion engine (e.g., OpenCV or the Facial Emotion Recognition (FER) library). Based on the analysis results, the server generates feedback according to the user's emotional state (e.g., "Let's take a short break") and sends it to the device. The device then displays this feedback to the user.

[1500] Examples of concrete examples and prompts

[1501] As a concrete example, let's consider an operation in which a user selects a mathematics module and requests a set of beginner-level arithmetic problems. The server generates problems such as "2 + 2 = ?" based on past learning data and sends them to the device. If the user answers "4," the server determines that this is the correct answer and generates feedback saying, "That's correct. Please proceed to the next problem." The device displays this feedback, and also recognizes the user's state of concentration via the emotion engine, generating and displaying feedback encouraging the user to continue learning.

[1502] Examples of prompts might include "Design a system that returns analysis results in real time after the user enters their answer" or "Explain the process of generating a customized problem set and displaying it to the user."

[1503] In this way, the educational support system according to the present invention can provide an optimal educational experience in real time based on the user's learning situation and emotional state.

[1504] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1505] Step 1: Authenticate the user

[1506] 1.1 Terminal:

[1507] You open a login screen, enter your username and password, and click the "Login" button. The computer program sends this information to the server using JavaScript.

[1508] Input: Username, Password

[1509] Output: Authentication information sent to the server

[1510] 1.2 Server:

[1511] The authentication information received from the device is checked against a database (e.g., MySQL). The Python Flask framework is used to compare the authentication information with the database, and if a match is found, authentication is successful. The user ID is stored in the session and training data is loaded. If authentication fails, an error message is generated.

[1512] Input: Authentication information (user name, password)

[1513] Output: Authentication result (success / failure), user data (if authentication is successful)

[1514] 1.3 Terminal:

[1515] Receives the authentication result from the server and displays the dashboard if successful, or an error message if unsuccessful.

[1516] Input: Authentication result (success / failure)

[1517] Output: Dashboard (if successful), Error message (if unsuccessful)

[1518] Step 2: Selecting a learning module and generating a problem set

[1519] 2.1 User:

[1520] From the dashboard, select the module you want to study. Your selection is collected using JavaScript.

[1521] Input: Module selection information

[1522] Output: Selections sent to the server

[1523] 2.2 Terminal:

[1524] Send the selection to the server. The selection is sent asynchronously using an AJAX request.

[1525] Input: Selection information

[1526] Output: Selections sent to the server

[1527] 2.3 Server:

[1528] Receives the selection information and analyzes the user's past training data. Analyzes the training data using Python and scikit-learn to generate a customized problem set. The generated problem set is sent to the device in JSON format.

[1529] Input: Selection information, past learning data

[1530] Output: Problem set in JSON format

[1531] 2.4 Terminal:

[1532] The problem set received from the server is displayed to the user. Problems are dynamically generated using JavaScript and HTML and displayed on the screen.

[1533] Input: Problem set (JSON format)

[1534] Output: The displayed problem set

[1535] Step 3: Answers and feedback

[1536] 3.1 User:

[1537] Enter your answer to the question and click the "Submit" button. The answer will be entered into the form and collected by JavaScript.

[1538] Input: Answer

[1539] Output: Answer sent to the server

[1540] 3.2 Terminal:

[1541] Send the user's answers to the server in real time using an AJAX request.

[1542] Input: Answer

[1543] Output: Answer sent to the server

[1544] 3.3 Server:

[1545] The received answers are analyzed and judged to be correct (using Python), and then individual feedback is generated using AI technologies such as TensorFlow and PyTorch and sent to the device.

[1546] Input: Answers, past learning data

[1547] Output: Individual feedback

[1548] 3.4 Terminal:

[1549] Display the feedback received from the server to the user. Use JavaScript to display the feedback message on the screen.

[1550] Input: Feedback

[1551] Output: Displayed feedback

[1552] Step 4: Evaluate progress and propose a new learning plan

[1553] 4.1 Server:

[1554] All user answers and learning history are stored in a database, and progress is evaluated. Based on the evaluation results, new learning plans and additional learning materials are generated and sent to the device in JSON format.

[1555] Input: Answer results, learning history

[1556] Output: New learning plan, additional materials (JSON format)

[1557] 4.2 Terminal:

[1558] Display new learning plans and additional learning materials received from the server to the user. Plans and learning materials are dynamically generated using JavaScript and HTML and displayed on the screen.

[1559] Input: New learning plan, additional materials (JSON format)

[1560] Output: New learning plan displayed, additional materials

[1561] Step 5: Emotion recognition and feedback by the emotion engine

[1562] 5.1 Terminal:

[1563] Emotional information such as facial expressions and tone of voice of the user is collected through a camera and microphone. OpenCV and Python libraries are used for the emotion engine.

[1564] Input: facial expression data, voice data

[1565] Output: Emotion information sent to the server

[1566] 5.2 Server:

[1567] The emotion engine analyzes the emotion information and identifies the user's emotional state. Based on the analysis results, it generates appropriate feedback and sends it to the device.

[1568] Input: Emotion information

[1569] Output: Feedback (based on emotional state)

[1570] 5.3 Terminal:

[1571] Display feedback to the user based on the emotional state received from the server. Use JavaScript to display the feedback message on the screen.

[1572] Input: Feedback (based on emotional state)

[1573] Output: Displayed feedback

[1574] (Application example 2)

[1575] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1576] Conventional education and training support systems have difficulty adapting flexibly to the learning progress and emotional state of individual users and workers, and have often provided uniform learning plans and training procedures. As a result, optimal feedback and training support tailored to each individual's level of understanding and emotional state has not been provided, leading to a decline in learning efficiency and work efficiency.

[1577] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving a user's authentication information and verifying it against a database for authentication; means for analyzing the user's past learning data and generating a customized problem set; means for analyzing the user's answers and generating individual feedback; means for saving the user's answers and learning history and evaluating progress; means for suggesting new learning plans and additional learning materials; means for analyzing the worker's past training data and presenting customized work procedures; means including an emotion engine for recognizing emotional states and generating feedback; and means for evaluating progress and suggesting the next training or work procedures. This makes it possible to provide each user or worker with an optimal learning and training experience tailored to their learning situation and emotional state.

[1578] "User authentication information" refers to information used to identify a person accessing a system and verify their identity.

[1579] A "database" is a collection of information used to store and manage user authentication information, learning data, and training data.

[1580] A "customized problem set" is a collection of problems optimized for each individual user, generated by analyzing the user's past learning data and answer patterns.

[1581] "Individual feedback" is advice and evaluation specific to a user that is generated based on the user's answers and progress.

[1582] "Progress" is information that indicates the progress and results of a user's learning or training.

[1583] "New study plans and additional learning materials" are new study plans and supplementary learning materials that are suggested based on the user's progress and level of understanding.

[1584] A "work procedure" is a series of steps or methods for accomplishing a particular task.

[1585] An "emotion engine" is a system that analyzes emotional information such as a user's facial expression and tone of voice to recognize the user's emotional state.

[1586] "Feedback" refers to evaluations and advice provided in response to a user's actions or input.

[1587] "Training data" refers to data that includes records of training that workers have previously undergone and the learning outcomes based on that training.

[1588] The present invention is a real-time training support system for factory workers, which uses smart glasses to provide real-time assistance to workers while they are working. Detailed description of the embodiments of the present invention will be given below.

[1589] First, the system includes the following hardware and software components:

[1590] Hardware: Smart glasses (camera, display)

[1591] software:

[1592] OpenCV: Camera image capture and display

[1593] Dlib: A face detection and recognition model

[1594] EmotionRecognizer: A model for recognizing emotional states

[1595] Data Management Module: Acquire and suggest customized training data

[1596] Server communication module: Sending and receiving feedback

[1597] System processing flow

[1598] 1. User authentication:

[1599] A worker puts on the smart glasses and logs in to the system. The user's authentication information (username and password) is sent from the smart glasses device to the server and checked against the database. If authentication is successful, the worker's training data is loaded and the session begins.

[1600] 2. Providing training procedures:

[1601] Based on the worker's past training data, customized work instructions are generated and displayed on the smart glasses' display, and the instructions are adjusted based on each worker's skill level and past performance.

[1602] 3. Recognition of emotional states:

[1603] A camera mounted on the smart glasses captures the worker's facial expressions, which are then analyzed by EmotionRecognizer, which identifies the worker's emotional state, such as whether they are tired, focused, or confused.

[1604] 4. Generate personalized feedback:

[1605] Based on the recognized emotional state, the server generates appropriate feedback in real time, such as "Take a break" if the worker is tired, or "Revisit the procedure" if the worker is confused.

[1606] 5. Progress evaluation and training plan suggestions:

[1607] The server stores all the worker's training data and evaluates their progress, generating new training plans and additional work instructions to prepare them for the next job.

[1608] Specific examples

[1609] For example, when a worker is tightening a nut, smart glasses will display the work instructions and analyze the worker's facial expressions using a facial recognition camera. If the smart glasses detect that the worker is tired, they will provide feedback such as "Take a short break." If the glasses detect that the worker is confused by the work instructions, they will again provide specific instructions such as "Turn the screw clockwise."

[1610] Example prompts for generative AI models

[1611] "Generate optimal feedback when workers feel fatigued during work."

[1612] "Propose procedural support for workers if they are confused."

[1613] In this way, the system of the present invention can improve work efficiency and safety by providing optimal support in real time according to the condition of each individual worker.

[1614] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1615] Step 1:

[1616] The user puts on the smart glasses and enters their username and password on the login screen. The input information is sent from the device to the server. The server compares the received authentication information with a database and performs authentication. If authentication is successful, the worker's training data is read and the session begins. If authentication fails, an error message is sent to the device and displayed.

[1617] Step 2:

[1618] After the worker is authenticated, he or she selects a training module. The terminal sends the selection information to the server. The server analyzes the worker's past training data based on the received selection information. Based on the analysis results, the server generates a customized work procedure and sends it to the terminal. The terminal displays the generated work procedure on the display of the smart glasses.

[1619] Step 3:

[1620] A camera mounted on the smart glasses captures the worker's facial expressions. The captured video is sent from the device to a server. The server then uses an emotion engine (EmotionRecognizer) to analyze the video data and recognize the worker's emotional state (fatigue, concentration, confusion, etc.). The recognized emotional state is then used to evaluate progress.

[1621] Step 4:

[1622] The server generates appropriate feedback in real time based on the recognized emotional state. For example, if fatigue is recognized, feedback such as "Take a short break" is generated. The generated feedback is sent to the device and displayed on the smart glasses display.

[1623] Step 5:

[1624] The server stores all of the worker's training data and evaluates their progress. The evaluation results are used to generate the next training or additional work procedures. The server generates new training plans or additional work procedures and sends them to the device. The device displays the new training plans or additional work procedures on the smart glasses display.

[1625] Step 6:

[1626] While the worker is performing additional work steps, the camera in the smart glasses again captures their facial expressions and sends them to the server, which then uses the emotion engine to analyze their emotional state again and updates the feedback as needed, ensuring optimal support is always provided based on the worker's condition.

[1627] This allows users to receive optimal feedback and work procedures in real time based on their individual learning situation and emotional state.

[1628] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1629] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1630] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1631] [Fourth embodiment]

[1632] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1633] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1634] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1635] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1636] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1637] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1638] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1639] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1640] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1641] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1642] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1643] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1644] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1645] The present invention provides an educational support system for providing an optimal educational experience to each student. The system includes means for receiving user authentication information and authenticating the user by comparing it with a database, means for analyzing the user's past learning data and generating a customized problem set, means for analyzing the user's answers and generating individualized feedback, means for storing the user's answers and learning history and evaluating the user's progress, and means for suggesting new learning plans and additional learning materials.

[1646] System processing flow

[1647] Authenticating Users

[1648] 1. Device:

[1649] The user opens the login screen, enters their username and password, and clicks the "Login" button.

[1650] 2. Server:

[1651] Authentication information received from the terminal is verified against the database for authentication. If authentication is successful, the user's learning data is read and the user ID is saved in the session. If authentication fails, an error message is generated and sent to the terminal.

[1652] 3. Terminal:

[1653] Receives authentication results from the server, and if authentication is successful, displays the dashboard. If authentication fails, displays an error message.

[1654] Selecting learning modules and generating problem sets

[1655] 4. User:

[1656] Select the module you want to study from your dashboard.

[1657] 5. Terminal:

[1658] The user's selection information is sent to the server.

[1659] 6. Server:

[1660] Based on the received selection information, the system analyzes the user's past learning data, generates a customized question set, and transmits it to the terminal.

[1661] 7. Terminal:

[1662] The problem set received from the server is displayed to the user.

[1663] Answers and feedback

[1664] 8. Users:

[1665] Enter the answer to the question that appears.

[1666] 9. Terminal:

[1667] The user's answers are sent to the server in real time.

[1668] 10. Server:

[1669] The system analyzes the received answers and determines whether they are correct or not. It also uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[1670] 11. Terminal:

[1671] Display feedback received from the server to the user.

[1672] Evaluating progress and suggesting new learning plans

[1673] 12. Server:

[1674] It stores all of the user's answers and learning history, evaluates their progress, and generates and sends new learning plans and additional learning materials to the device.

[1675] 13. Terminal:

[1676] Display new learning plans and additional learning materials received from the server to the user.

[1677] Specific examples

[1678] Example 1: Math Problems

[1679] 1. Device:

[1680] A user selects a mathematics module and requests an entry-level problem set.

[1681] 2. Server:

[1682] A customized problem set including basic arithmetic operations is generated from past learning data and sent to the device.

[1683] 3. Terminal:

[1684] A user answers "4" to the question "2 + 2 = ?" and sends the answer to the server.

[1685] 4. Server:

[1686] The answer is analyzed, confirmed to be correct, and feedback is sent to the device saying, "That's correct. Please proceed to the next question."

[1687] 5. Terminal:

[1688] Display feedback received from the server to the user.

[1689] In this way, the system of the present invention provides an optimal learning experience in real time based on the user's individual needs and level of understanding, improving students' learning efficiency and allowing teachers and parents to easily track progress.

[1690] The processing flow will be explained below.

[1691] Step 1:

[1692] Terminal: The user opens the login screen, enters their username and password, and clicks the "Login" button.

[1693] Step 2:

[1694] Server: Authenticates the user by checking the authentication information received from the terminal against the database. If authentication is successful, reads the user's learning data and stores the user ID in the session. If authentication fails, generates an error message and sends it to the terminal.

[1695] Step 3:

[1696] Terminal: Receives authentication result from the server, and displays the dashboard if authentication is successful. If authentication fails, displays an error message.

[1697] Step 4:

[1698] Users: Select the module they want to study from the dashboard.

[1699] Step 5:

[1700] Terminal: Sends the user's selection information to the server.

[1701] Step 6:

[1702] Server: Based on the received selection information, the server analyzes the user's past learning data, generates a customized problem set, and sends it to the device.

[1703] Step 7:

[1704] Terminal: Displays the problem set received from the server to the user.

[1705] Step 8:

[1706] User: Enter the answer to the question displayed.

[1707] Step 9:

[1708] Terminal: Sends the user's answers to the server in real time.

[1709] Step 10:

[1710] Server: Analyzes the received answers and determines whether they are correct. Furthermore, it uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[1711] Step 11:

[1712] Terminal: Displays the feedback received from the server to the user.

[1713] Step 12:

[1714] Server: Stores all user answers and learning history, evaluates progress, and generates new learning plans and additional learning materials and sends them to the device.

[1715] Step 13:

[1716] Terminal: Displays new learning plans and additional learning materials received from the server to the user.

[1717] Step 14:

[1718] User: Proceed to the next learning step based on the new learning plan or additional materials.

[1719] Example 1

[1720] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1721] Conventional educational support systems face the challenge of providing an educational experience tailored to each user's individual learning needs and level of understanding. Specifically, they lack the functionality to effectively analyze a user's past learning data to generate optimal problem sets or provide individually customized feedback in real time. This makes it difficult to improve learning efficiency and accurately evaluate progress.

[1722] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1723] In this invention, the server includes means for receiving the user's authentication information and verifying it against a database, means for analyzing the user's past learning data to generate a customized problem set, and means for analyzing the user's answers to generate individual feedback, thereby providing an optimal educational experience based on the user's individual needs and level of understanding, improving learning efficiency, and enabling accurate assessment of progress.

[1724] "User authentication information" refers to information such as a username and password that a user enters when accessing a system.

[1725] A "database" refers to a storage device for systematically storing and managing user authentication information, learning data, answer results, etc.

[1726] "Matching" refers to the process of comparing received information with existing data to see if there is a match.

[1727] A "customized problem set" refers to a collection of study questions that are individually optimized based on the user's past study data and answer patterns.

[1728] "Analyzing answers" refers to the process of evaluating the answers entered by the user and determining whether they are correct or incorrect.

[1729] "Individual feedback" refers to study advice and comments for individual users that are generated based on the user's answer results and past study data.

[1730] "Storage" refers to the act of recording data on a storage device so that it can be used at a later time.

[1731] "Progress assessment" refers to the process of analyzing a user's learning history and answer results to determine their current level of understanding and progress.

[1732] "New learning plan" refers to a plan that suggests future learning content and methods based on the user's progress.

[1733] "Additional Learning Materials" refers to additional learning materials or resources provided to assist the user in their learning.

[1734] "Terminal" refers to an electronic device used by a user to access and operate the system.

[1735] "Computing device" refers to an electronic device for analyzing data and performing calculations.

[1736] "AI technology" refers to technology that uses artificial intelligence to analyze, predict, optimize, and perform other functions on data.

[1737] "Learning history" refers to a record of the user's learning activities, answer results, and feedback.

[1738] The present invention provides an educational support system for providing an optimal educational experience to each student. The system includes means for receiving user authentication information and authenticating the user by comparing it with a database, means for analyzing the user's past learning data and generating a customized problem set, means for analyzing the user's answers and generating individualized feedback, means for storing the user's answers and learning history and evaluating the user's progress, and means for suggesting new learning plans and additional learning materials.

[1739] The basic configuration of the entire system is as follows:

[1740] Receive user credentials: When a user accesses the system, they enter their username and password on a login screen, which is sent to the server and checked against a MySQL database.

[1741] Authentication by checking against database: The server uses a MySQL database to check the entered authentication information against the information in the database. If authentication is successful, the user ID is saved in the session and a successful authentication response is sent to the terminal.

[1742] Analyzing the user's past learning data: Based on the received selection information, the server analyzes the user's learning history using Python and SciPy libraries to generate a customized problem set.

[1743] Generate customized problem sets: Based on the module selected by the user, the server generates an optimal problem set based on the user's past learning data. This problem set is sent to the terminal and displayed to the user.

[1744] Analyzing user answers: When a user enters an answer to a question, the answer is sent to the server in real time. The server analyzes the received answer and determines whether it is correct or incorrect.

[1745] Generate personalized feedback: Based on the received answer results, the server uses AI technology (e.g., a generative AI model) to generate personalized feedback and send it to the device.

[1746] Save user's answer results and learning history: All user's answer results and learning history are saved in cloud storage such as AWS S3.

[1747] Progress evaluation: The server analyzes the stored data using Apache Spark and evaluates the user's progress. Based on the evaluation results, a new learning plan and additional learning materials are generated and sent to the device.

[1748] The following flow can be considered as a specific example of how this system can be used.

[1749] 1. Terminal: The user selects the mathematics module and requests a beginner level problem set (clicks the "Math" button, sets the level to "Beginner" and presses the "Start" button).

[1750] 2. Server: Reads past training data from a MySQL database, generates a customized problem set including basic arithmetic problems using Python and SciPy, and sends it to the device.

[1751] 3. Terminal: Displays the problem set received from the server to the user (for example, displays the problem "2 + 2 = ?").

[1752] 4. User: Answers "4" to the displayed question "2 + 2 = ?" and clicks the button to submit the answer.

[1753] 5. Terminal: The user's answer "4" is sent to the server in real time.

[1754] 6. Server: Analyzes the received answer and verifies that it is correct. Using the generative AI model, it generates feedback such as "That's correct. Please proceed to the next question." and sends it to the device.

[1755] 7. Terminal: Displays the feedback received from the server to the user.

[1756] Examples of prompts for a generative AI model include:

[1757] "You are an AI teaching assistant. Create a customized set of math problems for a beginner student who has previously struggled with basic arithmetic (addition and subtraction)."

[1758] In this way, the system provides users with an optimal educational experience based on their individual needs and level of understanding, thereby improving their learning efficiency.

[1759] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1760] Step 1:

[1761] Terminal: The user opens a login screen and enters their username and password. When they click the "Login" button, the information they entered is sent to the server. The input is the username and password, and the output is the transmission of authentication information to the server.

[1762] Step 2:

[1763] Server: Receives the sent authentication information. Checks the received data against the database. Specifically, it compares the entered username and password with the user information stored in the MySQL database. If authentication is successful, it saves the user ID in the session and generates a response indicating successful authentication. If authentication fails, it generates an error message. The input is authentication information and the output is the authentication result.

[1764] Step 3:

[1765] Terminal: Receives authentication results from the server. If authentication is successful, displays the user's dashboard. If authentication fails, displays an error message. The input is the authentication result from the server, and the output is displaying the dashboard or an error message.

[1766] Step 4:

[1767] User: Selects the module they wish to learn from the list of learning modules displayed on the dashboard. The user's selection operation sends the selection information from the terminal to the server. The input is the selection of the learning module, and the output is the transmission of the selection information to the server.

[1768] Step 5:

[1769] Server: Based on the received selection information, it reads past learning data from the database. It uses Python and the SciPy library to analyze the user's learning history. Based on this analysis, it generates a customized problem set. The generated problem set is sent to the device. The input is the selection information and past learning data, and the output is a customized problem set.

[1770] Step 6:

[1771] Terminal: Displays the problem set received from the server to the user. The input is the problem set from the server, and the output is the display of the problem set.

[1772] Step 7:

[1773] User: Enters the answer to the displayed question and clicks the button to send the answer. The answer entered by the user is sent from the terminal to the server. The input is the answer to the question, and the output is the transmission of the answer to the server.

[1774] Step 8:

[1775] Server: Analyzes the received answers. First, it determines whether the answer is correct or incorrect. Next, it uses AI technology to generate individual feedback based on the user's answer and past learning data. The generated feedback is sent to the device. The input is the user's answer, and the output is the correctness determination and feedback.

[1776] Step 9:

[1777] Terminal: Receives feedback sent from the server. Displays the received feedback to the user. The input is the feedback from the server, and the output is the display of the feedback.

[1778] Step 10:

[1779] Server: All of the user's answer results and learning history are stored in cloud storage. Specifically, a storage service such as AWS S3 is used. Progress is evaluated using Apache Spark based on the stored data. Based on the evaluation results, new learning plans and additional learning materials are generated and sent to the device. The input is all answer results and learning history, and the output is the stored data and the progress evaluation results.

[1780] Step 11:

[1781] Terminal: Receives new learning plans and additional learning materials sent from the server. Displays them to the user. The input is the new learning plans and additional learning materials from the server, and the output is their display.

[1782] (Application example 1)

[1783] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1784] Conventional training support systems have had the challenge of providing real-time feedback and progress visualization specific to improving factory workers' skills. It was particularly difficult to generate appropriate problem sets and provide individualized feedback for tasks using robots in factories. Furthermore, there was a lack of methods for providing customized learning plans based on the worker's level of skill acquisition.

[1785] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1786] In this invention, the server includes means for receiving a user's authentication information and verifying it against a database for authentication, means for analyzing the user's past learning data to generate a customized work task set, and means for analyzing the user's answers to generate individual feedback. This allows factory workers to receive real-time feedback while undergoing work skill training using robots. Furthermore, the server can evaluate the user's progress and provide new learning plans and additional learning materials, thereby enabling effective skill improvement.

[1787] "User authentication information" refers to information required when a user accesses a system, and is primarily composed of a user name and password.

[1788] A "database" is a system for systematically storing and managing various information such as authentication information and learning data.

[1789] A "customized work task set" is a collection of work tasks that are individually designed based on the user's past learning data and answer patterns.

[1790] "Individual feedback" is information such as an evaluation and areas for improvement for a specific user that is generated based on the user's answers and work performance.

[1791] "Progress" refers to a state that indicates the learning results and skill acquisition level that the user has achieved so far.

[1792] A "new learning plan" is a new learning or training method suggested based on the user's current skill level and learning progress.

[1793] "Additional Learning Materials" are new educational content recommended for users to further their learning.

[1794] "Factory workers" are employees who work in factories using robots.

[1795] "Skills practice" is practical training to improve specific work skills.

[1796] A "smartphone application" is a software program that runs on a smartphone and can be operated by a user through an interface.

[1797] "Real-time" means immediate response to user operations and inputs.

[1798] "Generative AI model technology" refers to artificial intelligence technology that generates optimal answers and feedback based on large amounts of data.

[1799] "Answer patterns" refer to the tendencies and methods of answering questions that a user has used in the past.

[1800] A "robot" is a mechanical device that can automatically perform programmed tasks.

[1801] This invention is an educational support system that allows factory workers to effectively practice their skills using robots. The system is composed of a server, a smartphone, and a factory robot as its main components. The functions of each component and their specific implementation methods are described below.

[1802] Authenticating Users

[1803] The server receives the user's authentication information and performs authentication by checking it against the database. If authentication is successful, the server reads the user's learning data and stores the user ID in the session. The smartphone application receives the authentication result, and if authentication is successful, displays the dashboard.

[1804] Generate a customized set of work tasks

[1805] When a user selects a specific work task (e.g., welding or assembly) from the dashboard, the smartphone sends that information to the server. Based on the received selection, the server analyzes the user's past learning data and answer patterns to generate a customized work task set. This task set is sent to the smartphone and used as a guide for the user to start working.

[1806] Generate answers and feedback

[1807] When a user performs a task and enters the answer into their smartphone, the smartphone sends the information to the server in real time. The server analyzes the received answer and determines whether it is correct or incorrect. Furthermore, using generative AI model technology, the server generates personalized feedback based on the user's answer and past learning data and sends it to the smartphone. The feedback includes important hints and specific advice to help the user improve their work.

[1808] Evaluating progress and suggesting new learning plans

[1809] The server stores the user's answers and learning history, evaluates their progress, and generates a new learning plan and additional learning materials to guide the user to the next step in their learning, which are then sent to the user's smartphone.

[1810] Hardware and software used

[1811] Hardware

[1812] Smartphone

[1813] server

[1814] Factory robots

[1815] software

[1816] Flask (a web application framework written in Python)

[1817] scikit-learn (library for evaluation calculations)

[1818] Specific examples

[1819] Below are some specific examples of how this educational support system can be used.

[1820] Log in

[1821] The user logs in by entering "user1" and "pass123" into the smartphone app.

[1822] The server verifies the authentication information and returns the username if successful.

[1823] Selecting and generating work tasks

[1824] The user selects the "welding task" on the smartphone app.

[1825] The server generates a customized set of "welding" tasks and sends them to the smartphone.

[1826] Examples: "weld_joint_a: OK", "weld_joint_b: OK", "weld_joint_c: Fail"

[1827] Task execution and feedback

[1828] The user performs the work task and inputs the results into the smartphone.

[1829] Input examples: "OK", "OK", "Fail"

[1830] The server analyzes the answers and generates feedback.

[1831] The user sees feedback on their smartphone: "Great job! Continue to the next skill step."

[1832] Prompt Sentence Examples

[1833] Using the examples below as a reference, please describe the application's processing in natural language so that it can be explained to a human.

[1834] The user logs in by entering "user1" and "pass123" on the smartphone app. If authentication is successful on the server, the user name is displayed.

[1835] Next, the user selects the "welding" task on the smartphone app, and the server generates a customized task set and sends it to the smartphone.

[1836] Example of generated task set: "weld_joint_a: OK", "weld_joint_b: OK", "weld_joint_c: Fail"

[1837] The user performs the task and enters the results into a smartphone app.

[1838] Input examples: "OK", "OK", "Fail"

[1839] The server analyzes the results and generates feedback, which is displayed on the smartphone: "Great job! Proceed to the next skill step."

[1840] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1841] Step 1: User authentication

[1842] Input: The user attempts to log in by entering their username and password into the smartphone app.

[1843] How it works: The smartphone app sends this authentication information to the server.

[1844] Data processing: The server compares the received authentication information with the data stored in the database.

[1845] Output: If authentication is successful, read the user information and save the user ID in the session. If authentication fails, generate an error message and send it to the terminal.

[1846] Step 2: View the dashboard

[1847] Input: The authentication result from the server (success or failure).

[1848] Behavior: The smartphone app displays a dashboard if authentication is successful, or an error message if authentication is unsuccessful.

[1849] Data processing: None.

[1850] Output: Display a dashboard or display an error message.

[1851] Step 3: Select a work task

[1852] Input: The user selects a specific work task (e.g., welding or assembly) from the dashboard.

[1853] Operation: The smartphone app sends the selected work task information to the server.

[1854] Data processing: None.

[1855] Output: Work task information sent to the server.

[1856] Step 4: Generate a customized set of work tasks

[1857] Input: The server receives the user's selected task information, past learning data, and answer patterns.

[1858] How it works: The server uses a generative AI model to analyze incoming data and generate a customized set of work tasks.

[1859] Data processing: Analysis of past learning data and answer patterns, and generation of task sets using a generative AI model.

[1860] Output: The generated customized task set is sent to your smartphone.

[1861] Step 5: Performing work tasks

[1862] Input: A customized set of work tasks sent from the server.

[1863] Action: The user performs the work task displayed on the smartphone app and enters the results into the app.

[1864] Data processing: None.

[1865] Output: User's answer result.

[1866] Step 6: Analyze answers and generate feedback

[1867] Input: User answer results sent from the smartphone app.

[1868] How it works: The server receives and analyzes the answers, uses a generative AI model to determine whether the answer is correct, and generates personalized feedback.

[1869] Data processing: Analysis of answer results and generation of feedback using a generative AI model.

[1870] Output: The generated feedback is sent to a smartphone app.

[1871] Step 7: View your feedback

[1872] Input: Feedback sent by the server.

[1873] What it does: The smartphone app displays the received feedback to the user.

[1874] Data processing: None.

[1875] Output: Display feedback.

[1876] Step 8: Evaluate progress and propose a new learning plan

[1877] Input: All of the user's answers and learning history.

[1878] How it works: The server analyzes this data, assesses progress, and uses generative AI models and algorithms to generate new learning plans and additional learning materials.

[1879] Data processing: Saving answer results and learning history, evaluating progress, and generating new learning plans and additional learning materials.

[1880] Output: Send new learning plans and additional learning materials to a smartphone app.

[1881] Step 9: View your learning plan

[1882] Input: New study plans and additional materials sent from the server.

[1883] How it works: The smartphone app displays the plan in place.

[1884] Data processing: None.

[1885] Output: Display of new learning plans and additional learning materials.

[1886] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1887] The present invention provides an educational support system for providing an optimal educational experience to each student. The system includes a means for receiving user authentication information and authenticating it against a database, a means for analyzing the user's past learning data and generating a customized problem set, a means for analyzing the user's answers and generating individualized feedback, a means for saving the user's answers and learning history and evaluating the user's progress, and a means for suggesting new learning plans and additional learning materials. The system further includes an emotion engine that recognizes the user's emotions and generates feedback based on the user's emotional state to improve learning efficiency.

[1888] System processing flow

[1889] Authenticating Users

[1890] 1. Device:

[1891] The user opens the login screen, enters their username and password, and clicks the "Login" button.

[1892] 2. Server:

[1893] Authentication information received from the terminal is verified against the database for authentication. If authentication is successful, the user's learning data is read and the user ID is saved in the session. If authentication fails, an error message is generated and sent to the terminal.

[1894] 3. Terminal:

[1895] Receives authentication results from the server, and if authentication is successful, displays the dashboard. If authentication fails, displays an error message.

[1896] Selecting learning modules and generating problem sets

[1897] 4. User:

[1898] Select the module you want to study from your dashboard.

[1899] 5. Terminal:

[1900] The user's selection information is sent to the server.

[1901] 6. Server:

[1902] Based on the received selection information, the system analyzes the user's past learning data, generates a customized question set, and transmits it to the terminal.

[1903] 7. Terminal:

[1904] The problem set received from the server is displayed to the user.

[1905] Answers and feedback

[1906] 8. Users:

[1907] Enter the answer to the question that appears.

[1908] 9. Terminal:

[1909] The user's answers are sent to the server in real time.

[1910] 10. Server:

[1911] The system analyzes the received answers and determines whether they are correct or not. It also uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[1912] 11. Terminal:

[1913] Display feedback received from the server to the user.

[1914] Evaluating progress and suggesting new learning plans

[1915] 12. Server:

[1916] It stores all of the user's answers and learning history, evaluates their progress, and generates and sends new learning plans and additional learning materials to the device.

[1917] 13. Terminal:

[1918] Display new learning plans and additional learning materials received from the server to the user.

[1919] Introducing the Emotion Engine

[1920] The emotion engine has the ability to recognize the user's emotional state by analyzing the user's facial expressions, tone of voice, and other biometric information. Using this information, the following process is carried out to improve learning efficiency.

[1921] Emotion recognition and feedback with emotion engine

[1922] 1. Device:

[1923] Emotional information such as the user's facial expressions and tone of voice is collected through a camera and microphone.

[1924] 2. Server:

[1925] The emotion engine analyzes this emotion information and identifies the user's emotional state (e.g., excitement, fatigue, confusion, etc.).

[1926] 3. Server:

[1927] It generates feedback based on the emotional state obtained from the emotion engine. For example, if the user is tired, it generates feedback such as "Take a short break."

[1928] 4. Terminal:

[1929] Feedback based on the emotional state received from the server is displayed to the user.

[1930] Responding to emotional states

[1931] The emotional engine can adjust the difficulty and format of questions based on the user's emotional state. For example, if the user is confused, it can simplify the questions or provide hints. The emotional state obtained from the emotional engine is also added to the training data and reflected in the assessment of progress and comprehension.

[1932] Specific examples

[1933] Example 1: Math problems and emotional feedback

[1934] 1. Device:

[1935] A user selects a mathematics module and requests an entry-level problem set.

[1936] 2. Server:

[1937] A customized problem set including basic arithmetic operations is generated from past learning data and sent to the device.

[1938] 3. Terminal:

[1939] A user answers "4" to the question "2 + 2 = ?" and sends the answer to the server.

[1940] 4. Server:

[1941] The answer is analyzed, confirmed to be correct, and feedback is generated saying, "That's correct. Please proceed to the next question."

[1942] 5. Terminal:

[1943] Display feedback received from the server to the user.

[1944] 6. Terminal (Emotion Engine):

[1945] The camera analyzes the user's facial expressions and detects when the user is concentrating.

[1946] 7. Server (Emotion Engine):

[1947] It checks the user's concentration and generates feedback to encourage them to continue learning.

[1948] 8. Terminal:

[1949] Display feedback received from the server to the user.

[1950] In this way, the system of the present invention provides an optimal learning experience in real time based on the user's individual needs and level of understanding. Furthermore, the introduction of an emotion engine enables customization based on the user's emotional state, further improving learning efficiency. This results in improved learning outcomes for students, and allows teachers and parents to easily track progress.

[1951] The processing flow will be explained below.

[1952] System processing flow

[1953] Authenticating Users

[1954] Step 1:

[1955] Terminal: The user opens the login screen, enters their username and password, and clicks the "Login" button.

[1956] Step 2:

[1957] Server: Authenticates the user by checking the authentication information received from the terminal against the database. If authentication is successful, reads the user's learning data and stores the user ID in the session. If authentication fails, generates an error message and sends it to the terminal.

[1958] Step 3:

[1959] Terminal: Receives authentication result from the server, and displays the dashboard if authentication is successful. If authentication fails, displays an error message.

[1960] Selecting learning modules and generating problem sets

[1961] Step 4:

[1962] Users: Select the module they want to study from the dashboard.

[1963] Step 5:

[1964] Terminal: Sends the user's selection information to the server.

[1965] Step 6:

[1966] Server: Based on the received selection information, the server analyzes the user's past learning data, generates a customized problem set, and sends it to the device.

[1967] Step 7:

[1968] Terminal: Displays the problem set received from the server to the user.

[1969] Answers and feedback

[1970] Step 8:

[1971] User: Enter the answer to the question displayed.

[1972] Step 9:

[1973] Terminal: Sends the user's answers to the server in real time.

[1974] Step 10:

[1975] Server: Analyzes the received answers and determines whether they are correct. Furthermore, it uses AI technology to generate individual feedback based on the user's answers and past learning data, and sends it to the device.

[1976] Step 11:

[1977] Terminal: Displays the feedback received from the server to the user.

[1978] Evaluating progress and suggesting new learning plans

[1979] Step 12:

[1980] Server: Stores all user answers and learning history, evaluates progress, and generates new learning plans and additional learning materials and sends them to the device.

[1981] Step 13:

[1982] Terminal: Displays new learning plans and additional learning materials received from the server to the user.

[1983] Introducing the Emotion Engine

[1984] Emotion recognition and feedback with emotion engine

[1985] Step 14:

[1986] Device: Collects emotional information such as the user's facial expressions and tone of voice through a camera and microphone.

[1987] Step 15:

[1988] Server: The emotion engine analyzes this emotion information and identifies the user's emotional state (e.g., excitement, fatigue, confusion, etc.).

[1989] Step 16:

[1990] Server: Generates feedback based on the emotional state obtained from the emotion engine. For example, if the user is tired, it generates feedback such as "Take a short break."

[1991] Step 17:

[1992] Terminal: Displays feedback to the user based on the emotional state received from the server.

[1993] Responding to emotional states

[1994] Step 18:

[1995] Server: Using the emotion engine, adjust the difficulty and format of questions according to the user's emotional state. For example, if the user is confused, simplify the question or provide hints.

[1996] Step 19:

[1997] Server: The emotional state obtained from the emotion engine is added to the training data and reflected in the assessment of progress and understanding.

[1998] Specific examples

[1999] Example 1: Math problems and emotional feedback

[2000] Step 1:

[2001] Terminal: The user selects a mathematics module and requests an entry-level problem set.

[2002] Step 2:

[2003] Server: Generates a customized problem set, including basic arithmetic operations, from past learning data and sends it to the device.

[2004] Step 3:

[2005] Terminal: The user answers "4" to the question "2 + 2 = ?" and sends the answer to the server.

[2006] Step 4:

[2007] Server: Analyzes the answer, verifies it is correct, and generates feedback such as "That's correct. Please proceed to the next question."

[2008] Step 5:

[2009] Terminal: Displays the feedback received from the server to the user.

[2010] Step 6:

[2011] Terminal (emotion engine): The camera analyzes the user's facial expressions and detects when the user is concentrating.

[2012] Step 7:

[2013] Server (emotion engine): Checks the user's concentration and generates feedback to encourage them to continue learning.

[2014] Step 8:

[2015] Terminal: Displays the feedback received from the server to the user.

[2016] In this way, the system of the present invention provides an optimal learning experience in real time based on the user's individual needs and level of understanding. Furthermore, the introduction of an emotion engine enables customization based on the user's emotional state, further improving learning efficiency. This results in improved learning outcomes for students, and allows teachers and parents to easily track progress.

[2017] Example 2

[2018] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2019] Conventional educational support systems have struggled to provide an optimized educational experience for each individual user. In particular, it has been difficult to grasp a user's learning situation in real time and provide feedback based on that understanding, and customization to accommodate emotional states has been insufficient. As a result, learners are unable to properly understand their own learning situation, resulting in a decline in learning effectiveness.

[2020] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving user authentication information and verifying it against a database for authentication, means for analyzing the user's past learning data and generating a customized problem set, means for analyzing the user's answers and generating individual feedback, means for saving the user's answers and learning history and evaluating the user's progress, means for suggesting new learning plans and additional learning materials, means for collecting and analyzing the user's emotional information and generating feedback based on the user's emotional state, means for transmitting the user's answers in real time, and means for displaying the generated feedback to the user. This makes it possible to provide an optimal educational experience in real time based on the user's individual learning situation and emotional state.

[2021] "Means for receiving user authentication information and verifying it against a database" refers to the process of receiving user authentication information (such as username and password) sent from the terminal and comparing that information with the registered information in a database to verify the user's legitimacy.

[2022] "Means for analyzing a user's past learning data and generating a customized problem set" refers to the process of analyzing a user's learning history and achievements and generating individually optimized learning problems based on that.

[2023] The "means for analyzing the user's answers and generating personalized feedback" is a process for analyzing the answers submitted by the user and generating feedback that provides specific guidance based on the results.

[2024] "Means for saving the user's answers and learning history and evaluating the progress" refers to the process of recording the correctness of the user's answers and learning history, and evaluating the user's learning progress and level of understanding based on this.

[2025] The "means for suggesting new study plans and additional learning materials" is a process for providing the user with the next study content and supplementary learning materials based on the user's evaluation results.

[2026] "Means for generating feedback based on a user's emotional state by collecting and analyzing the user's emotional information" refers to a process of using a camera or microphone to collect the user's facial expressions and tone of voice, analyzing the user's emotional state based on that information, and providing appropriate feedback.

[2027] "Means for transmitting user answer results in real time" refers to a process in which the results are immediately transmitted to the server each time a user inputs an answer.

[2028] The "means for displaying the generated feedback to the user" is a process for displaying the feedback sent from the server on the screen of the terminal and providing it to the user.

[2029] The present invention is an educational support system for providing an optimal educational experience to each student. The system includes means for receiving user authentication information, authenticating the user by checking it against a database, analyzing the user's past learning data to generate a customized problem set, and analyzing the user's answers to generate personalized feedback. The system also stores the user's answers and learning history, evaluates the user's progress, and suggests new learning plans and additional learning materials. The system also incorporates an emotion engine for collecting and analyzing the user's emotional information and generating feedback based on the user's emotional state.

[2030] The entire system functions through the interaction of three parties: the server, the terminal, and the user. The specific configuration and operation will be explained below.

[2031] First, the user opens a login screen using the terminal and enters their username and password. This login information is sent to the server using JavaScript. The server uses the Python Flask framework to check the authentication information against a database (e.g., MySQL). If authentication is successful, the server loads the user's learning data and saves the user ID in the session. If authentication fails, a response including an error message is sent to the terminal. Based on the response from the server, the terminal displays a dashboard if the authentication was successful, or an error message if the authentication failed.

[2032] Next, the user selects the module they wish to learn from the dashboard. This selection information is sent to the server via JavaScript. Based on the received selection information, the server analyzes the user's past learning data using machine learning libraries such as scikit-learn. As a result, a problem set optimized for the user is generated and sent to the device. The device then dynamically generates the received problem set using HTML and JavaScript and displays it to the user.

[2033] When a user enters an answer to a question and clicks the submit button, the device sends the answer information to the server in real time. The server uses Python to analyze the answer and determine whether it is correct. It then uses AI technologies such as TensorFlow and PyTorch to generate individual feedback based on the analysis results and past learning data and sends it to the device. The device then displays the received feedback to the user.

[2034] The server stores all of the user's answers and learning history in a database. The server evaluates the user's progress based on this information. Based on the progress and evaluation results, it generates new learning plans and additional learning materials and sends them in JSON format to the device. The device then displays the received plans and learning materials to the user.

[2035] Furthermore, a means for recognizing the user's emotional state is incorporated using an emotion engine. The device collects the user's facial expressions and tone of voice through a camera and microphone, and analyzes them using an emotion engine (e.g., OpenCV or the Facial Emotion Recognition (FER) library). Based on the analysis results, the server generates feedback according to the user's emotional state (e.g., "Let's take a short break") and sends it to the device. The device then displays this feedback to the user.

[2036] Examples of concrete examples and prompts

[2037] As a concrete example, let's consider an operation in which a user selects a mathematics module and requests a set of beginner-level arithmetic problems. The server generates problems such as "2 + 2 = ?" based on past learning data and sends them to the device. If the user answers "4," the server determines that this is the correct answer and generates feedback saying, "That's correct. Please proceed to the next problem." The device displays this feedback, and also recognizes the user's state of concentration via the emotion engine, generating and displaying feedback encouraging the user to continue learning.

[2038] Examples of prompts might include "Design a system that returns analysis results in real time after the user enters their answer" or "Explain the process of generating a customized problem set and displaying it to the user."

[2039] In this way, the educational support system according to the present invention can provide an optimal educational experience in real time based on the user's learning situation and emotional state.

[2040] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2041] Step 1: Authenticate the user

[2042] 1.1 Terminal:

[2043] You open a login screen, enter your username and password, and click the "Login" button. The computer program sends this information to the server using JavaScript.

[2044] Input: Username, Password

[2045] Output: Authentication information sent to the server

[2046] 1.2 Server:

[2047] The authentication information received from the device is checked against a database (e.g., MySQL). The Python Flask framework is used to compare the authentication information with the database, and if a match is found, authentication is successful. The user ID is stored in the session and training data is loaded. If authentication fails, an error message is generated.

[2048] Input: Authentication information (user name, password)

[2049] Output: Authentication result (success / failure), user data (if authentication is successful)

[2050] 1.3 Terminal:

[2051] Receives the authentication result from the server and displays the dashboard if successful, or an error message if unsuccessful.

[2052] Input: Authentication result (success / failure)

[2053] Output: Dashboard (if successful), Error message (if unsuccessful)

[2054] Step 2: Selecting a learning module and generating a problem set

[2055] 2.1 User:

[2056] From the dashboard, select the module you want to study. Your selection is collected using JavaScript.

[2057] Input: Module selection information

[2058] Output: Selections sent to the server

[2059] 2.2 Terminal:

[2060] Send the selection to the server. The selection is sent asynchronously using an AJAX request.

[2061] Input: Selection information

[2062] Output: Selections sent to the server

[2063] 2.3 Server:

[2064] Receives the selection information and analyzes the user's past training data. Analyzes the training data using Python and scikit-learn to generate a customized problem set. The generated problem set is sent to the device in JSON format.

[2065] Input: Selection information, past learning data

[2066] Output: Problem set in JSON format

[2067] 2.4 Terminal:

[2068] The problem set received from the server is displayed to the user. Problems are dynamically generated using JavaScript and HTML and displayed on the screen.

[2069] Input: Problem set (JSON format)

[2070] Output: The displayed problem set

[2071] Step 3: Answers and feedback

[2072] 3.1 User:

[2073] Enter your answer to the question and click the "Submit" button. The answer will be entered into the form and collected by JavaScript.

[2074] Input: Answer

[2075] Output: Answer sent to the server

[2076] 3.2 Terminal:

[2077] Send the user's answers to the server in real time using an AJAX request.

[2078] Input: Answer

[2079] Output: Answer sent to the server

[2080] 3.3 Server:

[2081] The received answers are analyzed and judged to be correct (using Python), and then individual feedback is generated using AI technologies such as TensorFlow and PyTorch and sent to the device.

[2082] Input: Answers, past learning data

[2083] Output: Individual feedback

[2084] 3.4 Terminal:

[2085] Display the feedback received from the server to the user. Use JavaScript to display the feedback message on the screen.

[2086] Input: Feedback

[2087] Output: Displayed feedback

[2088] Step 4: Evaluate progress and propose a new learning plan

[2089] 4.1 Server:

[2090] All user answers and learning history are stored in a database, and progress is evaluated. Based on the evaluation results, new learning plans and additional learning materials are generated and sent to the device in JSON format.

[2091] Input: Answer results, learning history

[2092] Output: New learning plan, additional materials (JSON format)

[2093] 4.2 Terminal:

[2094] Display new learning plans and additional learning materials received from the server to the user. Plans and learning materials are dynamically generated using JavaScript and HTML and displayed on the screen.

[2095] Input: New learning plan, additional materials (JSON format)

[2096] Output: New learning plan displayed, additional materials

[2097] Step 5: Emotion recognition and feedback by the emotion engine

[2098] 5.1 Terminal:

[2099] Emotional information such as facial expressions and tone of voice of the user is collected through a camera and microphone. OpenCV and Python libraries are used for the emotion engine.

[2100] Input: facial expression data, voice data

[2101] Output: Emotion information sent to the server

[2102] 5.2 Server:

[2103] The emotion engine analyzes the emotion information and identifies the user's emotional state. Based on the analysis results, it generates appropriate feedback and sends it to the device.

[2104] Input: Emotion information

[2105] Output: Feedback (based on emotional state)

[2106] 5.3 Terminal:

[2107] Display feedback to the user based on the emotional state received from the server. Use JavaScript to display the feedback message on the screen.

[2108] Input: Feedback (based on emotional state)

[2109] Output: Displayed feedback

[2110] (Application example 2)

[2111] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2112] Conventional education and training support systems have difficulty adapting flexibly to the learning progress and emotional state of individual users and workers, and have often provided uniform learning plans and training procedures. As a result, optimal feedback and training support tailored to each individual's level of understanding and emotional state has not been provided, leading to a decline in learning efficiency and work efficiency.

[2113] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving a user's authentication information and verifying it against a database for authentication; means for analyzing the user's past learning data and generating a customized problem set; means for analyzing the user's answers and generating individual feedback; means for saving the user's answers and learning history and evaluating progress; means for suggesting new learning plans and additional learning materials; means for analyzing the worker's past training data and presenting customized work procedures; means including an emotion engine for recognizing emotional states and generating feedback; and means for evaluating progress and suggesting the next training or work procedures. This makes it possible to provide each user or worker with an optimal learning and training experience tailored to their learning situation and emotional state.

[2114] "User authentication information" refers to information used to identify a person accessing a system and verify their identity.

[2115] A "database" is a collection of information used to store and manage user authentication information, learning data, and training data.

[2116] A "customized problem set" is a collection of problems optimized for each individual user, generated by analyzing the user's past learning data and answer patterns.

[2117] "Individual feedback" is advice and evaluation specific to a user that is generated based on the user's answers and progress.

[2118] "Progress" is information that indicates the progress and results of a user's learning or training.

[2119] "New study plans and additional learning materials" are new study plans and supplementary learning materials that are suggested based on the user's progress and level of understanding.

[2120] A "work procedure" is a series of steps or methods for accomplishing a particular task.

[2121] An "emotion engine" is a system that analyzes emotional information such as a user's facial expression and tone of voice to recognize the user's emotional state.

[2122] "Feedback" refers to evaluations and advice provided in response to a user's actions or input.

[2123] "Training data" refers to data that includes records of training that workers have previously undergone and the learning outcomes based on that training.

[2124] The present invention is a real-time training support system for factory workers, which uses smart glasses to provide real-time assistance to workers while they are working. Detailed description of the embodiments of the present invention will be given below.

[2125] First, the system includes the following hardware and software components:

[2126] Hardware: Smart glasses (camera, display)

[2127] software:

[2128] OpenCV: Camera image capture and display

[2129] Dlib: A face detection and recognition model

[2130] EmotionRecognizer: A model for recognizing emotional states

[2131] Data Management Module: Acquire and suggest customized training data

[2132] Server communication module: Sending and receiving feedback

[2133] System processing flow

[2134] 1. User authentication:

[2135] A worker puts on the smart glasses and logs in to the system. The user's authentication information (username and password) is sent from the smart glasses device to the server and checked against the database. If authentication is successful, the worker's training data is loaded and the session begins.

[2136] 2. Providing training procedures:

[2137] Based on the worker's past training data, customized work instructions are generated and displayed on the smart glasses' display, and the instructions are adjusted based on each worker's skill level and past performance.

[2138] 3. Recognition of emotional states:

[2139] A camera mounted on the smart glasses captures the worker's facial expressions, which are then analyzed by EmotionRecognizer, which identifies the worker's emotional state, such as whether they are tired, focused, or confused.

[2140] 4. Generate personalized feedback:

[2141] Based on the recognized emotional state, the server generates appropriate feedback in real time, such as "Take a break" if the worker is tired, or "Revisit the procedure" if the worker is confused.

[2142] 5. Progress evaluation and training plan suggestions:

[2143] The server stores all the worker's training data and evaluates their progress, generating new training plans and additional work instructions to prepare them for the next job.

[2144] Specific examples

[2145] For example, when a worker is tightening a nut, smart glasses will display the work instructions and analyze the worker's facial expressions using a facial recognition camera. If the smart glasses detect that the worker is tired, they will provide feedback such as "Take a short break." If the glasses detect that the worker is confused by the work instructions, they will again provide specific instructions such as "Turn the screw clockwise."

[2146] Example prompts for generative AI models

[2147] "Generate optimal feedback when workers feel fatigued during work."

[2148] "Propose procedural support for workers if they are confused."

[2149] In this way, the system of the present invention can improve work efficiency and safety by providing optimal support in real time according to the condition of each individual worker.

[2150] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2151] Step 1:

[2152] The user puts on the smart glasses and enters their username and password on the login screen. The input information is sent from the device to the server. The server compares the received authentication information with a database and performs authentication. If authentication is successful, the worker's training data is read and the session begins. If authentication fails, an error message is sent to the device and displayed.

[2153] Step 2:

[2154] After the worker is authenticated, he or she selects a training module. The terminal sends the selection information to the server. The server analyzes the worker's past training data based on the received selection information. Based on the analysis results, the server generates a customized work procedure and sends it to the terminal. The terminal displays the generated work procedure on the display of the smart glasses.

[2155] Step 3:

[2156] A camera mounted on the smart glasses captures the worker's facial expressions. The captured video is sent from the device to a server. The server then uses an emotion engine (EmotionRecognizer) to analyze the video data and recognize the worker's emotional state (fatigue, concentration, confusion, etc.). The recognized emotional state is then used to evaluate progress.

[2157] Step 4:

[2158] The server generates appropriate feedback in real time based on the recognized emotional state. For example, if fatigue is recognized, feedback such as "Take a short break" is generated. The generated feedback is sent to the device and displayed on the smart glasses display.

[2159] Step 5:

[2160] The server stores all of the worker's training data and evaluates their progress. The evaluation results are used to generate the next training or additional work procedures. The server generates new training plans or additional work procedures and sends them to the device. The device displays the new training plans or additional work procedures on the smart glasses display.

[2161] Step 6:

[2162] While the worker is performing additional work steps, the camera in the smart glasses again captures their facial expressions and sends them to the server, which then uses the emotion engine to analyze their emotional state again and updates the feedback as needed, ensuring optimal support is always provided based on the worker's condition.

[2163] This allows users to receive optimal feedback and work procedures in real time based on their individual learning situation and emotional state.

[2164] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2165] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2166] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2167] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2168] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2169] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2170] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2171] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2172] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2173] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2174] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2175] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2176] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2177] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2178] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2179] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2180] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2181] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a ...

Claims

1. means for receiving user authentication information and verifying it against a database; A means for analyzing a user's past learning data and generating a customized problem set; means for analyzing the user's answers and generating personalized feedback; A means to store the user's answer results and learning history and evaluate their progress; A system that includes a means to suggest new learning plans and additional learning materials.

2. 2. The system of claim 1, further comprising means for generating questions based on the user's past answer patterns when generating the customized question set.

3. The system of claim 1, further comprising means for utilizing AI technology to analyze the level of understanding when assessing the user's progress.

Images