System

Abstract

A system is provided.SOLUTION: Means for inputting user information, means for generating a customized avatar based on the input user information, means for displaying the generated customized avatar, means for receiving a request for the user to fine-tune the avatar, means for regenerating the avatar based on the fine-tuning request, means for storing the finally determined avatar, means for selecting a learning session by the user, means for providing learning content based on the selected learning session, and means for setting a question based on the provided learning content, A system, comprising: means for providing commentary; means for receiving a user's answer and determining whether the answer is correct or incorrect; means for providing feedback based on the determination; and means for managing the user's learning progress.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's world, many children have limited opportunities to receive a full education due to various reasons, such as financial constraints, busy parents, or a reluctance to attend cram schools. Furthermore, children who primarily study at home often lack the ability to get help resolving homework questions. It is also difficult to provide equal educational opportunities to children who are shy and find it difficult to ask questions. There is a need to resolve these issues and provide an environment where everyone can receive a high-quality education equally. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for inputting user information, a means for generating a custom avatar based on the input user information, a means for displaying the generated custom avatar, a means for the user to request avatar fine-tuning, a means for regenerating the avatar based on the fine-tuning request, a means for saving the finalized avatar, a means for the user to select a study session, a means for providing study content based on the selected study session, a means for asking questions and providing explanations based on the provided study content, a means for receiving the user's answers and determining whether they are correct or incorrect, a means for providing feedback based on the results of the evaluation, and a means for managing the user's study progress. The system further includes a means for photographing paper questions using a camera function and converting them into text data using OCR technology, and a means for the generated custom avatar to provide explanations and feedback on the questions using a voice function, thereby enabling efficient and effective individualized learning support for children. This allows children to easily ask questions and receive high-quality feedback without financial or time constraints, thereby providing equal educational opportunities for all children.

[0006] "User Information" is data about the user's name, age, grade level, preferred subjects, and desired avatar characteristics.

[0007] A "custom avatar" is a virtual character that resembles a user's personal tutor and is generated based on user information.

[0008] A "study session" is a session of study timeframe and content selected by a user.

[0009] "Learning Content" includes educational materials, problem sets, instructional videos, etc. provided under a learning session.

[0010] The "camera function" is a function that uses the camera built into the device to take a photo of a question written on paper.

[0011] "OCR technology" stands for optical character recognition technology, which is a technology for extracting text data from image data.

[0012] "Answer evaluation" is the process of comparing the answer entered by the user with the correct answer and determining whether it is correct or incorrect.

[0013] "Feedback" refers to the evaluation and correction guidance provided to the user based on the results of their answers.

[0014] "Study progress" is data that indicates the user's learning situation and progress in grades.

[0015] "Regeneration" is the process of recreating an avatar's appearance and characteristics based on a user's request.

[0016] The "voice function" is a function in which the avatar provides explanations and feedback about the problem to the user via voice. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to a system that provides a custom avatar that functions as a personal tutor and delivers an individually optimized learning experience. The system includes a series of processes for inputting and managing user information, generating and adjusting the custom avatar, providing educational content throughout the learning session, and evaluating the user's answers and providing feedback.

[0039] 1. Initial Setup and User Profile Creation

[0040] First, a user downloads the application and accesses the user registration screen when they first launch it. Here, the user enters their name, age, grade, preferred subject, and desired avatar characteristics (such as gender, appearance, and voice). The device then sends the collected user data to the server, and an appropriate user profile is generated.

[0041] 2. Avatar Creation and Customization

[0042] Based on the generated user profile, the server uses a custom avatar generation algorithm to automatically construct an avatar with the specified characteristics. The generated avatar is sent to the device, where the user can review the avatar's appearance and request fine-tuning if necessary. The fine-tuning request is sent back to the server, which then re-adjusts the avatar. This process is repeated until a final avatar that the user is satisfied with is determined and stored on the server.

[0043] 3. Start your study session

[0044] When a user selects a study session in the application, the device sends a request to the server to provide learning content for the selected subject. The server sends the appropriate problem sets and explanations to the device, which then presents them to the user via an avatar. For example, if a user selects a study session on multiplication in arithmetic, relevant problems and explanations are provided.

[0045] 4. Reading the problem and explaining it

[0046] When a user uploads a photo of a written or handwritten problem with their camera, the device uses OCR technology to convert it into text data. The converted data is sent to a server, which analyzes the problem and generates an answer and detailed explanation. The generated answer and explanation are then sent to the device, where an avatar uses voice functions to explain the answer to the user.

[0047] 5. Answer assessment and feedback

[0048] When a user enters an answer to a question, the device sends the answer to the server. The server compares the answer with the correct answer and returns the result to the device. The device then presents the result to the user, and an avatar provides audio feedback. Through feedback such as "That's right!" or "This is wrong," the user can identify areas for improvement.

[0049] 6. Learning progress management

[0050] The server records the user's answers for each learning session and manages their progress. Progress data is reflected in the next learning session and feedback. The device provides this management information to the user and guides them to the next learning session. The device is also designed to allow for easy schedule changes and adjustments.

[0051] Specific examples

[0052] For example, if a fourth-grade user wants to learn multiplication, he / she first completes the avatar setup and then selects the multiplication learning session. Here, the avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback saying, "That's correct! Well done!" Through this process, the user can check their understanding while progressing through the learning process.

[0053] As described above, through this invention, children can receive high-quality, individually optimized educational support regardless of their family environment or economic situation.

[0054] The processing flow will be explained below.

[0055] Step 1:

[0056] The user downloads the application and launches it for the first time, which displays the user registration screen.

[0057] Step 2:

[0058] The user enters user information such as name, age, grade, preferred subjects, and desired avatar characteristics.

[0059] Step 3:

[0060] The device sends the entered user information to the server, which then creates a user profile.

[0061] Step 4:

[0062] The server generates a custom avatar based on the user profile using an avatar generation algorithm.

[0063] Step 5:

[0064] The server sends the generated custom avatar data to the device.

[0065] Step 6:

[0066] The device displays the custom avatar on the screen, allowing the user to see what the avatar looks like.

[0067] Step 7:

[0068] If a user wants to make minor adjustments to the appearance of their avatar, they make an adjustment request.

[0069] Step 8:

[0070] The device sends a fine-tuning request to the server, which regenerates the avatar based on the adjustment request and sends the data back to the device.

[0071] Step 9:

[0072] Once the user is satisfied with the appearance of the avatar, the avatar is finally finalized and the terminal transmits the finalized data to the server.

[0073] Step 10:

[0074] The server stores the determined avatar information in the user profile.

[0075] Step 11:

[0076] Users select a study session (e.g., "Math Multiplication") from the study schedule screen within the app.

[0077] Step 12:

[0078] The terminal requests the selected learning session information from the server.

[0079] Step 13:

[0080] The server sends the appropriate learning content (problem sets and explanatory materials) to the device.

[0081] Step 14:

[0082] The terminal presents the learning content to the user through the avatar and initiates a learning session.

[0083] Step 15:

[0084] Users can write the question on paper and take a photo of it using the device's camera, or they can enter the question manually.

[0085] Step 16:

[0086] The device uses OCR technology to convert the photographed question into text data, which is then sent to a server.

[0087] Step 17:

[0088] The server analyzes the question text and generates appropriate answers and explanations.

[0089] Step 18:

[0090] The server sends the generated answers and explanations to the terminal.

[0091] Step 19:

[0092] The device uses an avatar to provide audio commentary and teach the user.

[0093] Step 20:

[0094] The user enters the answer into the terminal.

[0095] Step 21:

[0096] The terminal sends the answer to the server.

[0097] Step 22:

[0098] The server determines whether the answer is correct or incorrect and sends the result to the terminal.

[0099] Step 23:

[0100] The device displays the results and an avatar provides audible feedback such as "That's right!" or "This is wrong."

[0101] Step 24:

[0102] The server records the user's answers and updates the learning progress data.

[0103] Step 25:

[0104] The device will prompt the user for the next study session or provide options to reschedule.

[0105] Step 26:

[0106] The user selects or schedules the next study session, which causes the system to start the study process again.

[0107] Through these steps, users can study efficiently and effectively.

[0108] Example 1

[0109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0110] Conventional online learning systems have struggled to provide a personalized educational experience for each user, and have struggled to provide smooth feedback to maximize learning outcomes. Furthermore, managing learning progress can be unintuitive for users, making it difficult to develop long-term learning plans and maintain motivation. Furthermore, they are unable to provide detailed explanations of paper-based questions and their answers in real time, preventing efficient learning.

[0111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0112] In this invention, the server includes means for inputting user information, means for generating a custom avatar based on the input user information, means for displaying the generated custom avatar, means for receiving a request from the user to fine-tune the avatar, means for regenerating the avatar based on the fine-tuning request, means for saving the finalized avatar, means for the user to select a study session, means for providing study content based on the selected study session, means for asking questions and providing explanations based on the provided study content, means for receiving the user's answers and determining whether they are correct or incorrect, means for providing feedback based on the determination results, means for managing the user's study progress, means for generating answers and detailed explanations using artificial intelligence, and means for displaying study progress management information in a dashboard format, thereby enabling the provision of a personalized learning experience, real-time feedback, and intuitive study progress management.

[0113] "User information" refers to information such as name, age, grade, preferred subjects, and desired avatar characteristics that a user enters when using an application.

[0114] The "custom avatar generation means" refers to an algorithm or program for automatically creating a custom avatar that matches the characteristics of the user based on input user information.

[0115] The "avatar display means" refers to an interface or device for visually presenting the generated custom avatar to the user.

[0116] The "means for receiving a fine-tuning request" is an interface or program that allows a user to request fine-tuning of characteristics such as appearance or voice of a custom avatar.

[0117] "Avatar regeneration means" refers to an algorithm or program for regenerating an avatar based on the user's fine-tuning requests.

[0118] The "avatar storage means" refers to a program or storage for saving the finalized custom avatar in a database or the like.

[0119] A "learning session selection means" is an interface or program that allows a user to select a particular learning session.

[0120] The "learning content providing means" refers to a system or program that provides appropriate learning content to a user based on the selected learning session.

[0121] A "question-setting means" is a program or interface that presents questions to users based on the learning content provided and provides explanations.

[0122] The "answer receiving means" is an interface for users to input answers and a program for sending the answers to the server.

[0123] The "correctness determination means" is an algorithm or program for determining whether the user's answer is correct or incorrect.

[0124] "Feedback providing means" refers to a program or system that provides feedback to the user in the form of voice or text based on the result of determining whether the answer is correct or incorrect.

[0125] A "learning progress management means" is a system or program that records and manages a user's learning progress and uses the information to help with the next learning session.

[0126] "Artificial intelligence" refers to machine learning algorithms and neural network technology that analyzes users' answers and generates answers and detailed explanations.

[0127] The "means for displaying in a dashboard format" refers to an interface or program for displaying learning progress management information in a visually easy-to-understand dashboard format.

[0128] MODE FOR CARRYING OUT THE INVENTION

[0129] The present invention provides a system that provides a custom avatar that functions as a personal tutor and delivers an individually optimized learning experience. This system includes a series of processes for inputting and managing user information, generating and adjusting the custom avatar, providing educational content throughout the learning session, and evaluating the user's answers and providing feedback.

[0130] Hardware and software used

[0131] Server: Manages user information, generates and stores custom avatars, provides learning content, judges answers, generates feedback, and manages learning progress.

[0132] Device: Enter user information, view a custom avatar, select a study session, enter and submit answers, view feedback, and view study progress.

[0133] OCR library (e.g. Tesseract): Converts paper questions into text data.

[0134] Generative AI models (e.g., machine learning algorithms, neural networks): generate answers and detailed explanations.

[0135] Initial Setup and User Profile Creation

[0136] First, the user downloads the application and accesses the user registration screen when launching it for the first time. Here, the user enters their name, age, grade, preferred subjects, and desired avatar characteristics (such as gender, appearance, and voice). The device collects this information and sends it to the server. The server then creates a user profile based on the received information and sends it to the device.

[0137] Avatar Creation and Customization

[0138] The server uses a custom avatar generation algorithm to automatically construct an avatar with the specified characteristics. The generated avatar is sent to the device, where the user can review its appearance and request fine-tuning if necessary. A retuning request is sent back to the server, which then retunes the avatar. This process is repeated until the user finally finds an avatar they are happy with and stores it on the server.

[0139] Start a study session

[0140] When a user selects a study session in the application, the device sends a request to the server to provide study content for the selected subject. The server sends the appropriate problem sets and explanations to the device, which then presents them to the user through an avatar.

[0141] Problem reading and explanation

[0142] When a user uploads a photo of a written or handwritten problem with their camera, the device converts it into text using OCR technology. The converted data is sent to a server, which analyzes the problem and generates an answer and detailed explanation. The generated answer and explanation are then sent to the device, where an avatar uses voice functions to explain the answer to the user.

[0143] Answer assessment and feedback

[0144] When a user enters an answer to a question, the device sends the answer to the server. The server compares the answer with the correct answer and sends the result back to the device. The device then presents the result to the user, and an avatar provides audio feedback. Through feedback such as "That's right!" or "This is wrong," the user can identify areas for improvement.

[0145] Learning progress management

[0146] The server records the user's answers for each learning session and manages their progress. Progress data is reflected in the next learning session and feedback. The device provides this management information to the user and guides them to the next learning session. The device is also designed to allow for easy schedule changes and adjustments.

[0147] Specific examples

[0148] For example, if a fourth-grade user wants to learn multiplication, he / she first completes the avatar setup and then selects the multiplication learning session. Here, the avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback saying, "That's correct! Well done!" Through this process, the user can check their understanding while progressing through the learning process.

[0149] Prompt Sentence Examples

[0150] Prompt statement:

[0151] "Analyze a math problem solved by a user and generate the answer and a detailed explanation. The problem is '12 x 8' and the answer is '96'."

[0152] In this way, this invention provides an individually optimized learning experience and effective feedback, enabling children to receive high-quality educational support at home.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Step 1: Enter and submit user information

[0155] Input: User information such as name, age, grade, preferred subject, desired avatar characteristics

[0156] Specific operation: The user downloads the application, accesses the user registration screen when launching it for the first time, and enters the required information.

[0157] Data processing / calculation: The terminal temporarily stores the input information in its memory.

[0158] Output: Collected user information

[0159] Step 2: Generate a user profile

[0160] Input: User information

[0161] Specific operation: The device sends the collected user data to the server.

[0162] Data processing / calculation: The server creates a new user profile in the database based on the received information and enters the corresponding data into each field.

[0163] Output: Generated user profile

[0164] Step 3: Generate and submit a custom avatar

[0165] Input: User Profile

[0166] What Happens: The server uses a custom avatar generation algorithm to build an avatar based on the user profile.

[0167] Data processing / calculation: Use an AI-based avatar generation model to generate an avatar with specified characteristics.

[0168] Output: The generated avatar

[0169] Step 4: Review your avatar and request tweaks

[0170] Input: Generated avatar

[0171] Specific operation: The generated avatar is sent to the device, and the user can check its appearance and request fine-tuning if necessary.

[0172] Data processing / calculation: The device temporarily stores the user's fine-tuning request in memory.

[0173] Output: Tweak request

[0174] Step 5: Regenerate your avatar

[0175] Input: Tweak Request

[0176] Specific operation: The device sends a fine-tuning request to the server, which adjusts and regenerates an avatar with the specified characteristics.

[0177] Data processing / calculation: The server receives the fine-tuning request and regenerates the avatar based on the avatar generation algorithm.

[0178] Output: Regenerated avatar

[0179] Step 6: Save the final avatar

[0180] Input: Regenerated avatar

[0181] Specific operation: The server saves the final avatar that the user is satisfied with in the database.

[0182] Data processing / calculation: The regenerated avatar data is saved in the server database.

[0183] Output: Final saved avatar

[0184] Step 7: Select and request a study session

[0185] Input: User's learning session selection information

[0186] Specific operation: A user selects a learning session in the application. The device sends a content request to the server based on the selected subject.

[0187] Data processing / calculation: Based on the information of the selected learning session, a request is generated and sent to the server.

[0188] Output: Learning content request

[0189] Step 8: Provide learning content

[0190] Input: Learning content request

[0191] Specific operation: The server selects an appropriate problem set and explanatory materials and sends them to the terminal.

[0192] Data processing / calculation: Retrieve problem sets and explanatory materials from the database and compose the selected content.

[0193] Output: Problem set and explanations

[0194] Step 9: Questions and Answers

[0195] Input: Problem set and explanation materials

[0196] Specific operation: The device displays the problem to the user through the avatar and provides an explanation. The user then inputs the answer.

[0197] Data processing / calculation: Display the problem set and receive user answer input.

[0198] Output: User's answer

[0199] Step 10: Answer assessment

[0200] Input: User's answer

[0201] Specific operation: The terminal sends the user's answer to the server, which determines whether the answer is correct or not.

[0202] Data processing / calculation: The answer is compared with the correct data to generate a correct or incorrect result.

[0203] Output: Judgment result

[0204] Step 11: Provide feedback

[0205] Input: Judgment result

[0206] Specific operation: The server sends the judgment result to the terminal, and the terminal provides feedback to the user through the avatar.

[0207] Data processing / calculation: The feedback content is constructed based on the judgment results.

[0208] Output: Feedback

[0209] Step 12: Manage your learning progress

[0210] Input: Feedback and Answer Results

[0211] Specific operation: The server records the user's answers for each learning session and manages the progress. The device displays this management information in a dashboard format.

[0212] Data processing / calculation: Answer results and feedback are stored in a database, and progress is visualized.

[0213] Output: Progress management data and dashboard display

[0214] (Application example 1)

[0215] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0216] Conventional learning support and training systems have struggled to provide individually optimized support for each user. Training systems for new employees and field workers also struggled to provide real-time guidance and feedback tailored to on-site work. As a result, there was a challenge in being unable to provide appropriate support tailored to each user's level of understanding and progress.

[0217] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0218] In this invention, the server includes means for inputting user information, means for generating a custom avatar based on the input user information, means for displaying the generated custom avatar, means for the user to receive a request to fine-tune the avatar, means for regenerating the avatar based on the fine-tuning request, means for saving the finalized avatar, means for the user to select a study session, means for providing study content based on the selected study session, means for asking questions and providing explanations based on the provided study content, means for receiving the user's answers and determining whether they are correct or incorrect, means for providing feedback based on the determination results, means for managing the user's study progress, means for a new employee or field worker to create their own profile and for providing training content based on the profile, and means for recording the user's operations and results with a camera and providing feedback. This enables individually optimized study support and training.

[0219] "Means for inputting user information" refers to a device or program that allows a user to input information such as name, age, and work experience.

[0220] The "means for generating a custom avatar based on input user information" refers to a device or program that generates an avatar using a specific algorithm based on information input by the user.

[0221] The "means for displaying the generated custom avatar" is a device or program for visually displaying the generated avatar on a screen.

[0222] A "means for accepting user requests to fine-tune an avatar" is a device or program that allows a user to input requests to change characteristics such as the appearance or voice of the avatar.

[0223] The "means for regenerating an avatar based on a fine-tuning request" refers to a device or program that adjusts the characteristics of the avatar and regenerates it based on a user request.

[0224] The "means for saving the finalized avatar" is a device or program for saving the finalized avatar that satisfies the user in a database or the like.

[0225] A "means for user selection of a learning session" is a device or program that allows a user to select the learning content or training session they wish to use.

[0226] The "means for providing learning content based on a selected learning session" refers to a device or program for providing a user with learning materials and questions according to the selected learning session.

[0227] "Means for asking questions and providing explanations based on the provided learning content" refers to a device or program that presents questions to the user based on the learning content and provides explanations for the answers.

[0228] The "means for receiving the user's answer and determining whether it is correct" is a device or program for receiving the answer entered by the user and determining whether it is correct.

[0229] The "means for providing feedback based on the judgment result" is a device or program that provides appropriate feedback to the user based on the result of the judgment of whether the answer is correct or incorrect.

[0230] "Means for managing a user's learning progress" refers to a device or program for recording and managing a user's learning history and progress.

[0231] A "means for new employees or field workers to create their own profile" is a device or program that allows new employees or field workers to enter information such as their name, work experience, and learning goals to create a profile.

[0232] The "means for providing training content based on a profile" refers to a device or program for providing individually optimized training materials and content based on the created profile information.

[0233] "Means for recording the operations and results performed by the user with a camera and providing feedback" refers to a device or program that uses a camera to capture the operations performed by the user, analyzes the data, and provides appropriate feedback.

[0234] This invention is a system that provides a custom avatar that acts as a personal tutor and delivers an individually optimized learning experience. In particular, it includes a system that provides practical training in real time for new employees and field workers. Specific implementation methods of the invention are as follows.

[0235] First, the user downloads a dedicated application onto their smartphone or tablet. When they first launch the application, they access the user registration screen and enter their name, age, work experience, and other user information. This information is then sent from the device to the server.

[0236] The server then uses a custom avatar generation algorithm to build an avatar based on the user's profile. The generated avatar is sent to the device, where the user can review its appearance and request tweaks if necessary. The tweaked information is sent back to the server, and the avatar is regenerated. This process is repeated until the user is satisfied, and the final avatar is saved on the server.

[0237] The user then selects a learning session on the dedicated application. The training content is based on practical applications such as machine operation and safety management. For example, if a new employee is receiving training on machine operation in a factory, the server sends the selected training content to the terminal and provides it to the user via an avatar.

[0238] During training, the user's operations are recorded using the device's camera function. Using OCR technology (using Tesseract.js), the recorded operations are converted into text data and sent to the server. The server generates the analysis results, and the avatar provides feedback using the voice function. For example, if the user performs an incorrect operation, the avatar will say, "This is incorrect. Please do it this way next time."

[0239] The server then receives the user's answer and determines whether it is correct. The result of the determination is immediately provided to the user as feedback via the avatar. For example, if the user makes a mistake in the operating procedure, the server will provide feedback such as, "This is an error. The correct way is this."

[0240] Finally, the server has the function of managing the user's learning progress. It records the results and answer history of each training session and reflects them in the next learning session. It also provides new training content according to the user's progress, supporting continuous skill development.

[0241] Examples:

[0242] For example, when a new employee receives training on operating machinery in a factory, the user creates a profile on a dedicated tablet and customizes the avatar's appearance and voice. The server then explains safe operating procedures for the machine via the avatar, and the user performs the procedures. The operations recorded by the camera are converted into text data using OCR technology (Tesseract.js) and analyzed by the server. Based on the results, the avatar provides feedback, such as "This is incorrect."

[0243] Example prompts to input to a generative AI model:

[0244] Create a new worker profile in the system. Input data includes name, age, and work experience. Follow the steps below to provide training content for operating factory robots. The camera will capture the user's operation results, convert them into text data using OCR technology, and provide feedback via an avatar.

[0245] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0246] Step 1:

[0247] Users use their smartphones or tablets to download a dedicated application.

[0248] Input: User operation (application download)

[0249] Action: Install application

[0250] Output: Application initial launch screen

[0251] Step 2:

[0252] A user starts the application and accesses the user registration screen, where they enter their user information such as name, age, and work experience.

[0253] Input: User information (name, age, work experience, etc.)

[0254] Process: Enter and submit user information

[0255] Output: A user profile is generated on the server.

[0256] Step 3:

[0257] The server uses a custom avatar generation algorithm to generate a custom avatar based on the user profile, and the generated avatar is sent to the device.

[0258] Input: User profile information

[0259] Processing: Custom avatar generation (Unity3D, TensorFlow)

[0260] Output: Generated custom avatar data

[0261] Step 4:

[0262] The custom avatar is displayed on the device, allowing the user to check its appearance, voice, etc., and request tweaks as needed.

[0263] Input: Generated avatar data

[0264] Action: Display a custom avatar

[0265] Output: Avatar confirmation screen

[0266] Step 5:

[0267] The device accepts the user's fine-tuning request and sends it to the server, which then fine-tunes the avatar, regenerates it, and sends it to the device.

[0268] Input: Tweak request from user

[0269] Action: Regenerate custom avatar

[0270] Output: Fine-tuned custom avatar data

[0271] Step 6:

[0272] The final custom avatar is saved on the server and the user selects a learning or training session.

[0273] Input: Confirmed custom avatar data, user session selection

[0274] Actions: Save custom avatar, select learning or training session

[0275] Output: Session start screen

[0276] Step 7:

[0277] The server sends appropriate content to the terminal based on the selected learning or training session.

[0278] Input: Selected information for your study or training session

[0279] Processing: Selection and delivery of training content (using Django)

[0280] Output: Training content

[0281] Step 8:

[0282] The user performs the training and records the operations and results using the device's camera.

[0283] Input: The action taken by the user

[0284] Process: Recording the operation (using a camera)

[0285] Output: Recorded operation data

[0286] Step 9:

[0287] The device converts the recorded operation data into text data using OCR technology (Tesseract.js) and sends it to the server.

[0288] Input: Recorded operation data

[0289] Processing: Converting operation data into text (OCR technology)

[0290] Output: Text data

[0291] Step 10:

[0292] The server analyzes the text data, and the avatar provides audio feedback based on the results.

[0293] Input: Text data

[0294] Processing: Data analysis and feedback generation (using Scikit-learn)

[0295] Output: Generate feedback content

[0296] Step 11:

[0297] The device provides the user with audible feedback via an avatar.

[0298] Input: Feedback

[0299] Processing: Audio feedback

[0300] Output: Audio feedback

[0301] Step 12:

[0302] The server manages the user's learning progress, recording the results of each training session and reflecting them in the next session.

[0303] Input: Results of each training session

[0304] Processing: Data management of learning progress (using MongoDB)

[0305] Output: Learning progress data and what to expect in the next session

[0306] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0307] The present invention relates to a personal tutor avatar system for assisting users in their learning, specifically a system that combines an emotion engine to recognize the user's emotional state and adjust dialogue and feedback accordingly to provide a more effective learning experience. This system includes processes for inputting user information, generating and displaying a custom avatar, managing a learning session, and recognizing the user's emotions to adjust dialogue content.

[0308] 1. Initial Setup and User Profile Creation

[0309] First, a user downloads the application and accesses the user registration screen when they first launch it. Here, the user enters their name, age, grade, preferred subject, and desired avatar characteristics (such as gender, appearance, and voice). The device then sends the collected user data to the server, and an appropriate user profile is generated.

[0310] 2. Avatar Creation and Customization

[0311] Based on the generated user profile, the server uses a custom avatar generation algorithm to automatically construct an avatar with the specified characteristics. The generated avatar is sent to the device, where the user can review the avatar's appearance and request fine-tuning if necessary. The fine-tuning request is sent back to the server, which then re-adjusts the avatar. This process is repeated until a final avatar that the user is satisfied with is determined and stored on the server.

[0312] 3. Start your study session

[0313] When a user selects a study session in the application, the device sends a request to the server to provide learning content for the selected subject. The server sends the appropriate problem sets and explanations to the device, which then presents them to the user via an avatar. For example, if a user selects a study session on multiplication in arithmetic, relevant problems and explanations are provided.

[0314] 4. Use of Emotion Engine

[0315] During the learning session, the device's built-in camera and microphone collect the user's facial expressions and voice data. This data is sent to the server in real time and analyzed by the emotion engine. Based on the analysis results, the server recognizes the user's emotional state (e.g., "happiness," "sadness," "anger," "surprise," etc.).

[0316] 5. Reading the problem and explaining it

[0317] When a user uploads a photo of a written or handwritten problem with their camera, the device uses OCR technology to convert it into text data. The converted data is sent to a server, which analyzes the problem and generates an answer and detailed explanation. The generated answer and explanation are then sent to the device, where an avatar uses voice functions to explain the answer to the user.

[0318] 6. Answer assessment and feedback

[0319] When a user enters an answer to a question, the device sends the answer to the server. The server compares it with the correct answer and returns the result to the device. The device then presents the result to the user, and an avatar provides audio feedback. Furthermore, the avatar's dialogue content and tone can be adjusted based on the user's recognized emotions. For example, if the user is confused, the avatar will use encouraging words such as "It's okay, let me explain it again."

[0320] 7. Learning progress management

[0321] The server records the user's answers for each study session and manages their progress. Progress data is reflected in the next study session and feedback. It also records recognized emotional data and uses this information to optimize study. For example, if a user frequently feels frustrated with a particular problem, it can provide additional support materials related to that problem. The device provides this management information to the user and guides them through the next study session. It is also designed to allow for easy schedule changes and adjustments.

[0322] Specific examples

[0323] For example, if a fourth-grade user wants to learn multiplication, he / she first completes the avatar setup and then selects the multiplication learning session. Here, the avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback, saying, "That's correct! Well done!" At the same time, if the user looks confused, the avatar will provide additional support, saying, "Shall I explain it again?" Through this process, the user can progress through their learning while checking their understanding.

[0324] As described above, through this invention, children can receive high-quality, individually optimized educational support regardless of their family environment or financial situation. The use of an emotion engine further improves the learning experience by recognizing the user's emotional state and providing appropriate feedback accordingly.

[0325] The processing flow will be explained below.

[0326] Step 1:

[0327] The user downloads the application and launches it for the first time, which displays the user registration screen.

[0328] Step 2:

[0329] The user enters user information such as name, age, grade, preferred subjects, and desired avatar characteristics.

[0330] Step 3:

[0331] The device sends the entered user information to the server, which then creates a user profile.

[0332] Step 4:

[0333] The server generates a custom avatar based on the user profile using an avatar generation algorithm.

[0334] Step 5:

[0335] The server sends the generated custom avatar data to the device.

[0336] Step 6:

[0337] The device displays the custom avatar on the screen, allowing the user to see what the avatar looks like.

[0338] Step 7:

[0339] If a user wants to make minor adjustments to the appearance of their avatar, they make an adjustment request.

[0340] Step 8:

[0341] The device sends a fine-tuning request to the server, which regenerates the avatar based on the adjustment request and sends the data back to the device.

[0342] Step 9:

[0343] Once the user is satisfied with the appearance of the avatar, the avatar is finally finalized and the terminal transmits the finalized data to the server.

[0344] Step 10:

[0345] The server stores the determined avatar information in the user profile.

[0346] Step 11:

[0347] Users select a study session (e.g., "Math Multiplication") from the study schedule screen within the app.

[0348] Step 12:

[0349] The terminal requests the selected learning session information from the server.

[0350] Step 13:

[0351] The server sends the appropriate learning content (problem sets and explanatory materials) to the device.

[0352] Step 14:

[0353] The terminal presents the learning content to the user through the avatar and initiates a learning session.

[0354] Step 15:

[0355] The device's built-in camera and microphone are used to collect the user's facial expressions and voice data in real time.

[0356] Step 16:

[0357] The collected facial and voice data is sent to a server, which uses an emotion engine to analyze the data and recognize the user's emotional state.

[0358] Step 17:

[0359] When a user takes a photo of a question written on paper or entered manually and uploads it, the device uses OCR technology to convert it into text data, which is then sent to the server.

[0360] Step 18:

[0361] The server analyzes the question text and generates appropriate answers and explanations.

[0362] Step 19:

[0363] The server sends the generated answers and explanations to the terminal.

[0364] Step 20:

[0365] The device provides audio commentary from an avatar, educating the user and adjusting the content and tone of the dialogue based on the user's emotional state.

[0366] Step 21:

[0367] The user enters the answer into the terminal.

[0368] Step 22:

[0369] The terminal sends the answer to the server.

[0370] Step 23:

[0371] The server determines whether the answer is correct or incorrect and sends the result to the terminal.

[0372] Step 24:

[0373] The device displays the results and an avatar provides audible feedback such as "That's right!" or "This is wrong."

[0374] Step 25:

[0375] The server records the user's answers and updates their learning progress data, along with their emotional state.

[0376] Step 26:

[0377] The device will prompt the user for the next study session or provide options to reschedule.

[0378] Step 27:

[0379] The user selects or schedules the next study session, which causes the system to start the study process again.

[0380] Through these steps, users can learn efficiently and effectively. By utilizing the emotion engine, appropriate feedback and support are provided that takes into account the user's emotional state, resulting in a more personalized learning experience.

[0381] Example 2

[0382] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0383] Conventional learning support systems have the problem of being unable to provide an effective learning experience that takes into account the user's emotional state. In addition, they are unable to provide customized feedback optimized for each individual user, which can reduce the effectiveness of learning.

[0384] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting facial expression and voice data of the user and analyzing it with an emotion engine, means for adjusting the dialogue content based on the analysis results and providing emotionally sensitive feedback, and means for managing the user's learning progress. This allows for an effective learning experience by providing detailed feedback according to the user's emotional state.

[0385] - "User Information" refers to personal information about a User, such as the User's name, age, grade, preferred subjects, and desired avatar characteristics.

[0386] "Custom avatar" refers to an avatar created specifically for a user based on user information.

[0387] "Study Session" means a study session offered based on a subject or topic selected by a User.

[0388] "Study Content" refers to the problem sets and explanatory materials provided based on a study session.

[0389] "Questions" refer to assignments or quizzes presented to users within the learning content.

[0390] "Answer" refers to the answer entered by the user in response to a question.

[0391] "Explanation" refers to explanations or supplementary information for the answers to questions.

[0392] "Feedback" refers to the evaluation and advice provided to users based on whether their answers are correct or incorrect.

[0393] "Facial expression data" refers to data collected by capturing a user's facial expressions with a camera.

[0394] "Voice data" refers to data collected by capturing the user's voice with a microphone.

[0395] An "emotion engine" refers to algorithms or software that analyzes facial expression and voice data to recognize the user's emotional state.

[0396] "Learning progress" refers to a user's progress, results, and achievements in learning.

[0397] "Means of collection" refers to devices or methods for obtaining data from users, such as cameras or microphones.

[0398] "Means of analysis" refers to the techniques and methods for processing collected data and analyzing emotions and learning trends.

[0399] "Means for adjusting dialogue content" refers to a mechanism for appropriately changing the avatar's speech and tone depending on the user's emotional state.

[0400] "Camera function" refers to a device and its functions for taking images and videos.

[0401] "OCR technology" refers to the technology that recognizes characters in an image and converts them into text data.

[0402] The present invention relates to a personal tutor avatar system for supporting users in their learning, specifically a system that combines an emotion engine to recognize the user's emotional state and adjust dialogue and feedback accordingly to provide a more effective learning experience. This system includes processes for inputting user information, generating and displaying a custom avatar, managing a learning session, and adjusting dialogue content based on the user's emotions.

[0403] First, the user downloads the learning support application and accesses the user registration screen when launching it for the first time. Here, the user enters their name, age, grade, preferred subjects, and desired avatar characteristics (gender, appearance, voice, etc.). This information is saved on the device. The device temporarily stores the information entered by the user and sends it to the server as soon as registration is complete. A message indicating registration is complete is displayed, and the device prepares to proceed to the next step.

[0404] The server generates an individual user profile based on the received user information. The generated profile is stored in a database and used in subsequent processes. The server then activates a custom avatar generation algorithm to automatically generate an avatar with the specified characteristics based on the user profile. The generated avatar data is sent to the device, which receives the data and displays the avatar on its screen.

[0405] The user checks the appearance of the avatar and requests fine-tuning if necessary. This request is sent from the device to the server, and the server regenerates the avatar reflecting the fine-tuning and sends the data back to the device. This process is repeated until the user is satisfied, and once the final avatar is finalized, the avatar data is saved on the server.

[0406] When a user selects a learning session, the device sends a request to the server to provide learning content for the selected subject. The server receives the request, prepares the corresponding problem set and explanatory materials, and sends the data to the device. Based on the received data, the device presents the learning session for the selected subject through an avatar. For example, in a learning session on multiplication in arithmetic, problems such as "12 x 8" are presented.

[0407] During a learning session, the device uses a camera and microphone to collect the user's facial and voice data. This data is sent to the server in real time and analyzed by the emotion engine. The server then analyzes the received facial and voice data using the emotion engine to recognize the user's emotional state (e.g., "happiness," "sadness," "anger," "surprise," etc.).

[0408] When a user writes a question on paper, takes a photo of it with a camera, and uploads it, the device uses OCR technology to convert the image data into text data. This converted text data is then sent to a server. The server analyzes the received text data and generates the answer to the question and a detailed explanation. The generated answer and explanation are then sent back to the device, which then provides them to the user through an avatar. The avatar uses voice functionality to provide explanations.

[0409] When a user enters an answer to a question and presses the send button, the device sends the answer data to the server. The server compares the received answer with the correct answer and sends the result back to the device. The result includes a correct / incorrect determination and detailed feedback. The device receives the result, and an avatar gives the user audio feedback. The content and tone of the feedback can be adjusted depending on the user's emotional state, and may use encouraging words such as "Shall I explain it again?"

[0410] The server records the user's answers for each study session and manages the user's study progress, including the results for each session and the user's emotional data. The server uses the recorded progress data to prepare for optimizing the next study session. The device notifies the user when the next study session is ready and helps the user easily manage their study progress and schedule.

[0411] Specific examples

[0412] For example, if a fourth-grade user wants to learn multiplication, he or she first sets up the avatar and then selects the multiplication learning session. The avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback, saying, "That's right! Well done!" Furthermore, if the user shows a confused expression, the avatar offers additional support by asking, "Shall I explain it again?"

[0413] Example of input prompt for generative AI model

[0414] "If the user's facial expression looks confused, have the avatar generate encouraging feedback, such as 'Shall I explain it again?' or 'Please let me know if you don't understand.'"

[0415] As described above, the present invention makes it possible to individually optimize the learning experience of each user and provide more effective learning support.

[0416] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0417] Step 1: User registration and initial setup

[0418] First, the user downloads the learning support application and accesses the user registration screen when the application is launched for the first time. The user enters their name, age, grade, preferred subjects, and desired avatar characteristics (gender, appearance, voice, etc.). This information is saved on the device. The device temporarily stores the user information and sends it to the server as soon as registration is complete. The server saves the received information in a database and returns a confirmation response to the device. The device then displays a message indicating that user registration has been completed.

[0419] Input: User information (name, age, grade, preferred subject, avatar characteristics)

[0420] Output: User information sent to the server, user registration complete message

[0421] Step 2: Create and save a user profile

[0422] The server generates a user profile based on the received user information. The generated profile is saved in a database for use in subsequent processes. A confirmation response is sent to the device confirming that the profile has been saved.

[0423] Input: User information

[0424] Output: Generated user profile, confirmation that it was saved by the server

[0425] Step 3: Avatar Creation and Customization

[0426] The server runs a custom avatar generation algorithm to automatically generate an avatar with the specified characteristics based on the user profile. The generated avatar data is sent to the device, where the user can view the displayed avatar. If the user requests fine-tuning as needed, this request is sent back to the server, and the avatar is regenerated. Finally, once the user is satisfied with the avatar, the avatar data is saved on the server.

[0427] Input: User profile, Tweak request

[0428] Output: Custom avatar, Retuned avatar

[0429] Specific actions: Adjusting the avatar's gender, appearance, voice, and other characteristics

[0430] Step 4: Start your study session

[0431] When a user selects a study session, the device sends a request to the server to provide study content according to the selected subject. The server receives the request, prepares the corresponding problem sets and explanation materials, and sends the data to the device. The device then presents the study session through an avatar based on the received data.

[0432] Input: User selects a study session

[0433] Output: Learning content (question sets, explanatory materials)

[0434] Specific actions: Displaying learning content and presenting questions

[0435] Step 5: Emotion Recognition with the Emotion Engine

[0436] During a learning session, the device uses a camera and microphone to collect facial and voice data from the user. This data is sent in real time to a server where it is analyzed by an emotion engine. The server recognizes the user's emotional state and adjusts feedback accordingly.

[0437] Input: facial expression data, voice data

[0438] Output: Analysis results (user's emotional state)

[0439] Specific actions: taking pictures of facial expressions with a camera and collecting audio with a microphone

[0440] Step 6: Reading the problem and generating an explanation

[0441] When a user takes a photo of a paper problem and uploads it, the device uses OCR technology to convert the image data into text. This converted data is sent to a server, which analyzes the problem, generates answers and explanations, and sends them back to the device. The device then provides these to the user through an avatar.

[0442] Input: Image data of the problem

[0443] Output: OCR text data, answers and explanations

[0444] Specific actions: taking a photo with a camera, OCR conversion, and explanation by an avatar

[0445] Step 7: Answers and feedback

[0446] The user inputs the answer to the question and sends it to the device. The device then sends the answer data to the server, which checks the answer against the correct answer and returns a judgment result. The device then presents the result to the user, and an avatar provides audio feedback. The content and tone of the feedback are adjusted according to the user's emotional state.

[0447] Input: User-provided answer data

[0448] Output: Matching result and feedback (correct / incorrect)

[0449] Specific actions: Entering answers, checking, and voice feedback

[0450] Step 8: Track your progress and prepare for the next session

[0451] The server records the user's answers for each study session and manages their study progress, including their grades and emotional state for each session. Based on the recorded progress data, the server optimizes the next study session and notifies the device when the next study session is ready. It also makes it easy to change the progress and schedule.

[0452] Input: Answer result, emotional state

[0453] Output: Tracked progress data, next session notifications

[0454] Specific actions: Recording results, managing emotional data, notifying preparation for the next session

[0455] (Application example 2)

[0456] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0457] Conventional learning support systems do not take into account the user's emotional state, and therefore have the problem of being unable to provide appropriate support when the user becomes confused or frustrated. This reduces the effectiveness of learning and leads to a lack of sustained user motivation. Furthermore, there is a lack of technology that can analyze the user's emotions in real time and appropriately adjust feedback based on the results.

[0458] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for inputting user information; means for generating a custom avatar based on the input user information; means for displaying the generated custom avatar; means for accepting a user request to fine-tune the avatar; means for regenerating the avatar based on the fine-tuning request; means for saving the finalized avatar; means for the user to select a study session; means for providing study content based on the selected study session; means for asking questions and providing explanations based on the provided study content; means for receiving the user's answers and determining whether they are correct or incorrect; means for providing feedback based on the determination results; means for managing the user's study progress; means for collecting the user's facial expression and voice data using a camera and microphone function; means for analyzing the user's emotions based on the collected data; and means for adjusting the feedback content based on the analysis results. This makes it possible to provide appropriate feedback according to the user's emotional state, improve learning effectiveness, and maintain the user's motivation.

[0459] "User information" refers to basic data about the user, including name, age, grade, preferred subjects, and avatar characteristics.

[0460] A "custom avatar" is a character with a unique appearance and voice that is generated based on user input.

[0461] A "study session" is a period of time or a particular learning activity during which a user studies.

[0462] "Study Content" means digital content, including study materials, problem sets, and explanatory materials, that a user works through during a study session.

[0463] "Facial expression and voice data" refers to data that records a user's facial expressions, tone of voice, and the content of their statements.

[0464] "Emotion analysis" refers to the process of identifying a user's emotional state (such as joy, sadness, anger, surprise, etc.) based on collected facial and voice data.

[0465] "Feedback" refers to advice, additional explanations, encouraging messages, etc. provided based on the user's learning progress and the accuracy of their answers.

[0466] "Progress management" is the process of recording and tracking a user's learning progress and grades to help plan future learning.

[0467] The present invention provides a personal tutor avatar system for supporting users in their learning. In particular, the system combines an emotion engine to recognize the user's emotional state and adjust dialogue and feedback based on that to provide a more effective learning experience.

[0468] The system is primarily composed of a server and a terminal. The server manages user information and performs key processes such as generating custom avatars, managing learning sessions, analyzing emotions, and adjusting feedback. The terminal is the device used by the user (smartphone, tablet, etc.) and is responsible for interacting with the user.

[0469] 1. Initial Setup

[0470] When a user starts the application for the first time, they download it and enter their user information (such as name, age, grade, preferred subjects, and avatar characteristics). The device sends this data to the server, and a user profile is created.

[0471] 2. Custom Avatar Creation

[0472] The server generates a custom avatar based on the information entered by the user. This avatar is tailored to the user's visual and auditory preferences. The generated avatar is sent to the device, where the user can review and fine-tune it. The avatar is regenerated upon the user's request, and the final avatar is stored on the server.

[0473] 3. Start your study session

[0474] When a user selects a learning session, the device requests it from the server, which then provides the device with the appropriate learning content based on the selected subject. For example, if a user selects a math session, the server provides relevant questions and explanations.

[0475] 4. Use of Emotion Engine

[0476] The device uses a camera and microphone to collect the user's facial and voice data. The collected data is sent to the server in real time, and the server analyzes the user's emotions using an emotion engine (e.g., the emotion analysis model in the Transformers library). Based on the analysis results, the server adjusts the feedback content and provides appropriate assistance to the user.

[0477] 5. Answer assessment and feedback

[0478] When a user enters an answer to a question, the device sends the answer to the server. The server determines whether the answer is correct and returns feedback to the device. The avatar provides feedback to the user in voice and text. For example, if the user's expression looks confused, encouraging feedback such as "Shall I explain it again?" is provided.

[0479] Specific examples

[0480] For example, if a fourth-grade user wants to learn multiplication, he or she can select a multiplication learning session after completing the avatar setup. Here, the avatar will present the problem "12 x 8." If the user enters "96" as the answer, the system will immediately check the answer and provide audio feedback saying, "That's correct! Well done!" At the same time, if the user shows a confused expression, the avatar will provide additional support by saying, "Shall I explain it again?"

[0481] Prompt Sentence Examples

[0482] Analyze what emotions users are expressing. Classify emotions such as "Happy," "Sadness," "Anger," "Surprise," etc.

[0483] This system provides appropriate feedback according to the user's emotional state, improving learning effectiveness and maintaining user motivation.

[0484] As described above, through the present invention, the learning assistance system can dynamically analyze the user's emotions and provide appropriate feedback based on the results, thereby improving the learning experience.

[0485] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0486] Step 1:

[0487] When a user downloads the application and launches it for the first time, they enter their user information, including their name, age, grade, preferred subjects, and avatar characteristics. The device then sends this data to a server, which creates a user profile.

[0488] Input: User information (name, age, grade, subject, avatar characteristics)

[0489] Output: User profile (stored on the server)

[0490] Specific operation: The user enters information into the input form on the terminal and presses the send button. The terminal then sends the information to the server.

[0491] Step 2:

[0492] The server generates a custom avatar based on the received user information, which is then sent to the device for the user to view.

[0493] Input: User Profile

[0494] Output: Custom avatar

[0495] What happens: The server generates an avatar using a custom avatar generation algorithm and sends it to the device, where a preview of the avatar is displayed.

[0496] Step 3:

[0497] The user reviews the avatar and sends a tweak request if necessary. The server receives the request and regenerates the avatar. Once the tweaks are complete, the final avatar is saved.

[0498] Input: Tweak Request

[0499] Output: Final custom avatar

[0500] Specific behavior: The user makes minor adjustments to the avatar's appearance or voice on their device and sends a new request to the server. The server then regenerates the avatar and saves the finalized avatar.

[0501] Step 4:

[0502] When a user selects a learning session, the device sends a request to the server, which provides the device with learning content based on the selected subject.

[0503] Input: Select study session

[0504] Output: Learning content

[0505] How it works: The user selects a learning session from a menu on their device, and the request is sent to the server, which provides the relevant learning materials and questions.

[0506] Step 5:

[0507] During a training session, the device uses a camera and microphone to collect facial and voice data from the user, which is then sent in real time to a server that uses a generative AI model to analyze emotions.

[0508] Input: facial expression and voice data

[0509] Output: Emotion analysis results

[0510] How it works: The device collects data in real time using the camera and microphone, and sends the data to a server, which then analyzes the data using a sentiment analysis model.

[0511] Step 6:

[0512] Based on the analysis results, the server adjusts the feedback content, which is provided to the user through an avatar in the form of visual and audio feedback.

[0513] Input: Sentiment analysis results

[0514] Output: Regulated Feedback

[0515] Specific operation: The server determines the feedback content based on the results of emotion analysis and conveys it to the user via the avatar. For example, if the user is confused, the avatar will say, "Shall I explain it again?"

[0516] Step 7:

[0517] When the user enters the answer to a question, the device sends the answer to the server, which determines whether the answer is correct and returns the result to the device. The avatar then provides feedback to the user via voice and text.

[0518] Input: User's answer

[0519] Output: Judgment results and feedback

[0520] How it works: The user enters and submits an answer on the device. The server evaluates the answer and returns the result to the device. The avatar provides feedback based on the result.

[0521] Step 8:

[0522] The device reports the user's learning progress to the server, which records the progress data and allows the user to plan their next learning session.

[0523] Input: Learning progress data

[0524] Output: Progress record and suggestions for next study session

[0525] Specific operation: The device sends learning progress data to the server, which records the data. The next learning plan is presented to the user.

[0526] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0527] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0528] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0529] [Second embodiment]

[0530] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0531] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0532] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0533] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0534] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0535] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0536] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0537] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0538] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0539] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0540] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0541] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0542] The present invention relates to a system that provides a custom avatar that functions as a personal tutor and delivers an individually optimized learning experience. The system includes a series of processes for inputting and managing user information, generating and adjusting the custom avatar, providing educational content throughout the learning session, and evaluating the user's answers and providing feedback.

[0543] 1. Initial Setup and User Profile Creation

[0544] First, a user downloads the application and accesses the user registration screen when they first launch it. Here, the user enters their name, age, grade, preferred subject, and desired avatar characteristics (such as gender, appearance, and voice). The device then sends the collected user data to the server, and an appropriate user profile is generated.

[0545] 2. Avatar Creation and Customization

[0546] Based on the generated user profile, the server uses a custom avatar generation algorithm to automatically construct an avatar with the specified characteristics. The generated avatar is sent to the device, where the user can review the avatar's appearance and request fine-tuning if necessary. The fine-tuning request is sent back to the server, which then re-adjusts the avatar. This process is repeated until a final avatar that the user is satisfied with is determined and stored on the server.

[0547] 3. Start your study session

[0548] When a user selects a study session in the application, the device sends a request to the server to provide learning content for the selected subject. The server sends the appropriate problem sets and explanations to the device, which then presents them to the user via an avatar. For example, if a user selects a study session on multiplication in arithmetic, relevant problems and explanations are provided.

[0549] 4. Reading the problem and explaining it

[0550] When a user uploads a photo of a written or handwritten problem with their camera, the device uses OCR technology to convert it into text data. The converted data is sent to a server, which analyzes the problem and generates an answer and detailed explanation. The generated answer and explanation are then sent to the device, where an avatar uses voice functions to explain the answer to the user.

[0551] 5. Answer assessment and feedback

[0552] When a user enters an answer to a question, the device sends the answer to the server. The server compares the answer with the correct answer and returns the result to the device. The device then presents the result to the user, and an avatar provides audio feedback. Through feedback such as "That's right!" or "This is wrong," the user can identify areas for improvement.

[0553] 6. Learning progress management

[0554] The server records the user's answers for each learning session and manages their progress. Progress data is reflected in the next learning session and feedback. The device provides this management information to the user and guides them to the next learning session. The device is also designed to allow for easy schedule changes and adjustments.

[0555] Specific examples

[0556] For example, if a fourth-grade user wants to learn multiplication, he / she first completes the avatar setup and then selects the multiplication learning session. Here, the avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback saying, "That's correct! Well done!" Through this process, the user can check their understanding while progressing through the learning process.

[0557] As described above, through this invention, children can receive high-quality, individually optimized educational support regardless of their family environment or economic situation.

[0558] The processing flow will be explained below.

[0559] Step 1:

[0560] The user downloads the application and launches it for the first time, which displays the user registration screen.

[0561] Step 2:

[0562] The user enters user information such as name, age, grade, preferred subjects, and desired avatar characteristics.

[0563] Step 3:

[0564] The device sends the entered user information to the server, which then creates a user profile.

[0565] Step 4:

[0566] The server generates a custom avatar based on the user profile using an avatar generation algorithm.

[0567] Step 5:

[0568] The server sends the generated custom avatar data to the device.

[0569] Step 6:

[0570] The device displays the custom avatar on the screen, allowing the user to see what the avatar looks like.

[0571] Step 7:

[0572] If a user wants to make minor adjustments to the appearance of their avatar, they make an adjustment request.

[0573] Step 8:

[0574] The device sends a fine-tuning request to the server, which regenerates the avatar based on the adjustment request and sends the data back to the device.

[0575] Step 9:

[0576] Once the user is satisfied with the appearance of the avatar, the avatar is finally finalized and the terminal transmits the finalized data to the server.

[0577] Step 10:

[0578] The server stores the determined avatar information in the user profile.

[0579] Step 11:

[0580] Users select a study session (e.g., "Math Multiplication") from the study schedule screen within the app.

[0581] Step 12:

[0582] The terminal requests the selected learning session information from the server.

[0583] Step 13:

[0584] The server sends the appropriate learning content (problem sets and explanatory materials) to the device.

[0585] Step 14:

[0586] The terminal presents the learning content to the user through the avatar and initiates a learning session.

[0587] Step 15:

[0588] Users can write the question on paper and take a photo of it using the device's camera, or they can enter the question manually.

[0589] Step 16:

[0590] The device uses OCR technology to convert the photographed question into text data, which is then sent to a server.

[0591] Step 17:

[0592] The server analyzes the question text and generates appropriate answers and explanations.

[0593] Step 18:

[0594] The server sends the generated answers and explanations to the terminal.

[0595] Step 19:

[0596] The device uses an avatar to provide audio commentary and teach the user.

[0597] Step 20:

[0598] The user enters the answer into the terminal.

[0599] Step 21:

[0600] The terminal sends the answer to the server.

[0601] Step 22:

[0602] The server determines whether the answer is correct or incorrect and sends the result to the terminal.

[0603] Step 23:

[0604] The device displays the results and an avatar provides audible feedback such as "That's right!" or "This is wrong."

[0605] Step 24:

[0606] The server records the user's answers and updates the learning progress data.

[0607] Step 25:

[0608] The device will prompt the user for the next study session or provide options to reschedule.

[0609] Step 26:

[0610] The user selects or schedules the next study session, which causes the system to start the study process again.

[0611] Through these steps, users can study efficiently and effectively.

[0612] Example 1

[0613] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0614] Conventional online learning systems have struggled to provide a personalized educational experience for each user, and have struggled to provide smooth feedback to maximize learning outcomes. Furthermore, managing learning progress can be unintuitive for users, making it difficult to develop long-term learning plans and maintain motivation. Furthermore, they are unable to provide detailed explanations of paper-based questions and their answers in real time, preventing efficient learning.

[0615] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0616] In this invention, the server includes means for inputting user information, means for generating a custom avatar based on the input user information, means for displaying the generated custom avatar, means for receiving a request from the user to fine-tune the avatar, means for regenerating the avatar based on the fine-tuning request, means for saving the finalized avatar, means for the user to select a study session, means for providing study content based on the selected study session, means for asking questions and providing explanations based on the provided study content, means for receiving the user's answers and determining whether they are correct or incorrect, means for providing feedback based on the determination results, means for managing the user's study progress, means for generating answers and detailed explanations using artificial intelligence, and means for displaying study progress management information in a dashboard format, thereby enabling the provision of a personalized learning experience, real-time feedback, and intuitive study progress management.

[0617] "User information" refers to information such as name, age, grade, preferred subjects, and desired avatar characteristics that a user enters when using an application.

[0618] The "custom avatar generation means" refers to an algorithm or program for automatically creating a custom avatar that matches the characteristics of the user based on input user information.

[0619] The "avatar display means" refers to an interface or device for visually presenting the generated custom avatar to the user.

[0620] The "means for receiving a fine-tuning request" is an interface or program that allows a user to request fine-tuning of characteristics such as appearance or voice of a custom avatar.

[0621] "Avatar regeneration means" refers to an algorithm or program for regenerating an avatar based on the user's fine-tuning requests.

[0622] The "avatar storage means" refers to a program or storage for saving the finalized custom avatar in a database or the like.

[0623] A "learning session selection means" is an interface or program that allows a user to select a particular learning session.

[0624] The "learning content providing means" refers to a system or program that provides appropriate learning content to a user based on the selected learning session.

[0625] A "question-setting means" is a program or interface that presents questions to users based on the learning content provided and provides explanations.

[0626] The "answer receiving means" is an interface for users to input answers and a program for sending the answers to the server.

[0627] The "correctness determination means" is an algorithm or program for determining whether the user's answer is correct or incorrect.

[0628] "Feedback providing means" refers to a program or system that provides feedback to the user in the form of voice or text based on the result of determining whether the answer is correct or incorrect.

[0629] A "learning progress management means" is a system or program that records and manages a user's learning progress and uses the information to help with the next learning session.

[0630] "Artificial intelligence" refers to machine learning algorithms and neural network technology that analyzes users' answers and generates answers and detailed explanations.

[0631] The "means for displaying in a dashboard format" refers to an interface or program for displaying learning progress management information in a visually easy-to-understand dashboard format.

[0632] MODE FOR CARRYING OUT THE INVENTION

[0633] The present invention provides a system that provides a custom avatar that functions as a personal tutor and delivers an individually optimized learning experience. This system includes a series of processes for inputting and managing user information, generating and adjusting the custom avatar, providing educational content throughout the learning session, and evaluating the user's answers and providing feedback.

[0634] Hardware and software used

[0635] Server: Manages user information, generates and stores custom avatars, provides learning content, judges answers, generates feedback, and manages learning progress.

[0636] Device: Enter user information, view a custom avatar, select a study session, enter and submit answers, view feedback, and view study progress.

[0637] OCR library (e.g. Tesseract): Converts paper questions into text data.

[0638] Generative AI models (e.g., machine learning algorithms, neural networks): generate answers and detailed explanations.

[0639] Initial Setup and User Profile Creation

[0640] First, the user downloads the application and accesses the user registration screen when launching it for the first time. Here, the user enters their name, age, grade, preferred subjects, and desired avatar characteristics (such as gender, appearance, and voice). The device collects this information and sends it to the server. The server then creates a user profile based on the received information and sends it to the device.

[0641] Avatar Creation and Customization

[0642] The server uses a custom avatar generation algorithm to automatically construct an avatar with the specified characteristics. The generated avatar is sent to the device, where the user can review its appearance and request fine-tuning if necessary. A retuning request is sent back to the server, which then retunes the avatar. This process is repeated until the user finally finds an avatar they are happy with and stores it on the server.

[0643] Start a study session

[0644] When a user selects a study session in the application, the device sends a request to the server to provide study content for the selected subject. The server sends the appropriate problem sets and explanations to the device, which then presents them to the user through an avatar.

[0645] Problem reading and explanation

[0646] When a user uploads a photo of a written or handwritten problem with their camera, the device converts it into text using OCR technology. The converted data is sent to a server, which analyzes the problem and generates an answer and detailed explanation. The generated answer and explanation are then sent to the device, where an avatar uses voice functions to explain the answer to the user.

[0647] Answer assessment and feedback

[0648] When a user enters an answer to a question, the device sends the answer to the server. The server compares the answer with the correct answer and sends the result back to the device. The device then presents the result to the user, and an avatar provides audio feedback. Through feedback such as "That's right!" or "This is wrong," the user can identify areas for improvement.

[0649] Learning progress management

[0650] The server records the user's answers for each learning session and manages their progress. Progress data is reflected in the next learning session and feedback. The device provides this management information to the user and guides them to the next learning session. The device is also designed to allow for easy schedule changes and adjustments.

[0651] Specific examples

[0652] For example, if a fourth-grade user wants to learn multiplication, he / she first completes the avatar setup and then selects the multiplication learning session. Here, the avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback saying, "That's correct! Well done!" Through this process, the user can check their understanding while progressing through the learning process.

[0653] Prompt Sentence Examples

[0654] Prompt statement:

[0655] "Analyze a math problem solved by a user and generate the answer and a detailed explanation. The problem is '12 x 8' and the answer is '96'."

[0656] In this way, this invention provides an individually optimized learning experience and effective feedback, enabling children to receive high-quality educational support at home.

[0657] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0658] Step 1: Enter and submit user information

[0659] Input: User information such as name, age, grade, preferred subject, desired avatar characteristics

[0660] Specific operation: The user downloads the application, accesses the user registration screen when launching it for the first time, and enters the required information.

[0661] Data processing / calculation: The terminal temporarily stores the input information in its memory.

[0662] Output: Collected user information

[0663] Step 2: Generate a user profile

[0664] Input: User information

[0665] Specific operation: The device sends the collected user data to the server.

[0666] Data processing / calculation: The server creates a new user profile in the database based on the received information and enters the corresponding data into each field.

[0667] Output: Generated user profile

[0668] Step 3: Generate and submit a custom avatar

[0669] Input: User Profile

[0670] What Happens: The server uses a custom avatar generation algorithm to build an avatar based on the user profile.

[0671] Data processing / calculation: Use an AI-based avatar generation model to generate an avatar with specified characteristics.

[0672] Output: The generated avatar

[0673] Step 4: Review your avatar and request tweaks

[0674] Input: Generated avatar

[0675] Specific operation: The generated avatar is sent to the device, and the user can check its appearance and request fine-tuning if necessary.

[0676] Data processing / calculation: The device temporarily stores the user's fine-tuning request in memory.

[0677] Output: Tweak request

[0678] Step 5: Regenerate your avatar

[0679] Input: Tweak Request

[0680] Specific operation: The device sends a fine-tuning request to the server, which adjusts and regenerates an avatar with the specified characteristics.

[0681] Data processing / calculation: The server receives the fine-tuning request and regenerates the avatar based on the avatar generation algorithm.

[0682] Output: Regenerated avatar

[0683] Step 6: Save the final avatar

[0684] Input: Regenerated avatar

[0685] Specific operation: The server saves the final avatar that the user is satisfied with in the database.

[0686] Data processing / calculation: The regenerated avatar data is saved in the server database.

[0687] Output: Final saved avatar

[0688] Step 7: Select and request a study session

[0689] Input: User's learning session selection information

[0690] Specific operation: A user selects a learning session in the application. The device sends a content request to the server based on the selected subject.

[0691] Data processing / calculation: Based on the information of the selected learning session, a request is generated and sent to the server.

[0692] Output: Learning content request

[0693] Step 8: Provide learning content

[0694] Input: Learning content request

[0695] Specific operation: The server selects an appropriate problem set and explanatory materials and sends them to the terminal.

[0696] Data processing / calculation: Retrieve problem sets and explanatory materials from the database and compose the selected content.

[0697] Output: Problem set and explanations

[0698] Step 9: Questions and Answers

[0699] Input: Problem set and explanation materials

[0700] Specific operation: The device displays the problem to the user through the avatar and provides an explanation. The user then inputs the answer.

[0701] Data processing / calculation: Display the problem set and receive user answer input.

[0702] Output: User's answer

[0703] Step 10: Answer assessment

[0704] Input: User's answer

[0705] Specific operation: The terminal sends the user's answer to the server, which determines whether the answer is correct or not.

[0706] Data processing / calculation: The answer is compared with the correct data to generate a correct or incorrect result.

[0707] Output: Judgment result

[0708] Step 11: Provide feedback

[0709] Input: Judgment result

[0710] Specific operation: The server sends the judgment result to the terminal, and the terminal provides feedback to the user through the avatar.

[0711] Data processing / calculation: The feedback content is constructed based on the judgment results.

[0712] Output: Feedback

[0713] Step 12: Manage your learning progress

[0714] Input: Feedback and Answer Results

[0715] Specific operation: The server records the user's answers for each learning session and manages the progress. The device displays this management information in a dashboard format.

[0716] Data processing / calculation: Answer results and feedback are stored in a database, and progress is visualized.

[0717] Output: Progress management data and dashboard display

[0718] (Application example 1)

[0719] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0720] Conventional learning support and training systems have struggled to provide individually optimized support for each user. Training systems for new employees and field workers also struggled to provide real-time guidance and feedback tailored to on-site work. As a result, there was a challenge in being unable to provide appropriate support tailored to each user's level of understanding and progress.

[0721] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0722] In this invention, the server includes means for inputting user information, means for generating a custom avatar based on the input user information, means for displaying the generated custom avatar, means for the user to receive a request to fine-tune the avatar, means for regenerating the avatar based on the fine-tuning request, means for saving the finalized avatar, means for the user to select a study session, means for providing study content based on the selected study session, means for asking questions and providing explanations based on the provided study content, means for receiving the user's answers and determining whether they are correct or incorrect, means for providing feedback based on the determination results, means for managing the user's study progress, means for a new employee or field worker to create their own profile and for providing training content based on the profile, and means for recording the user's operations and results with a camera and providing feedback. This enables individually optimized study support and training.

[0723] "Means for inputting user information" refers to a device or program that allows a user to input information such as name, age, and work experience.

[0724] The "means for generating a custom avatar based on input user information" refers to a device or program that generates an avatar using a specific algorithm based on information input by the user.

[0725] The "means for displaying the generated custom avatar" is a device or program for visually displaying the generated avatar on a screen.

[0726] A "means for accepting user requests to fine-tune an avatar" is a device or program that allows a user to input requests to change characteristics such as the appearance or voice of the avatar.

[0727] The "means for regenerating an avatar based on a fine-tuning request" refers to a device or program that adjusts the characteristics of the avatar and regenerates it based on a user request.

[0728] The "means for saving the finalized avatar" is a device or program for saving the finalized avatar that satisfies the user in a database or the like.

[0729] A "means for user selection of a learning session" is a device or program that allows a user to select the learning content or training session they wish to use.

[0730] The "means for providing learning content based on a selected learning session" refers to a device or program for providing a user with learning materials and questions according to the selected learning session.

[0731] "Means for asking questions and providing explanations based on the provided learning content" refers to a device or program that presents questions to the user based on the learning content and provides explanations for the answers.

[0732] The "means for receiving the user's answer and determining whether it is correct" is a device or program for receiving the answer entered by the user and determining whether it is correct.

[0733] The "means for providing feedback based on the judgment result" is a device or program that provides appropriate feedback to the user based on the result of the judgment of whether the answer is correct or incorrect.

[0734] "Means for managing a user's learning progress" refers to a device or program for recording and managing a user's learning history and progress.

[0735] A "means for new employees or field workers to create their own profile" is a device or program that allows new employees or field workers to enter information such as their name, work experience, and learning goals to create a profile.

[0736] The "means for providing training content based on a profile" refers to a device or program for providing individually optimized training materials and content based on the created profile information.

[0737] "Means for recording the operations and results performed by the user with a camera and providing feedback" refers to a device or program that uses a camera to capture the operations performed by the user, analyzes the data, and provides appropriate feedback.

[0738] This invention is a system that provides a custom avatar that acts as a personal tutor and delivers an individually optimized learning experience. In particular, it includes a system that provides practical training in real time for new employees and field workers. Specific implementation methods of the invention are as follows.

[0739] First, the user downloads a dedicated application onto their smartphone or tablet. When they first launch the application, they access the user registration screen and enter their name, age, work experience, and other user information. This information is then sent from the device to the server.

[0740] The server then uses a custom avatar generation algorithm to build an avatar based on the user's profile. The generated avatar is sent to the device, where the user can review its appearance and request tweaks if necessary. The tweaked information is sent back to the server, and the avatar is regenerated. This process is repeated until the user is satisfied, and the final avatar is saved on the server.

[0741] The user then selects a learning session on the dedicated application. The training content is based on practical applications such as machine operation and safety management. For example, if a new employee is receiving training on machine operation in a factory, the server sends the selected training content to the terminal and provides it to the user via an avatar.

[0742] During training, the user's operations are recorded using the device's camera function. Using OCR technology (using Tesseract.js), the recorded operations are converted into text data and sent to the server. The server generates the analysis results, and the avatar provides feedback using the voice function. For example, if the user performs an incorrect operation, the avatar will say, "This is incorrect. Please do it this way next time."

[0743] The server then receives the user's answer and determines whether it is correct. The result of the determination is immediately provided to the user as feedback via the avatar. For example, if the user makes a mistake in the operating procedure, the server will provide feedback such as, "This is an error. The correct way is this."

[0744] Finally, the server has the function of managing the user's learning progress. It records the results and answer history of each training session and reflects them in the next learning session. It also provides new training content according to the user's progress, supporting continuous skill development.

[0745] Examples:

[0746] For example, when a new employee receives training on operating machinery in a factory, the user creates a profile on a dedicated tablet and customizes the avatar's appearance and voice. The server then explains safe operating procedures for the machine via the avatar, and the user performs the procedures. The operations recorded by the camera are converted into text data using OCR technology (Tesseract.js) and analyzed by the server. Based on the results, the avatar provides feedback, such as "This is incorrect."

[0747] Example prompts to input to a generative AI model:

[0748] Create a new worker profile in the system. Input data includes name, age, and work experience. Follow the steps below to provide training content for operating factory robots. The camera will capture the user's operation results, convert them into text data using OCR technology, and provide feedback via an avatar.

[0749] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0750] Step 1:

[0751] Users use their smartphones or tablets to download a dedicated application.

[0752] Input: User operation (application download)

[0753] Action: Install application

[0754] Output: Application initial launch screen

[0755] Step 2:

[0756] A user starts the application and accesses the user registration screen, where they enter their user information such as name, age, and work experience.

[0757] Input: User information (name, age, work experience, etc.)

[0758] Process: Enter and submit user information

[0759] Output: A user profile is generated on the server.

[0760] Step 3:

[0761] The server uses a custom avatar generation algorithm to generate a custom avatar based on the user profile, and the generated avatar is sent to the device.

[0762] Input: User profile information

[0763] Processing: Custom avatar generation (Unity3D, TensorFlow)

[0764] Output: Generated custom avatar data

[0765] Step 4:

[0766] The custom avatar is displayed on the device, allowing the user to check its appearance, voice, etc., and request tweaks as needed.

[0767] Input: Generated avatar data

[0768] Action: Display a custom avatar

[0769] Output: Avatar confirmation screen

[0770] Step 5:

[0771] The device accepts the user's fine-tuning request and sends it to the server, which then fine-tunes the avatar, regenerates it, and sends it to the device.

[0772] Input: Tweak request from user

[0773] Action: Regenerate custom avatar

[0774] Output: Fine-tuned custom avatar data

[0775] Step 6:

[0776] The final custom avatar is saved on the server and the user selects a learning or training session.

[0777] Input: Confirmed custom avatar data, user session selection

[0778] Actions: Save custom avatar, select learning or training session

[0779] Output: Session start screen

[0780] Step 7:

[0781] The server sends appropriate content to the terminal based on the selected learning or training session.

[0782] Input: Selected information for your study or training session

[0783] Processing: Selection and delivery of training content (using Django)

[0784] Output: Training content

[0785] Step 8:

[0786] The user performs the training and records the operations and results using the device's camera.

[0787] Input: The action taken by the user

[0788] Process: Recording the operation (using a camera)

[0789] Output: Recorded operation data

[0790] Step 9:

[0791] The device converts the recorded operation data into text data using OCR technology (Tesseract.js) and sends it to the server.

[0792] Input: Recorded operation data

[0793] Processing: Converting operation data into text (OCR technology)

[0794] Output: Text data

[0795] Step 10:

[0796] The server analyzes the text data, and the avatar provides audio feedback based on the results.

[0797] Input: Text data

[0798] Processing: Data analysis and feedback generation (using Scikit-learn)

[0799] Output: Generate feedback content

[0800] Step 11:

[0801] The device provides the user with audible feedback via an avatar.

[0802] Input: Feedback

[0803] Processing: Audio feedback

[0804] Output: Audio feedback

[0805] Step 12:

[0806] The server manages the user's learning progress, recording the results of each training session and reflecting them in the next session.

[0807] Input: Results of each training session

[0808] Processing: Data management of learning progress (using MongoDB)

[0809] Output: Learning progress data and what to expect in the next session

[0810] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0811] The present invention relates to a personal tutor avatar system for assisting users in their learning, specifically a system that combines an emotion engine to recognize the user's emotional state and adjust dialogue and feedback accordingly to provide a more effective learning experience. This system includes processes for inputting user information, generating and displaying a custom avatar, managing a learning session, and recognizing the user's emotions to adjust dialogue content.

[0812] 1. Initial Setup and User Profile Creation

[0813] First, a user downloads the application and accesses the user registration screen when they first launch it. Here, the user enters their name, age, grade, preferred subject, and desired avatar characteristics (such as gender, appearance, and voice). The device then sends the collected user data to the server, and an appropriate user profile is generated.

[0814] 2. Avatar Creation and Customization

[0815] Based on the generated user profile, the server uses a custom avatar generation algorithm to automatically construct an avatar with the specified characteristics. The generated avatar is sent to the device, where the user can review the avatar's appearance and request fine-tuning if necessary. The fine-tuning request is sent back to the server, which then re-adjusts the avatar. This process is repeated until a final avatar that the user is satisfied with is determined and stored on the server.

[0816] 3. Start your study session

[0817] When a user selects a study session in the application, the device sends a request to the server to provide learning content for the selected subject. The server sends the appropriate problem sets and explanations to the device, which then presents them to the user via an avatar. For example, if a user selects a study session on multiplication in arithmetic, relevant problems and explanations are provided.

[0818] 4. Use of Emotion Engine

[0819] During the learning session, the device's built-in camera and microphone collect the user's facial expressions and voice data. This data is sent to the server in real time and analyzed by the emotion engine. Based on the analysis results, the server recognizes the user's emotional state (e.g., "happiness," "sadness," "anger," "surprise," etc.).

[0820] 5. Reading the problem and explaining it

[0821] When a user uploads a photo of a written or handwritten problem with their camera, the device uses OCR technology to convert it into text data. The converted data is sent to a server, which analyzes the problem and generates an answer and detailed explanation. The generated answer and explanation are then sent to the device, where an avatar uses voice functions to explain the answer to the user.

[0822] 6. Answer assessment and feedback

[0823] When a user enters an answer to a question, the device sends the answer to the server. The server compares it with the correct answer and returns the result to the device. The device then presents the result to the user, and an avatar provides audio feedback. Furthermore, the avatar's dialogue content and tone can be adjusted based on the user's recognized emotions. For example, if the user is confused, the avatar will use encouraging words such as "It's okay, let me explain it again."

[0824] 7. Learning progress management

[0825] The server records the user's answers for each study session and manages their progress. Progress data is reflected in the next study session and feedback. It also records recognized emotional data and uses this information to optimize study. For example, if a user frequently feels frustrated with a particular problem, it can provide additional support materials related to that problem. The device provides this management information to the user and guides them through the next study session. It is also designed to allow for easy schedule changes and adjustments.

[0826] Specific examples

[0827] For example, if a fourth-grade user wants to learn multiplication, he / she first completes the avatar setup and then selects the multiplication learning session. Here, the avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback, saying, "That's correct! Well done!" At the same time, if the user looks confused, the avatar will provide additional support, saying, "Shall I explain it again?" Through this process, the user can progress through their learning while checking their understanding.

[0828] As described above, through this invention, children can receive high-quality, individually optimized educational support regardless of their family environment or financial situation. The use of an emotion engine further improves the learning experience by recognizing the user's emotional state and providing appropriate feedback accordingly.

[0829] The processing flow will be explained below.

[0830] Step 1:

[0831] The user downloads the application and launches it for the first time, which displays the user registration screen.

[0832] Step 2:

[0833] The user enters user information such as name, age, grade, preferred subjects, and desired avatar characteristics.

[0834] Step 3:

[0835] The device sends the entered user information to the server, which then creates a user profile.

[0836] Step 4:

[0837] The server generates a custom avatar based on the user profile using an avatar generation algorithm.

[0838] Step 5:

[0839] The server sends the generated custom avatar data to the device.

[0840] Step 6:

[0841] The device displays the custom avatar on the screen, allowing the user to see what the avatar looks like.

[0842] Step 7:

[0843] If a user wants to make minor adjustments to the appearance of their avatar, they make an adjustment request.

[0844] Step 8:

[0845] The device sends a fine-tuning request to the server, which regenerates the avatar based on the adjustment request and sends the data back to the device.

[0846] Step 9:

[0847] Once the user is satisfied with the appearance of the avatar, the avatar is finally finalized and the terminal transmits the finalized data to the server.

[0848] Step 10:

[0849] The server stores the determined avatar information in the user profile.

[0850] Step 11:

[0851] Users select a study session (e.g., "Math Multiplication") from the study schedule screen within the app.

[0852] Step 12:

[0853] The terminal requests the selected learning session information from the server.

[0854] Step 13:

[0855] The server sends the appropriate learning content (problem sets and explanatory materials) to the device.

[0856] Step 14:

[0857] The terminal presents the learning content to the user through the avatar and initiates a learning session.

[0858] Step 15:

[0859] The device's built-in camera and microphone are used to collect the user's facial expressions and voice data in real time.

[0860] Step 16:

[0861] The collected facial and voice data is sent to a server, which uses an emotion engine to analyze the data and recognize the user's emotional state.

[0862] Step 17:

[0863] When a user takes a photo of a question written on paper or entered manually and uploads it, the device uses OCR technology to convert it into text data, which is then sent to the server.

[0864] Step 18:

[0865] The server analyzes the question text and generates appropriate answers and explanations.

[0866] Step 19:

[0867] The server sends the generated answers and explanations to the terminal.

[0868] Step 20:

[0869] The device provides audio commentary from an avatar, educating the user and adjusting the content and tone of the dialogue based on the user's emotional state.

[0870] Step 21:

[0871] The user enters the answer into the terminal.

[0872] Step 22:

[0873] The terminal sends the answer to the server.

[0874] Step 23:

[0875] The server determines whether the answer is correct or incorrect and sends the result to the terminal.

[0876] Step 24:

[0877] The device displays the results and an avatar provides audible feedback such as "That's right!" or "This is wrong."

[0878] Step 25:

[0879] The server records the user's answers and updates their learning progress data, along with their emotional state.

[0880] Step 26:

[0881] The device will prompt the user for the next study session or provide options to reschedule.

[0882] Step 27:

[0883] The user selects or schedules the next study session, which causes the system to start the study process again.

[0884] Through these steps, users can learn efficiently and effectively. By utilizing the emotion engine, appropriate feedback and support are provided that takes into account the user's emotional state, resulting in a more personalized learning experience.

[0885] Example 2

[0886] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0887] Conventional learning support systems have the problem of being unable to provide an effective learning experience that takes into account the user's emotional state. In addition, they are unable to provide customized feedback optimized for each individual user, which can reduce the effectiveness of learning.

[0888] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting facial expression and voice data of the user and analyzing it with an emotion engine, means for adjusting the dialogue content based on the analysis results and providing emotionally sensitive feedback, and means for managing the user's learning progress. This allows for an effective learning experience by providing detailed feedback according to the user's emotional state.

[0889] - "User Information" refers to personal information about a User, such as the User's name, age, grade, preferred subjects, and desired avatar characteristics.

[0890] "Custom avatar" refers to an avatar created specifically for a user based on user information.

[0891] "Study Session" means a study session offered based on a subject or topic selected by a User.

[0892] "Study Content" refers to the problem sets and explanatory materials provided based on a study session.

[0893] "Questions" refer to assignments or quizzes presented to users within the learning content.

[0894] "Answer" refers to the answer entered by the user in response to a question.

[0895] "Explanation" refers to explanations or supplementary information for the answers to questions.

[0896] "Feedback" refers to the evaluation and advice provided to users based on whether their answers are correct or incorrect.

[0897] "Facial expression data" refers to data collected by capturing a user's facial expressions with a camera.

[0898] "Voice data" refers to data collected by capturing the user's voice with a microphone.

[0899] An "emotion engine" refers to algorithms or software that analyzes facial expression and voice data to recognize the user's emotional state.

[0900] "Learning progress" refers to a user's progress, results, and achievements in learning.

[0901] "Means of collection" refers to devices or methods for obtaining data from users, such as cameras or microphones.

[0902] "Means of analysis" refers to the techniques and methods for processing collected data and analyzing emotions and learning trends.

[0903] "Means for adjusting dialogue content" refers to a mechanism for appropriately changing the avatar's speech and tone depending on the user's emotional state.

[0904] "Camera function" refers to a device and its functions for taking images and videos.

[0905] "OCR technology" refers to the technology that recognizes characters in an image and converts them into text data.

[0906] The present invention relates to a personal tutor avatar system for supporting users in their learning, specifically a system that combines an emotion engine to recognize the user's emotional state and adjust dialogue and feedback accordingly to provide a more effective learning experience. This system includes processes for inputting user information, generating and displaying a custom avatar, managing a learning session, and adjusting dialogue content based on the user's emotions.

[0907] First, the user downloads the learning support application and accesses the user registration screen when launching it for the first time. Here, the user enters their name, age, grade, preferred subjects, and desired avatar characteristics (gender, appearance, voice, etc.). This information is saved on the device. The device temporarily stores the information entered by the user and sends it to the server as soon as registration is complete. A message indicating registration is complete is displayed, and the device prepares to proceed to the next step.

[0908] The server generates an individual user profile based on the received user information. The generated profile is stored in a database and used in subsequent processes. The server then activates a custom avatar generation algorithm to automatically generate an avatar with the specified characteristics based on the user profile. The generated avatar data is sent to the device, which receives the data and displays the avatar on its screen.

[0909] The user checks the appearance of the avatar and requests fine-tuning if necessary. This request is sent from the device to the server, and the server regenerates the avatar reflecting the fine-tuning and sends the data back to the device. This process is repeated until the user is satisfied, and once the final avatar is finalized, the avatar data is saved on the server.

[0910] When a user selects a learning session, the device sends a request to the server to provide learning content for the selected subject. The server receives the request, prepares the corresponding problem set and explanatory materials, and sends the data to the device. Based on the received data, the device presents the learning session for the selected subject through an avatar. For example, in a learning session on multiplication in arithmetic, problems such as "12 x 8" are presented.

[0911] During a learning session, the device uses a camera and microphone to collect the user's facial and voice data. This data is sent to the server in real time and analyzed by the emotion engine. The server then analyzes the received facial and voice data using the emotion engine to recognize the user's emotional state (e.g., "happiness," "sadness," "anger," "surprise," etc.).

[0912] When a user writes a question on paper, takes a photo of it with a camera, and uploads it, the device uses OCR technology to convert the image data into text data. This converted text data is then sent to a server. The server analyzes the received text data and generates the answer to the question and a detailed explanation. The generated answer and explanation are then sent back to the device, which then provides them to the user through an avatar. The avatar uses voice functionality to provide explanations.

[0913] When a user enters an answer to a question and presses the send button, the device sends the answer data to the server. The server compares the received answer with the correct answer and sends the result back to the device. The result includes a correct / incorrect determination and detailed feedback. The device receives the result, and an avatar gives the user audio feedback. The content and tone of the feedback can be adjusted depending on the user's emotional state, and may use encouraging words such as "Shall I explain it again?"

[0914] The server records the user's answers for each study session and manages the user's study progress, including the results for each session and the user's emotional data. The server uses the recorded progress data to prepare for optimizing the next study session. The device notifies the user when the next study session is ready and helps the user easily manage their study progress and schedule.

[0915] Specific examples

[0916] For example, if a fourth-grade user wants to learn multiplication, he or she first sets up the avatar and then selects the multiplication learning session. The avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback, saying, "That's right! Well done!" Furthermore, if the user shows a confused expression, the avatar offers additional support by asking, "Shall I explain it again?"

[0917] Example of input prompt for generative AI model

[0918] "If the user's facial expression looks confused, have the avatar generate encouraging feedback, such as 'Shall I explain it again?' or 'Please let me know if you don't understand.'"

[0919] As described above, the present invention makes it possible to individually optimize the learning experience of each user and provide more effective learning support.

[0920] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0921] Step 1: User registration and initial setup

[0922] First, the user downloads the learning support application and accesses the user registration screen when the application is launched for the first time. The user enters their name, age, grade, preferred subjects, and desired avatar characteristics (gender, appearance, voice, etc.). This information is saved on the device. The device temporarily stores the user information and sends it to the server as soon as registration is complete. The server saves the received information in a database and returns a confirmation response to the device. The device then displays a message indicating that user registration has been completed.

[0923] Input: User information (name, age, grade, preferred subject, avatar characteristics)

[0924] Output: User information sent to the server, user registration complete message

[0925] Step 2: Create and save a user profile

[0926] The server generates a user profile based on the received user information. The generated profile is saved in a database for use in subsequent processes. A confirmation response is sent to the device confirming that the profile has been saved.

[0927] Input: User information

[0928] Output: Generated user profile, confirmation that it was saved by the server

[0929] Step 3: Avatar Creation and Customization

[0930] The server runs a custom avatar generation algorithm to automatically generate an avatar with the specified characteristics based on the user profile. The generated avatar data is sent to the device, where the user can view the displayed avatar. If the user requests fine-tuning as needed, this request is sent back to the server, and the avatar is regenerated. Finally, once the user is satisfied with the avatar, the avatar data is saved on the server.

[0931] Input: User profile, Tweak request

[0932] Output: Custom avatar, Retuned avatar

[0933] Specific actions: Adjusting the avatar's gender, appearance, voice, and other characteristics

[0934] Step 4: Start your study session

[0935] When a user selects a study session, the device sends a request to the server to provide study content according to the selected subject. The server receives the request, prepares the corresponding problem sets and explanation materials, and sends the data to the device. The device then presents the study session through an avatar based on the received data.

[0936] Input: User selects a study session

[0937] Output: Learning content (question sets, explanatory materials)

[0938] Specific actions: Displaying learning content and presenting questions

[0939] Step 5: Emotion Recognition with the Emotion Engine

[0940] During a learning session, the device uses a camera and microphone to collect facial and voice data from the user. This data is sent in real time to a server where it is analyzed by an emotion engine. The server recognizes the user's emotional state and adjusts feedback accordingly.

[0941] Input: facial expression data, voice data

[0942] Output: Analysis results (user's emotional state)

[0943] Specific actions: taking pictures of facial expressions with a camera and collecting audio with a microphone

[0944] Step 6: Reading the problem and generating an explanation

[0945] When a user takes a photo of a paper problem and uploads it, the device uses OCR technology to convert the image data into text. This converted data is sent to a server, which analyzes the problem, generates answers and explanations, and sends them back to the device. The device then provides these to the user through an avatar.

[0946] Input: Image data of the problem

[0947] Output: OCR text data, answers and explanations

[0948] Specific actions: taking a photo with a camera, OCR conversion, and explanation by an avatar

[0949] Step 7: Answers and feedback

[0950] The user inputs the answer to the question and sends it to the device. The device then sends the answer data to the server, which checks the answer against the correct answer and returns a judgment result. The device then presents the result to the user, and an avatar provides audio feedback. The content and tone of the feedback are adjusted according to the user's emotional state.

[0951] Input: User-provided answer data

[0952] Output: Matching result and feedback (correct / incorrect)

[0953] Specific actions: Entering answers, checking, and voice feedback

[0954] Step 8: Track your progress and prepare for the next session

[0955] The server records the user's answers for each study session and manages their study progress, including their grades and emotional state for each session. Based on the recorded progress data, the server optimizes the next study session and notifies the device when the next study session is ready. It also makes it easy to change the progress and schedule.

[0956] Input: Answer result, emotional state

[0957] Output: Tracked progress data, next session notifications

[0958] Specific actions: Recording results, managing emotional data, notifying preparation for the next session

[0959] (Application example 2)

[0960] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0961] Conventional learning support systems do not take into account the user's emotional state, and therefore have the problem of being unable to provide appropriate support when the user becomes confused or frustrated. This reduces the effectiveness of learning and leads to a lack of sustained user motivation. Furthermore, there is a lack of technology that can analyze the user's emotions in real time and appropriately adjust feedback based on the results.

[0962] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for inputting user information; means for generating a custom avatar based on the input user information; means for displaying the generated custom avatar; means for accepting a user request to fine-tune the avatar; means for regenerating the avatar based on the fine-tuning request; means for saving the finalized avatar; means for the user to select a study session; means for providing study content based on the selected study session; means for asking questions and providing explanations based on the provided study content; means for receiving the user's answers and determining whether they are correct or incorrect; means for providing feedback based on the determination results; means for managing the user's study progress; means for collecting the user's facial expression and voice data using a camera and microphone function; means for analyzing the user's emotions based on the collected data; and means for adjusting the feedback content based on the analysis results. This makes it possible to provide appropriate feedback according to the user's emotional state, improve learning effectiveness, and maintain the user's motivation.

[0963] "User information" refers to basic data about the user, including name, age, grade, preferred subjects, and avatar characteristics.

[0964] A "custom avatar" is a character with a unique appearance and voice that is generated based on user input.

[0965] A "study session" is a period of time or a particular learning activity during which a user studies.

[0966] "Study Content" means digital content, including study materials, problem sets, and explanatory materials, that a user works through during a study session.

[0967] "Facial expression and voice data" refers to data that records a user's facial expressions, tone of voice, and the content of their statements.

[0968] "Emotion analysis" refers to the process of identifying a user's emotional state (such as joy, sadness, anger, surprise, etc.) based on collected facial and voice data.

[0969] "Feedback" refers to advice, additional explanations, encouraging messages, etc. provided based on the user's learning progress and the accuracy of their answers.

[0970] "Progress management" is the process of recording and tracking a user's learning progress and grades to help plan future learning.

[0971] The present invention provides a personal tutor avatar system for supporting users in their learning. In particular, the system combines an emotion engine to recognize the user's emotional state and adjust dialogue and feedback based on that to provide a more effective learning experience.

[0972] The system is primarily composed of a server and a terminal. The server manages user information and performs key processes such as generating custom avatars, managing learning sessions, analyzing emotions, and adjusting feedback. The terminal is the device used by the user (smartphone, tablet, etc.) and is responsible for interacting with the user.

[0973] 1. Initial Setup

[0974] When a user starts the application for the first time, they download it and enter their user information (such as name, age, grade, preferred subjects, and avatar characteristics). The device sends this data to the server, and a user profile is created.

[0975] 2. Custom Avatar Creation

[0976] The server generates a custom avatar based on the information entered by the user. This avatar is tailored to the user's visual and auditory preferences. The generated avatar is sent to the device, where the user can review and fine-tune it. The avatar is regenerated upon the user's request, and the final avatar is stored on the server.

[0977] 3. Start your study session

[0978] When a user selects a learning session, the device requests it from the server, which then provides the device with the appropriate learning content based on the selected subject. For example, if a user selects a math session, the server provides relevant questions and explanations.

[0979] 4. Use of Emotion Engine

[0980] The device uses a camera and microphone to collect the user's facial and voice data. The collected data is sent to the server in real time, and the server analyzes the user's emotions using an emotion engine (e.g., the emotion analysis model in the Transformers library). Based on the analysis results, the server adjusts the feedback content and provides appropriate assistance to the user.

[0981] 5. Answer assessment and feedback

[0982] When a user enters an answer to a question, the device sends the answer to the server. The server determines whether the answer is correct and returns feedback to the device. The avatar provides feedback to the user in voice and text. For example, if the user's expression looks confused, encouraging feedback such as "Shall I explain it again?" is provided.

[0983] Specific examples

[0984] For example, if a fourth-grade user wants to learn multiplication, he or she can select a multiplication learning session after completing the avatar setup. Here, the avatar will present the problem "12 x 8." If the user enters "96" as the answer, the system will immediately check the answer and provide audio feedback saying, "That's correct! Well done!" At the same time, if the user shows a confused expression, the avatar will provide additional support by saying, "Shall I explain it again?"

[0985] Prompt Sentence Examples

[0986] Analyze what emotions users are expressing. Classify emotions such as "Happy," "Sadness," "Anger," "Surprise," etc.

[0987] This system provides appropriate feedback according to the user's emotional state, improving learning effectiveness and maintaining user motivation.

[0988] As described above, through the present invention, the learning assistance system can dynamically analyze the user's emotions and provide appropriate feedback based on the results, thereby improving the learning experience.

[0989] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0990] Step 1:

[0991] When a user downloads the application and launches it for the first time, they enter their user information, including their name, age, grade, preferred subjects, and avatar characteristics. The device then sends this data to a server, which creates a user profile.

[0992] Input: User information (name, age, grade, subject, avatar characteristics)

[0993] Output: User profile (stored on the server)

[0994] Specific operation: The user enters information into the input form on the terminal and presses the send button. The terminal then sends the information to the server.

[0995] Step 2:

[0996] The server generates a custom avatar based on the received user information, which is then sent to the device for the user to view.

[0997] Input: User Profile

[0998] Output: Custom avatar

[0999] What happens: The server generates an avatar using a custom avatar generation algorithm and sends it to the device, where a preview of the avatar is displayed.

[1000] Step 3:

[1001] The user reviews the avatar and sends a tweak request if necessary. The server receives the request and regenerates the avatar. Once the tweaks are complete, the final avatar is saved.

[1002] Input: Tweak Request

[1003] Output: Final custom avatar

[1004] Specific behavior: The user makes minor adjustments to the avatar's appearance or voice on their device and sends a new request to the server. The server then regenerates the avatar and saves the finalized avatar.

[1005] Step 4:

[1006] When a user selects a learning session, the device sends a request to the server, which provides the device with learning content based on the selected subject.

[1007] Input: Select study session

[1008] Output: Learning content

[1009] How it works: The user selects a learning session from a menu on their device, and the request is sent to the server, which provides the relevant learning materials and questions.

[1010] Step 5:

[1011] During a training session, the device uses a camera and microphone to collect facial and voice data from the user, which is then sent in real time to a server that uses a generative AI model to analyze emotions.

[1012] Input: facial expression and voice data

[1013] Output: Emotion analysis results

[1014] How it works: The device collects data in real time using the camera and microphone, and sends the data to a server, which then analyzes the data using a sentiment analysis model.

[1015] Step 6:

[1016] Based on the analysis results, the server adjusts the feedback content, which is provided to the user through an avatar in the form of visual and audio feedback.

[1017] Input: Sentiment analysis results

[1018] Output: Regulated Feedback

[1019] Specific operation: The server determines the feedback content based on the results of emotion analysis and conveys it to the user via the avatar. For example, if the user is confused, the avatar will say, "Shall I explain it again?"

[1020] Step 7:

[1021] When the user enters the answer to a question, the device sends the answer to the server, which determines whether the answer is correct and returns the result to the device. The avatar then provides feedback to the user via voice and text.

[1022] Input: User's answer

[1023] Output: Judgment results and feedback

[1024] How it works: The user enters and submits an answer on the device. The server evaluates the answer and returns the result to the device. The avatar provides feedback based on the result.

[1025] Step 8:

[1026] The device reports the user's learning progress to the server, which records the progress data and allows the user to plan their next learning session.

[1027] Input: Learning progress data

[1028] Output: Progress record and suggestions for next study session

[1029] Specific operation: The device sends learning progress data to the server, which records the data. The next learning plan is presented to the user.

[1030] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1031] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1032] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1033] [Third embodiment]

[1034] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1035] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1036] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1037] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1038] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1039] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1040] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1041] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1042] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1043] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1044] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1045] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1046] The present invention relates to a system that provides a custom avatar that functions as a personal tutor and delivers an individually optimized learning experience. The system includes a series of processes for inputting and managing user information, generating and adjusting the custom avatar, providing educational content throughout the learning session, and evaluating the user's answers and providing feedback.

[1047] 1. Initial Setup and User Profile Creation

[1048] First, a user downloads the application and accesses the user registration screen when they first launch it. Here, the user enters their name, age, grade, preferred subject, and desired avatar characteristics (such as gender, appearance, and voice). The device then sends the collected user data to the server, and an appropriate user profile is generated.

[1049] 2. Avatar Creation and Customization

[1050] Based on the generated user profile, the server uses a custom avatar generation algorithm to automatically construct an avatar with the specified characteristics. The generated avatar is sent to the device, where the user can review the avatar's appearance and request fine-tuning if necessary. The fine-tuning request is sent back to the server, which then re-adjusts the avatar. This process is repeated until a final avatar that the user is satisfied with is determined and stored on the server.

[1051] 3. Start your study session

[1052] When a user selects a study session in the application, the device sends a request to the server to provide learning content for the selected subject. The server sends the appropriate problem sets and explanations to the device, which then presents them to the user via an avatar. For example, if a user selects a study session on multiplication in arithmetic, relevant problems and explanations are provided.

[1053] 4. Reading the problem and explaining it

[1054] When a user uploads a photo of a written or handwritten problem with their camera, the device uses OCR technology to convert it into text data. The converted data is sent to a server, which analyzes the problem and generates an answer and detailed explanation. The generated answer and explanation are then sent to the device, where an avatar uses voice functions to explain the answer to the user.

[1055] 5. Answer assessment and feedback

[1056] When a user enters an answer to a question, the device sends the answer to the server. The server compares the answer with the correct answer and returns the result to the device. The device then presents the result to the user, and an avatar provides audio feedback. Through feedback such as "That's right!" or "This is wrong," the user can identify areas for improvement.

[1057] 6. Learning progress management

[1058] The server records the user's answers for each learning session and manages their progress. Progress data is reflected in the next learning session and feedback. The device provides this management information to the user and guides them to the next learning session. The device is also designed to allow for easy schedule changes and adjustments.

[1059] Specific examples

[1060] For example, if a fourth-grade user wants to learn multiplication, he / she first completes the avatar setup and then selects the multiplication learning session. Here, the avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback saying, "That's correct! Well done!" Through this process, the user can check their understanding while progressing through the learning process.

[1061] As described above, through this invention, children can receive high-quality, individually optimized educational support regardless of their family environment or economic situation.

[1062] The processing flow will be explained below.

[1063] Step 1:

[1064] The user downloads the application and launches it for the first time, which displays the user registration screen.

[1065] Step 2:

[1066] The user enters user information such as name, age, grade, preferred subjects, and desired avatar characteristics.

[1067] Step 3:

[1068] The device sends the entered user information to the server, which then creates a user profile.

[1069] Step 4:

[1070] The server generates a custom avatar based on the user profile using an avatar generation algorithm.

[1071] Step 5:

[1072] The server sends the generated custom avatar data to the device.

[1073] Step 6:

[1074] The device displays the custom avatar on the screen, allowing the user to see what the avatar looks like.

[1075] Step 7:

[1076] If a user wants to make minor adjustments to the appearance of their avatar, they make an adjustment request.

[1077] Step 8:

[1078] The device sends a fine-tuning request to the server, which regenerates the avatar based on the adjustment request and sends the data back to the device.

[1079] Step 9:

[1080] Once the user is satisfied with the appearance of the avatar, the avatar is finally finalized and the terminal transmits the finalized data to the server.

[1081] Step 10:

[1082] The server stores the determined avatar information in the user profile.

[1083] Step 11:

[1084] Users select a study session (e.g., "Math Multiplication") from the study schedule screen within the app.

[1085] Step 12:

[1086] The terminal requests the selected learning session information from the server.

[1087] Step 13:

[1088] The server sends the appropriate learning content (problem sets and explanatory materials) to the device.

[1089] Step 14:

[1090] The terminal presents the learning content to the user through the avatar and initiates a learning session.

[1091] Step 15:

[1092] Users can write the question on paper and take a photo of it using the device's camera, or they can enter the question manually.

[1093] Step 16:

[1094] The device uses OCR technology to convert the photographed question into text data, which is then sent to a server.

[1095] Step 17:

[1096] The server analyzes the question text and generates appropriate answers and explanations.

[1097] Step 18:

[1098] The server sends the generated answers and explanations to the terminal.

[1099] Step 19:

[1100] The device uses an avatar to provide audio commentary and teach the user.

[1101] Step 20:

[1102] The user enters the answer into the terminal.

[1103] Step 21:

[1104] The terminal sends the answer to the server.

[1105] Step 22:

[1106] The server determines whether the answer is correct or incorrect and sends the result to the terminal.

[1107] Step 23:

[1108] The device displays the results and an avatar provides audible feedback such as "That's right!" or "This is wrong."

[1109] Step 24:

[1110] The server records the user's answers and updates the learning progress data.

[1111] Step 25:

[1112] The device will prompt the user for the next study session or provide options to reschedule.

[1113] Step 26:

[1114] The user selects or schedules the next study session, which causes the system to start the study process again.

[1115] Through these steps, users can study efficiently and effectively.

[1116] Example 1

[1117] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1118] Conventional online learning systems have struggled to provide a personalized educational experience for each user, and have struggled to provide smooth feedback to maximize learning outcomes. Furthermore, managing learning progress can be unintuitive for users, making it difficult to develop long-term learning plans and maintain motivation. Furthermore, they are unable to provide detailed explanations of paper-based questions and their answers in real time, preventing efficient learning.

[1119] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1120] In this invention, the server includes means for inputting user information, means for generating a custom avatar based on the input user information, means for displaying the generated custom avatar, means for receiving a user request to fine-tune the avatar, means for regenerating the avatar based on the fine-tuning request, means for saving the finalized avatar, means for the user to select a study session, means for providing study content based on the selected study session, means for asking questions and providing explanations based on the provided study content, means for receiving the user's answers and determining whether they are correct or incorrect, means for providing feedback based on the determination results, means for managing the user's study progress, means for generating answers and detailed explanations using artificial intelligence, and means for displaying study progress management information in a dashboard format, thereby enabling the provision of a personalized learning experience, real-time feedback, and intuitive study progress management.

[1121] "User information" refers to information such as name, age, grade, preferred subjects, and desired avatar characteristics that a user enters when using an application.

[1122] The "custom avatar generation means" refers to an algorithm or program for automatically creating a custom avatar that matches the characteristics of the user based on input user information.

[1123] The "avatar display means" refers to an interface or device for visually presenting the generated custom avatar to the user.

[1124] The "means for receiving a fine-tuning request" is an interface or program that allows a user to request fine-tuning of characteristics such as appearance or voice of a custom avatar.

[1125] "Avatar regeneration means" refers to an algorithm or program for regenerating an avatar based on the user's fine-tuning requests.

[1126] The "avatar storage means" refers to a program or storage for saving the finalized custom avatar in a database or the like.

[1127] A "learning session selection means" is an interface or program that allows a user to select a particular learning session.

[1128] The "learning content providing means" refers to a system or program that provides appropriate learning content to a user based on the selected learning session.

[1129] "Question-setting means" refers to a program or interface that presents questions to users based on the learning content provided and provides explanations.

[1130] The "answer receiving means" refers to an interface that allows the user to input an answer, and a program that transmits the answer to the server.

[1131] The "correctness determination means" is an algorithm or program for determining whether the user's answer is correct or incorrect.

[1132] "Feedback providing means" refers to a program or system that provides feedback to the user in the form of voice or text based on the result of determining whether the answer is correct or incorrect.

[1133] A "learning progress management means" is a system or program that records and manages a user's learning progress and uses the information to help with the next learning session.

[1134] "Artificial intelligence" refers to machine learning algorithms and neural network technology that analyzes users' answers and generates answers and detailed explanations.

[1135] The "means for displaying in a dashboard format" refers to an interface or program for displaying learning progress management information in a visually easy-to-understand dashboard format.

[1136] MODE FOR CARRYING OUT THE INVENTION

[1137] The present invention provides a system that provides a custom avatar that functions as a personal tutor and delivers an individually optimized learning experience. This system includes a series of processes for inputting and managing user information, generating and adjusting the custom avatar, providing educational content throughout the learning session, and evaluating the user's answers and providing feedback.

[1138] Hardware and software used

[1139] Server: Manages user information, generates and stores custom avatars, provides learning content, judges answers, generates feedback, and manages learning progress.

[1140] Device: Enter user information, view a custom avatar, select a study session, enter and submit answers, view feedback, and view study progress.

[1141] OCR library (e.g. Tesseract): Converts paper questions into text data.

[1142] Generative AI models (e.g., machine learning algorithms, neural networks): generate answers and detailed explanations.

[1143] Initial Setup and User Profile Creation

[1144] First, the user downloads the application and accesses the user registration screen when launching it for the first time. Here, the user enters their name, age, grade, preferred subjects, and desired avatar characteristics (such as gender, appearance, and voice). The device collects this information and sends it to the server. The server then creates a user profile based on the received information and sends it to the device.

[1145] Avatar Creation and Customization

[1146] The server uses a custom avatar generation algorithm to automatically construct an avatar with the specified characteristics. The generated avatar is sent to the device, where the user can review its appearance and request fine-tuning if necessary. A retuning request is sent back to the server, which then retunes the avatar. This process is repeated until the user finally finds an avatar they are happy with and stores it on the server.

[1147] Start a study session

[1148] When a user selects a study session in the application, the device sends a request to the server to provide study content for the selected subject. The server sends the appropriate problem sets and explanations to the device, which then presents them to the user through an avatar.

[1149] Problem reading and explanation

[1150] When a user uploads a photo of a written or handwritten problem with their camera, the device converts it into text using OCR technology. The converted data is sent to a server, which analyzes the problem and generates an answer and detailed explanation. The generated answer and explanation are then sent to the device, where an avatar uses voice functions to explain the answer to the user.

[1151] Answer assessment and feedback

[1152] When a user enters an answer to a question, the device sends the answer to the server. The server compares the answer with the correct answer and sends the result back to the device. The device then presents the result to the user, and an avatar provides audio feedback. Through feedback such as "That's right!" or "This is wrong," the user can identify areas for improvement.

[1153] Learning progress management

[1154] The server records the user's answers for each learning session and manages their progress. Progress data is reflected in the next learning session and feedback. The device provides this management information to the user and guides them to the next learning session. The device is also designed to allow for easy schedule changes and adjustments.

[1155] Specific examples

[1156] For example, if a fourth-grade user wants to learn multiplication, he / she first completes the avatar setup and then selects the multiplication learning session. Here, the avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback saying, "That's correct! Well done!" Through this process, the user can check their understanding while progressing through the learning process.

[1157] Prompt Sentence Examples

[1158] Prompt statement:

[1159] "Analyze a math problem solved by a user and generate the answer and a detailed explanation. The problem is '12 x 8' and the answer is '96'."

[1160] In this way, this invention provides an individually optimized learning experience and effective feedback, enabling children to receive high-quality educational support at home.

[1161] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1162] Step 1: Enter and submit user information

[1163] Input: User information such as name, age, grade, preferred subject, desired avatar characteristics

[1164] Specific operation: The user downloads the application, accesses the user registration screen when launching it for the first time, and enters the required information.

[1165] Data processing / calculation: The terminal temporarily stores the input information in its memory.

[1166] Output: Collected user information

[1167] Step 2: Generate a user profile

[1168] Input: User information

[1169] Specific operation: The device sends the collected user data to the server.

[1170] Data processing / calculation: The server creates a new user profile in the database based on the received information and enters the corresponding data into each field.

[1171] Output: Generated user profile

[1172] Step 3: Generate and submit a custom avatar

[1173] Input: User Profile

[1174] What Happens: The server uses a custom avatar generation algorithm to build an avatar based on the user profile.

[1175] Data processing / calculation: Use an AI-based avatar generation model to generate an avatar with specified characteristics.

[1176] Output: The generated avatar

[1177] Step 4: Review your avatar and request tweaks

[1178] Input: Generated avatar

[1179] Specific operation: The generated avatar is sent to the device, and the user can check its appearance and request fine-tuning if necessary.

[1180] Data processing / calculation: The device temporarily stores the user's fine-tuning request in memory.

[1181] Output: Tweak request

[1182] Step 5: Regenerate your avatar

[1183] Input: Tweak Request

[1184] Specific operation: The device sends a fine-tuning request to the server, which adjusts and regenerates an avatar with the specified characteristics.

[1185] Data processing / calculation: The server receives the fine-tuning request and regenerates the avatar based on the avatar generation algorithm.

[1186] Output: Regenerated avatar

[1187] Step 6: Save the final avatar

[1188] Input: Regenerated avatar

[1189] Specific operation: The server saves the final avatar that the user is satisfied with in the database.

[1190] Data processing / calculation: The regenerated avatar data is saved in the server database.

[1191] Output: Final saved avatar

[1192] Step 7: Select and request a study session

[1193] Input: User's learning session selection information

[1194] Specific operation: A user selects a learning session in the application. The device sends a content request to the server based on the selected subject.

[1195] Data processing / calculation: Based on the information of the selected learning session, a request is generated and sent to the server.

[1196] Output: Learning content request

[1197] Step 8: Provide learning content

[1198] Input: Learning content request

[1199] Specific operation: The server selects an appropriate problem set and explanatory materials and sends them to the terminal.

[1200] Data processing / calculation: Retrieve problem sets and explanatory materials from the database and compose the selected content.

[1201] Output: Problem set and explanations

[1202] Step 9: Questions and Answers

[1203] Input: Problem set and explanation materials

[1204] Specific operation: The device displays the problem to the user through the avatar and provides an explanation, after which the user inputs the answer.

[1205] Data processing / calculation: Display the problem set and receive user answer input.

[1206] Output: User's answer

[1207] Step 10: Answer assessment

[1208] Input: User's answer

[1209] Specific operation: The terminal sends the user's answer to the server, which determines whether the answer is correct or not.

[1210] Data processing / calculation: The answer is compared with the correct data to generate a correct or incorrect result.

[1211] Output: Judgment result

[1212] Step 11: Provide feedback

[1213] Input: Judgment result

[1214] Specific operation: The server sends the judgment result to the terminal, and the terminal provides feedback to the user through the avatar.

[1215] Data processing / calculation: The feedback content is constructed based on the judgment results.

[1216] Output: Feedback

[1217] Step 12: Manage your learning progress

[1218] Input: Feedback and Answer Results

[1219] Specific operation: The server records the user's answers for each learning session and manages the progress. The device displays this management information in a dashboard format.

[1220] Data processing / calculation: Answer results and feedback are stored in a database, and progress is visualized.

[1221] Output: Progress management data and dashboard display

[1222] (Application example 1)

[1223] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1224] Conventional learning support and training systems have struggled to provide individually optimized support for each user. Training systems for new employees and field workers also struggled to provide real-time guidance and feedback tailored to on-site work. As a result, there was a challenge in being unable to provide appropriate support tailored to each user's level of understanding and progress.

[1225] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1226] In this invention, the server includes means for inputting user information, means for generating a custom avatar based on the input user information, means for displaying the generated custom avatar, means for the user to receive a request to fine-tune the avatar, means for regenerating the avatar based on the fine-tuning request, means for saving the finalized avatar, means for the user to select a study session, means for providing study content based on the selected study session, means for asking questions and providing explanations based on the provided study content, means for receiving the user's answers and determining whether they are correct or incorrect, means for providing feedback based on the determination results, means for managing the user's study progress, means for a new employee or field worker to create their own profile and for providing training content based on the profile, and means for recording the user's operations and results with a camera and providing feedback. This enables individually optimized study support and training.

[1227] "Means for inputting user information" refers to a device or program that allows a user to input information such as name, age, and work experience.

[1228] The "means for generating a custom avatar based on input user information" refers to a device or program that generates an avatar using a specific algorithm based on information input by the user.

[1229] The "means for displaying the generated custom avatar" is a device or program for visually displaying the generated avatar on a screen.

[1230] A "means for accepting user requests to fine-tune an avatar" is a device or program that allows a user to input requests to change characteristics such as the appearance or voice of the avatar.

[1231] The "means for regenerating an avatar based on a fine-tuning request" refers to a device or program that adjusts the characteristics of the avatar and regenerates it based on a user request.

[1232] The "means for saving the finalized avatar" is a device or program for saving the finalized avatar that satisfies the user in a database or the like.

[1233] A "means for user selection of a learning session" is a device or program that allows a user to select the learning content or training session they wish to use.

[1234] The "means for providing learning content based on a selected learning session" refers to a device or program for providing a user with learning materials and questions according to the selected learning session.

[1235] "Means for asking questions and providing explanations based on the provided learning content" refers to a device or program that presents questions to the user based on the learning content and provides explanations for the answers.

[1236] The "means for receiving the user's answer and determining whether it is correct" is a device or program for receiving the answer entered by the user and determining whether it is correct.

[1237] The "means for providing feedback based on the judgment result" is a device or program that provides appropriate feedback to the user based on the result of the judgment of whether the answer is correct or incorrect.

[1238] "Means for managing a user's learning progress" refers to a device or program for recording and managing a user's learning history and progress.

[1239] A "means for new employees or field workers to create their own profile" is a device or program that allows new employees or field workers to enter information such as their name, work experience, and learning goals to create a profile.

[1240] The "means for providing training content based on a profile" refers to a device or program for providing individually optimized training materials and content based on the created profile information.

[1241] "Means for recording the operations and results performed by the user with a camera and providing feedback" refers to a device or program that uses a camera to capture the operations performed by the user, analyzes the data, and provides appropriate feedback.

[1242] This invention is a system that provides a custom avatar that acts as a personal tutor and delivers an individually optimized learning experience. In particular, it includes a system that provides practical training in real time for new employees and field workers. Specific implementation methods of the invention are as follows.

[1243] First, the user downloads a dedicated application onto their smartphone or tablet. When they first launch the application, they access the user registration screen and enter their user information, such as their name, age, and work experience. This information is then sent from the device to the server.

[1244] The server then uses a custom avatar generation algorithm to build an avatar based on the user's profile. The generated avatar is sent to the device, where the user can review its appearance and request tweaks if necessary. The tweaked information is sent back to the server, and the avatar is regenerated. This process is repeated until the user is satisfied, and the final avatar is saved on the server.

[1245] The user then selects a learning session on the dedicated application. The training content is based on practical applications such as machine operation and safety management. For example, if a new employee is receiving training on machine operation in a factory, the server sends the selected training content to the terminal and provides it to the user via an avatar.

[1246] During training, the user's operations are recorded using the device's camera function. Using OCR technology (using Tesseract.js), the recorded operations are converted into text data and sent to the server. The server generates the analysis results, and the avatar provides feedback using the voice function. For example, if the user performs an incorrect operation, the avatar will say, "This is incorrect. Please do it this way next time."

[1247] The server then receives the user's answer and determines whether it is correct. The result of the determination is immediately provided to the user as feedback via the avatar. For example, if the user makes a mistake in the operating procedure, the server will provide feedback such as, "This is an error. The correct way is this."

[1248] Finally, the server has the function of managing the user's learning progress. It records the results and answer history of each training session and reflects them in the next learning session. It also provides new training content according to the user's progress, supporting continuous skill development.

[1249] Examples:

[1250] For example, when a new employee receives training on operating machinery in a factory, the user creates a profile on a dedicated tablet and customizes the avatar's appearance and voice. The server then explains safe operating procedures for the machine via the avatar, and the user performs the procedures. The operations recorded by the camera are converted into text data using OCR technology (Tesseract.js) and analyzed by the server. Based on the results, the avatar provides feedback, such as "This is incorrect."

[1251] Example prompts to input to a generative AI model:

[1252] Create a new worker profile in the system. Input data includes name, age, and work experience. Follow the steps below to provide training content for operating factory robots. The camera will capture the user's operation results, convert them into text data using OCR technology, and provide feedback via an avatar.

[1253] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1254] Step 1:

[1255] Users use their smartphones or tablets to download a dedicated application.

[1256] Input: User operation (application download)

[1257] Action: Install application

[1258] Output: Application initial launch screen

[1259] Step 2:

[1260] A user starts the application and accesses the user registration screen, where they enter their user information such as name, age, and work experience.

[1261] Input: User information (name, age, work experience, etc.)

[1262] Process: Enter and submit user information

[1263] Output: A user profile is generated on the server.

[1264] Step 3:

[1265] The server uses a custom avatar generation algorithm to generate a custom avatar based on the user profile, and the generated avatar is sent to the device.

[1266] Input: User profile information

[1267] Processing: Custom avatar generation (Unity3D, TensorFlow)

[1268] Output: Generated custom avatar data

[1269] Step 4:

[1270] The custom avatar is displayed on the device, allowing the user to check its appearance, voice, etc., and request tweaks as needed.

[1271] Input: Generated avatar data

[1272] Action: Display a custom avatar

[1273] Output: Avatar confirmation screen

[1274] Step 5:

[1275] The device accepts the user's fine-tuning request and sends it to the server, which then fine-tunes the avatar, regenerates it, and sends it to the device.

[1276] Input: Tweak request from user

[1277] Action: Regenerate custom avatar

[1278] Output: Fine-tuned custom avatar data

[1279] Step 6:

[1280] The final custom avatar is saved on the server and the user selects a learning or training session.

[1281] Input: Confirmed custom avatar data, user session selection

[1282] Actions: Save custom avatar, select learning or training session

[1283] Output: Session start screen

[1284] Step 7:

[1285] The server sends appropriate content to the terminal based on the selected learning or training session.

[1286] Input: Selected information for your study or training session

[1287] Processing: Selection and delivery of training content (using Django)

[1288] Output: Training content

[1289] Step 8:

[1290] The user performs the training and records the operations and results using the device's camera.

[1291] Input: The action taken by the user

[1292] Process: Recording the operation (using a camera)

[1293] Output: Recorded operation data

[1294] Step 9:

[1295] The device converts the recorded operation data into text data using OCR technology (Tesseract.js) and sends it to the server.

[1296] Input: Recorded operation data

[1297] Processing: Converting operation data into text (OCR technology)

[1298] Output: Text data

[1299] Step 10:

[1300] The server analyzes the text data, and the avatar provides audio feedback based on the results.

[1301] Input: Text data

[1302] Processing: Data analysis and feedback generation (using Scikit-learn)

[1303] Output: Generate feedback content

[1304] Step 11:

[1305] The device provides the user with audible feedback via an avatar.

[1306] Input: Feedback

[1307] Processing: Audio feedback

[1308] Output: Audio feedback

[1309] Step 12:

[1310] The server manages the user's learning progress, recording the results of each training session and reflecting them in the next session.

[1311] Input: Results of each training session

[1312] Processing: Data management of learning progress (using MongoDB)

[1313] Output: Learning progress data and what to expect in the next session

[1314] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1315] The present invention relates to a personal tutor avatar system for assisting users in their learning, specifically a system that combines an emotion engine to recognize the user's emotional state and adjust dialogue and feedback accordingly to provide a more effective learning experience. This system includes processes for inputting user information, generating and displaying a custom avatar, managing a learning session, and recognizing the user's emotions to adjust dialogue content.

[1316] 1. Initial Setup and User Profile Creation

[1317] First, a user downloads the application and accesses the user registration screen when they first launch it. Here, the user enters their name, age, grade, preferred subject, and desired avatar characteristics (such as gender, appearance, and voice). The device then sends the collected user data to the server, and an appropriate user profile is generated.

[1318] 2. Avatar Creation and Customization

[1319] Based on the generated user profile, the server uses a custom avatar generation algorithm to automatically construct an avatar with the specified characteristics. The generated avatar is sent to the device, where the user can review the avatar's appearance and request fine-tuning if necessary. The fine-tuning request is sent back to the server, which then re-adjusts the avatar. This process is repeated until a final avatar that the user is satisfied with is determined and stored on the server.

[1320] 3. Start your study session

[1321] When a user selects a study session in the application, the device sends a request to the server to provide learning content for the selected subject. The server sends the appropriate problem sets and explanations to the device, which then presents them to the user via an avatar. For example, if a user selects a study session on multiplication in arithmetic, relevant problems and explanations are provided.

[1322] 4. Use of Emotion Engine

[1323] During the learning session, the device's built-in camera and microphone collect the user's facial expressions and voice data. This data is sent to the server in real time and analyzed by the emotion engine. Based on the analysis results, the server recognizes the user's emotional state (e.g., "happiness," "sadness," "anger," "surprise," etc.).

[1324] 5. Reading the problem and explaining it

[1325] When a user uploads a photo of a written or handwritten problem with their camera, the device uses OCR technology to convert it into text data. The converted data is sent to a server, which analyzes the problem and generates an answer and detailed explanation. The generated answer and explanation are then sent to the device, where an avatar uses voice functions to explain the answer to the user.

[1326] 6. Answer assessment and feedback

[1327] When a user enters an answer to a question, the device sends the answer to the server. The server compares it with the correct answer and returns the result to the device. The device then presents the result to the user, and an avatar provides audio feedback. Furthermore, the avatar's dialogue content and tone can be adjusted based on the user's recognized emotions. For example, if the user is confused, the avatar will use encouraging words such as "It's okay, let me explain it again."

[1328] 7. Learning progress management

[1329] The server records the user's answers for each study session and manages their progress. Progress data is reflected in the next study session and feedback. It also records recognized emotional data and uses this information to optimize study. For example, if a user frequently feels frustrated with a particular problem, it can provide additional support materials related to that problem. The device provides this management information to the user and guides them through the next study session. It is also designed to allow for easy schedule changes and adjustments.

[1330] Specific examples

[1331] For example, if a fourth-grade user wants to learn multiplication, he / she first completes the avatar setup and then selects the multiplication learning session. Here, the avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback, saying, "That's correct! Well done!" At the same time, if the user looks confused, the avatar will provide additional support, saying, "Shall I explain it again?" Through this process, the user can progress through their learning while checking their understanding.

[1332] As described above, through this invention, children can receive high-quality, individually optimized educational support regardless of their family environment or financial situation. The use of an emotion engine further improves the learning experience by recognizing the user's emotional state and providing appropriate feedback accordingly.

[1333] The processing flow will be explained below.

[1334] Step 1:

[1335] The user downloads the application and launches it for the first time, which displays the user registration screen.

[1336] Step 2:

[1337] The user enters user information such as name, age, grade, preferred subjects, and desired avatar characteristics.

[1338] Step 3:

[1339] The device sends the entered user information to the server, which then creates a user profile.

[1340] Step 4:

[1341] The server generates a custom avatar based on the user profile using an avatar generation algorithm.

[1342] Step 5:

[1343] The server sends the generated custom avatar data to the device.

[1344] Step 6:

[1345] The device displays the custom avatar on the screen, allowing the user to see what the avatar looks like.

[1346] Step 7:

[1347] If a user wants to make minor adjustments to the appearance of their avatar, they make an adjustment request.

[1348] Step 8:

[1349] The device sends a fine-tuning request to the server, which regenerates the avatar based on the adjustment request and sends the data back to the device.

[1350] Step 9:

[1351] Once the user is satisfied with the appearance of the avatar, the avatar is finally finalized and the terminal transmits the finalized data to the server.

[1352] Step 10:

[1353] The server stores the determined avatar information in the user profile.

[1354] Step 11:

[1355] Users select a study session (e.g., "Math Multiplication") from the study schedule screen within the app.

[1356] Step 12:

[1357] The terminal requests the selected learning session information from the server.

[1358] Step 13:

[1359] The server sends the appropriate learning content (problem sets and explanatory materials) to the device.

[1360] Step 14:

[1361] The terminal presents the learning content to the user through the avatar and initiates a learning session.

[1362] Step 15:

[1363] The device's built-in camera and microphone are used to collect the user's facial expressions and voice data in real time.

[1364] Step 16:

[1365] The collected facial and voice data is sent to a server, which uses an emotion engine to analyze the data and recognize the user's emotional state.

[1366] Step 17:

[1367] When a user takes a photo of a question written on paper or entered manually and uploads it, the device uses OCR technology to convert it into text data, which is then sent to the server.

[1368] Step 18:

[1369] The server analyzes the question text and generates appropriate answers and explanations.

[1370] Step 19:

[1371] The server sends the generated answers and explanations to the terminal.

[1372] Step 20:

[1373] The device provides audio commentary from an avatar, educating the user and adjusting the content and tone of the dialogue based on the user's emotional state.

[1374] Step 21:

[1375] The user enters the answer into the terminal.

[1376] Step 22:

[1377] The terminal sends the answer to the server.

[1378] Step 23:

[1379] The server determines whether the answer is correct or incorrect and sends the result to the terminal.

[1380] Step 24:

[1381] The device displays the results and an avatar provides audible feedback such as "That's right!" or "This is wrong."

[1382] Step 25:

[1383] The server records the user's answers and updates their learning progress data, along with their emotional state.

[1384] Step 26:

[1385] The device will prompt the user for the next study session or provide options to reschedule.

[1386] Step 27:

[1387] The user selects or schedules the next study session, which causes the system to start the study process again.

[1388] Through these steps, users can learn efficiently and effectively. By utilizing the emotion engine, appropriate feedback and support are provided that takes into account the user's emotional state, resulting in a more personalized learning experience.

[1389] Example 2

[1390] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1391] Conventional learning support systems have the problem of being unable to provide an effective learning experience that takes into account the user's emotional state. In addition, they are unable to provide customized feedback optimized for each individual user, which can reduce the effectiveness of learning.

[1392] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting facial expression and voice data of the user and analyzing it with an emotion engine, means for adjusting the dialogue content based on the analysis results and providing emotionally sensitive feedback, and means for managing the user's learning progress. This allows for an effective learning experience by providing detailed feedback according to the user's emotional state.

[1393] - "User Information" refers to personal information about a User, such as the User's name, age, grade, preferred subjects, and desired avatar characteristics.

[1394] "Custom avatar" refers to an avatar created specifically for a user based on user information.

[1395] "Study Session" means a study session offered based on a subject or topic selected by a User.

[1396] "Study Content" refers to the problem sets and explanatory materials provided based on a study session.

[1397] "Questions" refer to assignments or quizzes presented to users within the learning content.

[1398] "Answer" refers to the answer entered by the user in response to a question.

[1399] "Explanation" refers to explanations or supplementary information for the answers to questions.

[1400] "Feedback" refers to the evaluation and advice provided to users based on whether their answers are correct or incorrect.

[1401] "Facial expression data" refers to data collected by capturing a user's facial expressions with a camera.

[1402] "Voice data" refers to data collected by capturing the user's voice with a microphone.

[1403] An "emotion engine" refers to algorithms or software that analyzes facial expression and voice data to recognize the user's emotional state.

[1404] "Learning progress" refers to a user's progress, results, and achievements in learning.

[1405] "Means of collection" refers to devices or methods for obtaining data from users, such as cameras or microphones.

[1406] "Means of analysis" refers to the techniques and methods for processing collected data and analyzing emotions and learning trends.

[1407] "Means for adjusting dialogue content" refers to a mechanism for appropriately changing the avatar's speech and tone depending on the user's emotional state.

[1408] "Camera function" refers to a device and its functions for taking images and videos.

[1409] "OCR technology" refers to the technology that recognizes characters in an image and converts them into text data.

[1410] The present invention relates to a personal tutor avatar system for supporting users in their learning, specifically a system that combines an emotion engine to recognize the user's emotional state and adjust dialogue and feedback accordingly to provide a more effective learning experience. This system includes processes for inputting user information, generating and displaying a custom avatar, managing a learning session, and adjusting dialogue content based on the user's emotions.

[1411] First, the user downloads the learning support application and accesses the user registration screen when launching it for the first time. Here, the user enters their name, age, grade, preferred subjects, and desired avatar characteristics (gender, appearance, voice, etc.). This information is saved on the device. The device temporarily stores the information entered by the user and sends it to the server as soon as registration is complete. A message indicating registration is complete is displayed, and the device prepares to proceed to the next step.

[1412] The server generates an individual user profile based on the received user information. The generated profile is stored in a database and used in subsequent processes. The server then activates a custom avatar generation algorithm to automatically generate an avatar with the specified characteristics based on the user profile. The generated avatar data is sent to the device, which receives the data and displays the avatar on its screen.

[1413] The user checks the appearance of the avatar and requests fine-tuning if necessary. This request is sent from the device to the server, and the server regenerates the avatar reflecting the fine-tuning and sends the data back to the device. This process is repeated until the user is satisfied, and once the final avatar is finalized, the avatar data is saved on the server.

[1414] When a user selects a learning session, the device sends a request to the server to provide learning content for the selected subject. The server receives the request, prepares the corresponding problem set and explanatory materials, and sends the data to the device. Based on the received data, the device presents the learning session for the selected subject through an avatar. For example, in a learning session on multiplication in arithmetic, problems such as "12 x 8" are presented.

[1415] During a learning session, the device uses a camera and microphone to collect the user's facial and voice data. This data is sent to the server in real time and analyzed by the emotion engine. The server then analyzes the received facial and voice data using the emotion engine to recognize the user's emotional state (e.g., "happiness," "sadness," "anger," "surprise," etc.).

[1416] When a user writes a question on paper, takes a photo of it with a camera, and uploads it, the device uses OCR technology to convert the image data into text data. This converted text data is then sent to a server. The server analyzes the received text data and generates the answer to the question and a detailed explanation. The generated answer and explanation are then sent back to the device, which then provides them to the user through an avatar. The avatar uses voice functionality to provide explanations.

[1417] When a user enters an answer to a question and presses the send button, the device sends the answer data to the server. The server compares the received answer with the correct answer and sends the result back to the device. The result includes a correct / incorrect determination and detailed feedback. The device receives the result, and an avatar gives the user audio feedback. The content and tone of the feedback can be adjusted depending on the user's emotional state, and may use encouraging words such as "Shall I explain it again?"

[1418] The server records the user's answers for each study session and manages the user's study progress, including the results for each session and the user's emotional data. The server uses the recorded progress data to prepare for optimizing the next study session. The device notifies the user when the next study session is ready and helps the user easily manage their study progress and schedule.

[1419] Specific examples

[1420] For example, if a fourth-grade user wants to learn multiplication, he or she first sets up the avatar and then selects the multiplication learning session. The avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback, saying, "That's right! Well done!" Furthermore, if the user shows a confused expression, the avatar offers additional support by asking, "Shall I explain it again?"

[1421] Example of input prompt for generative AI model

[1422] "If the user's facial expression looks confused, have the avatar generate encouraging feedback, such as 'Shall I explain it again?' or 'Please let me know if you don't understand.'"

[1423] As described above, the present invention makes it possible to individually optimize the learning experience of each user and provide more effective learning support.

[1424] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1425] Step 1: User registration and initial setup

[1426] First, the user downloads the learning support application and accesses the user registration screen when the application is launched for the first time. The user enters their name, age, grade, preferred subjects, and desired avatar characteristics (gender, appearance, voice, etc.). This information is saved on the device. The device temporarily stores the user information and sends it to the server as soon as registration is complete. The server saves the received information in a database and returns a confirmation response to the device. The device then displays a message indicating that user registration has been completed.

[1427] Input: User information (name, age, grade, preferred subject, avatar characteristics)

[1428] Output: User information sent to the server, user registration complete message

[1429] Step 2: Create and save a user profile

[1430] The server generates a user profile based on the received user information. The generated profile is saved in a database for use in subsequent processes. A confirmation response is sent to the device confirming that the profile has been saved.

[1431] Input: User information

[1432] Output: Generated user profile, confirmation that it was saved by the server

[1433] Step 3: Avatar Creation and Customization

[1434] The server runs a custom avatar generation algorithm to automatically generate an avatar with the specified characteristics based on the user profile. The generated avatar data is sent to the device, where the user can view the displayed avatar. If the user requests fine-tuning as needed, this request is sent back to the server, and the avatar is regenerated. Finally, once the user is satisfied with the avatar, the avatar data is saved on the server.

[1435] Input: User profile, Tweak request

[1436] Output: Custom avatar, Retuned avatar

[1437] Specific actions: Adjusting the avatar's gender, appearance, voice, and other characteristics

[1438] Step 4: Start your study session

[1439] When a user selects a study session, the device sends a request to the server to provide study content according to the selected subject. The server receives the request, prepares the corresponding problem sets and explanation materials, and sends the data to the device. The device then presents the study session through an avatar based on the received data.

[1440] Input: User selects a study session

[1441] Output: Learning content (question sets, explanatory materials)

[1442] Specific actions: Displaying learning content and presenting questions

[1443] Step 5: Emotion Recognition with the Emotion Engine

[1444] During a learning session, the device uses a camera and microphone to collect facial and voice data from the user. This data is sent in real time to a server where it is analyzed by an emotion engine. The server recognizes the user's emotional state and adjusts feedback accordingly.

[1445] Input: facial expression data, voice data

[1446] Output: Analysis results (user's emotional state)

[1447] Specific actions: taking pictures of facial expressions with a camera and collecting audio with a microphone

[1448] Step 6: Reading the problem and generating an explanation

[1449] When a user takes a photo of a paper problem and uploads it, the device uses OCR technology to convert the image data into text. This converted data is sent to a server, which analyzes the problem, generates answers and explanations, and sends them back to the device. The device then provides these to the user through an avatar.

[1450] Input: Image data of the problem

[1451] Output: OCR text data, answers and explanations

[1452] Specific actions: taking a photo with a camera, OCR conversion, and explanation by an avatar

[1453] Step 7: Answers and feedback

[1454] The user inputs the answer to the question and sends it to the device. The device then sends the answer data to the server, which checks the answer against the correct answer and returns a judgment result. The device then presents the result to the user, and an avatar provides audio feedback. The content and tone of the feedback are adjusted according to the user's emotional state.

[1455] Input: User-provided answer data

[1456] Output: Matching result and feedback (correct / incorrect)

[1457] Specific actions: Entering answers, checking, and voice feedback

[1458] Step 8: Track your progress and prepare for the next session

[1459] The server records the user's answers for each study session and manages their study progress, including their grades and emotional state for each session. Based on the recorded progress data, the server optimizes the next study session and notifies the device when the next study session is ready. It also makes it easy to change the progress and schedule.

[1460] Input: Answer result, emotional state

[1461] Output: Tracked progress data, next session notifications

[1462] Specific actions: Recording results, managing emotional data, notifying preparation for the next session

[1463] (Application example 2)

[1464] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1465] Conventional learning support systems do not take into account the user's emotional state, and therefore have the problem of being unable to provide appropriate support when the user becomes confused or frustrated. This reduces the effectiveness of learning and leads to a lack of sustained user motivation. Furthermore, there is a lack of technology that can analyze the user's emotions in real time and appropriately adjust feedback based on the results.

[1466] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for inputting user information; means for generating a custom avatar based on the input user information; means for displaying the generated custom avatar; means for accepting a user request to fine-tune the avatar; means for regenerating the avatar based on the fine-tuning request; means for saving the finalized avatar; means for the user to select a study session; means for providing study content based on the selected study session; means for asking questions and providing explanations based on the provided study content; means for receiving the user's answers and determining whether they are correct or incorrect; means for providing feedback based on the determination results; means for managing the user's study progress; means for collecting the user's facial expression and voice data using a camera and microphone function; means for analyzing the user's emotions based on the collected data; and means for adjusting the feedback content based on the analysis results. This makes it possible to provide appropriate feedback according to the user's emotional state, improve learning effectiveness, and maintain the user's motivation.

[1467] "User information" refers to basic data about the user, including name, age, grade, preferred subjects, and avatar characteristics.

[1468] A "custom avatar" is a character with a unique appearance and voice that is generated based on user input.

[1469] A "study session" is a period of time or a particular learning activity during which a user studies.

[1470] "Study Content" means digital content, including study materials, problem sets, and explanatory materials, that a user works through during a study session.

[1471] "Facial expression and voice data" refers to data that records a user's facial expressions, tone of voice, and the content of their statements.

[1472] "Emotion analysis" refers to the process of identifying a user's emotional state (such as joy, sadness, anger, surprise, etc.) based on collected facial and voice data.

[1473] "Feedback" refers to advice, additional explanations, encouraging messages, etc. provided based on the user's learning progress and the accuracy of their answers.

[1474] "Progress management" is the process of recording and tracking a user's learning progress and grades to help plan future learning.

[1475] The present invention provides a personal tutor avatar system for supporting users in their learning. In particular, the system combines an emotion engine to recognize the user's emotional state and adjust dialogue and feedback based on that to provide a more effective learning experience.

[1476] The system is primarily composed of a server and a terminal. The server manages user information and performs key processes such as generating custom avatars, managing learning sessions, analyzing emotions, and adjusting feedback. The terminal is the device used by the user (smartphone, tablet, etc.) and is responsible for interacting with the user.

[1477] 1. Initial Setup

[1478] When a user starts the application for the first time, they download it and enter their user information (such as name, age, grade, preferred subjects, and avatar characteristics). The device sends this data to the server, and a user profile is created.

[1479] 2. Custom Avatar Creation

[1480] The server generates a custom avatar based on the information entered by the user. This avatar is tailored to the user's visual and auditory preferences. The generated avatar is sent to the device, where the user can review and fine-tune it. The avatar is regenerated upon the user's request, and the final avatar is stored on the server.

[1481] 3. Start your study session

[1482] When a user selects a learning session, the device requests it from the server, which then provides the device with the appropriate learning content based on the selected subject. For example, if a user selects a math session, the server provides relevant questions and explanations.

[1483] 4. Use of Emotion Engine

[1484] The device uses a camera and microphone to collect the user's facial and voice data. The collected data is sent to the server in real time, and the server analyzes the user's emotions using an emotion engine (e.g., the emotion analysis model in the Transformers library). Based on the analysis results, the server adjusts the feedback content and provides appropriate assistance to the user.

[1485] 5. Answer assessment and feedback

[1486] When a user enters an answer to a question, the device sends the answer to the server. The server determines whether the answer is correct and returns feedback to the device. The avatar provides feedback to the user in voice and text. For example, if the user's expression looks confused, encouraging feedback such as "Shall I explain it again?" is provided.

[1487] Specific examples

[1488] For example, if a fourth-grade user wants to learn multiplication, he or she can select a multiplication learning session after completing the avatar setup. Here, the avatar will present the problem "12 x 8." If the user enters "96" as the answer, the system will immediately check the answer and provide audio feedback saying, "That's correct! Well done!" At the same time, if the user shows a confused expression, the avatar will provide additional support by saying, "Shall I explain it again?"

[1489] Prompt Sentence Examples

[1490] Analyze what emotions users are expressing. Classify emotions such as "Happy," "Sadness," "Anger," "Surprise," etc.

[1491] This system provides appropriate feedback according to the user's emotional state, improving learning effectiveness and maintaining user motivation.

[1492] As described above, through the present invention, the learning assistance system can dynamically analyze the user's emotions and provide appropriate feedback based on the results, thereby improving the learning experience.

[1493] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1494] Step 1:

[1495] When a user downloads the application and launches it for the first time, they enter their user information, including their name, age, grade, preferred subjects, and avatar characteristics. The device then sends this data to a server, which creates a user profile.

[1496] Input: User information (name, age, grade, subject, avatar characteristics)

[1497] Output: User profile (stored on the server)

[1498] Specific operation: The user enters information into the input form on the terminal and presses the send button. The terminal then sends the information to the server.

[1499] Step 2:

[1500] The server generates a custom avatar based on the received user information, which is then sent to the device for the user to view.

[1501] Input: User Profile

[1502] Output: Custom avatar

[1503] What happens: The server generates an avatar using a custom avatar generation algorithm and sends it to the device, where a preview of the avatar is displayed.

[1504] Step 3:

[1505] The user reviews the avatar and sends a tweak request if necessary. The server receives the request and regenerates the avatar. Once the tweaks are complete, the final avatar is saved.

[1506] Input: Tweak Request

[1507] Output: Final custom avatar

[1508] Specific behavior: The user makes minor adjustments to the avatar's appearance or voice on their device and sends a new request to the server. The server then regenerates the avatar and saves the finalized avatar.

[1509] Step 4:

[1510] When a user selects a learning session, the device sends a request to the server, which provides the device with learning content based on the selected subject.

[1511] Input: Select study session

[1512] Output: Learning content

[1513] How it works: The user selects a learning session from a menu on their device, and the request is sent to the server, which provides the relevant learning materials and questions.

[1514] Step 5:

[1515] During a training session, the device uses a camera and microphone to collect facial and voice data from the user, which is then sent in real time to a server that uses a generative AI model to analyze emotions.

[1516] Input: facial expression and voice data

[1517] Output: Emotion analysis results

[1518] How it works: The device collects data in real time using the camera and microphone, and sends the data to a server, which then analyzes the data using a sentiment analysis model.

[1519] Step 6:

[1520] Based on the analysis results, the server adjusts the feedback content, which is provided to the user through an avatar in the form of visual and audio feedback.

[1521] Input: Sentiment analysis results

[1522] Output: Regulated Feedback

[1523] Specific operation: The server determines the feedback content based on the results of emotion analysis and conveys it to the user via the avatar. For example, if the user is confused, the avatar will say, "Shall I explain it again?"

[1524] Step 7:

[1525] When the user enters the answer to a question, the device sends the answer to the server, which determines whether the answer is correct and returns the result to the device. The avatar then provides feedback to the user via voice and text.

[1526] Input: User's answer

[1527] Output: Judgment results and feedback

[1528] How it works: The user enters and submits an answer on the device. The server evaluates the answer and returns the result to the device. The avatar provides feedback based on the result.

[1529] Step 8:

[1530] The device reports the user's learning progress to the server, which records the progress data and allows the user to plan their next learning session.

[1531] Input: Learning progress data

[1532] Output: Progress record and suggestions for next study session

[1533] Specific operation: The device sends learning progress data to the server, which records the data. The next learning plan is presented to the user.

[1534] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1535] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1536] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1537] [Fourth embodiment]

[1538] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1539] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1540] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1541] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1542] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1543] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1544] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1545] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1546] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1547] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1548] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1549] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1550] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1551] The present invention relates to a system that provides a custom avatar that functions as a personal tutor and delivers an individually optimized learning experience. The system includes a series of processes for inputting and managing user information, generating and adjusting the custom avatar, providing educational content throughout the learning session, and evaluating the user's answers and providing feedback.

[1552] 1. Initial Setup and User Profile Creation

[1553] First, a user downloads the application and accesses the user registration screen when they first launch it. Here, the user enters their name, age, grade, preferred subject, and desired avatar characteristics (such as gender, appearance, and voice). The device then sends the collected user data to the server, and an appropriate user profile is generated.

[1554] 2. Avatar Creation and Customization

[1555] Based on the generated user profile, the server uses a custom avatar generation algorithm to automatically construct an avatar with the specified characteristics. The generated avatar is sent to the device, where the user can review the avatar's appearance and request fine-tuning if necessary. The fine-tuning request is sent back to the server, which then re-adjusts the avatar. This process is repeated until a final avatar that the user is satisfied with is determined and stored on the server.

[1556] 3. Start your study session

[1557] When a user selects a study session in the application, the device sends a request to the server to provide learning content for the selected subject. The server sends the appropriate problem sets and explanations to the device, which then presents them to the user via an avatar. For example, if a user selects a study session on multiplication in arithmetic, relevant problems and explanations are provided.

[1558] 4. Reading the problem and explaining it

[1559] When a user uploads a photo of a written or handwritten problem with their camera, the device uses OCR technology to convert it into text data. The converted data is sent to a server, which analyzes the problem and generates an answer and detailed explanation. The generated answer and explanation are then sent to the device, where an avatar uses voice functions to explain the answer to the user.

[1560] 5. Answer assessment and feedback

[1561] When a user enters an answer to a question, the device sends the answer to the server. The server compares the answer with the correct answer and returns the result to the device. The device then presents the result to the user, and an avatar provides audio feedback. Through feedback such as "That's right!" or "This is wrong," the user can identify areas for improvement.

[1562] 6. Learning progress management

[1563] The server records the user's answers for each learning session and manages their progress. Progress data is reflected in the next learning session and feedback. The device provides this management information to the user and guides them to the next learning session. The device is also designed to allow for easy schedule changes and adjustments.

[1564] Specific examples

[1565] For example, if a fourth-grade user wants to learn multiplication, he / she first completes the avatar setup and then selects the multiplication learning session. Here, the avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback saying, "That's correct! Well done!" Through this process, the user can check their understanding while progressing through the learning process.

[1566] As described above, through this invention, children can receive high-quality, individually optimized educational support regardless of their family environment or economic situation.

[1567] The processing flow will be explained below.

[1568] Step 1:

[1569] The user downloads the application and launches it for the first time, which displays the user registration screen.

[1570] Step 2:

[1571] The user enters user information such as name, age, grade, preferred subjects, and desired avatar characteristics.

[1572] Step 3:

[1573] The device sends the entered user information to the server, which then creates a user profile.

[1574] Step 4:

[1575] The server generates a custom avatar based on the user profile using an avatar generation algorithm.

[1576] Step 5:

[1577] The server sends the generated custom avatar data to the device.

[1578] Step 6:

[1579] The device displays the custom avatar on the screen, allowing the user to see what the avatar looks like.

[1580] Step 7:

[1581] If a user wants to make minor adjustments to the appearance of their avatar, they make an adjustment request.

[1582] Step 8:

[1583] The device sends a fine-tuning request to the server, which regenerates the avatar based on the adjustment request and sends the data back to the device.

[1584] Step 9:

[1585] Once the user is satisfied with the appearance of the avatar, the avatar is finally finalized and the terminal transmits the finalized data to the server.

[1586] Step 10:

[1587] The server stores the determined avatar information in the user profile.

[1588] Step 11:

[1589] Users select a study session (e.g., "Math Multiplication") from the study schedule screen within the app.

[1590] Step 12:

[1591] The terminal requests the selected learning session information from the server.

[1592] Step 13:

[1593] The server sends the appropriate learning content (problem sets and explanatory materials) to the device.

[1594] Step 14:

[1595] The terminal presents the learning content to the user through the avatar and initiates a learning session.

[1596] Step 15:

[1597] Users can write the question on paper and take a photo of it using the device's camera, or they can enter the question manually.

[1598] Step 16:

[1599] The device uses OCR technology to convert the photographed question into text data, which is then sent to a server.

[1600] Step 17:

[1601] The server analyzes the question text and generates appropriate answers and explanations.

[1602] Step 18:

[1603] The server sends the generated answers and explanations to the terminal.

[1604] Step 19:

[1605] The device uses an avatar to provide audio commentary and teach the user.

[1606] Step 20:

[1607] The user enters the answer into the terminal.

[1608] Step 21:

[1609] The terminal sends the answer to the server.

[1610] Step 22:

[1611] The server determines whether the answer is correct or incorrect and sends the result to the terminal.

[1612] Step 23:

[1613] The device displays the results and an avatar provides audible feedback such as "That's right!" or "This is wrong."

[1614] Step 24:

[1615] The server records the user's answers and updates the learning progress data.

[1616] Step 25:

[1617] The device will prompt the user for the next study session or provide options to reschedule.

[1618] Step 26:

[1619] The user selects or schedules the next study session, which causes the system to start the study process again.

[1620] Through these steps, users can study efficiently and effectively.

[1621] Example 1

[1622] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1623] Conventional online learning systems have struggled to provide a personalized educational experience for each user, and have struggled to provide smooth feedback to maximize learning outcomes. Furthermore, managing learning progress can be unintuitive for users, making it difficult to develop long-term learning plans and maintain motivation. Furthermore, they are unable to provide detailed explanations of paper-based questions and their answers in real time, preventing efficient learning.

[1624] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1625] In this invention, the server includes means for inputting user information, means for generating a custom avatar based on the input user information, means for displaying the generated custom avatar, means for receiving a user request to fine-tune the avatar, means for regenerating the avatar based on the fine-tuning request, means for saving the finalized avatar, means for the user to select a study session, means for providing study content based on the selected study session, means for asking questions and providing explanations based on the provided study content, means for receiving the user's answers and determining whether they are correct or incorrect, means for providing feedback based on the determination results, means for managing the user's study progress, means for generating answers and detailed explanations using artificial intelligence, and means for displaying study progress management information in a dashboard format, thereby enabling the provision of a personalized learning experience, real-time feedback, and intuitive study progress management.

[1626] "User information" refers to information such as name, age, grade, preferred subjects, and desired avatar characteristics that a user enters when using an application.

[1627] The "custom avatar generation means" refers to an algorithm or program for automatically creating a custom avatar that matches the characteristics of the user based on input user information.

[1628] The "avatar display means" refers to an interface or device for visually presenting the generated custom avatar to the user.

[1629] The "means for receiving a fine-tuning request" is an interface or program that allows a user to request fine-tuning of characteristics such as appearance or voice of a custom avatar.

[1630] "Avatar regeneration means" refers to an algorithm or program for regenerating an avatar based on the user's fine-tuning requests.

[1631] The "avatar storage means" refers to a program or storage for saving the finalized custom avatar in a database or the like.

[1632] A "learning session selection means" is an interface or program that allows a user to select a particular learning session.

[1633] The "learning content providing means" refers to a system or program that provides appropriate learning content to a user based on the selected learning session.

[1634] "Question-setting means" refers to a program or interface that presents questions to users based on the learning content provided and provides explanations.

[1635] The "answer receiving means" refers to an interface that allows the user to input an answer, and a program that transmits the answer to the server.

[1636] The "correctness determination means" is an algorithm or program for determining whether the user's answer is correct or incorrect.

[1637] "Feedback providing means" refers to a program or system that provides feedback to the user in the form of voice or text based on the result of determining whether the answer is correct or incorrect.

[1638] A "learning progress management means" is a system or program that records and manages a user's learning progress and uses the information to help with the next learning session.

[1639] "Artificial intelligence" refers to machine learning algorithms and neural network technology that analyzes users' answers and generates answers and detailed explanations.

[1640] The "means for displaying in a dashboard format" refers to an interface or program for displaying learning progress management information in a visually easy-to-understand dashboard format.

[1641] MODE FOR CARRYING OUT THE INVENTION

[1642] The present invention provides a system that provides a custom avatar that functions as a personal tutor and delivers an individually optimized learning experience. This system includes a series of processes for inputting and managing user information, generating and adjusting the custom avatar, providing educational content throughout the learning session, and evaluating the user's answers and providing feedback.

[1643] Hardware and software used

[1644] Server: Manages user information, generates and stores custom avatars, provides learning content, judges answers, generates feedback, and manages learning progress.

[1645] Device: Enter user information, view a custom avatar, select a study session, enter and submit answers, view feedback, and view study progress.

[1646] OCR library (e.g. Tesseract): Converts paper questions into text data.

[1647] Generative AI models (e.g., machine learning algorithms, neural networks): generate answers and detailed explanations.

[1648] Initial Setup and User Profile Creation

[1649] First, the user downloads the application and accesses the user registration screen when launching it for the first time. Here, the user enters their name, age, grade, preferred subjects, and desired avatar characteristics (such as gender, appearance, and voice). The device collects this information and sends it to the server. The server then creates a user profile based on the received information and sends it to the device.

[1650] Avatar Creation and Customization

[1651] The server uses a custom avatar generation algorithm to automatically construct an avatar with the specified characteristics. The generated avatar is sent to the device, where the user can review its appearance and request fine-tuning if necessary. A retuning request is sent back to the server, which then retunes the avatar. This process is repeated until the user finally finds an avatar they are happy with and stores it on the server.

[1652] Start a study session

[1653] When a user selects a study session in the application, the device sends a request to the server to provide study content for the selected subject. The server sends the appropriate problem sets and explanations to the device, which then presents them to the user through an avatar.

[1654] Problem reading and explanation

[1655] When a user uploads a photo of a written or handwritten problem with their camera, the device converts it into text using OCR technology. The converted data is sent to a server, which analyzes the problem and generates an answer and detailed explanation. The generated answer and explanation are then sent to the device, where an avatar uses voice functions to explain the answer to the user.

[1656] Answer assessment and feedback

[1657] When a user enters an answer to a question, the device sends the answer to the server. The server compares the answer with the correct answer and sends the result back to the device. The device then presents the result to the user, and an avatar provides audio feedback. Through feedback such as "That's right!" or "This is wrong," the user can identify areas for improvement.

[1658] Learning progress management

[1659] The server records the user's answers for each learning session and manages their progress. Progress data is reflected in the next learning session and feedback. The device provides this management information to the user and guides them to the next learning session. The device is also designed to allow for easy schedule changes and adjustments.

[1660] Specific examples

[1661] For example, if a fourth-grade user wants to learn multiplication, he / she first completes the avatar setup and then selects the multiplication learning session. Here, the avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback saying, "That's correct! Well done!" Through this process, the user can check their understanding while progressing through the learning process.

[1662] Prompt Sentence Examples

[1663] Prompt statement:

[1664] "Analyze a math problem solved by a user and generate the answer and a detailed explanation. The problem is '12 x 8' and the answer is '96'."

[1665] In this way, this invention provides an individually optimized learning experience and effective feedback, enabling children to receive high-quality educational support at home.

[1666] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1667] Step 1: Enter and submit user information

[1668] Input: User information such as name, age, grade, preferred subject, desired avatar characteristics

[1669] Specific operation: The user downloads the application, accesses the user registration screen when launching it for the first time, and enters the required information.

[1670] Data processing / calculation: The terminal temporarily stores the input information in its memory.

[1671] Output: Collected user information

[1672] Step 2: Generate a user profile

[1673] Input: User information

[1674] Specific operation: The device sends the collected user data to the server.

[1675] Data processing / calculation: The server creates a new user profile in the database based on the received information and enters the corresponding data into each field.

[1676] Output: Generated user profile

[1677] Step 3: Generate and submit a custom avatar

[1678] Input: User Profile

[1679] What Happens: The server uses a custom avatar generation algorithm to build an avatar based on the user profile.

[1680] Data processing / calculation: Use an AI-based avatar generation model to generate an avatar with specified characteristics.

[1681] Output: The generated avatar

[1682] Step 4: Review your avatar and request tweaks

[1683] Input: Generated avatar

[1684] Specific operation: The generated avatar is sent to the device, and the user can check its appearance and request fine-tuning if necessary.

[1685] Data processing / calculation: The device temporarily stores the user's fine-tuning request in memory.

[1686] Output: Tweak request

[1687] Step 5: Regenerate your avatar

[1688] Input: Tweak Request

[1689] Specific operation: The device sends a fine-tuning request to the server, which adjusts and regenerates an avatar with the specified characteristics.

[1690] Data processing / calculation: The server receives the fine-tuning request and regenerates the avatar based on the avatar generation algorithm.

[1691] Output: Regenerated avatar

[1692] Step 6: Save the final avatar

[1693] Input: Regenerated avatar

[1694] Specific operation: The server saves the final avatar that the user is satisfied with in the database.

[1695] Data processing / calculation: The regenerated avatar data is saved in the server database.

[1696] Output: Final saved avatar

[1697] Step 7: Select and request a study session

[1698] Input: User's learning session selection information

[1699] Specific operation: A user selects a learning session in the application. The device sends a content request to the server based on the selected subject.

[1700] Data processing / calculation: Based on the information of the selected learning session, a request is generated and sent to the server.

[1701] Output: Learning content request

[1702] Step 8: Provide learning content

[1703] Input: Learning content request

[1704] Specific operation: The server selects an appropriate problem set and explanatory materials and sends them to the terminal.

[1705] Data processing / calculation: Retrieve problem sets and explanatory materials from the database and compose the selected content.

[1706] Output: Problem set and explanations

[1707] Step 9: Questions and Answers

[1708] Input: Problem set and explanation materials

[1709] Specific operation: The device displays the problem to the user through the avatar and provides an explanation, after which the user inputs the answer.

[1710] Data processing / calculation: Display the problem set and receive user answer input.

[1711] Output: User's answer

[1712] Step 10: Answer assessment

[1713] Input: User's answer

[1714] Specific operation: The terminal sends the user's answer to the server, which determines whether the answer is correct or not.

[1715] Data processing / calculation: The answer is compared with the correct data to generate a correct or incorrect result.

[1716] Output: Judgment result

[1717] Step 11: Provide feedback

[1718] Input: Judgment result

[1719] Specific operation: The server sends the judgment result to the terminal, and the terminal provides feedback to the user through the avatar.

[1720] Data processing / calculation: The feedback content is constructed based on the judgment results.

[1721] Output: Feedback

[1722] Step 12: Manage your learning progress

[1723] Input: Feedback and Answer Results

[1724] Specific operation: The server records the user's answers for each learning session and manages the progress. The device displays this management information in a dashboard format.

[1725] Data processing / calculation: Answer results and feedback are stored in a database, and progress is visualized.

[1726] Output: Progress management data and dashboard display

[1727] (Application example 1)

[1728] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1729] Conventional learning support and training systems have struggled to provide individually optimized support for each user. Training systems for new employees and field workers also struggled to provide real-time guidance and feedback tailored to on-site work. As a result, there was a challenge in being unable to provide appropriate support tailored to each user's level of understanding and progress.

[1730] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1731] In this invention, the server includes means for inputting user information, means for generating a custom avatar based on the input user information, means for displaying the generated custom avatar, means for the user to receive a request to fine-tune the avatar, means for regenerating the avatar based on the fine-tuning request, means for saving the finalized avatar, means for the user to select a study session, means for providing study content based on the selected study session, means for asking questions and providing explanations based on the provided study content, means for receiving the user's answers and determining whether they are correct or incorrect, means for providing feedback based on the determination results, means for managing the user's study progress, means for a new employee or field worker to create their own profile and for providing training content based on the profile, and means for recording the user's operations and results with a camera and providing feedback. This enables individually optimized study support and training.

[1732] "Means for inputting user information" refers to a device or program that allows a user to input information such as name, age, and work experience.

[1733] The "means for generating a custom avatar based on input user information" refers to a device or program that generates an avatar using a specific algorithm based on information input by the user.

[1734] The "means for displaying the generated custom avatar" is a device or program for visually displaying the generated avatar on a screen.

[1735] A "means for accepting user requests to fine-tune an avatar" is a device or program that allows a user to input requests to change characteristics such as the appearance or voice of the avatar.

[1736] The "means for regenerating an avatar based on a fine-tuning request" refers to a device or program that adjusts the characteristics of the avatar and regenerates it based on a user request.

[1737] The "means for saving the finalized avatar" is a device or program for saving the finalized avatar that satisfies the user in a database or the like.

[1738] A "means for user selection of a learning session" is a device or program that allows a user to select the learning content or training session they wish to use.

[1739] The "means for providing learning content based on a selected learning session" refers to a device or program for providing a user with learning materials and questions according to the selected learning session.

[1740] "Means for asking questions and providing explanations based on the provided learning content" refers to a device or program that presents questions to the user based on the learning content and provides explanations for the answers.

[1741] The "means for receiving the user's answer and determining whether it is correct" is a device or program for receiving the answer entered by the user and determining whether it is correct.

[1742] The "means for providing feedback based on the judgment result" is a device or program that provides appropriate feedback to the user based on the result of the judgment of whether the answer is correct or incorrect.

[1743] "Means for managing a user's learning progress" refers to a device or program for recording and managing a user's learning history and progress.

[1744] A "means for new employees or field workers to create their own profile" is a device or program that allows new employees or field workers to enter information such as their name, work experience, and learning goals to create a profile.

[1745] The "means for providing training content based on a profile" refers to a device or program for providing individually optimized training materials and content based on the created profile information.

[1746] "Means for recording the operations and results performed by the user with a camera and providing feedback" refers to a device or program that uses a camera to capture the operations performed by the user, analyzes the data, and provides appropriate feedback.

[1747] This invention is a system that provides a custom avatar that acts as a personal tutor and delivers an individually optimized learning experience. In particular, it includes a system that provides practical training in real time for new employees and field workers. Specific implementation methods of the invention are as follows.

[1748] First, the user downloads a dedicated application onto their smartphone or tablet. When they first launch the application, they access the user registration screen and enter their user information, such as their name, age, and work experience. This information is then sent from the device to the server.

[1749] The server then uses a custom avatar generation algorithm to build an avatar based on the user's profile. The generated avatar is sent to the device, where the user can review its appearance and request tweaks if necessary. The tweaked information is sent back to the server, and the avatar is regenerated. This process is repeated until the user is satisfied, and the final avatar is saved on the server.

[1750] The user then selects a learning session on the dedicated application. The training content is based on practical applications such as machine operation and safety management. For example, if a new employee is receiving training on machine operation in a factory, the server sends the selected training content to the terminal and provides it to the user via an avatar.

[1751] During training, the user's operations are recorded using the device's camera function. Using OCR technology (using Tesseract.js), the recorded operations are converted into text data and sent to the server. The server generates the analysis results, and the avatar provides feedback using the voice function. For example, if the user performs an incorrect operation, the avatar will say, "This is incorrect. Please do it this way next time."

[1752] The server then receives the user's answer and determines whether it is correct. The result of the determination is immediately provided to the user as feedback via the avatar. For example, if the user makes a mistake in the operating procedure, the server will provide feedback such as, "This is an error. The correct way is this."

[1753] Finally, the server has the function of managing the user's learning progress. It records the results and answer history of each training session and reflects them in the next learning session. It also provides new training content according to the user's progress, supporting continuous skill development.

[1754] Examples:

[1755] For example, when a new employee receives training on operating machinery in a factory, the user creates a profile on a dedicated tablet and customizes the avatar's appearance and voice. The server then explains safe operating procedures for the machine via the avatar, and the user performs the procedures. The operations recorded by the camera are converted into text data using OCR technology (Tesseract.js) and analyzed by the server. Based on the results, the avatar provides feedback, such as "This is incorrect."

[1756] Example prompts to input to the generative AI model:

[1757] Create a new worker profile in the system. Input data includes name, age, and work experience. Follow the steps below to provide training content for operating factory robots. The system will capture user operation results with a camera, convert them into text data using OCR technology, analyze them, and provide feedback via an avatar.

[1758] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1759] Step 1:

[1760] Users use their smartphones or tablets to download a dedicated application.

[1761] Input: User operation (application download)

[1762] Action: Install application

[1763] Output: Application initial launch screen

[1764] Step 2:

[1765] A user starts the application and accesses the user registration screen, where they enter user information such as their name, age, and work experience.

[1766] Input: User information (name, age, work experience, etc.)

[1767] Process: Enter and submit user information

[1768] Output: A user profile is generated on the server.

[1769] Step 3:

[1770] The server uses a custom avatar generation algorithm to generate a custom avatar based on the user profile, and the generated avatar is sent to the device.

[1771] Input: User profile information

[1772] Processing: Custom avatar generation (Unity3D, TensorFlow)

[1773] Output: Generated custom avatar data

[1774] Step 4:

[1775] The custom avatar is displayed on the device, allowing the user to check its appearance, voice, etc., and request tweaks as needed.

[1776] Input: Generated avatar data

[1777] Action: Display a custom avatar

[1778] Output: Avatar confirmation screen

[1779] Step 5:

[1780] The device accepts the user's fine-tuning request and sends it to the server, which then fine-tunes the avatar, regenerates it, and sends it to the device.

[1781] Input: Tweak request from user

[1782] Action: Regenerate custom avatar

[1783] Output: Fine-tuned custom avatar data

[1784] Step 6:

[1785] The final custom avatar is saved on the server and the user selects a learning or training session.

[1786] Input: Confirmed custom avatar data, user session selection

[1787] Actions: Save custom avatar, select learning or training session

[1788] Output: Session start screen

[1789] Step 7:

[1790] The server sends appropriate content to the terminal based on the selected learning or training session.

[1791] Input: Selected information for your study or training session

[1792] Processing: Selection and delivery of training content (using Django)

[1793] Output: Training content

[1794] Step 8:

[1795] The user performs the training and records the operations and results using the device's camera.

[1796] Input: The action taken by the user

[1797] Process: Recording the operation (using a camera)

[1798] Output: Recorded operation data

[1799] Step 9:

[1800] The device converts the recorded operation data into text data using OCR technology (Tesseract.js) and sends it to the server.

[1801] Input: Recorded operation data

[1802] Processing: Converting operation data into text (OCR technology)

[1803] Output: Text data

[1804] Step 10:

[1805] The server analyzes the text data, and the avatar provides audio feedback based on the results.

[1806] Input: Text data

[1807] Processing: Data analysis and feedback generation (using Scikit-learn)

[1808] Output: Generate feedback content

[1809] Step 11:

[1810] The device provides the user with audible feedback via an avatar.

[1811] Input: Feedback

[1812] Processing: Audio feedback

[1813] Output: Audio feedback

[1814] Step 12:

[1815] The server manages the user's learning progress, recording the results of each training session and reflecting them in the next session.

[1816] Input: Results of each training session

[1817] Processing: Data management of learning progress (using MongoDB)

[1818] Output: Learning progress data and what to expect in the next session

[1819] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1820] The present invention relates to a personal tutor avatar system for assisting users in their learning, specifically a system that combines an emotion engine to recognize the user's emotional state and adjust dialogue and feedback accordingly to provide a more effective learning experience. This system includes processes for inputting user information, generating and displaying a custom avatar, managing a learning session, and recognizing the user's emotions to adjust dialogue content.

[1821] 1. Initial Setup and User Profile Creation

[1822] First, a user downloads the application and accesses the user registration screen when they first launch it. Here, the user enters their name, age, grade, preferred subject, and desired avatar characteristics (such as gender, appearance, and voice). The device then sends the collected user data to the server, and an appropriate user profile is generated.

[1823] 2. Avatar Creation and Customization

[1824] Based on the generated user profile, the server uses a custom avatar generation algorithm to automatically construct an avatar with the specified characteristics. The generated avatar is sent to the device, where the user can review the avatar's appearance and request fine-tuning if necessary. The fine-tuning request is sent back to the server, which then re-adjusts the avatar. This process is repeated until a final avatar that the user is satisfied with is determined and stored on the server.

[1825] 3. Start your study session

[1826] When a user selects a study session in the application, the device sends a request to the server to provide learning content for the selected subject. The server sends the appropriate problem sets and explanations to the device, which then presents them to the user via an avatar. For example, if a user selects a study session on multiplication in arithmetic, relevant problems and explanations are provided.

[1827] 4. Use of Emotion Engine

[1828] During the learning session, the device's built-in camera and microphone collect the user's facial expressions and voice data. This data is sent to the server in real time and analyzed by the emotion engine. Based on the analysis results, the server recognizes the user's emotional state (e.g., "happiness," "sadness," "anger," "surprise," etc.).

[1829] 5. Reading the problem and explaining it

[1830] When a user uploads a photo of a written or handwritten problem with their camera, the device uses OCR technology to convert it into text data. The converted data is sent to a server, which analyzes the problem and generates an answer and detailed explanation. The generated answer and explanation are then sent to the device, where an avatar uses voice functions to explain the answer to the user.

[1831] 6. Answer assessment and feedback

[1832] When a user enters an answer to a question, the device sends the answer to the server. The server compares it with the correct answer and returns the result to the device. The device then presents the result to the user, and an avatar provides audio feedback. Furthermore, the avatar's dialogue content and tone can be adjusted based on the user's recognized emotions. For example, if the user is confused, the avatar will use encouraging words such as "It's okay, let me explain it again."

[1833] 7. Learning progress management

[1834] The server records the user's answers for each study session and manages their progress. Progress data is reflected in the next study session and feedback. It also records recognized emotional data and uses this information to optimize study. For example, if a user frequently feels frustrated with a particular problem, it can provide additional support materials related to that problem. The device provides this management information to the user and guides them through the next study session. It is also designed to allow for easy schedule changes and adjustments.

[1835] Specific examples

[1836] For example, if a fourth-grade user wants to learn multiplication, he / she first completes the avatar setup and then selects the multiplication learning session. Here, the avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback, saying, "That's correct! Well done!" At the same time, if the user looks confused, the avatar will provide additional support, saying, "Shall I explain it again?" Through this process, the user can progress through their learning while checking their understanding.

[1837] As described above, through this invention, children can receive high-quality, individually optimized educational support regardless of their family environment or financial situation. The use of an emotion engine further improves the learning experience by recognizing the user's emotional state and providing appropriate feedback accordingly.

[1838] The processing flow will be explained below.

[1839] Step 1:

[1840] The user downloads the application and launches it for the first time, which displays the user registration screen.

[1841] Step 2:

[1842] The user enters user information such as name, age, grade, preferred subjects, and desired avatar characteristics.

[1843] Step 3:

[1844] The device sends the entered user information to the server, which then creates a user profile.

[1845] Step 4:

[1846] The server generates a custom avatar based on the user profile using an avatar generation algorithm.

[1847] Step 5:

[1848] The server sends the generated custom avatar data to the device.

[1849] Step 6:

[1850] The device displays the custom avatar on the screen, allowing the user to see what the avatar looks like.

[1851] Step 7:

[1852] If a user wants to make minor adjustments to the appearance of their avatar, they make an adjustment request.

[1853] Step 8:

[1854] The device sends a fine-tuning request to the server, which regenerates the avatar based on the adjustment request and sends the data back to the device.

[1855] Step 9:

[1856] Once the user is satisfied with the appearance of the avatar, the avatar is finally finalized and the terminal transmits the finalized data to the server.

[1857] Step 10:

[1858] The server stores the determined avatar information in the user profile.

[1859] Step 11:

[1860] Users select a study session (e.g., "Math Multiplication") from the study schedule screen within the app.

[1861] Step 12:

[1862] The terminal requests the selected learning session information from the server.

[1863] Step 13:

[1864] The server sends the appropriate learning content (problem sets and explanatory materials) to the device.

[1865] Step 14:

[1866] The terminal presents the learning content to the user through the avatar and initiates a learning session.

[1867] Step 15:

[1868] The device's built-in camera and microphone are used to collect the user's facial expressions and voice data in real time.

[1869] Step 16:

[1870] The collected facial and voice data is sent to a server, which uses an emotion engine to analyze the data and recognize the user's emotional state.

[1871] Step 17:

[1872] When a user takes a photo of a question written on paper or entered manually and uploads it, the device uses OCR technology to convert it into text data, which is then sent to the server.

[1873] Step 18:

[1874] The server analyzes the question text and generates appropriate answers and explanations.

[1875] Step 19:

[1876] The server sends the generated answers and explanations to the terminal.

[1877] Step 20:

[1878] The device provides audio commentary from an avatar, educating the user and adjusting the content and tone of the dialogue based on the user's emotional state.

[1879] Step 21:

[1880] The user enters the answer into the terminal.

[1881] Step 22:

[1882] The terminal sends the answer to the server.

[1883] Step 23:

[1884] The server determines whether the answer is correct or incorrect and sends the result to the terminal.

[1885] Step 24:

[1886] The device displays the results and an avatar provides audible feedback such as "That's right!" or "This is wrong."

[1887] Step 25:

[1888] The server records the user's answers and updates their learning progress data, along with their emotional state.

[1889] Step 26:

[1890] The device will prompt the user for the next study session or provide options to reschedule.

[1891] Step 27:

[1892] The user selects or schedules the next study session, which causes the system to start the study process again.

[1893] Through these steps, users can learn efficiently and effectively. By utilizing the emotion engine, appropriate feedback and support are provided that takes into account the user's emotional state, resulting in a more personalized learning experience.

[1894] Example 2

[1895] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1896] Conventional learning support systems have the problem of being unable to provide an effective learning experience that takes into account the user's emotional state. In addition, they are unable to provide customized feedback optimized for each individual user, which can reduce the effectiveness of learning.

[1897] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting facial expression and voice data of the user and analyzing it with an emotion engine, means for adjusting the dialogue content based on the analysis results and providing emotionally sensitive feedback, and means for managing the user's learning progress. This allows for an effective learning experience by providing detailed feedback according to the user's emotional state.

[1898] - "User Information" refers to personal information about a User, such as the User's name, age, grade, preferred subjects, and desired avatar characteristics.

[1899] "Custom avatar" refers to an avatar created specifically for a user based on user information.

[1900] "Study Session" means a study session offered based on a subject or topic selected by a User.

[1901] "Study Content" refers to the problem sets and explanatory materials provided based on a study session.

[1902] "Questions" refer to assignments or quizzes presented to users within the learning content.

[1903] "Answer" refers to the answer entered by the user in response to a question.

[1904] "Explanation" refers to explanations or supplementary information for the answers to questions.

[1905] "Feedback" refers to the evaluation and advice provided to users based on whether their answers are correct or incorrect.

[1906] "Facial expression data" refers to data collected by capturing a user's facial expressions with a camera.

[1907] "Voice data" refers to data collected by capturing the user's voice with a microphone.

[1908] An "emotion engine" refers to algorithms or software that analyzes facial expression and voice data to recognize the user's emotional state.

[1909] "Learning progress" refers to a user's progress, results, and achievements in learning.

[1910] "Means of collection" refers to devices or methods for obtaining data from users, such as cameras or microphones.

[1911] "Means of analysis" refers to the techniques and methods for processing collected data and analyzing emotions and learning trends.

[1912] "Means for adjusting dialogue content" refers to a mechanism for appropriately changing the avatar's speech and tone depending on the user's emotional state.

[1913] "Camera function" refers to a device and its functions for taking images and videos.

[1914] "OCR technology" refers to the technology that recognizes characters in an image and converts them into text data.

[1915] The present invention relates to a personal tutor avatar system for supporting users in their learning, specifically a system that combines an emotion engine to recognize the user's emotional state and adjust dialogue and feedback accordingly to provide a more effective learning experience. This system includes processes for inputting user information, generating and displaying a custom avatar, managing a learning session, and recognizing the user's emotions to adjust dialogue content.

[1916] First, the user downloads the learning support application and accesses the user registration screen when launching it for the first time. Here, the user enters their name, age, grade, preferred subjects, and desired avatar characteristics (gender, appearance, voice, etc.). This information is saved on the device. The device temporarily stores the information entered by the user and sends it to the server as soon as registration is complete. A message indicating registration is complete is displayed, and the device prepares to proceed to the next step.

[1917] The server generates an individual user profile based on the received user information. The generated profile is stored in a database and used in subsequent processes. The server then activates a custom avatar generation algorithm to automatically generate an avatar with the specified characteristics based on the user profile. The generated avatar data is sent to the device, which receives the data and displays the avatar on its screen.

[1918] The user checks the appearance of the avatar and requests fine-tuning if necessary. This request is sent from the device to the server, and the server regenerates the avatar reflecting the fine-tuning and sends the data back to the device. This process is repeated until the user is satisfied, and once the final avatar is finalized, the avatar data is saved on the server.

[1919] When a user selects a learning session, the device sends a request to the server to provide learning content for the selected subject. The server receives the request, prepares the corresponding problem set and explanatory materials, and sends the data to the device. Based on the received data, the device presents the learning session for the selected subject through an avatar. For example, in a learning session on multiplication in arithmetic, problems such as "12 x 8" are presented.

[1920] During a learning session, the device uses a camera and microphone to collect the user's facial and voice data. This data is sent to the server in real time and analyzed by the emotion engine. The server then analyzes the received facial and voice data using the emotion engine to recognize the user's emotional state (e.g., "happiness," "sadness," "anger," "surprise," etc.).

[1921] When a user writes a question on paper, takes a photo of it with a camera, and uploads it, the device uses OCR technology to convert the image data into text data. This converted text data is then sent to a server. The server analyzes the received text data and generates the answer to the question and a detailed explanation. The generated answer and explanation are then sent back to the device, which then provides them to the user through an avatar. The avatar uses voice functionality to provide explanations.

[1922] When a user enters an answer to a question and presses the send button, the device sends the answer data to the server. The server compares the received answer with the correct answer and sends the result back to the device. The result includes a correct / incorrect determination and detailed feedback. The device receives the result, and an avatar gives the user audio feedback. The content and tone of the feedback can be adjusted depending on the user's emotional state, and may use encouraging words such as "Shall I explain it again?"

[1923] The server records the user's answers for each study session and manages the user's study progress, including the results for each session and the user's emotional data. The server uses the recorded progress data to prepare for optimizing the next study session. The device notifies the user when the next study session is ready and helps the user easily manage their study progress and schedule.

[1924] Specific examples

[1925] For example, if a fourth-grade user wants to learn multiplication, he or she first sets up the avatar and then selects the multiplication learning session. The avatar presents the problem "12 x 8." When the user enters the answer (96), the system immediately checks the answer and provides voice feedback, saying, "That's right! Well done!" Furthermore, if the user shows a confused expression, the avatar offers additional support by asking, "Shall I explain it again?"

[1926] Example of input prompt for generative AI model

[1927] "If the user's facial expression looks confused, have the avatar generate encouraging feedback, such as 'Shall I explain it again?' or 'Please let me know if you don't understand.'"

[1928] As described above, the present invention makes it possible to individually optimize the learning experience of each user and provide more effective learning support.

[1929] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1930] Step 1: User registration and initial setup

[1931] First, the user downloads the learning support application and accesses the user registration screen when the application is launched for the first time. The user enters their name, age, grade, preferred subjects, and desired avatar characteristics (gender, appearance, voice, etc.). This information is saved on the device. The device temporarily stores the user information and sends it to the server as soon as registration is complete. The server saves the received information in a database and returns a confirmation response to the device. The device then displays a message indicating that user registration has been completed.

[1932] Input: User information (name, age, grade, preferred subject, avatar characteristics)

[1933] Output: User information sent to the server, user registration complete message

[1934] Step 2: Create and save a user profile

[1935] The server generates a user profile based on the received user information. The generated profile is saved in a database for use in subsequent processes. A confirmation response is sent to the device confirming that the profile has been saved.

[1936] Input: User information

[1937] Output: Generated user profile, confirmation that it was saved by the server

[1938] Step 3: Avatar Creation and Customization

[1939] The server runs a custom avatar generation algorithm to automatically generate an avatar with the specified characteristics based on the user profile. The generated avatar data is sent to the device, where the user can view the displayed avatar. If the user requests fine-tuning as needed, this request is sent back to the server, and the avatar is regenerated. Finally, once the user is satisfied with the avatar, the avatar data is saved on the server.

[1940] Input: User profile, fine-tuning request

[1941] Output: Custom avatar, Retuned avatar

[1942] Specific actions: Adjusting the avatar's gender, appearance, voice, and other characteristics

[1943] Step 4: Start your study session

[1944] When a user selects a study session, the device sends a request to the server to provide study content according to the selected subject. The server receives the request, prepares the corresponding problem sets and explanation materials, and sends the data to the device. The device then presents the study session through an avatar based on the received data.

[1945] Input: User selects a study session

[1946] Output: Learning content (question sets, explanatory materials)

[1947] Specific actions: Displaying learning content and presenting questions

[1948] Step 5: Emotion Recognition with the Emotion Engine

[1949] During a learning session, the device uses a camera and microphone to collect facial and voice data from the user. This data is sent in real time to a server where it is analyzed by an emotion engine. The server recognizes the user's emotional state and adjusts feedback accordingly.

[1950] Input: facial expression data, voice data

[1951] Output: Analysis results (user's emotional state)

[1952] Specific actions: taking pictures of facial expressions with a camera and collecting audio with a microphone

[1953] Step 6: Reading the problem and generating an explanation

[1954] When a user takes a photo of a paper problem and uploads it, the device uses OCR technology to convert the image data into text. This converted data is sent to a server, which analyzes the problem, generates answers and explanations, and sends them back to the device. The device then provides these to the user through an avatar.

[1955] Input: Image data of the problem

[1956] Output: OCR text data, answers and explanations

[1957] Specific actions: taking a photo with a camera, OCR conversion, and explanation by an avatar

[1958] Step 7: Answers and feedback

[1959] The user inputs the answer to the question and sends it to the device. The device then sends the answer data to the server, which checks the answer against the correct answer and returns a judgment result. The device then presents the result to the user, and an avatar provides audio feedback. The content and tone of the feedback are adjusted according to the user's emotional state.

[1960] Input: User-provided answer data

[1961] Output: Matching result and feedback (correct / incorrect)

[1962] Specific actions: Entering answers, checking, and voice feedback

[1963] Step 8: Track your progress and prepare for the next session

[1964] The server records the user's answers for each study session and manages their study progress, including their grades and emotional state for each session. Based on the recorded progress data, the server optimizes the next study session and notifies the device when the next study session is ready. It also makes it easy to change the progress and schedule.

[1965] Input: Answer result, emotional state

[1966] Output: Tracked progress data, next session notifications

[1967] Specific actions: Recording results, managing emotional data, notifying preparation for the next session

[1968] (Application example 2)

[1969] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1970] Conventional learning support systems do not take into account the user's emotional state, and therefore have the problem of being unable to provide appropriate support when the user becomes confused or frustrated. This reduces the effectiveness of learning and leads to a lack of sustained user motivation. Furthermore, there is a lack of technology that can analyze the user's emotions in real time and appropriately adjust feedback based on the results.

[1971] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for inputting user information; means for generating a custom avatar based on the input user information; means for displaying the generated custom avatar; means for accepting a user request to fine-tune the avatar; means for regenerating the avatar based on the fine-tuning request; means for saving the finalized avatar; means for the user to select a study session; means for providing study content based on the selected study session; means for asking questions and providing explanations based on the provided study content; means for receiving the user's answers and determining whether they are correct or incorrect; means for providing feedback based on the determination results; means for managing the user's study progress; means for collecting the user's facial expression and voice data using a camera and microphone function; means for analyzing the user's emotions based on the collected data; and means for adjusting the feedback content based on the analysis results. This makes it possible to provide appropriate feedback according to the user's emotional state, improve learning effectiveness, and maintain the user's motivation.

[1972] "User information" refers to basic data about the user, including name, age, grade, preferred subjects, and avatar characteristics.

[1973] A "custom avatar" is a character with a unique appearance and voice that is generated based on user inpu...

Claims

1. a means for inputting user information; means for generating a custom avatar based on input user information; a means for displaying the generated custom avatar; a means for users to request tweaks to their avatars; means for regenerating the avatar based on the tweak request; A means to save the finalized avatar; a means for a user to select a learning session; means for providing learning content based on the selected learning session; A means of asking questions and providing explanations based on the learning content provided; a means for receiving the user's answer and determining whether it is correct; a means for providing feedback based on the determination; a means of managing the user's learning progress; A system including:

2. The system according to claim 1 , further comprising means for taking a photograph of the paper problem using a camera function and converting the photograph into text data using OCR technology.

3. The system of claim 1 , further comprising means for the generated custom avatar to use a voice function to provide problem explanation and feedback.

Images