System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to create an account and select a language and avatar by default; means for generating a personalized study plan based on the user's age, grade, and study goals; means for analyzing unknown photos taken by the user and generating commentary; means for displaying the generated commentary and materials to the user; and means for recording the user's study progress and adjusting the study plan for the next day.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, many families lack the time for parents to support their children's learning, and many families find it difficult to send their children to cram schools for financial reasons. Under these circumstances, it is difficult for children to effectively learn at a level that suits their own level of understanding. Furthermore, existing learning systems lack sufficient multilingual support and individual avatar settings, and lack the ingenuity to increase motivation to learn. Furthermore, current private tutoring services are expensive, making them economically inaccessible to many families. Under these circumstances, there is a challenge in that not all children have equal access to high-quality education. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes a means for a user to create an account and select a language and avatar in the initial settings; a means for generating an individualized learning plan based on the user's age, grade, and learning goals; a means for analyzing photos of unclear points taken by the user and generating explanations; a means for displaying the generated explanations and learning materials to the user; and a means for recording the user's learning progress and adjusting the learning plan for the next day. Furthermore, by including a means for storing the user's setting information and learning plan in a database and a means for playing audio for the explanations and learning materials, an effective and attractive learning environment can be provided. This makes it possible for anyone to receive a high-quality education regardless of financial constraints or family circumstances.

[0006] "Users" refers to students, their parents, and teachers who use the system.

[0007] "Account" refers to the registration information required for a User to access and use the System.

[0008] "Initial setup" refers to the process by which a user sets personal settings such as language and avatar.

[0009] "Language" refers to a setting that specifies the language used by the user.

[0010] "Avatar" refers to a character representation that can be selected based on a user's personality and preferences.

[0011] "Study plan" refers to a specific schedule of study content generated by the system based on the user's age, grade, and learning goals.

[0012] "Photo" refers to an image taken with a smartphone or other device to record parts that the user does not understand.

[0013] "Analysis" refers to the process by which the system identifies and understands the content of the photos it receives.

[0014] "Explanation" refers to the explanation or answer that the system generates for a problem.

[0015] "Teaching materials" refers to learning resources such as texts, videos, and VR content that users use to advance their studies.

[0016] "Display" refers to the act of a system visually presenting information to a user.

[0017] "Progress" refers to the user's learning progress.

[0018] "Recording" refers to the act of the system saving a user's learning history and progress information.

[0019] "Adjustment" refers to the act of changing and optimizing a study plan based on a user's progress.

[0020] "Database" refers to the storage device where the system stores user information, study plans, and progress data.

[0021] "Audio playback" refers to the system's ability to provide audio commentary and teaching materials. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] The present invention is a tutoring system that uses AI to help users effectively advance their learning. This system functions through the interaction of a server, a terminal, and a user. A specific embodiment of this system will be described below.

[0044] User registration and initial settings

[0045] First, the user creates an account using a device (smartphone, tablet, PC, etc.). When creating an account, they enter basic information such as their name, email address, and password. The server then stores the user's information in a database and creates the account.

[0046] Next, the user selects the language and avatar as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This information is also sent from the device to the server and stored in the database.

[0047] Generate a lesson plan

[0048] The server retrieves information about the user's age, grade level, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[0049] Start learning

[0050] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, videos, VR content, etc.) corresponding to the day's learning content. For example, for the first math class, learning materials for learning number concepts are provided using VR. The device displays this learning material to the user, and learning begins.

[0051] Questions and Explanations

[0052] If a user comes across a question during their study, they can take a photo of the problem with their smartphone. The photo is then sent to a server via their device. The server then uses an image analysis algorithm to analyze the photo and determine the nature of the question. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via their device.

[0053] Track your progress and adjust your study plan

[0054] Once the learning is complete, the device sends a progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[0055] Specific examples

[0056] For example, when 8-year-old User A uses the system for the first time, he or she creates an account, selects Japanese as the initial language, and sets a favorite avatar. After that, math and science teaching materials are displayed on the device according to the learning plan sent from the server. User A begins learning number concepts in VR, and if there is anything they don't understand, they take a photo and send it to the server. The server analyzes the information and returns an explanation, allowing User A to deepen their understanding. After completing the learning, progress information is sent to the server, and the next day's learning plan is fine-tuned.

[0057] In this way, the AI-based tutoring system can provide high-quality education efficiently and effectively, without being affected by financial constraints or family circumstances.

[0058] The processing flow will be explained below.

[0059] Step 1:

[0060] The user launches the application and is directed to the account creation screen.

[0061] Step 2:

[0062] The device prompts the user to enter the required information (name, email address, password, etc.).

[0063] Step 3:

[0064] The user enters the information and presses the submit button.

[0065] Step 4:

[0066] The device sends the entered information to the server.

[0067] Step 5:

[0068] The server stores the received information in a database and creates an account.

[0069] Step 6:

[0070] The server sends the user an initial setup page.

[0071] Step 7:

[0072] The device will display an initial setup page, prompting the user to select a language and avatar.

[0073] Step 8:

[0074] Users can choose their preferred language and avatar.

[0075] Step 9:

[0076] The device sends the user's configuration information to the server.

[0077] Step 10:

[0078] The server stores the received configuration information in a database.

[0079] Step 11:

[0080] The server retrieves information about the user's age, grade level, and learning goals from a database.

[0081] Step 12:

[0082] The server generates an individualized learning plan based on the information it obtains.

[0083] Step 13:

[0084] The server generates a learning plan and sends it to the device.

[0085] Step 14:

[0086] The device displays the learning plan to the user.

[0087] Step 15:

[0088] The user presses the start button to begin learning.

[0089] Step 16:

[0090] The device sends a start request to the server.

[0091] Step 17:

[0092] The server prepares the day's learning content based on the user's request.

[0093] Step 18:

[0094] The server sends the corresponding educational materials (text, videos, VR content, etc.) to the device.

[0095] Step 19:

[0096] The device displays the teaching materials to the user.

[0097] Step 20:

[0098] As users continue their studies, if they come across something they don't understand, they can take a photo of it with their smartphone.

[0099] Step 21:

[0100] The device takes a photo and sends it to the server.

[0101] Step 22:

[0102] The server then uses an image analysis algorithm to analyze the photos it receives and determine the nature of the problem.

[0103] Step 23:

[0104] The server generates an explanation for the problem and creates explanatory content in text or audio.

[0105] Step 24:

[0106] The server sends the generated commentary to the device.

[0107] Step 25:

[0108] The device will display the explanation to the user.

[0109] Step 26:

[0110] Users continue to learn.

[0111] Step 27:

[0112] The device sends progress information to the server at the end of the learning process.

[0113] Step 28:

[0114] The server stores the received progress data in a database.

[0115] Step 29:

[0116] The server adjusts the next day's study plan based on the progress data.

[0117] Step 30:

[0118] The server generates a new learning plan and sends it to the device.

[0119] Step 31:

[0120] The device will notify the user of the new study plan.

[0121] Step 32:

[0122] The user can then review the new plan and prepare for the next day's study.

[0123] Example 1

[0124] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] Conventional learning systems lack the ability to respond to individual learning needs, making it difficult to maximize users' learning effectiveness. Furthermore, they lack the means to provide quick and accurate explanations when users encounter questions during their studies. Furthermore, they do not dynamically adjust learning plans according to learning progress, making it difficult to continuously improve learning.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0127] In this invention, the server includes: a means for a user to create an account and select a language and avatar in the initial settings; a means for generating an individualized learning plan based on the user's age, grade, and learning goals; a means for analyzing images of unclear points taken by the user and generating explanations; a means for displaying the generated explanations and learning materials to the user; and a means for recording the user's learning progress and adjusting the learning plan for the next day. This enables effective learning support tailored to individual learning needs. Furthermore, by enabling users to quickly resolve unclear points, they can deepen their understanding of the learning. Furthermore, by dynamically adjusting the learning plan according to their learning progress, continuous improvement in learning is possible.

[0128] "User" refers to an individual who uses the system.

[0129] "Account" refers to information for user authentication, including personal information required for a user to access and use the system.

[0130] "Initial settings" refers to the language, avatar, and other settings that a user selects when they first start using the system.

[0131] "Language" refers to the language used by the user to display and operate the system.

[0132] "Avatar" refers to the character or image a user chooses to represent themselves on the system.

[0133] A "study plan" refers to a schedule of study content for a specific period of time that is generated based on the user's age, grade level, and learning goals.

[0134] "Image analysis" refers to the process of analyzing an image taken by a user and understanding its contents.

[0135] "Explanation" refers to content that explains the problem or issue in a way that is easy for users to understand.

[0136] "Learning materials" refers to materials such as texts, videos, and VR content provided for users to study.

[0137] "Progress" refers to the portion of the learning experience that the user has achieved and the current situation.

[0138] "Adjustment" refers to the process of changing and updating the learning plan according to the user's learning progress.

[0139] A "server" refers to a computer system that receives a user request, processes it, and returns a response.

[0140] "Terminal" refers to a device such as a smartphone, tablet, or PC that a user uses to access and operate the system.

[0141] "Database" refers to a system for storing and managing user information, study plans, etc.

[0142] A "generative AI model" refers to an algorithm or system that uses machine learning and artificial intelligence techniques to generate personalized learning plans and explanations for users.

[0143] A "prompt" refers to text that is an instruction or question entered into a generative AI model.

[0144] This invention is a tutoring system that uses AI to support users in effectively progressing with their studies. This system functions through the interaction between a server, a terminal, and a user.

[0145] User registration and initial settings

[0146] First, a user creates an account using a device such as a smartphone, tablet, or PC. When creating an account, they enter basic information such as their name, email address, and password. The entered information is sent from the device to a server, which then stores the information in a database and creates the account.

[0147] Next, the user selects the language and avatar to use as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This setting information is also sent from the device to the server and stored in the database.

[0148] Generate a lesson plan

[0149] The server retrieves information about the user's age, grade level, and learning goals from a database. It then uses a generative AI model to generate a personalized learning plan for the user. The generative AI model is an algorithm that uses machine learning and artificial intelligence techniques to derive appropriate learning content based on specific prompts. For example, an 8-year-old user might be scheduled to learn basic math problems and fundamental science concepts.

[0150] The generated lesson plan is sent to the terminal and displayed to the user.

[0151] Start learning

[0152] The user presses the start button on the device to begin learning. This request is sent from the device to the server. The server prepares the learning materials (text, videos, VR content, etc.) corresponding to the learning content for that day. For example, for the first math class, materials for learning number concepts are provided using VR. The prepared materials are sent to the device, which then displays them to the user.

[0153] Questions and Explanations

[0154] If a user encounters a problem during learning, they can take a photo of the problem with their smartphone. The image is then sent to the server via the device. The server then analyzes the photo using an image analysis algorithm (e.g., OpenCV, TensorFlow) to determine the problem.

[0155] Based on the analysis results, the server uses a generative AI model to generate an explanation. The generated explanation is provided in text or audio format and displayed to the user through the device. For example, it provides a step-by-step solution to a difficult math problem. Examples of prompts used in this process include:

[0156] "Analyze user-taken photos of math problems and provide step-by-step solutions."

[0157] Track your progress and adjust your study plan

[0158] Once the learning is complete, the device sends a progress report to the server. The server stores the progress data in a database and uses the generative AI model to adjust the next day's learning plan. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[0159] In this way, the present invention provides an AI-based tutoring system that provides effective education tailored to individual learning needs.

[0160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0161] Step 1:

[0162] A user creates an account using a device such as a smartphone, tablet, or PC. First, the user enters basic information such as their name, email address, and password, and submits it. The device then sends the entered information to the server. The server stores the received information in a database and creates the user's account.

[0163] (Input) User name, email address, and password.

[0164] (Output) The user account is saved in the database.

[0165] (Specific operation) When a user enters information into an input form on a terminal and presses the send button, the information is sent to the server as an HTTP request, and the server processes the information to save it in a database.

[0166] Step 2:

[0167] Next, the user selects the language and avatar to use as the default settings. The information selected by the user is then sent from the device to the server and stored in a database.

[0168] (Input) Selected language, avatar.

[0169] (Output) Language and avatar settings are saved in the database.

[0170] (Specific operation) The user selects a language and avatar using drop-down menus or radio buttons, and then presses the Settings button. The setting information is sent to the server as an HTTP request, and the information is saved in a database on the server side.

[0171] Step 3:

[0172] The server retrieves information about the user's age, grade level, and learning goals from a database and uses a generative AI model to generate an individualized lesson plan. For example, an 8-year-old user would receive a lesson plan that includes basic math and science content. The generated lesson plan is then sent to the device and displayed to the user.

[0173] (Input) User's age, grade level, and learning goals.

[0174] (Output) Individualized learning plan.

[0175] (Specific operation) The server executes a database query to obtain user information, and generates a learning plan using prompts for the generative AI model. The generated learning plan is sent to the device as an HTTP response, and the device displays it to the user.

[0176] Step 4:

[0177] The user presses the start button on the device to begin learning. This request is sent from the device to the server. The server prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content, and the prepared materials are sent to the device and displayed to the user.

[0178] (Input) Press the start button.

[0179] (Output) Learning materials.

[0180] (Specific operation) When the user presses the start button, an HTTP request is sent to the server. The server selects learning materials according to the learning content and sends them to the terminal as an HTTP response. The terminal displays the received learning material data.

[0181] Step 5:

[0182] When a user encounters a problem during learning, they can take a photo of the problem with their smartphone and send it to the server via their device. The server then uses an image analysis algorithm to analyze the photo and generates an explanation using a generative AI model. The explanation is then displayed to the user via their device in text or audio format.

[0183] (Input) The captured image of the problem.

[0184] (Output) Explanatory text or audio.

[0185] (Specific operation) The user takes a photo of the problem with their smartphone, and the device sends it to the server as an HTTP request. The server analyzes the image using an image analysis library and applies a generative AI model to generate an explanation. The explanation data is sent to the device as an HTTP response, and the device displays it.

[0186] Step 6:

[0187] Once the learning is complete, the device sends a progress report to the server. The server stores the progress data in a database and adjusts the next day's learning plan using the generative AI model. The new adjusted learning plan is sent to the device and notified to the user.

[0188] (Input) Learning progress data.

[0189] (Output) The new adjusted lesson plan.

[0190] (Specific operation) The device sends learning progress information to the server via an HTTP request, and the server saves the progress data in a database. The next day's learning plan is readjusted using the generative AI model, and the new plan is sent to the device as an HTTP response to notify the user.

[0191] (Application example 1)

[0192] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0193] Conventional tutoring systems have difficulty meeting the individual needs of learners, making it difficult to provide efficient and effective learning support. Furthermore, providing operational guidance and problem-solving for factory engineers requires individualized support, making it difficult to implement. The objective of this invention is to provide an effective and efficient learning and operational guidance system that allows learners and engineers to receive support based on their individual needs.

[0194] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0195] In this invention, the server includes means for a user to create an account and select a language and avatar in the initial settings, means for generating an individualized learning plan based on the user's age, grade, and learning goals, means for analyzing photos of unclear parts taken by the user and generating an explanation, means for displaying the generated explanation and learning materials to the user, means for recording the user's learning progress and adjusting the user's learning plan for the next day, means for a technician to create an account and perform initial settings, means for generating individualized operating instructions based on the technician's skill level and past work history, means for analyzing photos and videos taken by the technician when a problem occurs and generating an explanation, means for displaying the generated explanation and instruction content to the technician, and means for recording operation progress and adjusting the next instruction content. This allows users to receive individualized learning support, and technicians to receive effective operating instructions and problem-solving support.

[0196] "User" refers to a person who creates an account to use the system and receives a learning plan and instruction on how to use it.

[0197] "Create an account" refers to the process by which a user enters their information to register with the system and configure their personal settings.

[0198] "Initial Settings" refers to the basic settings that a user makes when accessing the system, including selecting a language and an avatar.

[0199] "Language" refers to the language used by the user when learning or receiving instruction.

[0200] "Avatar" refers to the character or image a user chooses to represent themselves within the system.

[0201] "Age" refers to the age of the user or technician based on their date of birth.

[0202] "Grade" refers to the educational stage to which a learner currently belongs.

[0203] "Learning goal" refers to the goal regarding the learning content or skills that the user wants to achieve.

[0204] "Individualized Learning Plan" refers to a personalized learning schedule based on a user's age, grade level, and learning goals.

[0205] "Photography" refers to the act of taking photos or videos using the system.

[0206] "Analysis" refers to processing captured images and video, performing calculations and evaluations to identify problems.

[0207] "Explanation" refers to content that provides step-by-step explanations and procedures for solving a problem.

[0208] "Teaching materials" refers to educational materials such as textbooks, videos, and VR content used by users to further their studies.

[0209] "Study progress" refers to data that indicates the achievement status of a user as they progress with their studies.

[0210] "Adjustment" refers to the act of changing and optimizing learning plans and instructional content based on progress data.

[0211] "Technician" refers to someone who uses the system to operate and maintain robots and machinery in a factory or other location.

[0212] "Skill level" refers to an indicator of an engineer's technical proficiency and ability.

[0213] "Operational instruction" refers to instructions and guidelines for engineers to operate robots and machines properly.

[0214] "Photos and videos at the time of the problem" refers to visual data taken by technicians to record malfunctions in machines or robots.

[0215] "Instruction content" refers to the specific work procedures and solutions provided to engineers.

[0216] "Progress" refers to data that shows the progress a technician makes as they carry out operation and maintenance tasks.

[0217] To implement the present invention, the system functions mainly through the interaction of a server, a terminal, and a user. Specifically, the system is configured as follows:

[0218] First, a user creates an account using a device (smartphone, tablet, PC, etc.) and performs initial setup. This initial setup includes selecting a language (e.g., Japanese or English) and an avatar. This information is sent from the device to the server and stored in the server's database.

[0219] The server generates an individualized learning plan based on the user's age, grade level, and learning goal data. This learning plan may include, for example, basic math problems or basic science concepts. The generated learning plan is sent to the device and displayed to the user.

[0220] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content. The device displays this material to the user, and learning begins.

[0221] If a user encounters a problem during their study, they can take a photo of the problem using their device's camera. This photo is then sent from the device to a server, which analyzes it using an image analysis algorithm. Based on the analysis results, a generative AI model then generates an explanation for the problem and provides it to the user in text or audio.

[0222] Meanwhile, factory engineers can also use the system. Engineers create an account and perform initial setup. The server generates individual operating instructions based on the engineer's skill level and past work history. If a problem occurs during operation, the engineer takes a photo or video with their device and sends it to the server. The server analyzes the image, and the generative AI model generates the cause of the problem and a solution.

[0223] For example, if an engineer encounters an error while operating a factory robot, they can use a smartphone app to take a photo of the problem and send it to the AI ​​via the app. The AI ​​will then analyze the image and provide the engineer with the cause and solution of the problem in real time. This process enables effective operation guidance and problem-solving support.

[0224] Servers and terminals use software such as Python, OpenCV (image processing), TensorFlow (AI model), and SQLite (database management), and data processing and calculation are carried out through image processing and AI analysis.

[0225] Specific prompt examples:

[0226] "An error occurred while operating the factory robot. Please analyze the picture below and tell me the cause and how to solve it."

[0227] (upload photo here)

[0228] Through this system, users can receive individual learning support, and engineers can receive effective operation instruction and problem-solving support.

[0229] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0230] Step 1:

[0231] The user uses a device to create an account and perform initial settings. Input data includes the user's name, email address, password, language, and avatar information. The device sends this information to the server, which stores it in a database. Specifically, the user enters the required information on the device's registration screen and presses the send button.

[0232] Step 2:

[0233] The server generates an individualized learning plan based on the user's age, grade level, and learning goal data. The input data is the user's basic information, and the output is an individualized learning plan. To generate this, the server refers to past learning data and standard curricula to determine the optimal learning content for the user. Specifically, an algorithm within the server analyzes the user's information and generates a plan.

[0234] Step 3:

[0235] The user presses the start button on the device to begin learning. The input data is the user's start request, and the output is the display of learning materials (text, video, VR content, etc.) corresponding to the day's learning content. The device sends a request to the server to display this, and the server provides learning materials based on the day's plan. In concrete terms, the user presses the start button and the learning materials are displayed on the device.

[0236] Step 4:

[0237] If a user encounters a problem during learning, they use the device's camera to take a photo of the problem. The input data is the photo, and the output is an explanation or solution for the problem. The device sends the photo to a server, which then analyzes the problem using an image analysis algorithm (such as OpenCV). Specifically, the user takes a photo and uploads it from the device to the server.

[0238] Step 5:

[0239] The server uses the generative AI model to generate an explanation for the problem based on the image analysis results. The input data is the image analysis results, and the output is the generated explanation. The server provides this to the user in text or audio format. Specifically, the AI ​​model generates an explanation based on the analysis results, and sends it from the server to the device in text or audio format.

[0240] Step 6:

[0241] When the learning is completed, the device sends learning progress information to the server. The input data is the learning progress information, and the output is the adjusted learning plan for the next day. The server analyzes the received progress data and adjusts the learning plan for the next day. Specifically, the device sends the learning progress to the server, and the server generates a newly adjusted learning plan.

[0242] Step 7:

[0243] Engineers working in factories also use terminals to create accounts and perform initial setup. The input data is the engineer's name, skill level, and past work history, and the output is an individual operation training plan. The terminal sends this to the server, which then stores the engineer's information in a database. Specifically, the engineer enters information on the terminal's registration screen and presses the send button.

[0244] Step 8:

[0245] If a problem occurs during operation, the technician takes a photo or video with the device and sends it to the server. The input data is the photo or video, and the output is the cause of the problem and a solution. The server performs image analysis, and a generative AI model generates the cause of the problem and a solution. Specifically, the technician takes a photo or video and uploads it from the device to the server.

[0246] Step 9:

[0247] The server displays the generated explanations and instruction content to the engineer based on the analysis results. The input data is the analysis results, and the output is the generated explanations and instruction content. The server provides this to the engineer in text or audio format. Specifically, the generative AI model generates explanations based on the analysis results, and sends them from the server to the terminal in text or audio format.

[0248] Step 10:

[0249] When the operation is completed, the terminal sends operation progress information to the server. The input data is the operation progress information, and the output is the adjustment result of the next operation instruction content. The server analyzes the received progress data and adjusts the next operation instruction content. Specifically, the terminal sends the operation progress to the server, and the server generates newly adjusted instruction content.

[0250] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0251] The present invention is a tutoring system that uses AI to support users' learning. In particular, it aims to improve learning efficiency and motivation by incorporating an emotion engine that recognizes the user's emotions. This system functions through the interaction between a server, a terminal, and the user. A specific embodiment of this system will be described below.

[0252] User registration and initial settings

[0253] First, a user creates an account using a device (smartphone, tablet, PC, etc.). When creating an account, basic information such as name, email address, and password is entered and submitted, which is then sent to the server. The server stores the received information in a database and creates the account.

[0254] Next, the user selects the language and avatar as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This information is also sent from the device to the server and stored in the database.

[0255] Generate a lesson plan

[0256] The server retrieves information about the user's age, grade level, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[0257] Start learning

[0258] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content and sends them to the device. For example, for the first math class, learning materials for learning number concepts are provided using VR. The device displays this learning material to the user, and learning begins.

[0259] Questions and Explanations

[0260] If a user comes across a question during their study, they can take a photo of the problem with their smartphone. The photo is then sent to a server via their device. The server then uses an image analysis algorithm to analyze the photo and determine the nature of the question. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via their device.

[0261] Track your progress and adjust your study plan

[0262] Once the learning is complete, the device sends a progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[0263] Incorporating an emotion engine

[0264] The system also includes an emotion engine that recognizes the user's emotions. The emotion engine uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. For example, if the user loses concentration or feels stressed while studying, the emotion engine will detect this situation.

[0265] Dynamic adjustment based on emotions

[0266] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data: for example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material, or it can speed up the progress if positive emotions are recognized.

[0267] Modifying materials based on emotions

[0268] Based on the emotions recognized by the emotion engine, the server can also change the content of the displayed learning materials and explanations. For example, if the user is having fun, the server can display learning materials incorporating game elements to further stimulate interest. In this way, the server can provide an optimal learning environment according to the user's emotional state.

[0269] Specific examples

[0270] For example, if 8-year-old User A feels tired, the emotion engine will recognize this and the server will change the learning materials to make them more relaxing. Also, if User A feels stressed during a particular unit, the server will check the user's progress and adjust the difficulty of that unit to be less difficult. This makes the user's learning experience more effective and enjoyable.

[0271] This invention enables a tutoring system that utilizes AI to respond flexibly to the user's emotional state, maximizing learning efficiency and motivation.

[0272] The processing flow will be explained below.

[0273] Step 1:

[0274] The user launches the application and is directed to the account creation screen.

[0275] Step 2:

[0276] The device prompts the user to enter the required information (name, email address, password, etc.).

[0277] Step 3:

[0278] The user enters the information and presses the submit button.

[0279] Step 4:

[0280] The device sends the entered information to the server.

[0281] Step 5:

[0282] The server stores the received information in a database and creates an account.

[0283] Step 6:

[0284] The server sends the user an initial setup page.

[0285] Step 7:

[0286] The device will display an initial setup page, prompting the user to select a language and avatar.

[0287] Step 8:

[0288] Users can choose their preferred language and avatar.

[0289] Step 9:

[0290] The device sends the user's configuration information to the server.

[0291] Step 10:

[0292] The server stores the received configuration information in a database.

[0293] Step 11:

[0294] The server retrieves information about the user's age, grade level, and learning goals from a database.

[0295] Step 12:

[0296] The server generates an individualized learning plan based on the information it obtains.

[0297] Step 13:

[0298] The server generates a learning plan and sends it to the device.

[0299] Step 14:

[0300] The device displays the learning plan to the user.

[0301] Step 15:

[0302] The user presses the start button to begin learning.

[0303] Step 16:

[0304] The device sends a start request to the server.

[0305] Step 17:

[0306] The server prepares the day's learning content based on the user's request.

[0307] Step 18:

[0308] The server sends the corresponding educational materials (text, videos, VR content, etc.) to the device.

[0309] Step 19:

[0310] The device displays the teaching materials to the user.

[0311] Step 20:

[0312] As users continue their studies, if they come across something they don't understand, they can take a photo of it with their smartphone.

[0313] Step 21:

[0314] The device takes a photo and sends it to the server.

[0315] Step 22:

[0316] The server then uses an image analysis algorithm to analyze the photos it receives and determine the nature of the problem.

[0317] Step 23:

[0318] The server generates an explanation for the problem and creates explanatory content in text or audio.

[0319] Step 24:

[0320] The server sends the generated commentary to the device.

[0321] Step 25:

[0322] The device will display the explanation to the user.

[0323] Step 26:

[0324] Users continue to learn.

[0325] Step 27:

[0326] The device sends progress information to the server at the end of the learning process.

[0327] Step 28:

[0328] The server stores the received progress data in a database.

[0329] Step 29:

[0330] The server adjusts the next day's study plan based on the progress data.

[0331] Step 30:

[0332] The system also includes an emotion engine that recognizes the user's emotions. During training, the device sends the user's facial expressions and voice to the emotion engine.

[0333] Step 31:

[0334] The emotion engine uses facial recognition and voice analysis technology to analyze the user's emotions.

[0335] Step 32:

[0336] The emotion engine sends the analysis results to the server.

[0337] Step 33:

[0338] The server dynamically adjusts according to the user's emotions based on data from the emotion engine.

[0339] Step 34:

[0340] For example, if a user is feeling stressed, the server can reduce the difficulty of the study plan or switch to more engaging material.

[0341] Step 35:

[0342] The server changes the content of the teaching materials and explanations displayed based on the data from the emotion engine.

[0343] Step 36:

[0344] For example, if the user is having fun, the server will display educational materials that incorporate game elements on the device.

[0345] Step 37:

[0346] The device displays tailored learning materials and explanations to the user.

[0347] Step 38:

[0348] After the learning session ends, the device sends the emotion engine analysis data and learning progress data to the server.

[0349] Step 39:

[0350] The server stores this data in a database and adjusts the next day's study plan accordingly.

[0351] Step 40:

[0352] The server sends the new study plan to the device, which then notifies the user.

[0353] Step 41:

[0354] Users can review their new study plan and prepare for the next day's study.

[0355] Example 2

[0356] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0357] Conventional learning support systems have difficulty in fully responding to the individual learning needs of users, limiting their ability to improve learning efficiency and motivation. Furthermore, because they are unable to dynamically adjust learning plans or change learning materials taking into account the user's emotional state, they provide a uniform learning process, resulting in a suboptimal learning experience.

[0358] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0359] In this invention, the server includes means for a user to create an account and select a language and avatar in the initial settings, means for generating an individual study plan based on the user's age, grade, and study goals, means for preparing and displaying study materials in response to a user's request at the start of study, means for analyzing photos of unclear points taken by the user and generating explanations, means for displaying the generated explanations and study materials to the user, means for recording the user's study progress and adjusting the study plan for the next day, means for analyzing the user's facial expressions and tone of voice and dynamically adjusting the study plan based on the user's emotional state, and means for changing the content of the study materials and explanations to be displayed based on the user's emotional state. This makes it possible to provide a flexible study process that meets the individual learning needs of users and corresponds to their emotional state.

[0360] "User" refers to a person who uses the tutoring system to study.

[0361] "Account" refers to a user profile that contains authentication information for a user to access and use the System.

[0362] "Initial settings" refers to basic setting operations such as language selection and avatar settings that users perform before using the system.

[0363] "Language" refers to the interface language that the user uses within the system.

[0364] "Avatar" refers to a character or icon that a user selects within a virtual environment.

[0365] A "study plan" refers to a series of study schedules and content created based on a user's age, grade level, and learning goals.

[0366] "Teaching materials" refers to educational materials such as textbooks, videos, and VR content used by users to study.

[0367] "Explanation" refers to text or audio materials that explain questions and learning content in a way that is easy for users to understand.

[0368] "Study progress" refers to the progress and learning status achieved by the user in the course of their studies.

[0369] An "emotion engine" refers to technology that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[0370] "Database" refers to a system for centrally managing and storing data such as user settings and study plans.

[0371] "Generative AI model" refers to artificial intelligence technology for automatically generating study plans based on user information.

[0372] A "prompt sentence" refers to an input sentence that gives specific instructions to a generative AI model.

[0373] MODE FOR CARRYING OUT THE INVENTION

[0374] This invention relates to a tutoring system that uses AI to support users' learning. In particular, it aims to improve learning efficiency and motivation by incorporating an emotion engine that recognizes users' emotions. This system functions through the interaction of a server, a terminal, and a user.

[0375] User registration and initial settings

[0376] First, a user creates an account using a device such as a smartphone, tablet, or PC. When creating an account, basic information such as name, email address, and password is entered and submitted, which is then sent to the server. The server stores the received information in a database and creates the account.

[0377] Next, the user selects the language and avatar as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This information is also sent from the device to the server and stored in the database.

[0378] Generate a lesson plan

[0379] The server retrieves information about the user's age, grade level, and learning goals from the database and uses a generative AI model to generate a personalized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[0380] Start learning

[0381] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content and sends them to the device. For example, for the first math class, learning materials for learning number concepts are provided using VR. The device displays this learning material to the user, and learning begins.

[0382] Questions and Explanations

[0383] If a user comes across a question during their study, they can take a photo of the problem with their smartphone. The photo is then sent to a server via their device. The server then uses an image analysis algorithm to analyze the photo and determine the nature of the question. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via their device.

[0384] Track your progress and adjust your study plan

[0385] Once the learning is complete, the device sends a progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[0386] Incorporating an emotion engine

[0387] The system also includes an emotion engine that recognizes the user's emotions. The emotion engine uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. For example, if the user loses concentration or feels stressed while studying, the emotion engine will detect this situation.

[0388] Dynamic adjustment based on emotions

[0389] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data: for example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material, or it can speed up the progress if positive emotions are recognized.

[0390] Modifying materials based on emotions

[0391] Based on the emotions recognized by the emotion engine, the server can also change the content of the displayed learning materials and explanations. For example, if the user is having fun, the server can display learning materials incorporating game elements to further stimulate interest. In this way, the server can provide an optimal learning environment according to the user's emotional state.

[0392] Specific examples

[0393] For example, if 8-year-old User A feels tired, the emotion engine will recognize this and the server will change the learning materials to make them more relaxing. Also, if User A feels stressed during a particular unit, the server will check the user's progress and adjust the difficulty of that unit to be less difficult. This makes the user's learning experience more effective and enjoyable.

[0394] Sample prompt sentence

[0395] Example user profile:

[0396] Age: 8

[0397] Grade: 3rd grade

[0398] Learning Objectives: Learning basic mathematical operations and basic science concepts

[0399] Prompt sentence to input to the generative AI model:

[0400] "Create a daily lesson plan for third graders that includes basic math operations (e.g., addition, subtraction) and basic science concepts (e.g., familiar natural phenomena)."

[0401] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0402] Step 1:

[0403] A user creates an account using a terminal. As input, the user enters basic information such as name, email address, and password into the terminal and presses the "Submit" button. The terminal sends this information to the server. The output is that the user is presented with a message confirming successful account registration.

[0404] Step 2:

[0405] The server stores the received user information in a database. The input is user information sent from the terminal, such as name, email address, and password. The database adds this information as a new record. The output is that a new user record is created and saved in the database.

[0406] Step 3:

[0407] The user uses the device to perform initial setup. As input, the user selects the desired language and avatar and saves the settings. The device sends this information to the server. The output is that the initial setup information is sent to the server and stored in a database.

[0408] Step 4:

[0409] The server retrieves information about the user's age, grade, and learning goals from the database. The input is the user ID. The server uses the user ID as a key to retrieve related information from the database. The output is the user's age, grade, and learning goals.

[0410] Step 5:

[0411] The server uses the generative AI model to create an individualized learning plan. The input is the information about the user's age, grade, and learning goals obtained in step 4. The generative AI model creates prompt sentences based on this information and automatically generates a learning plan. The output is the generation of an individualized learning plan.

[0412] Step 6:

[0413] The server sends the generated learning plan to the device. The input is the learning plan output from the generative AI model. The server sends this to the user's device. The output is the learning plan displayed on the device.

[0414] Step 7:

[0415] The user presses the start button on the terminal to start learning. The input is the user's operation. The terminal notifies the server of this operation. The output is that a request to start learning is sent to the server.

[0416] Step 8:

[0417] The server prepares learning materials corresponding to the day's learning content. The input is a request to start learning. The server selects appropriate learning materials from the database and sends them to the terminal. The output is the learning materials displayed on the terminal.

[0418] Step 9:

[0419] If a user has a question while studying, they can use the camera to take a photo of the question. The input is the visual information of the question. The user takes a photo using the camera and saves it on the device. The output is the photographed photo data.

[0420] Step 10:

[0421] The device sends the captured photo data to the server. The input is a photo taken by the user. The device uploads this photo to the server. The output is the photo data stored on the server.

[0422] Step 11:

[0423] The server analyzes the photo using an image analysis algorithm. The input is the photo data sent from the device. The server analyzes the image and identifies the problem content. The output is the analyzed problem content.

[0424] Step 12:

[0425] The server generates an explanation based on the analysis results. The input is the problem content obtained by image analysis. The server creates an explanation using the explanation generation model. The output is the generated explanation.

[0426] Step 13:

[0427] The server sends the generated comment to the terminal. The input is the generated comment. The server sends this information to the terminal. The output is the comment displayed on the terminal.

[0428] Step 14:

[0429] The terminal displays the explanatory text to the user. The input is the explanatory text sent by the server. The terminal displays it in an appropriate format for the user. The output is the explanatory text that is presented to the user.

[0430] Step 15:

[0431] When the learning is completed, the terminal sends the learning progress data to the server. The input is the user's learning progress information. The terminal uploads it to the server. The output is the learning progress data stored on the server.

[0432] Step 16:

[0433] The server stores the received learning progress data in a database. The input is the progress data sent from the terminal. The server stores this in a database. The output is the learning progress data recorded in the database.

[0434] Step 17:

[0435] The server adjusts the learning plan for the next day. The input is the saved learning progress data. The server creates a new learning plan using a generative AI model based on the progress data. The output is the adjusted learning plan.

[0436] Step 18:

[0437] The server sends the new lesson plan to the device. The input is the adjusted lesson plan. The server sends this to the device. The output is the new lesson plan displayed on the device.

[0438] Step 19:

[0439] The device analyzes the user's facial expressions and tone of voice using an emotion engine. The input is the user's facial expressions and voice information. The device captures this data with a camera and microphone and analyzes it with the emotion engine. The output is the analyzed emotion data.

[0440] Step 20:

[0441] The emotion engine sends the analysis results to the server. The input is the analyzed emotion data. The emotion engine sends this data to the server. The output is the emotion data stored on the server.

[0442] Step 21:

[0443] The server dynamically adjusts the learning plan based on the analysis results. The input is the emotion data sent from the emotion engine. The server dynamically changes the learning plan using a generative AI model based on this. The output is the adjusted learning plan.

[0444] Step 22:

[0445] The server sends the adjusted lesson plan to the device. The input is the new lesson plan. The server sends it to the device. The output is the adjusted lesson plan displayed on the user's device.

[0446] Step 23:

[0447] The server changes the content of the learning materials and explanations displayed based on the results of the emotion engine. The input is the adjusted learning plan and emotional data. The server takes the emotional state into consideration and selects the most appropriate learning materials and explanations, which it then sends to the device. The output is the updated learning materials and explanations that are displayed to the user.

[0448] (Application example 2)

[0449] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0450] Conventional learning support systems provide fixed learning plans without considering the user's emotional state, making it difficult to maximize the user's learning efficiency and motivation. Furthermore, when the user is stressed or fatigued, they do not provide appropriate support, which reduces the effectiveness of learning.

[0451] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to create an account and select a language and avatar in the initial settings, means for generating an individual study plan based on the user's age, grade, and learning goals, means for analyzing photos of unclear parts taken by the user and generating explanations, means for displaying the generated explanations and learning materials to the user, means for recording the user's study progress and adjusting the study plan for the next day, means for recognizing the user's emotions using facial recognition technology and voice analysis technology, means for dynamically adjusting the study content based on the recognized emotional data, and means for providing feedback according to the user's emotional state. This makes it possible to maximize the user's learning efficiency and motivation and flexibly adjust the study plan according to the user's individual emotional state.

[0452] The "user account creation means" is a means for a user to create an account for using the system and select a language and avatar as initial settings.

[0453] A "study plan generator" is a means for generating an individualized study plan based on the user's age, grade level, and learning goals.

[0454] The "photo analysis means" is a means for analyzing a photo of an unknown location taken by a user and generating an explanation based on the analysis.

[0455] The "teaching material display means" is a means for displaying the generated explanations and teaching materials to the user.

[0456] The "progress recording means" is a means for recording the user's learning progress and adjusting the next day's learning plan based on that progress.

[0457] The "emotion recognition means" is a means for recognizing the user's emotions using face recognition technology or voice analysis technology.

[0458] The "dynamic adjustment means" is a means for dynamically adjusting the learning content based on the recognized emotion data.

[0459] A "feedback providing means" is a means for providing feedback to a user according to their emotional state.

[0460] The "database storage means" is a means for storing user setting information, study plans, and emotional data in a database.

[0461] The "audio playback means" is a means for playing back audio of the generated commentary or teaching material.

[0462] This invention is a tutoring system that uses AI to support users' learning, and in particular, improves learning efficiency and motivation by incorporating an emotion engine that recognizes users' emotions. This system functions through the interaction of a server, terminals, and users.

[0463] Specifically, this is carried out as follows.

[0464] User account creation method

[0465] The server provides a means for users to create an account using a device (smartphone, tablet, PC, etc.). When creating an account, the user enters basic information such as name, email address, and password and sends it to the server. The server stores the received information in a database and creates the account. The user then selects the language and avatar as initial settings. For example, the user might select Japanese and set a popular anime character as their avatar.

[0466] Learning plan generation tool

[0467] The server retrieves information about the user's age, grade, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[0468] Photo analysis tools

[0469] If a user encounters a problem while studying, they can use their device to take a photo of the problem. The photo is then sent to the server via the device. The server then uses an image analysis algorithm to analyze the photo and determine the problem's content. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via the device.

[0470] emotion recognition means

[0471] The system also includes an emotion engine that recognizes the user's emotions. The server uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. For example, if the user loses concentration or feels stressed while studying, the emotion engine will detect this.

[0472] Dynamic Adjustment Means

[0473] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data: for example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material, or it can speed up the progress if positive emotions are recognized.

[0474] Feedback methods

[0475] Once the learning is complete, the device sends a progress report to the server. The server stores the progress data in a database and adjusts the learning plan for the next day. Feedback can be provided based on the user's emotional state, creating an appropriate learning environment. For example, if the user has a negative reaction to a particular problem, the device can skip the next time the learning is completed or approach it differently.

[0476] Specific examples

[0477] For example, let's say a new employee at Store A is undergoing training using smart glasses. If the employee shows signs of fatigue, the emotion engine will detect this and display a notification on their smart device suggesting a break. Also, if the employee is performing a task with interest, a more challenging task will automatically be suggested as the next step.

[0478] Example prompts for generative AI models

[0479] Describe the process flow for real-time emotion recognition when a new employee is participating in training using smart glasses. Show how the system analyzes the employee's emotional state from their facial expressions and tone of voice, and dynamically adjusts the training accordingly.

[0480] The present invention enables flexible adjustment of study plans and feedback according to the user's emotional state, which is expected to improve study efficiency and motivation.

[0481] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0482] Step 1:

[0483] A user creates an account using a device (smartphone, tablet, smart glasses, etc.). The user enters basic information such as name, email address, and password, and sends that information to the server. The server stores the received information in a database and creates an account. In this process, the user's basic information is given as input, and an account is generated as output.

[0484] Step 2:

[0485] The user selects the language and avatar as the initial settings. For example, they may select Japanese and set a popular anime character as their avatar. This setting information is also sent from the device to the server and stored in the database. In this process, the selected language and avatar information are given as input, and the saved setting information is obtained as output.

[0486] Step 3:

[0487] The server retrieves information about the user's age, grade, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user will be scheduled to learn basic math problems and basic science concepts. This generated learning plan is sent to the device and displayed to the user. In this process, the user's profile information is given as input and an individualized learning plan is generated as output.

[0488] Step 4:

[0489] The user begins studying on their device. If they come across a question they don't understand, they take a photo of the question with their smart device. The photo is sent to the server via the device. The server analyzes the photo using an image analysis algorithm to determine the question's content. Based on the results of this analysis, an explanation is generated and the content is provided as text or audio. In this process, the photo of the question is given as input, and the generated explanation is provided as output.

[0490] Step 5:

[0491] Learning progress is recorded, and when learning is completed, the device sends a learning progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. Based on the progress data, a new learning plan is generated and sent to the device, and the user is notified. In this process, learning progress data is given as input, and the next day's learning plan is adjusted as output.

[0492] Step 6:

[0493] Furthermore, the system is equipped with an emotion engine that recognizes the user's emotions. The server uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. The user's emotional state is confirmed during the learning process, and the data is updated as necessary. During this process, the user's facial expression images and voice data are given as input, and the analyzed emotional state is output.

[0494] Step 7:

[0495] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data. For example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material. It can also speed up the learning process if positive emotions are recognized. This process takes emotion data as input and generates an adjusted learning plan as output.

[0496] Step 8:

[0497] The device provides feedback based on the user's emotional state. For example, if the user has a negative reaction to a particular problem, feedback is provided to change the user's approach to that problem in the next learning session. In this process, emotional state and learning progress are given as inputs, and feedback is provided as output.

[0498] This makes it possible to maximize the user's learning efficiency and motivation.

[0499] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0500] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0501] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0502] [Second embodiment]

[0503] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0504] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0505] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0506] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0507] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0508] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0509] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0510] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0511] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0512] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0513] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0514] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0515] The present invention is a tutoring system that uses AI to help users effectively advance their learning. This system functions through the interaction of a server, a terminal, and a user. A specific embodiment of this system will be described below.

[0516] User registration and initial settings

[0517] First, the user creates an account using a device (smartphone, tablet, PC, etc.). When creating an account, they enter basic information such as their name, email address, and password. The server then stores the user's information in a database and creates the account.

[0518] Next, the user selects the language and avatar as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This information is also sent from the device to the server and stored in the database.

[0519] Generate a lesson plan

[0520] The server retrieves information about the user's age, grade level, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[0521] Start learning

[0522] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, videos, VR content, etc.) corresponding to the day's learning content. For example, for the first math class, learning materials for learning number concepts are provided using VR. The device displays this learning material to the user, and learning begins.

[0523] Questions and Explanations

[0524] If a user comes across a question during their study, they can take a photo of the problem with their smartphone. The photo is then sent to a server via their device. The server then uses an image analysis algorithm to analyze the photo and determine the nature of the question. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via their device.

[0525] Track your progress and adjust your study plan

[0526] Once the learning is complete, the device sends a progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[0527] Specific examples

[0528] For example, when 8-year-old User A uses the system for the first time, he or she creates an account, selects Japanese as the initial language, and sets a favorite avatar. After that, math and science teaching materials are displayed on the device according to the learning plan sent from the server. User A begins learning number concepts in VR, and if there is anything they don't understand, they take a photo and send it to the server. The server analyzes the information and returns an explanation, allowing User A to deepen their understanding. After completing the learning, progress information is sent to the server, and the next day's learning plan is fine-tuned.

[0529] In this way, the AI-based tutoring system can provide high-quality education efficiently and effectively, without being affected by financial constraints or family circumstances.

[0530] The processing flow will be explained below.

[0531] Step 1:

[0532] The user launches the application and is directed to the account creation screen.

[0533] Step 2:

[0534] The device prompts the user to enter the required information (name, email address, password, etc.).

[0535] Step 3:

[0536] The user enters the information and presses the submit button.

[0537] Step 4:

[0538] The device sends the entered information to the server.

[0539] Step 5:

[0540] The server stores the received information in a database and creates an account.

[0541] Step 6:

[0542] The server sends the user an initial setup page.

[0543] Step 7:

[0544] The device will display an initial setup page, prompting the user to select a language and avatar.

[0545] Step 8:

[0546] Users can choose their preferred language and avatar.

[0547] Step 9:

[0548] The device sends the user's configuration information to the server.

[0549] Step 10:

[0550] The server stores the received configuration information in a database.

[0551] Step 11:

[0552] The server retrieves information about the user's age, grade level, and learning goals from a database.

[0553] Step 12:

[0554] The server generates an individualized learning plan based on the information it obtains.

[0555] Step 13:

[0556] The server generates a learning plan and sends it to the device.

[0557] Step 14:

[0558] The device displays the learning plan to the user.

[0559] Step 15:

[0560] The user presses the start button to begin learning.

[0561] Step 16:

[0562] The device sends a start request to the server.

[0563] Step 17:

[0564] The server prepares the day's learning content based on the user's request.

[0565] Step 18:

[0566] The server sends the corresponding educational materials (text, videos, VR content, etc.) to the device.

[0567] Step 19:

[0568] The device displays the teaching materials to the user.

[0569] Step 20:

[0570] As users continue their studies, if they come across something they don't understand, they can take a photo of it with their smartphone.

[0571] Step 21:

[0572] The device takes a photo and sends it to the server.

[0573] Step 22:

[0574] The server then uses an image analysis algorithm to analyze the photos it receives and determine the nature of the problem.

[0575] Step 23:

[0576] The server generates an explanation for the problem and creates explanatory content in text or audio.

[0577] Step 24:

[0578] The server sends the generated commentary to the device.

[0579] Step 25:

[0580] The device will display the explanation to the user.

[0581] Step 26:

[0582] Users continue to learn.

[0583] Step 27:

[0584] The device sends progress information to the server at the end of the learning process.

[0585] Step 28:

[0586] The server stores the received progress data in a database.

[0587] Step 29:

[0588] The server adjusts the next day's study plan based on the progress data.

[0589] Step 30:

[0590] The server generates a new learning plan and sends it to the device.

[0591] Step 31:

[0592] The device will notify the user of the new study plan.

[0593] Step 32:

[0594] The user can then review the new plan and prepare for the next day's study.

[0595] Example 1

[0596] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0597] Conventional learning systems lack the ability to respond to individual learning needs, making it difficult to maximize users' learning effectiveness. Furthermore, they lack the means to provide quick and accurate explanations when users encounter questions during their studies. Furthermore, they do not dynamically adjust learning plans according to learning progress, making it difficult to continuously improve learning.

[0598] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0599] In this invention, the server includes: a means for a user to create an account and select a language and avatar in the initial settings; a means for generating an individualized learning plan based on the user's age, grade, and learning goals; a means for analyzing images of unclear points taken by the user and generating explanations; a means for displaying the generated explanations and learning materials to the user; and a means for recording the user's learning progress and adjusting the learning plan for the next day. This enables effective learning support tailored to individual learning needs. Furthermore, by enabling users to quickly resolve unclear points, they can deepen their understanding of the learning. Furthermore, by dynamically adjusting the learning plan according to their learning progress, continuous improvement in learning is possible.

[0600] "User" refers to an individual who uses the system.

[0601] "Account" refers to information for user authentication, including personal information required for a user to access and use the system.

[0602] "Initial settings" refers to the language, avatar, and other settings that a user selects when they first start using the system.

[0603] "Language" refers to the language used by the user to display and operate the system.

[0604] "Avatar" refers to the character or image a user chooses to represent themselves on the system.

[0605] A "study plan" refers to a schedule of study content for a specific period of time that is generated based on the user's age, grade level, and learning goals.

[0606] "Image analysis" refers to the process of analyzing an image taken by a user and understanding its contents.

[0607] "Explanation" refers to content that explains the problem or issue in a way that is easy for users to understand.

[0608] "Learning materials" refers to materials such as texts, videos, and VR content provided for users to study.

[0609] "Progress" refers to the portion of the learning experience that the user has achieved and the current situation.

[0610] "Adjustment" refers to the process of changing and updating the learning plan according to the user's learning progress.

[0611] A "server" refers to a computer system that receives a user request, processes it, and returns a response.

[0612] "Terminal" refers to a device such as a smartphone, tablet, or PC that a user uses to access and operate the system.

[0613] "Database" refers to a system for storing and managing user information, study plans, etc.

[0614] A "generative AI model" refers to an algorithm or system that uses machine learning and artificial intelligence techniques to generate personalized learning plans and explanations for users.

[0615] A "prompt" refers to text that is an instruction or question entered into a generative AI model.

[0616] This invention is a tutoring system that uses AI to support users in effectively progressing with their studies. This system functions through the interaction between a server, a terminal, and a user.

[0617] User registration and initial settings

[0618] First, a user creates an account using a device such as a smartphone, tablet, or PC. When creating an account, they enter basic information such as their name, email address, and password. The entered information is sent from the device to a server, which then stores the information in a database and creates the account.

[0619] Next, the user selects the language and avatar to use as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This setting information is also sent from the device to the server and stored in the database.

[0620] Generate a lesson plan

[0621] The server retrieves information about the user's age, grade level, and learning goals from a database. It then uses a generative AI model to generate a personalized learning plan for the user. The generative AI model is an algorithm that uses machine learning and artificial intelligence techniques to derive appropriate learning content based on specific prompts. For example, an 8-year-old user might be scheduled to learn basic math problems and fundamental science concepts.

[0622] The generated lesson plan is sent to the terminal and displayed to the user.

[0623] Start learning

[0624] The user presses the start button on the device to begin learning. This request is sent from the device to the server. The server prepares the learning materials (text, videos, VR content, etc.) corresponding to the learning content for that day. For example, for the first math class, materials for learning number concepts are provided using VR. The prepared materials are sent to the device, which then displays them to the user.

[0625] Questions and Explanations

[0626] If a user encounters a problem during learning, they can take a photo of the problem with their smartphone. The image is then sent to the server via the device. The server then analyzes the photo using an image analysis algorithm (e.g., OpenCV, TensorFlow) to determine the problem.

[0627] Based on the analysis results, the server uses a generative AI model to generate an explanation. The generated explanation is provided in text or audio format and displayed to the user through the device. For example, it provides a step-by-step solution to a difficult math problem. Examples of prompts used in this process include:

[0628] "Analyze user-taken photos of math problems and provide step-by-step solutions."

[0629] Track your progress and adjust your study plan

[0630] Once the learning is complete, the device sends a progress report to the server. The server stores the progress data in a database and uses the generative AI model to adjust the next day's learning plan. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[0631] In this way, the present invention provides an AI-based tutoring system that provides effective education tailored to individual learning needs.

[0632] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0633] Step 1:

[0634] A user creates an account using a device such as a smartphone, tablet, or PC. First, the user enters basic information such as their name, email address, and password, and submits it. The device then sends the entered information to the server. The server stores the received information in a database and creates the user's account.

[0635] (Input) User name, email address, and password.

[0636] (Output) The user account is saved in the database.

[0637] (Specific operation) When a user enters information into an input form on a terminal and presses the send button, the information is sent to the server as an HTTP request, and the server processes the information to save it in a database.

[0638] Step 2:

[0639] Next, the user selects the language and avatar to use as the default settings. The information selected by the user is then sent from the device to the server and stored in a database.

[0640] (Input) Selected language, avatar.

[0641] (Output) Language and avatar settings are saved in the database.

[0642] (Specific operation) The user selects a language and avatar using drop-down menus or radio buttons, and then presses the Settings button. The setting information is sent to the server as an HTTP request, and the information is saved in a database on the server side.

[0643] Step 3:

[0644] The server retrieves information about the user's age, grade level, and learning goals from a database and uses a generative AI model to generate an individualized lesson plan. For example, an 8-year-old user would receive a lesson plan that includes basic math and science content. The generated lesson plan is then sent to the device and displayed to the user.

[0645] (Input) User's age, grade level, and learning goals.

[0646] (Output) Individualized learning plan.

[0647] (Specific operation) The server executes a database query to obtain user information, and generates a learning plan using prompts for the generative AI model. The generated learning plan is sent to the device as an HTTP response, and the device displays it to the user.

[0648] Step 4:

[0649] The user presses the start button on the device to begin learning. This request is sent from the device to the server. The server prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content, and the prepared materials are sent to the device and displayed to the user.

[0650] (Input) Press the start button.

[0651] (Output) Learning materials.

[0652] (Specific operation) When the user presses the start button, an HTTP request is sent to the server. The server selects learning materials according to the learning content and sends them to the terminal as an HTTP response. The terminal displays the received learning material data.

[0653] Step 5:

[0654] When a user encounters a problem during learning, they can take a photo of the problem with their smartphone and send it to the server via their device. The server then uses an image analysis algorithm to analyze the photo and generates an explanation using a generative AI model. The explanation is then displayed to the user via their device in text or audio format.

[0655] (Input) The captured image of the problem.

[0656] (Output) Explanatory text or audio.

[0657] (Specific operation) The user takes a photo of the problem with their smartphone, and the device sends it to the server as an HTTP request. The server analyzes the image using an image analysis library and applies a generative AI model to generate an explanation. The explanation data is sent to the device as an HTTP response, and the device displays it.

[0658] Step 6:

[0659] Once the learning is complete, the device sends a progress report to the server. The server stores the progress data in a database and adjusts the next day's learning plan using the generative AI model. The new adjusted learning plan is sent to the device and notified to the user.

[0660] (Input) Learning progress data.

[0661] (Output) The new adjusted lesson plan.

[0662] (Specific operation) The device sends learning progress information to the server via an HTTP request, and the server saves the progress data in a database. The next day's learning plan is readjusted using the generative AI model, and the new plan is sent to the device as an HTTP response to notify the user.

[0663] (Application example 1)

[0664] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0665] Conventional tutoring systems have difficulty meeting the individual needs of learners, making it difficult to provide efficient and effective learning support. Furthermore, providing operational guidance and problem-solving for factory engineers requires individualized support, making it difficult to implement. The objective of this invention is to provide an effective and efficient learning and operational guidance system that allows learners and engineers to receive support based on their individual needs.

[0666] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0667] In this invention, the server includes means for a user to create an account and select a language and avatar in the initial settings, means for generating an individualized learning plan based on the user's age, grade, and learning goals, means for analyzing photos of unclear parts taken by the user and generating an explanation, means for displaying the generated explanation and learning materials to the user, means for recording the user's learning progress and adjusting the user's learning plan for the next day, means for a technician to create an account and perform initial settings, means for generating individualized operating instructions based on the technician's skill level and past work history, means for analyzing photos and videos taken by the technician when a problem occurs and generating an explanation, means for displaying the generated explanation and instruction content to the technician, and means for recording operation progress and adjusting the next instruction content. This allows users to receive individualized learning support, and technicians to receive effective operating instructions and problem-solving support.

[0668] "User" refers to a person who creates an account to use the system and receives a learning plan and instruction on how to use it.

[0669] "Create an account" refers to the process by which a user enters their information to register with the system and configure their personal settings.

[0670] "Initial Settings" refers to the basic settings that a user makes when accessing the system, including selecting a language and an avatar.

[0671] "Language" refers to the language used by the user when learning or receiving instruction.

[0672] "Avatar" refers to the character or image a user chooses to represent themselves within the system.

[0673] "Age" refers to the age of the user or technician based on their date of birth.

[0674] "Grade" refers to the educational stage to which a learner currently belongs.

[0675] "Learning goal" refers to the goal regarding the learning content or skills that the user wants to achieve.

[0676] "Individualized Learning Plan" refers to a personalized learning schedule based on a user's age, grade level, and learning goals.

[0677] "Photography" refers to the act of taking photos or videos using the system.

[0678] "Analysis" refers to processing captured images and video, performing calculations and evaluations to identify problems.

[0679] "Explanation" refers to content that provides step-by-step explanations and procedures for solving a problem.

[0680] "Teaching materials" refers to educational materials such as textbooks, videos, and VR content used by users to further their studies.

[0681] "Study progress" refers to data that indicates the achievement status of a user as they progress with their studies.

[0682] "Adjustment" refers to the act of changing and optimizing learning plans and instructional content based on progress data.

[0683] "Technician" refers to someone who uses the system to operate and maintain robots and machinery in a factory or other location.

[0684] "Skill level" refers to an indicator of an engineer's technical proficiency and ability.

[0685] "Operational instruction" refers to instructions and guidelines for engineers to operate robots and machines properly.

[0686] "Photos and videos at the time of the problem" refers to visual data taken by technicians to record malfunctions in machines or robots.

[0687] "Instruction content" refers to the specific work procedures and solutions provided to engineers.

[0688] "Progress" refers to data that shows the progress a technician makes as they carry out operation and maintenance tasks.

[0689] To implement the present invention, the system functions mainly through the interaction of a server, a terminal, and a user. Specifically, the system is configured as follows:

[0690] First, a user creates an account using a device (smartphone, tablet, PC, etc.) and performs initial setup. This initial setup includes selecting a language (e.g., Japanese or English) and an avatar. This information is sent from the device to the server and stored in the server's database.

[0691] The server generates an individualized learning plan based on the user's age, grade level, and learning goal data. This learning plan may include, for example, basic math problems or basic science concepts. The generated learning plan is sent to the device and displayed to the user.

[0692] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content. The device displays this material to the user, and learning begins.

[0693] If a user encounters a problem during their study, they can take a photo of the problem using their device's camera. This photo is then sent from the device to a server, which analyzes it using an image analysis algorithm. Based on the analysis results, a generative AI model then generates an explanation for the problem and provides it to the user in text or audio.

[0694] Meanwhile, factory engineers can also use the system. Engineers create an account and perform initial setup. The server generates individual operating instructions based on the engineer's skill level and past work history. If a problem occurs during operation, the engineer takes a photo or video with their device and sends it to the server. The server analyzes the image, and the generative AI model generates the cause of the problem and a solution.

[0695] For example, if an engineer encounters an error while operating a factory robot, they can use a smartphone app to take a photo of the problem and send it to the AI ​​via the app. The AI ​​will then analyze the image and provide the engineer with the cause and solution of the problem in real time. This process enables effective operation guidance and problem-solving support.

[0696] Servers and terminals use software such as Python, OpenCV (image processing), TensorFlow (AI model), and SQLite (database management), and data processing and calculation are carried out through image processing and AI analysis.

[0697] Specific prompt examples:

[0698] "An error occurred while operating the factory robot. Please analyze the picture below and tell me the cause and how to solve it."

[0699] (upload photo here)

[0700] Through this system, users can receive individual learning support, and engineers can receive effective operation instruction and problem-solving support.

[0701] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0702] Step 1:

[0703] The user uses a device to create an account and perform initial settings. Input data includes the user's name, email address, password, language, and avatar information. The device sends this information to the server, which stores it in a database. Specifically, the user enters the required information on the device's registration screen and presses the send button.

[0704] Step 2:

[0705] The server generates an individualized learning plan based on the user's age, grade level, and learning goal data. The input data is the user's basic information, and the output is an individualized learning plan. To generate this, the server refers to past learning data and standard curricula to determine the optimal learning content for the user. Specifically, an algorithm within the server analyzes the user's information and generates a plan.

[0706] Step 3:

[0707] The user presses the start button on the device to begin learning. The input data is the user's start request, and the output is the display of learning materials (text, video, VR content, etc.) corresponding to the day's learning content. The device sends a request to the server to display this, and the server provides learning materials based on the day's plan. In concrete terms, the user presses the start button and the learning materials are displayed on the device.

[0708] Step 4:

[0709] If a user encounters a problem during learning, they use the device's camera to take a photo of the problem. The input data is the photo, and the output is an explanation or solution for the problem. The device sends the photo to a server, which then analyzes the problem using an image analysis algorithm (such as OpenCV). Specifically, the user takes a photo and uploads it from the device to the server.

[0710] Step 5:

[0711] The server uses the generative AI model to generate an explanation for the problem based on the image analysis results. The input data is the image analysis results, and the output is the generated explanation. The server provides this to the user in text or audio format. Specifically, the AI ​​model generates an explanation based on the analysis results, and sends it from the server to the device in text or audio format.

[0712] Step 6:

[0713] When the learning is completed, the device sends learning progress information to the server. The input data is the learning progress information, and the output is the adjusted learning plan for the next day. The server analyzes the received progress data and adjusts the learning plan for the next day. Specifically, the device sends the learning progress to the server, and the server generates a newly adjusted learning plan.

[0714] Step 7:

[0715] Engineers working in factories also use terminals to create accounts and perform initial setup. The input data is the engineer's name, skill level, and past work history, and the output is an individual operation training plan. The terminal sends this to the server, which then stores the engineer's information in a database. Specifically, the engineer enters information on the terminal's registration screen and presses the send button.

[0716] Step 8:

[0717] If a problem occurs during operation, the technician takes a photo or video with the device and sends it to the server. The input data is the photo or video, and the output is the cause of the problem and a solution. The server performs image analysis, and a generative AI model generates the cause of the problem and a solution. Specifically, the technician takes a photo or video and uploads it from the device to the server.

[0718] Step 9:

[0719] The server displays the generated explanations and instruction content to the engineer based on the analysis results. The input data is the analysis results, and the output is the generated explanations and instruction content. The server provides this to the engineer in text or audio format. Specifically, the generative AI model generates explanations based on the analysis results, and sends them from the server to the terminal in text or audio format.

[0720] Step 10:

[0721] When the operation is completed, the terminal sends operation progress information to the server. The input data is the operation progress information, and the output is the adjustment result of the next operation instruction content. The server analyzes the received progress data and adjusts the next operation instruction content. Specifically, the terminal sends the operation progress to the server, and the server generates newly adjusted instruction content.

[0722] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0723] The present invention is a tutoring system that uses AI to support users' learning. In particular, it aims to improve learning efficiency and motivation by incorporating an emotion engine that recognizes the user's emotions. This system functions through the interaction between a server, a terminal, and the user. A specific embodiment of this system will be described below.

[0724] User registration and initial settings

[0725] First, a user creates an account using a device (smartphone, tablet, PC, etc.). When creating an account, basic information such as name, email address, and password is entered and submitted, which is then sent to the server. The server stores the received information in a database and creates the account.

[0726] Next, the user selects the language and avatar as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This information is also sent from the device to the server and stored in the database.

[0727] Generate a lesson plan

[0728] The server retrieves information about the user's age, grade level, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[0729] Start learning

[0730] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content and sends them to the device. For example, for the first math class, learning materials for learning number concepts are provided using VR. The device displays this learning material to the user, and learning begins.

[0731] Questions and Explanations

[0732] If a user comes across a question during their study, they can take a photo of the problem with their smartphone. The photo is then sent to a server via their device. The server then uses an image analysis algorithm to analyze the photo and determine the nature of the question. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via their device.

[0733] Track your progress and adjust your study plan

[0734] Once the learning is complete, the device sends a progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[0735] Incorporating an emotion engine

[0736] The system also includes an emotion engine that recognizes the user's emotions. The emotion engine uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. For example, if the user loses concentration or feels stressed while studying, the emotion engine will detect this situation.

[0737] Dynamic adjustment based on emotions

[0738] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data: for example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material, or it can speed up the progress if positive emotions are recognized.

[0739] Modifying materials based on emotions

[0740] Based on the emotions recognized by the emotion engine, the server can also change the content of the displayed learning materials and explanations. For example, if the user is having fun, the server can display learning materials incorporating game elements to further stimulate interest. In this way, the server can provide an optimal learning environment according to the user's emotional state.

[0741] Specific examples

[0742] For example, if 8-year-old User A feels tired, the emotion engine will recognize this and the server will change the learning materials to make them more relaxing. Also, if User A feels stressed during a particular unit, the server will check the user's progress and adjust the difficulty of that unit to be less difficult. This makes the user's learning experience more effective and enjoyable.

[0743] This invention enables a tutoring system that utilizes AI to respond flexibly to the user's emotional state, maximizing learning efficiency and motivation.

[0744] The processing flow will be explained below.

[0745] Step 1:

[0746] The user launches the application and is directed to the account creation screen.

[0747] Step 2:

[0748] The device prompts the user to enter the required information (name, email address, password, etc.).

[0749] Step 3:

[0750] The user enters the information and presses the submit button.

[0751] Step 4:

[0752] The device sends the entered information to the server.

[0753] Step 5:

[0754] The server stores the received information in a database and creates an account.

[0755] Step 6:

[0756] The server sends the user an initial setup page.

[0757] Step 7:

[0758] The device will display an initial setup page, prompting the user to select a language and avatar.

[0759] Step 8:

[0760] Users can choose their preferred language and avatar.

[0761] Step 9:

[0762] The device sends the user's configuration information to the server.

[0763] Step 10:

[0764] The server stores the received configuration information in a database.

[0765] Step 11:

[0766] The server retrieves information about the user's age, grade level, and learning goals from a database.

[0767] Step 12:

[0768] The server generates an individualized learning plan based on the information it obtains.

[0769] Step 13:

[0770] The server generates a learning plan and sends it to the device.

[0771] Step 14:

[0772] The device displays the learning plan to the user.

[0773] Step 15:

[0774] The user presses the start button to begin learning.

[0775] Step 16:

[0776] The device sends a start request to the server.

[0777] Step 17:

[0778] The server prepares the day's learning content based on the user's request.

[0779] Step 18:

[0780] The server sends the corresponding educational materials (text, videos, VR content, etc.) to the device.

[0781] Step 19:

[0782] The device displays the teaching materials to the user.

[0783] Step 20:

[0784] As users continue their studies, if they come across something they don't understand, they can take a photo of it with their smartphone.

[0785] Step 21:

[0786] The device takes a photo and sends it to the server.

[0787] Step 22:

[0788] The server then uses an image analysis algorithm to analyze the photos it receives and determine the nature of the problem.

[0789] Step 23:

[0790] The server generates an explanation for the problem and creates explanatory content in text or audio.

[0791] Step 24:

[0792] The server sends the generated commentary to the device.

[0793] Step 25:

[0794] The device will display the explanation to the user.

[0795] Step 26:

[0796] Users continue to learn.

[0797] Step 27:

[0798] The device sends progress information to the server at the end of the learning process.

[0799] Step 28:

[0800] The server stores the received progress data in a database.

[0801] Step 29:

[0802] The server adjusts the next day's study plan based on the progress data.

[0803] Step 30:

[0804] The system also includes an emotion engine that recognizes the user's emotions. During training, the device sends the user's facial expressions and voice to the emotion engine.

[0805] Step 31:

[0806] The emotion engine uses facial recognition and voice analysis technology to analyze the user's emotions.

[0807] Step 32:

[0808] The emotion engine sends the analysis results to the server.

[0809] Step 33:

[0810] The server dynamically adjusts according to the user's emotions based on data from the emotion engine.

[0811] Step 34:

[0812] For example, if a user is feeling stressed, the server can reduce the difficulty of the study plan or switch to more engaging material.

[0813] Step 35:

[0814] The server changes the content of the teaching materials and explanations displayed based on the data from the emotion engine.

[0815] Step 36:

[0816] For example, if the user is having fun, the server will display educational materials that incorporate game elements on the device.

[0817] Step 37:

[0818] The device displays tailored learning materials and explanations to the user.

[0819] Step 38:

[0820] After the learning session ends, the device sends the emotion engine analysis data and learning progress data to the server.

[0821] Step 39:

[0822] The server stores this data in a database and adjusts the next day's study plan accordingly.

[0823] Step 40:

[0824] The server sends the new study plan to the device, which then notifies the user.

[0825] Step 41:

[0826] Users can review their new study plan and prepare for the next day's study.

[0827] Example 2

[0828] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0829] Conventional learning support systems have difficulty in fully responding to the individual learning needs of users, limiting their ability to improve learning efficiency and motivation. Furthermore, because they are unable to dynamically adjust learning plans or change learning materials taking into account the user's emotional state, they provide a uniform learning process, resulting in a suboptimal learning experience.

[0830] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0831] In this invention, the server includes means for a user to create an account and select a language and avatar in the initial settings, means for generating an individual study plan based on the user's age, grade, and study goals, means for preparing and displaying study materials in response to a user's request at the start of study, means for analyzing photos of unclear points taken by the user and generating explanations, means for displaying the generated explanations and study materials to the user, means for recording the user's study progress and adjusting the study plan for the next day, means for analyzing the user's facial expressions and tone of voice and dynamically adjusting the study plan based on the user's emotional state, and means for changing the content of the study materials and explanations to be displayed based on the user's emotional state. This makes it possible to provide a flexible study process that meets the individual learning needs of users and corresponds to their emotional state.

[0832] "User" refers to a person who uses the tutoring system to study.

[0833] "Account" refers to a user profile that contains authentication information for a user to access and use the System.

[0834] "Initial settings" refers to basic setting operations such as language selection and avatar settings that users perform before using the system.

[0835] "Language" refers to the interface language that the user uses within the system.

[0836] "Avatar" refers to a character or icon that a user selects within a virtual environment.

[0837] A "study plan" refers to a series of study schedules and content created based on a user's age, grade level, and learning goals.

[0838] "Teaching materials" refers to educational materials such as textbooks, videos, and VR content used by users to study.

[0839] "Explanation" refers to text or audio materials that explain questions and learning content in a way that is easy for users to understand.

[0840] "Study progress" refers to the progress and learning status achieved by the user in the course of their studies.

[0841] An "emotion engine" refers to technology that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[0842] "Database" refers to a system for centrally managing and storing data such as user settings and study plans.

[0843] "Generative AI model" refers to artificial intelligence technology for automatically generating study plans based on user information.

[0844] A "prompt sentence" refers to an input sentence that gives specific instructions to a generative AI model.

[0845] MODE FOR CARRYING OUT THE INVENTION

[0846] This invention relates to a tutoring system that uses AI to support users' learning. In particular, it aims to improve learning efficiency and motivation by incorporating an emotion engine that recognizes users' emotions. This system functions through the interaction of a server, a terminal, and a user.

[0847] User registration and initial settings

[0848] First, a user creates an account using a device such as a smartphone, tablet, or PC. When creating an account, basic information such as name, email address, and password is entered and submitted, which is then sent to the server. The server stores the received information in a database and creates the account.

[0849] Next, the user selects the language and avatar as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This information is also sent from the device to the server and stored in the database.

[0850] Generate a lesson plan

[0851] The server retrieves information about the user's age, grade level, and learning goals from the database and uses a generative AI model to generate a personalized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[0852] Start learning

[0853] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content and sends them to the device. For example, for the first math class, learning materials for learning number concepts are provided using VR. The device displays this learning material to the user, and learning begins.

[0854] Questions and Explanations

[0855] If a user comes across a question during their study, they can take a photo of the problem with their smartphone. The photo is then sent to a server via their device. The server then uses an image analysis algorithm to analyze the photo and determine the nature of the question. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via their device.

[0856] Track your progress and adjust your study plan

[0857] Once the learning is complete, the device sends a progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[0858] Incorporating an emotion engine

[0859] The system also includes an emotion engine that recognizes the user's emotions. The emotion engine uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. For example, if the user loses concentration or feels stressed while studying, the emotion engine will detect this situation.

[0860] Dynamic adjustment based on emotions

[0861] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data: for example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material, or it can speed up the progress if positive emotions are recognized.

[0862] Modifying materials based on emotions

[0863] Based on the emotions recognized by the emotion engine, the server can also change the content of the displayed learning materials and explanations. For example, if the user is having fun, the server can display learning materials incorporating game elements to further stimulate interest. In this way, the server can provide an optimal learning environment according to the user's emotional state.

[0864] Specific examples

[0865] For example, if 8-year-old User A feels tired, the emotion engine will recognize this and the server will change the learning materials to make them more relaxing. Also, if User A feels stressed during a particular unit, the server will check the user's progress and adjust the difficulty of that unit to be less difficult. This makes the user's learning experience more effective and enjoyable.

[0866] Sample prompt sentence

[0867] Example user profile:

[0868] Age: 8

[0869] Grade: 3rd grade

[0870] Learning Objectives: Learning basic mathematical operations and basic science concepts

[0871] Prompt sentence to input to the generative AI model:

[0872] "Create a daily lesson plan for third graders that includes basic math operations (e.g., addition, subtraction) and basic science concepts (e.g., familiar natural phenomena)."

[0873] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0874] Step 1:

[0875] A user creates an account using a terminal. As input, the user enters basic information such as name, email address, and password into the terminal and presses the "Submit" button. The terminal sends this information to the server. The output is that the user is presented with a message confirming successful account registration.

[0876] Step 2:

[0877] The server stores the received user information in a database. The input is user information sent from the terminal, such as name, email address, and password. The database adds this information as a new record. The output is that a new user record is created and saved in the database.

[0878] Step 3:

[0879] The user uses the device to perform initial setup. As input, the user selects the desired language and avatar and saves the settings. The device sends this information to the server. The output is that the initial setup information is sent to the server and stored in a database.

[0880] Step 4:

[0881] The server retrieves information about the user's age, grade, and learning goals from the database. The input is the user ID. The server uses the user ID as a key to retrieve related information from the database. The output is the user's age, grade, and learning goals.

[0882] Step 5:

[0883] The server uses the generative AI model to create an individualized learning plan. The input is the information about the user's age, grade, and learning goals obtained in step 4. The generative AI model creates prompt sentences based on this information and automatically generates a learning plan. The output is the generation of an individualized learning plan.

[0884] Step 6:

[0885] The server sends the generated learning plan to the device. The input is the learning plan output from the generative AI model. The server sends this to the user's device. The output is the learning plan displayed on the device.

[0886] Step 7:

[0887] The user presses the start button on the terminal to start learning. The input is the user's operation. The terminal notifies the server of this operation. The output is that a request to start learning is sent to the server.

[0888] Step 8:

[0889] The server prepares learning materials corresponding to the day's learning content. The input is a request to start learning. The server selects appropriate learning materials from the database and sends them to the terminal. The output is the learning materials displayed on the terminal.

[0890] Step 9:

[0891] If a user has a question while studying, they can use the camera to take a photo of the question. The input is the visual information of the question. The user takes a photo using the camera and saves it on the device. The output is the photographed photo data.

[0892] Step 10:

[0893] The device sends the captured photo data to the server. The input is a photo taken by the user. The device uploads this photo to the server. The output is the photo data stored on the server.

[0894] Step 11:

[0895] The server analyzes the photo using an image analysis algorithm. The input is the photo data sent from the device. The server analyzes the image and identifies the problem content. The output is the analyzed problem content.

[0896] Step 12:

[0897] The server generates an explanation based on the analysis results. The input is the problem content obtained by image analysis. The server creates an explanation using the explanation generation model. The output is the generated explanation.

[0898] Step 13:

[0899] The server sends the generated comment to the terminal. The input is the generated comment. The server sends this information to the terminal. The output is the comment displayed on the terminal.

[0900] Step 14:

[0901] The terminal displays the explanatory text to the user. The input is the explanatory text sent by the server. The terminal displays it in an appropriate format for the user. The output is the explanatory text that is presented to the user.

[0902] Step 15:

[0903] When the learning is completed, the terminal sends the learning progress data to the server. The input is the user's learning progress information. The terminal uploads it to the server. The output is the learning progress data stored on the server.

[0904] Step 16:

[0905] The server stores the received learning progress data in a database. The input is the progress data sent from the terminal. The server stores this in a database. The output is the learning progress data recorded in the database.

[0906] Step 17:

[0907] The server adjusts the learning plan for the next day. The input is the saved learning progress data. The server creates a new learning plan using a generative AI model based on the progress data. The output is the adjusted learning plan.

[0908] Step 18:

[0909] The server sends the new lesson plan to the device. The input is the adjusted lesson plan. The server sends this to the device. The output is the new lesson plan displayed on the device.

[0910] Step 19:

[0911] The device analyzes the user's facial expressions and tone of voice using an emotion engine. The input is the user's facial expressions and voice information. The device captures this data with a camera and microphone and analyzes it with the emotion engine. The output is the analyzed emotion data.

[0912] Step 20:

[0913] The emotion engine sends the analysis results to the server. The input is the analyzed emotion data. The emotion engine sends this data to the server. The output is the emotion data stored on the server.

[0914] Step 21:

[0915] The server dynamically adjusts the learning plan based on the analysis results. The input is the emotion data sent from the emotion engine. The server dynamically changes the learning plan using a generative AI model based on this. The output is the adjusted learning plan.

[0916] Step 22:

[0917] The server sends the adjusted lesson plan to the device. The input is the new lesson plan. The server sends it to the device. The output is the adjusted lesson plan displayed on the user's device.

[0918] Step 23:

[0919] The server changes the content of the learning materials and explanations displayed based on the results of the emotion engine. The input is the adjusted learning plan and emotional data. The server takes the emotional state into consideration and selects the most appropriate learning materials and explanations, which it then sends to the device. The output is the updated learning materials and explanations that are displayed to the user.

[0920] (Application example 2)

[0921] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0922] Conventional learning support systems provide fixed learning plans without considering the user's emotional state, making it difficult to maximize the user's learning efficiency and motivation. Furthermore, when the user is stressed or fatigued, they do not provide appropriate support, which reduces the effectiveness of learning.

[0923] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to create an account and select a language and avatar in the initial settings, means for generating an individual study plan based on the user's age, grade, and learning goals, means for analyzing photos of unclear parts taken by the user and generating explanations, means for displaying the generated explanations and learning materials to the user, means for recording the user's study progress and adjusting the study plan for the next day, means for recognizing the user's emotions using facial recognition technology and voice analysis technology, means for dynamically adjusting the study content based on the recognized emotional data, and means for providing feedback according to the user's emotional state. This makes it possible to maximize the user's learning efficiency and motivation and flexibly adjust the study plan according to the user's individual emotional state.

[0924] The "user account creation means" is a means for a user to create an account for using the system and select a language and avatar as initial settings.

[0925] A "study plan generator" is a means for generating an individualized study plan based on the user's age, grade level, and learning goals.

[0926] The "photo analysis means" is a means for analyzing a photo of an unknown location taken by a user and generating an explanation based on the analysis.

[0927] The "teaching material display means" is a means for displaying the generated explanations and teaching materials to the user.

[0928] The "progress recording means" is a means for recording the user's learning progress and adjusting the next day's learning plan based on that progress.

[0929] The "emotion recognition means" is a means for recognizing the user's emotions using face recognition technology or voice analysis technology.

[0930] The "dynamic adjustment means" is a means for dynamically adjusting the learning content based on the recognized emotion data.

[0931] A "feedback providing means" is a means for providing feedback to a user according to their emotional state.

[0932] The "database storage means" is a means for storing user setting information, study plans, and emotional data in a database.

[0933] The "audio playback means" is a means for playing back audio of the generated commentary or teaching material.

[0934] This invention is a tutoring system that uses AI to support users' learning, and in particular, improves learning efficiency and motivation by incorporating an emotion engine that recognizes users' emotions. This system functions through the interaction of a server, terminals, and users.

[0935] Specifically, this is carried out as follows.

[0936] User account creation method

[0937] The server provides a means for users to create an account using a device (smartphone, tablet, PC, etc.). When creating an account, the user enters basic information such as name, email address, and password and sends it to the server. The server stores the received information in a database and creates the account. The user then selects the language and avatar as initial settings. For example, the user might select Japanese and set a popular anime character as their avatar.

[0938] Learning plan generation tool

[0939] The server retrieves information about the user's age, grade, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[0940] Photo analysis tools

[0941] If a user encounters a problem while studying, they can use their device to take a photo of the problem. The photo is then sent to the server via the device. The server then uses an image analysis algorithm to analyze the photo and determine the problem's content. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via the device.

[0942] emotion recognition means

[0943] The system also includes an emotion engine that recognizes the user's emotions. The server uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. For example, if the user loses concentration or feels stressed while studying, the emotion engine will detect this.

[0944] Dynamic Adjustment Means

[0945] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data: for example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material, or it can speed up the progress if positive emotions are recognized.

[0946] Feedback methods

[0947] Once the learning is complete, the device sends a progress report to the server. The server stores the progress data in a database and adjusts the learning plan for the next day. Feedback can be provided based on the user's emotional state, creating an appropriate learning environment. For example, if the user has a negative reaction to a particular problem, the device can skip the next time the learning is completed or approach it differently.

[0948] Specific examples

[0949] For example, let's say a new employee at Store A is undergoing training using smart glasses. If the employee shows signs of fatigue, the emotion engine will detect this and display a notification on their smart device suggesting a break. Also, if the employee is performing a task with interest, a more challenging task will automatically be suggested as the next step.

[0950] Example prompts for generative AI models

[0951] Describe the process flow for real-time emotion recognition when a new employee is participating in training using smart glasses. Show how the system analyzes the employee's emotional state from their facial expressions and tone of voice, and dynamically adjusts the training accordingly.

[0952] The present invention enables flexible adjustment of study plans and feedback according to the user's emotional state, which is expected to improve study efficiency and motivation.

[0953] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0954] Step 1:

[0955] A user creates an account using a device (smartphone, tablet, smart glasses, etc.). The user enters basic information such as name, email address, and password, and sends that information to the server. The server stores the received information in a database and creates an account. In this process, the user's basic information is given as input, and an account is generated as output.

[0956] Step 2:

[0957] The user selects the language and avatar as the initial settings. For example, they may select Japanese and set a popular anime character as their avatar. This setting information is also sent from the device to the server and stored in the database. In this process, the selected language and avatar information are given as input, and the saved setting information is obtained as output.

[0958] Step 3:

[0959] The server retrieves information about the user's age, grade, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user will be scheduled to learn basic math problems and basic science concepts. This generated learning plan is sent to the device and displayed to the user. In this process, the user's profile information is given as input and an individualized learning plan is generated as output.

[0960] Step 4:

[0961] The user begins studying on their device. If they come across a question they don't understand, they take a photo of the question with their smart device. The photo is sent to the server via the device. The server analyzes the photo using an image analysis algorithm to determine the question's content. Based on the results of this analysis, an explanation is generated and the content is provided as text or audio. In this process, the photo of the question is given as input, and the generated explanation is provided as output.

[0962] Step 5:

[0963] Learning progress is recorded, and when learning is completed, the device sends a learning progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. Based on the progress data, a new learning plan is generated and sent to the device, and the user is notified. In this process, learning progress data is given as input, and the next day's learning plan is adjusted as output.

[0964] Step 6:

[0965] Furthermore, the system is equipped with an emotion engine that recognizes the user's emotions. The server uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. The user's emotional state is confirmed during the learning process, and the data is updated as necessary. During this process, the user's facial expression images and voice data are given as input, and the analyzed emotional state is output.

[0966] Step 7:

[0967] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data. For example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material. It can also speed up the learning process if positive emotions are recognized. This process takes emotion data as input and generates an adjusted learning plan as output.

[0968] Step 8:

[0969] The device provides feedback based on the user's emotional state. For example, if the user has a negative reaction to a particular problem, feedback is provided to change the user's approach to that problem in the next learning session. In this process, emotional state and learning progress are given as inputs, and feedback is provided as output.

[0970] This makes it possible to maximize the user's learning efficiency and motivation.

[0971] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0972] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0973] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0974] [Third embodiment]

[0975] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0976] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0977] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0978] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0979] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0980] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0981] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0982] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0983] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0984] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0985] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0986] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0987] The present invention is a tutoring system that uses AI to help users effectively advance their learning. This system functions through the interaction of a server, a terminal, and a user. A specific embodiment of this system will be described below.

[0988] User registration and initial settings

[0989] First, the user creates an account using a device (smartphone, tablet, PC, etc.). When creating an account, they enter basic information such as their name, email address, and password. The server then stores the user's information in a database and creates the account.

[0990] Next, the user selects the language and avatar as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This information is also sent from the device to the server and stored in the database.

[0991] Generate a lesson plan

[0992] The server retrieves information about the user's age, grade level, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[0993] Start learning

[0994] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, videos, VR content, etc.) corresponding to the day's learning content. For example, for the first math class, learning materials for learning number concepts are provided using VR. The device displays this learning material to the user, and learning begins.

[0995] Questions and Explanations

[0996] If a user comes across a question during their study, they can take a photo of the problem with their smartphone. The photo is then sent to a server via their device. The server then uses an image analysis algorithm to analyze the photo and determine the nature of the question. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via their device.

[0997] Track your progress and adjust your study plan

[0998] Once the learning is complete, the device sends a progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[0999] Specific examples

[1000] For example, when 8-year-old User A uses the system for the first time, he or she creates an account, selects Japanese as the initial language, and sets a favorite avatar. After that, math and science teaching materials are displayed on the device according to the learning plan sent from the server. User A begins learning number concepts in VR, and if there is anything they don't understand, they take a photo and send it to the server. The server analyzes the information and returns an explanation, allowing User A to deepen their understanding. After completing the learning, progress information is sent to the server, and the next day's learning plan is fine-tuned.

[1001] In this way, the AI-based tutoring system can provide high-quality education efficiently and effectively, without being affected by financial constraints or family circumstances.

[1002] The processing flow will be explained below.

[1003] Step 1:

[1004] The user launches the application and is directed to the account creation screen.

[1005] Step 2:

[1006] The device prompts the user to enter the required information (name, email address, password, etc.).

[1007] Step 3:

[1008] The user enters the information and presses the submit button.

[1009] Step 4:

[1010] The device sends the entered information to the server.

[1011] Step 5:

[1012] The server stores the received information in a database and creates an account.

[1013] Step 6:

[1014] The server sends the user an initial setup page.

[1015] Step 7:

[1016] The device will display an initial setup page, prompting the user to select a language and avatar.

[1017] Step 8:

[1018] Users can choose their preferred language and avatar.

[1019] Step 9:

[1020] The device sends the user's configuration information to the server.

[1021] Step 10:

[1022] The server stores the received configuration information in a database.

[1023] Step 11:

[1024] The server retrieves information about the user's age, grade level, and learning goals from a database.

[1025] Step 12:

[1026] The server generates an individualized learning plan based on the information it obtains.

[1027] Step 13:

[1028] The server generates a learning plan and sends it to the device.

[1029] Step 14:

[1030] The device displays the learning plan to the user.

[1031] Step 15:

[1032] The user presses the start button to begin learning.

[1033] Step 16:

[1034] The device sends a start request to the server.

[1035] Step 17:

[1036] The server prepares the day's learning content based on the user's request.

[1037] Step 18:

[1038] The server sends the corresponding educational materials (text, videos, VR content, etc.) to the device.

[1039] Step 19:

[1040] The device displays the teaching materials to the user.

[1041] Step 20:

[1042] As users continue their studies, if they come across something they don't understand, they can take a photo of it with their smartphone.

[1043] Step 21:

[1044] The device takes a photo and sends it to the server.

[1045] Step 22:

[1046] The server then uses an image analysis algorithm to analyze the photos it receives and determine the nature of the problem.

[1047] Step 23:

[1048] The server generates an explanation for the problem and creates explanatory content in text or audio.

[1049] Step 24:

[1050] The server sends the generated commentary to the device.

[1051] Step 25:

[1052] The device will display the explanation to the user.

[1053] Step 26:

[1054] Users continue to learn.

[1055] Step 27:

[1056] The device sends progress information to the server at the end of the learning process.

[1057] Step 28:

[1058] The server stores the received progress data in a database.

[1059] Step 29:

[1060] The server adjusts the next day's study plan based on the progress data.

[1061] Step 30:

[1062] The server generates a new learning plan and sends it to the device.

[1063] Step 31:

[1064] The device will notify the user of the new study plan.

[1065] Step 32:

[1066] The user can then review the new plan and prepare for the next day's study.

[1067] Example 1

[1068] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1069] Conventional learning systems lack the ability to respond to individual learning needs, making it difficult to maximize users' learning effectiveness. Furthermore, they lack the means to provide quick and accurate explanations when users encounter questions during their studies. Furthermore, they do not dynamically adjust learning plans according to learning progress, making it difficult to continuously improve learning.

[1070] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1071] In this invention, the server includes: a means for a user to create an account and select a language and avatar in the initial settings; a means for generating an individualized learning plan based on the user's age, grade, and learning goals; a means for analyzing images of unclear points taken by the user and generating explanations; a means for displaying the generated explanations and learning materials to the user; and a means for recording the user's learning progress and adjusting the learning plan for the next day. This enables effective learning support tailored to individual learning needs. Furthermore, by enabling users to quickly resolve unclear points, they can deepen their understanding of the learning. Furthermore, by dynamically adjusting the learning plan according to their learning progress, continuous improvement in learning is possible.

[1072] "User" refers to an individual who uses the system.

[1073] "Account" refers to information for user authentication, including personal information required for a user to access and use the system.

[1074] "Initial settings" refers to the language, avatar, and other settings that a user selects when they first start using the system.

[1075] "Language" refers to the language used by the user to display and operate the system.

[1076] "Avatar" refers to the character or image a user chooses to represent themselves on the system.

[1077] A "study plan" refers to a schedule of study content for a specific period of time that is generated based on the user's age, grade level, and learning goals.

[1078] "Image analysis" refers to the process of analyzing an image taken by a user and understanding its contents.

[1079] "Explanation" refers to content that explains the problem or issue in a way that is easy for users to understand.

[1080] "Learning materials" refers to materials such as texts, videos, and VR content provided for users to study.

[1081] "Progress" refers to the portion of the learning experience that the user has achieved and the current situation.

[1082] "Adjustment" refers to the process of changing and updating the learning plan according to the user's learning progress.

[1083] A "server" refers to a computer system that receives a user request, processes it, and returns a response.

[1084] "Terminal" refers to a device such as a smartphone, tablet, or PC that a user uses to access and operate the system.

[1085] "Database" refers to a system for storing and managing user information, study plans, etc.

[1086] A "generative AI model" refers to an algorithm or system that uses machine learning and artificial intelligence techniques to generate personalized learning plans and explanations for users.

[1087] A "prompt" refers to text that is an instruction or question entered into a generative AI model.

[1088] This invention is a tutoring system that uses AI to support users in effectively progressing with their studies. This system functions through the interaction between a server, a terminal, and a user.

[1089] User registration and initial settings

[1090] First, a user creates an account using a device such as a smartphone, tablet, or PC. When creating an account, they enter basic information such as their name, email address, and password. The entered information is sent from the device to a server, which then stores the information in a database and creates the account.

[1091] Next, the user selects the language and avatar to use as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This setting information is also sent from the device to the server and stored in the database.

[1092] Generate a lesson plan

[1093] The server retrieves information about the user's age, grade level, and learning goals from a database. It then uses a generative AI model to generate a personalized learning plan for the user. The generative AI model is an algorithm that uses machine learning and artificial intelligence techniques to derive appropriate learning content based on specific prompts. For example, an 8-year-old user might be scheduled to learn basic math problems and fundamental science concepts.

[1094] The generated lesson plan is sent to the terminal and displayed to the user.

[1095] Start learning

[1096] The user presses the start button on the device to begin learning. This request is sent from the device to the server. The server prepares the learning materials (text, videos, VR content, etc.) corresponding to the learning content for that day. For example, for the first math class, materials for learning number concepts are provided using VR. The prepared materials are sent to the device, which then displays them to the user.

[1097] Questions and Explanations

[1098] If a user encounters a problem during learning, they can take a photo of the problem with their smartphone. The image is then sent to the server via the device. The server then analyzes the photo using an image analysis algorithm (e.g., OpenCV, TensorFlow) to determine the problem.

[1099] Based on the analysis results, the server uses a generative AI model to generate an explanation. The generated explanation is provided in text or audio format and displayed to the user through the device. For example, it provides a step-by-step solution to a difficult math problem. Examples of prompts used in this process include:

[1100] "Analyze user-taken photos of math problems and provide step-by-step solutions."

[1101] Track your progress and adjust your study plan

[1102] Once the learning is complete, the device sends a progress report to the server. The server stores the progress data in a database and uses the generative AI model to adjust the next day's learning plan. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[1103] In this way, the present invention provides an AI-based tutoring system that provides effective education tailored to individual learning needs.

[1104] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1105] Step 1:

[1106] A user creates an account using a device such as a smartphone, tablet, or PC. First, the user enters basic information such as their name, email address, and password, and submits it. The device then sends the entered information to the server. The server stores the received information in a database and creates the user's account.

[1107] (Input) User name, email address, and password.

[1108] (Output) The user account is saved in the database.

[1109] (Specific operation) When a user enters information into an input form on a terminal and presses the send button, the information is sent to the server as an HTTP request, and the server processes the information to save it in a database.

[1110] Step 2:

[1111] Next, the user selects the language and avatar to use as the default settings. The information selected by the user is then sent from the device to the server and stored in a database.

[1112] (Input) Selected language, avatar.

[1113] (Output) Language and avatar settings are saved in the database.

[1114] (Specific operation) The user selects a language and avatar using drop-down menus or radio buttons, and then presses the Settings button. The setting information is sent to the server as an HTTP request, and the information is saved in a database on the server side.

[1115] Step 3:

[1116] The server retrieves information about the user's age, grade level, and learning goals from a database and uses a generative AI model to generate an individualized lesson plan. For example, an 8-year-old user would receive a lesson plan that includes basic math and science content. The generated lesson plan is then sent to the device and displayed to the user.

[1117] (Input) User's age, grade level, and learning goals.

[1118] (Output) Individualized learning plan.

[1119] (Specific operation) The server executes a database query to obtain user information, and generates a learning plan using prompts for the generative AI model. The generated learning plan is sent to the device as an HTTP response, and the device displays it to the user.

[1120] Step 4:

[1121] The user presses the start button on the device to begin learning. This request is sent from the device to the server. The server prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content, and the prepared materials are sent to the device and displayed to the user.

[1122] (Input) Press the start button.

[1123] (Output) Learning materials.

[1124] (Specific operation) When the user presses the start button, an HTTP request is sent to the server. The server selects learning materials according to the learning content and sends them to the terminal as an HTTP response. The terminal displays the received learning material data.

[1125] Step 5:

[1126] When a user encounters a problem during learning, they can take a photo of the problem with their smartphone and send it to the server via their device. The server then uses an image analysis algorithm to analyze the photo and generates an explanation using a generative AI model. The explanation is then displayed to the user via their device in text or audio format.

[1127] (Input) The captured image of the problem.

[1128] (Output) Explanatory text or audio.

[1129] (Specific operation) The user takes a photo of the problem with their smartphone, and the device sends it to the server as an HTTP request. The server analyzes the image using an image analysis library and applies a generative AI model to generate an explanation. The explanation data is sent to the device as an HTTP response, and the device displays it.

[1130] Step 6:

[1131] Once the learning is complete, the device sends a progress report to the server. The server stores the progress data in a database and adjusts the next day's learning plan using the generative AI model. The new adjusted learning plan is sent to the device and notified to the user.

[1132] (Input) Learning progress data.

[1133] (Output) The new adjusted lesson plan.

[1134] (Specific operation) The device sends learning progress information to the server via an HTTP request, and the server saves the progress data in a database. The next day's learning plan is readjusted using the generative AI model, and the new plan is sent to the device as an HTTP response to notify the user.

[1135] (Application example 1)

[1136] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1137] Conventional tutoring systems have difficulty meeting the individual needs of learners, making it difficult to provide efficient and effective learning support. Furthermore, providing operational guidance and problem-solving for factory engineers requires individualized support, making it difficult to implement. The objective of this invention is to provide an effective and efficient learning and operational guidance system that allows learners and engineers to receive support based on their individual needs.

[1138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1139] In this invention, the server includes means for a user to create an account and select a language and avatar in the initial settings, means for generating an individualized learning plan based on the user's age, grade, and learning goals, means for analyzing photos of unclear parts taken by the user and generating an explanation, means for displaying the generated explanation and learning materials to the user, means for recording the user's learning progress and adjusting the user's learning plan for the next day, means for a technician to create an account and perform initial settings, means for generating individualized operating instructions based on the technician's skill level and past work history, means for analyzing photos and videos taken by the technician when a problem occurs and generating an explanation, means for displaying the generated explanation and instruction content to the technician, and means for recording operation progress and adjusting the next instruction content. This allows users to receive individualized learning support, and technicians to receive effective operating instructions and problem-solving support.

[1140] "User" refers to a person who creates an account to use the system and receives a learning plan and instruction on how to use it.

[1141] "Create an account" refers to the process by which a user enters their information to register with the system and configure their personal settings.

[1142] "Initial Settings" refers to the basic settings that a user makes when accessing the system, including selecting a language and an avatar.

[1143] "Language" refers to the language used by the user when learning or receiving instruction.

[1144] "Avatar" refers to the character or image a user chooses to represent themselves within the system.

[1145] "Age" refers to the age of the user or technician based on their date of birth.

[1146] "Grade" refers to the educational stage to which a learner currently belongs.

[1147] "Learning goal" refers to the goal regarding the learning content or skills that the user wants to achieve.

[1148] "Individualized Learning Plan" refers to a personalized learning schedule based on a user's age, grade level, and learning goals.

[1149] "Photography" refers to the act of taking photos or videos using the system.

[1150] "Analysis" refers to processing captured images and video, performing calculations and evaluations to identify problems.

[1151] "Explanation" refers to content that provides step-by-step explanations and procedures for solving a problem.

[1152] "Teaching materials" refers to educational materials such as textbooks, videos, and VR content used by users to further their studies.

[1153] "Study progress" refers to data that indicates the achievement status of a user as they progress with their studies.

[1154] "Adjustment" refers to the act of changing and optimizing learning plans and instructional content based on progress data.

[1155] "Technician" refers to someone who uses the system to operate and maintain robots and machinery in a factory or other location.

[1156] "Skill level" refers to an indicator of an engineer's technical proficiency and ability.

[1157] "Operational instruction" refers to instructions and guidelines for engineers to operate robots and machines properly.

[1158] "Photos and videos at the time of the problem" refers to visual data taken by technicians to record malfunctions in machines or robots.

[1159] "Instruction content" refers to the specific work procedures and solutions provided to engineers.

[1160] "Progress" refers to data that shows the progress a technician makes as they carry out operation and maintenance tasks.

[1161] To implement the present invention, the system functions mainly through the interaction of a server, a terminal, and a user. Specifically, the system is configured as follows:

[1162] First, a user creates an account using a device (smartphone, tablet, PC, etc.) and performs initial setup. This initial setup includes selecting a language (e.g., Japanese or English) and an avatar. This information is sent from the device to the server and stored in the server's database.

[1163] The server generates an individualized learning plan based on the user's age, grade level, and learning goal data. This learning plan may include, for example, basic math problems or basic science concepts. The generated learning plan is sent to the device and displayed to the user.

[1164] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content. The device displays this material to the user, and learning begins.

[1165] If a user encounters a problem during their study, they can take a photo of the problem using their device's camera. This photo is then sent from the device to a server, which analyzes it using an image analysis algorithm. Based on the analysis results, a generative AI model then generates an explanation for the problem and provides it to the user in text or audio.

[1166] Meanwhile, factory engineers can also use the system. Engineers create an account and perform initial setup. The server generates individual operating instructions based on the engineer's skill level and past work history. If a problem occurs during operation, the engineer takes a photo or video with their device and sends it to the server. The server analyzes the image, and the generative AI model generates the cause of the problem and a solution.

[1167] For example, if an engineer encounters an error while operating a factory robot, they can use a smartphone app to take a photo of the problem and send it to the AI ​​via the app. The AI ​​will then analyze the image and provide the engineer with the cause and solution of the problem in real time. This process enables effective operation guidance and problem-solving support.

[1168] Servers and terminals use software such as Python, OpenCV (image processing), TensorFlow (AI model), and SQLite (database management), and data processing and calculation are carried out through image processing and AI analysis.

[1169] Specific prompt examples:

[1170] "An error occurred while operating the factory robot. Please analyze the picture below and tell me the cause and how to solve it."

[1171] (upload photo here)

[1172] Through this system, users can receive individual learning support, and engineers can receive effective operation instruction and problem-solving support.

[1173] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1174] Step 1:

[1175] The user uses a device to create an account and perform initial settings. Input data includes the user's name, email address, password, language, and avatar information. The device sends this information to the server, which stores it in a database. Specifically, the user enters the required information on the device's registration screen and presses the send button.

[1176] Step 2:

[1177] The server generates an individualized learning plan based on the user's age, grade level, and learning goal data. The input data is the user's basic information, and the output is an individualized learning plan. To generate this, the server refers to past learning data and standard curricula to determine the optimal learning content for the user. Specifically, an algorithm within the server analyzes the user's information and generates a plan.

[1178] Step 3:

[1179] The user presses the start button on the device to begin learning. The input data is the user's start request, and the output is the display of learning materials (text, video, VR content, etc.) corresponding to the day's learning content. The device sends a request to the server to display this, and the server provides learning materials based on the day's plan. In concrete terms, the user presses the start button and the learning materials are displayed on the device.

[1180] Step 4:

[1181] If a user encounters a problem during learning, they use the device's camera to take a photo of the problem. The input data is the photo, and the output is an explanation or solution for the problem. The device sends the photo to a server, which then analyzes the problem using an image analysis algorithm (such as OpenCV). Specifically, the user takes a photo and uploads it from the device to the server.

[1182] Step 5:

[1183] The server uses the generative AI model to generate an explanation for the problem based on the image analysis results. The input data is the image analysis results, and the output is the generated explanation. The server provides this to the user in text or audio format. Specifically, the AI ​​model generates an explanation based on the analysis results, and sends it from the server to the device in text or audio format.

[1184] Step 6:

[1185] When the learning is completed, the device sends learning progress information to the server. The input data is the learning progress information, and the output is the adjusted learning plan for the next day. The server analyzes the received progress data and adjusts the learning plan for the next day. Specifically, the device sends the learning progress to the server, and the server generates a newly adjusted learning plan.

[1186] Step 7:

[1187] Engineers working in factories also use terminals to create accounts and perform initial setup. The input data is the engineer's name, skill level, and past work history, and the output is an individual operation training plan. The terminal sends this to the server, which then stores the engineer's information in a database. Specifically, the engineer enters information on the terminal's registration screen and presses the send button.

[1188] Step 8:

[1189] If a problem occurs during operation, the technician takes a photo or video with the device and sends it to the server. The input data is the photo or video, and the output is the cause of the problem and a solution. The server performs image analysis, and a generative AI model generates the cause of the problem and a solution. Specifically, the technician takes a photo or video and uploads it from the device to the server.

[1190] Step 9:

[1191] The server displays the generated explanations and instruction content to the engineer based on the analysis results. The input data is the analysis results, and the output is the generated explanations and instruction content. The server provides this to the engineer in text or audio format. Specifically, the generative AI model generates explanations based on the analysis results, and sends them from the server to the terminal in text or audio format.

[1192] Step 10:

[1193] When the operation is completed, the terminal sends operation progress information to the server. The input data is the operation progress information, and the output is the adjustment result of the next operation instruction content. The server analyzes the received progress data and adjusts the next operation instruction content. Specifically, the terminal sends the operation progress to the server, and the server generates newly adjusted instruction content.

[1194] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1195] The present invention is a tutoring system that uses AI to support users' learning. In particular, it aims to improve learning efficiency and motivation by incorporating an emotion engine that recognizes the user's emotions. This system functions through the interaction between a server, a terminal, and the user. A specific embodiment of this system will be described below.

[1196] User registration and initial settings

[1197] First, a user creates an account using a device (smartphone, tablet, PC, etc.). When creating an account, basic information such as name, email address, and password is entered and submitted, which is then sent to the server. The server stores the received information in a database and creates the account.

[1198] Next, the user selects the language and avatar as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This information is also sent from the device to the server and stored in the database.

[1199] Generate a lesson plan

[1200] The server retrieves information about the user's age, grade level, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[1201] Start learning

[1202] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content and sends them to the device. For example, for the first math class, learning materials for learning number concepts are provided using VR. The device displays this learning material to the user, and learning begins.

[1203] Questions and Explanations

[1204] If a user comes across a question during their study, they can take a photo of the problem with their smartphone. The photo is then sent to a server via their device. The server then uses an image analysis algorithm to analyze the photo and determine the nature of the question. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via their device.

[1205] Track your progress and adjust your study plan

[1206] Once the learning is complete, the device sends a progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[1207] Incorporating an emotion engine

[1208] The system also includes an emotion engine that recognizes the user's emotions. The emotion engine uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. For example, if the user loses concentration or feels stressed while studying, the emotion engine will detect this situation.

[1209] Dynamic adjustment based on emotions

[1210] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data: for example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material, or it can speed up the progress if positive emotions are recognized.

[1211] Modifying materials based on emotions

[1212] Based on the emotions recognized by the emotion engine, the server can also change the content of the displayed learning materials and explanations. For example, if the user is having fun, the server can display learning materials incorporating game elements to further stimulate interest. In this way, the server can provide an optimal learning environment according to the user's emotional state.

[1213] Specific examples

[1214] For example, if 8-year-old User A feels tired, the emotion engine will recognize this and the server will change the learning materials to make them more relaxing. Also, if User A feels stressed during a particular unit, the server will check the user's progress and adjust the difficulty of that unit to be less difficult. This makes the user's learning experience more effective and enjoyable.

[1215] This invention enables a tutoring system that utilizes AI to respond flexibly to the user's emotional state, maximizing learning efficiency and motivation.

[1216] The processing flow will be explained below.

[1217] Step 1:

[1218] The user launches the application and is directed to the account creation screen.

[1219] Step 2:

[1220] The device prompts the user to enter the required information (name, email address, password, etc.).

[1221] Step 3:

[1222] The user enters the information and presses the submit button.

[1223] Step 4:

[1224] The device sends the entered information to the server.

[1225] Step 5:

[1226] The server stores the received information in a database and creates an account.

[1227] Step 6:

[1228] The server sends the user an initial setup page.

[1229] Step 7:

[1230] The device will display an initial setup page, prompting the user to select a language and avatar.

[1231] Step 8:

[1232] Users can choose their preferred language and avatar.

[1233] Step 9:

[1234] The device sends the user's configuration information to the server.

[1235] Step 10:

[1236] The server stores the received configuration information in a database.

[1237] Step 11:

[1238] The server retrieves information about the user's age, grade level, and learning goals from a database.

[1239] Step 12:

[1240] The server generates an individualized learning plan based on the information it obtains.

[1241] Step 13:

[1242] The server generates a learning plan and sends it to the device.

[1243] Step 14:

[1244] The device displays the learning plan to the user.

[1245] Step 15:

[1246] The user presses the start button to begin learning.

[1247] Step 16:

[1248] The device sends a start request to the server.

[1249] Step 17:

[1250] The server prepares the day's learning content based on the user's request.

[1251] Step 18:

[1252] The server sends the corresponding educational materials (text, videos, VR content, etc.) to the device.

[1253] Step 19:

[1254] The device displays the teaching materials to the user.

[1255] Step 20:

[1256] As users continue their studies, if they come across something they don't understand, they can take a photo of it with their smartphone.

[1257] Step 21:

[1258] The device takes a photo and sends it to the server.

[1259] Step 22:

[1260] The server then uses an image analysis algorithm to analyze the photos it receives and determine the nature of the problem.

[1261] Step 23:

[1262] The server generates an explanation for the problem and creates explanatory content in text or audio.

[1263] Step 24:

[1264] The server sends the generated commentary to the device.

[1265] Step 25:

[1266] The device will display the explanation to the user.

[1267] Step 26:

[1268] Users continue to learn.

[1269] Step 27:

[1270] The device sends progress information to the server at the end of the learning process.

[1271] Step 28:

[1272] The server stores the received progress data in a database.

[1273] Step 29:

[1274] The server adjusts the next day's study plan based on the progress data.

[1275] Step 30:

[1276] The system also includes an emotion engine that recognizes the user's emotions. During training, the device sends the user's facial expressions and voice to the emotion engine.

[1277] Step 31:

[1278] The emotion engine uses facial recognition and voice analysis technology to analyze the user's emotions.

[1279] Step 32:

[1280] The emotion engine sends the analysis results to the server.

[1281] Step 33:

[1282] The server dynamically adjusts according to the user's emotions based on data from the emotion engine.

[1283] Step 34:

[1284] For example, if a user is feeling stressed, the server can reduce the difficulty of the study plan or switch to more engaging material.

[1285] Step 35:

[1286] The server changes the content of the teaching materials and explanations displayed based on the data from the emotion engine.

[1287] Step 36:

[1288] For example, if the user is having fun, the server will display educational materials that incorporate game elements on the device.

[1289] Step 37:

[1290] The device displays tailored learning materials and explanations to the user.

[1291] Step 38:

[1292] After the learning session ends, the device sends the emotion engine analysis data and learning progress data to the server.

[1293] Step 39:

[1294] The server stores this data in a database and adjusts the next day's study plan accordingly.

[1295] Step 40:

[1296] The server sends the new study plan to the device, which then notifies the user.

[1297] Step 41:

[1298] Users can review their new study plan and prepare for the next day's study.

[1299] Example 2

[1300] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1301] Conventional learning support systems have difficulty in fully responding to the individual learning needs of users, limiting their ability to improve learning efficiency and motivation. Furthermore, because they are unable to dynamically adjust learning plans or change learning materials taking into account the user's emotional state, they provide a uniform learning process, resulting in a suboptimal learning experience.

[1302] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1303] In this invention, the server includes means for a user to create an account and select a language and avatar in the initial settings, means for generating an individual study plan based on the user's age, grade, and study goals, means for preparing and displaying study materials in response to a user's request at the start of study, means for analyzing photos of unclear points taken by the user and generating explanations, means for displaying the generated explanations and study materials to the user, means for recording the user's study progress and adjusting the study plan for the next day, means for analyzing the user's facial expressions and tone of voice and dynamically adjusting the study plan based on the user's emotional state, and means for changing the content of the study materials and explanations to be displayed based on the user's emotional state. This makes it possible to provide a flexible study process that meets the individual learning needs of users and corresponds to their emotional state.

[1304] "User" refers to a person who uses the tutoring system to study.

[1305] "Account" refers to a user profile that contains authentication information for a user to access and use the System.

[1306] "Initial settings" refers to basic setting operations such as language selection and avatar settings that users perform before using the system.

[1307] "Language" refers to the interface language that the user uses within the system.

[1308] "Avatar" refers to a character or icon that a user selects within a virtual environment.

[1309] A "study plan" refers to a series of study schedules and content created based on a user's age, grade level, and learning goals.

[1310] "Teaching materials" refers to educational materials such as textbooks, videos, and VR content used by users to study.

[1311] "Explanation" refers to text or audio materials that explain questions and learning content in a way that is easy for users to understand.

[1312] "Study progress" refers to the progress and learning status achieved by the user in the course of their studies.

[1313] An "emotion engine" refers to technology that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[1314] "Database" refers to a system for centrally managing and storing data such as user settings and study plans.

[1315] "Generative AI model" refers to artificial intelligence technology for automatically generating study plans based on user information.

[1316] A "prompt sentence" refers to an input sentence that gives specific instructions to a generative AI model.

[1317] MODE FOR CARRYING OUT THE INVENTION

[1318] This invention relates to a tutoring system that uses AI to support users' learning. In particular, it aims to improve learning efficiency and motivation by incorporating an emotion engine that recognizes users' emotions. This system functions through the interaction of a server, a terminal, and a user.

[1319] User registration and initial settings

[1320] First, a user creates an account using a device such as a smartphone, tablet, or PC. When creating an account, basic information such as name, email address, and password is entered and submitted, which is then sent to the server. The server stores the received information in a database and creates the account.

[1321] Next, the user selects the language and avatar as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This information is also sent from the device to the server and stored in the database.

[1322] Generate a lesson plan

[1323] The server retrieves information about the user's age, grade level, and learning goals from the database and uses a generative AI model to generate a personalized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[1324] Start learning

[1325] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content and sends them to the device. For example, for the first math class, learning materials for learning number concepts are provided using VR. The device displays this learning material to the user, and learning begins.

[1326] Questions and Explanations

[1327] If a user comes across a question during their study, they can take a photo of the problem with their smartphone. The photo is then sent to a server via their device. The server then uses an image analysis algorithm to analyze the photo and determine the nature of the question. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via their device.

[1328] Track your progress and adjust your study plan

[1329] Once the learning is complete, the device sends a progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[1330] Incorporating an emotion engine

[1331] The system also includes an emotion engine that recognizes the user's emotions. The emotion engine uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. For example, if the user loses concentration or feels stressed while studying, the emotion engine will detect this situation.

[1332] Dynamic adjustment based on emotions

[1333] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data: for example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material, or it can speed up the progress if positive emotions are recognized.

[1334] Modifying materials based on emotions

[1335] Based on the emotions recognized by the emotion engine, the server can also change the content of the displayed learning materials and explanations. For example, if the user is having fun, the server can display learning materials incorporating game elements to further stimulate interest. In this way, the server can provide an optimal learning environment according to the user's emotional state.

[1336] Specific examples

[1337] For example, if 8-year-old User A feels tired, the emotion engine will recognize this and the server will change the learning materials to make them more relaxing. Also, if User A feels stressed during a particular unit, the server will check the user's progress and adjust the difficulty of that unit to be less difficult. This makes the user's learning experience more effective and enjoyable.

[1338] Sample prompt sentence

[1339] Example user profile:

[1340] Age: 8

[1341] Grade: 3rd grade

[1342] Learning Objectives: Learning basic mathematical operations and basic science concepts

[1343] Prompt sentence to input to the generative AI model:

[1344] "Create a daily lesson plan for third graders that includes basic math operations (e.g., addition, subtraction) and basic science concepts (e.g., familiar natural phenomena)."

[1345] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1346] Step 1:

[1347] A user creates an account using a terminal. As input, the user enters basic information such as name, email address, and password into the terminal and presses the "Submit" button. The terminal sends this information to the server. The output is that the user is presented with a message confirming successful account registration.

[1348] Step 2:

[1349] The server stores the received user information in a database. The input is user information sent from the terminal, such as name, email address, and password. The database adds this information as a new record. The output is that a new user record is created and saved in the database.

[1350] Step 3:

[1351] The user uses the device to perform initial setup. As input, the user selects the desired language and avatar and saves the settings. The device sends this information to the server. The output is that the initial setup information is sent to the server and stored in a database.

[1352] Step 4:

[1353] The server retrieves information about the user's age, grade, and learning goals from the database. The input is the user ID. The server uses the user ID as a key to retrieve related information from the database. The output is the user's age, grade, and learning goals.

[1354] Step 5:

[1355] The server uses the generative AI model to create an individualized learning plan. The input is the information about the user's age, grade, and learning goals obtained in step 4. The generative AI model creates prompt sentences based on this information and automatically generates a learning plan. The output is the generation of an individualized learning plan.

[1356] Step 6:

[1357] The server sends the generated learning plan to the device. The input is the learning plan output from the generative AI model. The server sends this to the user's device. The output is the learning plan displayed on the device.

[1358] Step 7:

[1359] The user presses the start button on the terminal to start learning. The input is the user's operation. The terminal notifies the server of this operation. The output is that a request to start learning is sent to the server.

[1360] Step 8:

[1361] The server prepares learning materials corresponding to the day's learning content. The input is a request to start learning. The server selects appropriate learning materials from the database and sends them to the terminal. The output is the learning materials displayed on the terminal.

[1362] Step 9:

[1363] If a user has a question while studying, they can use the camera to take a photo of the question. The input is the visual information of the question. The user takes a photo using the camera and saves it on the device. The output is the photographed photo data.

[1364] Step 10:

[1365] The device sends the captured photo data to the server. The input is a photo taken by the user. The device uploads this photo to the server. The output is the photo data stored on the server.

[1366] Step 11:

[1367] The server analyzes the photo using an image analysis algorithm. The input is the photo data sent from the device. The server analyzes the image and identifies the problem content. The output is the analyzed problem content.

[1368] Step 12:

[1369] The server generates an explanation based on the analysis results. The input is the problem content obtained by image analysis. The server creates an explanation using the explanation generation model. The output is the generated explanation.

[1370] Step 13:

[1371] The server sends the generated comment to the terminal. The input is the generated comment. The server sends this information to the terminal. The output is the comment displayed on the terminal.

[1372] Step 14:

[1373] The terminal displays the explanatory text to the user. The input is the explanatory text sent by the server. The terminal displays it in an appropriate format for the user. The output is the explanatory text that is presented to the user.

[1374] Step 15:

[1375] When the learning is completed, the terminal sends the learning progress data to the server. The input is the user's learning progress information. The terminal uploads it to the server. The output is the learning progress data stored on the server.

[1376] Step 16:

[1377] The server stores the received learning progress data in a database. The input is the progress data sent from the terminal. The server stores this in a database. The output is the learning progress data recorded in the database.

[1378] Step 17:

[1379] The server adjusts the learning plan for the next day. The input is the saved learning progress data. The server creates a new learning plan using a generative AI model based on the progress data. The output is the adjusted learning plan.

[1380] Step 18:

[1381] The server sends the new lesson plan to the device. The input is the adjusted lesson plan. The server sends this to the device. The output is the new lesson plan displayed on the device.

[1382] Step 19:

[1383] The device analyzes the user's facial expressions and tone of voice using an emotion engine. The input is the user's facial expressions and voice information. The device captures this data with a camera and microphone and analyzes it with the emotion engine. The output is the analyzed emotion data.

[1384] Step 20:

[1385] The emotion engine sends the analysis results to the server. The input is the analyzed emotion data. The emotion engine sends this data to the server. The output is the emotion data stored on the server.

[1386] Step 21:

[1387] The server dynamically adjusts the learning plan based on the analysis results. The input is the emotion data sent from the emotion engine. The server dynamically changes the learning plan using a generative AI model based on this. The output is the adjusted learning plan.

[1388] Step 22:

[1389] The server sends the adjusted lesson plan to the device. The input is the new lesson plan. The server sends it to the device. The output is the adjusted lesson plan displayed on the user's device.

[1390] Step 23:

[1391] The server changes the content of the learning materials and explanations displayed based on the results of the emotion engine. The input is the adjusted learning plan and emotional data. The server takes the emotional state into consideration and selects the most appropriate learning materials and explanations, which it then sends to the device. The output is the updated learning materials and explanations that are displayed to the user.

[1392] (Application example 2)

[1393] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1394] Conventional learning support systems provide fixed learning plans without considering the user's emotional state, making it difficult to maximize the user's learning efficiency and motivation. Furthermore, when the user is stressed or fatigued, they do not provide appropriate support, which reduces the effectiveness of learning.

[1395] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to create an account and select a language and avatar in the initial settings, means for generating an individual study plan based on the user's age, grade, and learning goals, means for analyzing photos of unclear parts taken by the user and generating explanations, means for displaying the generated explanations and learning materials to the user, means for recording the user's study progress and adjusting the study plan for the next day, means for recognizing the user's emotions using facial recognition technology and voice analysis technology, means for dynamically adjusting the study content based on the recognized emotional data, and means for providing feedback according to the user's emotional state. This makes it possible to maximize the user's learning efficiency and motivation and flexibly adjust the study plan according to the user's individual emotional state.

[1396] The "user account creation means" is a means for a user to create an account for using the system and select a language and avatar as initial settings.

[1397] A "study plan generator" is a means for generating an individualized study plan based on the user's age, grade level, and learning goals.

[1398] The "photo analysis means" is a means for analyzing a photo of an unknown location taken by a user and generating an explanation based on the analysis.

[1399] The "teaching material display means" is a means for displaying the generated explanations and teaching materials to the user.

[1400] The "progress recording means" is a means for recording the user's learning progress and adjusting the next day's learning plan based on that progress.

[1401] The "emotion recognition means" is a means for recognizing the user's emotions using face recognition technology or voice analysis technology.

[1402] The "dynamic adjustment means" is a means for dynamically adjusting the learning content based on the recognized emotion data.

[1403] A "feedback providing means" is a means for providing feedback to a user according to their emotional state.

[1404] The "database storage means" is a means for storing user setting information, study plans, and emotional data in a database.

[1405] The "audio playback means" is a means for playing back audio of the generated commentary or teaching material.

[1406] This invention is a tutoring system that uses AI to support users' learning, and in particular, improves learning efficiency and motivation by incorporating an emotion engine that recognizes users' emotions. This system functions through the interaction of a server, terminals, and users.

[1407] Specifically, this is carried out as follows.

[1408] User account creation method

[1409] The server provides a means for users to create an account using a device (smartphone, tablet, PC, etc.). When creating an account, the user enters basic information such as name, email address, and password and sends it to the server. The server stores the received information in a database and creates the account. The user then selects the language and avatar as initial settings. For example, the user might select Japanese and set a popular anime character as their avatar.

[1410] Learning plan generation tool

[1411] The server retrieves information about the user's age, grade, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[1412] Photo analysis tools

[1413] If a user encounters a problem while studying, they can use their device to take a photo of the problem. The photo is then sent to the server via the device. The server then uses an image analysis algorithm to analyze the photo and determine the problem's content. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via the device.

[1414] emotion recognition means

[1415] The system also includes an emotion engine that recognizes the user's emotions. The server uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. For example, if the user loses concentration or feels stressed while studying, the emotion engine will detect this.

[1416] Dynamic Adjustment Means

[1417] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data: for example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material, or it can speed up the progress if positive emotions are recognized.

[1418] Feedback methods

[1419] Once the learning is complete, the device sends a progress report to the server. The server stores the progress data in a database and adjusts the learning plan for the next day. Feedback can be provided based on the user's emotional state, creating an appropriate learning environment. For example, if the user has a negative reaction to a particular problem, the device can skip the next time the learning is completed or approach it differently.

[1420] Specific examples

[1421] For example, let's say a new employee at Store A is undergoing training using smart glasses. If the employee shows signs of fatigue, the emotion engine will detect this and display a notification on their smart device suggesting a break. Also, if the employee is performing a task with interest, a more challenging task will automatically be suggested as the next step.

[1422] Example prompts for generative AI models

[1423] Describe the process flow for real-time emotion recognition when a new employee is participating in training using smart glasses. Show how the system analyzes the employee's emotional state from their facial expressions and tone of voice, and dynamically adjusts the training accordingly.

[1424] The present invention enables flexible adjustment of study plans and feedback according to the user's emotional state, which is expected to improve study efficiency and motivation.

[1425] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1426] Step 1:

[1427] A user creates an account using a device (smartphone, tablet, smart glasses, etc.). The user enters basic information such as name, email address, and password, and sends that information to the server. The server stores the received information in a database and creates an account. In this process, the user's basic information is given as input, and an account is generated as output.

[1428] Step 2:

[1429] The user selects the language and avatar as the initial settings. For example, they may select Japanese and set a popular anime character as their avatar. This setting information is also sent from the device to the server and stored in the database. In this process, the selected language and avatar information are given as input, and the saved setting information is obtained as output.

[1430] Step 3:

[1431] The server retrieves information about the user's age, grade, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user will be scheduled to learn basic math problems and basic science concepts. This generated learning plan is sent to the device and displayed to the user. In this process, the user's profile information is given as input and an individualized learning plan is generated as output.

[1432] Step 4:

[1433] The user begins studying on their device. If they come across a question they don't understand, they take a photo of the question with their smart device. The photo is sent to the server via the device. The server analyzes the photo using an image analysis algorithm to determine the question's content. Based on the results of this analysis, an explanation is generated and the content is provided as text or audio. In this process, the photo of the question is given as input, and the generated explanation is provided as output.

[1434] Step 5:

[1435] Learning progress is recorded, and when learning is completed, the device sends a learning progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. Based on the progress data, a new learning plan is generated and sent to the device, and the user is notified. In this process, learning progress data is given as input, and the next day's learning plan is adjusted as output.

[1436] Step 6:

[1437] Furthermore, the system is equipped with an emotion engine that recognizes the user's emotions. The server uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. The user's emotional state is confirmed during the learning process, and the data is updated as necessary. During this process, the user's facial expression images and voice data are given as input, and the analyzed emotional state is output.

[1438] Step 7:

[1439] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data. For example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material. It can also speed up the learning process if positive emotions are recognized. This process takes emotion data as input and generates an adjusted learning plan as output.

[1440] Step 8:

[1441] The device provides feedback based on the user's emotional state. For example, if the user has a negative reaction to a particular problem, feedback is provided to change the user's approach to that problem in the next learning session. In this process, emotional state and learning progress are given as inputs, and feedback is provided as output.

[1442] This makes it possible to maximize the user's learning efficiency and motivation.

[1443] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1444] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1445] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1446] [Fourth embodiment]

[1447] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1448] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1449] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1450] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1451] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1452] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1453] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1454] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1455] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1456] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1457] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1458] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1459] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1460] The present invention is a tutoring system that uses AI to help users effectively advance their learning. This system functions through the interaction of a server, a terminal, and a user. A specific embodiment of this system will be described below.

[1461] User registration and initial settings

[1462] First, the user creates an account using a device (smartphone, tablet, PC, etc.). When creating an account, they enter basic information such as their name, email address, and password. The server then stores the user's information in a database and creates the account.

[1463] Next, the user selects the language and avatar as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This information is also sent from the device to the server and stored in the database.

[1464] Generate a lesson plan

[1465] The server retrieves information about the user's age, grade level, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[1466] Start learning

[1467] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, videos, VR content, etc.) corresponding to the day's learning content. For example, for the first math class, learning materials for learning number concepts are provided using VR. The device displays this learning material to the user, and learning begins.

[1468] Questions and Explanations

[1469] If a user comes across a question during their study, they can take a photo of the problem with their smartphone. The photo is then sent to a server via their device. The server then uses an image analysis algorithm to analyze the photo and determine the nature of the question. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via their device.

[1470] Track your progress and adjust your study plan

[1471] Once the learning is complete, the device sends a progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[1472] Specific examples

[1473] For example, when 8-year-old User A uses the system for the first time, he or she creates an account, selects Japanese as the initial language, and sets a favorite avatar. After that, math and science teaching materials are displayed on the device according to the learning plan sent from the server. User A begins learning number concepts in VR, and if there is anything they don't understand, they take a photo and send it to the server. The server analyzes the information and returns an explanation, allowing User A to deepen their understanding. After completing the learning, progress information is sent to the server, and the next day's learning plan is fine-tuned.

[1474] In this way, the AI-based tutoring system can provide high-quality education efficiently and effectively, without being affected by financial constraints or family circumstances.

[1475] The processing flow will be explained below.

[1476] Step 1:

[1477] The user launches the application and is directed to the account creation screen.

[1478] Step 2:

[1479] The device prompts the user to enter the required information (name, email address, password, etc.).

[1480] Step 3:

[1481] The user enters the information and presses the submit button.

[1482] Step 4:

[1483] The device sends the entered information to the server.

[1484] Step 5:

[1485] The server stores the received information in a database and creates an account.

[1486] Step 6:

[1487] The server sends the user an initial setup page.

[1488] Step 7:

[1489] The device will display an initial setup page, prompting the user to select a language and avatar.

[1490] Step 8:

[1491] Users can choose their preferred language and avatar.

[1492] Step 9:

[1493] The device sends the user's configuration information to the server.

[1494] Step 10:

[1495] The server stores the received configuration information in a database.

[1496] Step 11:

[1497] The server retrieves information about the user's age, grade level, and learning goals from a database.

[1498] Step 12:

[1499] The server generates an individualized learning plan based on the information it obtains.

[1500] Step 13:

[1501] The server generates a learning plan and sends it to the device.

[1502] Step 14:

[1503] The device displays the learning plan to the user.

[1504] Step 15:

[1505] The user presses the start button to begin learning.

[1506] Step 16:

[1507] The device sends a start request to the server.

[1508] Step 17:

[1509] The server prepares the day's learning content based on the user's request.

[1510] Step 18:

[1511] The server sends the corresponding educational materials (text, videos, VR content, etc.) to the device.

[1512] Step 19:

[1513] The device displays the teaching materials to the user.

[1514] Step 20:

[1515] As users continue their studies, if they come across something they don't understand, they can take a photo of it with their smartphone.

[1516] Step 21:

[1517] The device takes a photo and sends it to the server.

[1518] Step 22:

[1519] The server then uses an image analysis algorithm to analyze the photos it receives and determine the nature of the problem.

[1520] Step 23:

[1521] The server generates an explanation for the problem and creates explanatory content in text or audio.

[1522] Step 24:

[1523] The server sends the generated commentary to the device.

[1524] Step 25:

[1525] The device will display the explanation to the user.

[1526] Step 26:

[1527] Users continue to learn.

[1528] Step 27:

[1529] The device sends progress information to the server at the end of the learning process.

[1530] Step 28:

[1531] The server stores the received progress data in a database.

[1532] Step 29:

[1533] The server adjusts the next day's study plan based on the progress data.

[1534] Step 30:

[1535] The server generates a new learning plan and sends it to the device.

[1536] Step 31:

[1537] The device will notify the user of the new study plan.

[1538] Step 32:

[1539] The user can then review the new plan and prepare for the next day's study.

[1540] Example 1

[1541] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1542] Conventional learning systems lack the ability to respond to individual learning needs, making it difficult to maximize users' learning effectiveness. Furthermore, they lack the means to provide quick and accurate explanations when users encounter questions during their studies. Furthermore, they do not dynamically adjust learning plans according to learning progress, making it difficult to continuously improve learning.

[1543] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1544] In this invention, the server includes: a means for a user to create an account and select a language and avatar in the initial settings; a means for generating an individualized learning plan based on the user's age, grade, and learning goals; a means for analyzing images of unclear points taken by the user and generating explanations; a means for displaying the generated explanations and learning materials to the user; and a means for recording the user's learning progress and adjusting the learning plan for the next day. This enables effective learning support tailored to individual learning needs. Furthermore, by enabling users to quickly resolve unclear points, they can deepen their understanding of the learning. Furthermore, by dynamically adjusting the learning plan according to their learning progress, continuous improvement in learning is possible.

[1545] "User" refers to an individual who uses the system.

[1546] "Account" refers to information for user authentication, including personal information required for a user to access and use the system.

[1547] "Initial settings" refers to the language, avatar, and other settings that a user selects when they first start using the system.

[1548] "Language" refers to the language used by the user to display and operate the system.

[1549] "Avatar" refers to the character or image a user chooses to represent themselves on the system.

[1550] A "study plan" refers to a schedule of study content for a specific period of time that is generated based on the user's age, grade level, and learning goals.

[1551] "Image analysis" refers to the process of analyzing an image taken by a user and understanding its contents.

[1552] "Explanation" refers to content that explains the problem or issue in a way that is easy for users to understand.

[1553] "Learning materials" refers to materials such as texts, videos, and VR content provided for users to study.

[1554] "Progress" refers to the portion of the learning experience that the user has achieved and the current situation.

[1555] "Adjustment" refers to the process of changing and updating the learning plan according to the user's learning progress.

[1556] A "server" refers to a computer system that receives a user request, processes it, and returns a response.

[1557] "Terminal" refers to a device such as a smartphone, tablet, or PC that a user uses to access and operate the system.

[1558] "Database" refers to a system for storing and managing user information, study plans, etc.

[1559] A "generative AI model" refers to an algorithm or system that uses machine learning and artificial intelligence techniques to generate personalized learning plans and explanations for users.

[1560] A "prompt" refers to text that is an instruction or question entered into a generative AI model.

[1561] This invention is a tutoring system that uses AI to support users in effectively progressing with their studies. This system functions through the interaction between a server, a terminal, and a user.

[1562] User registration and initial settings

[1563] First, a user creates an account using a device such as a smartphone, tablet, or PC. When creating an account, they enter basic information such as their name, email address, and password. The entered information is sent from the device to a server, which then stores the information in a database and creates the account.

[1564] Next, the user selects the language and avatar to use as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This setting information is also sent from the device to the server and stored in the database.

[1565] Generate a lesson plan

[1566] The server retrieves information about the user's age, grade level, and learning goals from a database. It then uses a generative AI model to generate a personalized learning plan for the user. The generative AI model is an algorithm that uses machine learning and artificial intelligence techniques to derive appropriate learning content based on specific prompts. For example, an 8-year-old user might be scheduled to learn basic math problems and fundamental science concepts.

[1567] The generated lesson plan is sent to the terminal and displayed to the user.

[1568] Start learning

[1569] The user presses the start button on the device to begin learning. This request is sent from the device to the server. The server prepares the learning materials (text, videos, VR content, etc.) corresponding to the learning content for that day. For example, for the first math class, materials for learning number concepts are provided using VR. The prepared materials are sent to the device, which then displays them to the user.

[1570] Questions and Explanations

[1571] If a user encounters a problem during learning, they can take a photo of the problem with their smartphone. The image is then sent to the server via the device. The server then analyzes the photo using an image analysis algorithm (e.g., OpenCV, TensorFlow) to determine the problem.

[1572] Based on the analysis results, the server uses a generative AI model to generate an explanation. The generated explanation is provided in text or audio format and displayed to the user through the device. For example, it provides a step-by-step solution to a difficult math problem. Examples of prompts used in this process include:

[1573] "Analyze user-taken photos of math problems and provide step-by-step solutions."

[1574] Track your progress and adjust your study plan

[1575] Once the learning is complete, the device sends a progress report to the server. The server stores the progress data in a database and uses the generative AI model to adjust the next day's learning plan. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[1576] In this way, the present invention provides an AI-based tutoring system that provides effective education tailored to individual learning needs.

[1577] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1578] Step 1:

[1579] A user creates an account using a device such as a smartphone, tablet, or PC. First, the user enters basic information such as their name, email address, and password, and submits it. The device then sends the entered information to the server. The server stores the received information in a database and creates the user's account.

[1580] (Input) User name, email address, and password.

[1581] (Output) The user account is saved in the database.

[1582] (Specific operation) When a user enters information into an input form on a terminal and presses the send button, the information is sent to the server as an HTTP request, and the server processes the information to save it in a database.

[1583] Step 2:

[1584] Next, the user selects the language and avatar to use as the default settings. The information selected by the user is then sent from the device to the server and stored in a database.

[1585] (Input) Selected language, avatar.

[1586] (Output) Language and avatar settings are saved in the database.

[1587] (Specific operation) The user selects a language and avatar using drop-down menus or radio buttons, and then presses the Settings button. The setting information is sent to the server as an HTTP request, and the information is saved in a database on the server side.

[1588] Step 3:

[1589] The server retrieves information about the user's age, grade level, and learning goals from a database and uses a generative AI model to generate an individualized lesson plan. For example, an 8-year-old user would receive a lesson plan that includes basic math and science content. The generated lesson plan is then sent to the device and displayed to the user.

[1590] (Input) User's age, grade level, and learning goals.

[1591] (Output) Individualized learning plan.

[1592] (Specific operation) The server executes a database query to obtain user information, and generates a learning plan using prompts for the generative AI model. The generated learning plan is sent to the device as an HTTP response, and the device displays it to the user.

[1593] Step 4:

[1594] The user presses the start button on the device to begin learning. This request is sent from the device to the server. The server prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content, and the prepared materials are sent to the device and displayed to the user.

[1595] (Input) Press the start button.

[1596] (Output) Learning materials.

[1597] (Specific operation) When the user presses the start button, an HTTP request is sent to the server. The server selects learning materials according to the learning content and sends them to the terminal as an HTTP response. The terminal displays the received learning material data.

[1598] Step 5:

[1599] When a user encounters a problem during learning, they can take a photo of the problem with their smartphone and send it to the server via their device. The server then uses an image analysis algorithm to analyze the photo and generates an explanation using a generative AI model. The explanation is then displayed to the user via their device in text or audio format.

[1600] (Input) The captured image of the problem.

[1601] (Output) Explanatory text or audio.

[1602] (Specific operation) The user takes a photo of the problem with their smartphone, and the device sends it to the server as an HTTP request. The server analyzes the image using an image analysis library and applies a generative AI model to generate an explanation. The explanation data is sent to the device as an HTTP response, and the device displays it.

[1603] Step 6:

[1604] Once the learning is complete, the device sends a progress report to the server. The server stores the progress data in a database and adjusts the next day's learning plan using the generative AI model. The new adjusted learning plan is sent to the device and notified to the user.

[1605] (Input) Learning progress data.

[1606] (Output) The new adjusted lesson plan.

[1607] (Specific operation) The device sends learning progress information to the server via an HTTP request, and the server saves the progress data in a database. The next day's learning plan is readjusted using the generative AI model, and the new plan is sent to the device as an HTTP response to notify the user.

[1608] (Application example 1)

[1609] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1610] Conventional tutoring systems have difficulty meeting the individual needs of learners, making it difficult to provide efficient and effective learning support. Furthermore, providing operational guidance and problem-solving for factory engineers requires individualized support, making it difficult to implement. The objective of this invention is to provide an effective and efficient learning and operational guidance system that allows learners and engineers to receive support based on their individual needs.

[1611] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1612] In this invention, the server includes means for a user to create an account and select a language and avatar in the initial settings, means for generating an individualized learning plan based on the user's age, grade, and learning goals, means for analyzing photos of unclear parts taken by the user and generating an explanation, means for displaying the generated explanation and learning materials to the user, means for recording the user's learning progress and adjusting the user's learning plan for the next day, means for a technician to create an account and perform initial settings, means for generating individualized operating instructions based on the technician's skill level and past work history, means for analyzing photos and videos taken by the technician when a problem occurs and generating an explanation, means for displaying the generated explanation and instruction content to the technician, and means for recording operation progress and adjusting the next instruction content. This allows users to receive individualized learning support, and technicians to receive effective operating instructions and problem-solving support.

[1613] "User" refers to a person who creates an account to use the system and receives a learning plan and instruction on how to use it.

[1614] "Create an account" refers to the process by which a user enters their information to register with the system and configure their personal settings.

[1615] "Initial Settings" refers to the basic settings that a user makes when accessing the system, including selecting a language and an avatar.

[1616] "Language" refers to the language used by the user when learning or receiving instruction.

[1617] "Avatar" refers to the character or image a user chooses to represent themselves within the system.

[1618] "Age" refers to the age of the user or technician based on their date of birth.

[1619] "Grade" refers to the educational stage to which a learner currently belongs.

[1620] "Learning goal" refers to the goal regarding the learning content or skills that the user wants to achieve.

[1621] "Individualized Learning Plan" refers to a personalized learning schedule based on a user's age, grade level, and learning goals.

[1622] "Photography" refers to the act of taking photos or videos using the system.

[1623] "Analysis" refers to processing captured images and video, performing calculations and evaluations to identify problems.

[1624] "Explanation" refers to content that provides step-by-step explanations and procedures for solving a problem.

[1625] "Teaching materials" refers to educational materials such as textbooks, videos, and VR content used by users to further their studies.

[1626] "Study progress" refers to data that indicates the achievement status of a user as they progress with their studies.

[1627] "Adjustment" refers to the act of changing and optimizing learning plans and instructional content based on progress data.

[1628] "Technician" refers to someone who uses the system to operate and maintain robots and machinery in a factory or other location.

[1629] "Skill level" refers to an indicator of an engineer's technical proficiency and ability.

[1630] "Operational instruction" refers to instructions and guidelines for engineers to operate robots and machines properly.

[1631] "Photos and videos at the time of the problem" refers to visual data taken by technicians to record malfunctions in machines or robots.

[1632] "Instruction content" refers to the specific work procedures and solutions provided to engineers.

[1633] "Progress" refers to data that shows the progress a technician makes as they carry out operation and maintenance tasks.

[1634] To implement the present invention, the system functions mainly through the interaction of a server, a terminal, and a user. Specifically, the system is configured as follows:

[1635] First, a user creates an account using a device (smartphone, tablet, PC, etc.) and performs initial setup. This initial setup includes selecting a language (e.g., Japanese or English) and an avatar. This information is sent from the device to the server and stored in the server's database.

[1636] The server generates an individualized learning plan based on the user's age, grade level, and learning goal data. This learning plan may include, for example, basic math problems or basic science concepts. The generated learning plan is sent to the device and displayed to the user.

[1637] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content. The device displays this material to the user, and learning begins.

[1638] If a user encounters a problem during their study, they can take a photo of the problem using their device's camera. This photo is then sent from the device to a server, which analyzes it using an image analysis algorithm. Based on the analysis results, a generative AI model then generates an explanation for the problem and provides it to the user in text or audio.

[1639] Meanwhile, factory engineers can also use the system. Engineers create an account and perform initial setup. The server generates individual operating instructions based on the engineer's skill level and past work history. If a problem occurs during operation, the engineer takes a photo or video with their device and sends it to the server. The server analyzes the image, and the generative AI model generates the cause of the problem and a solution.

[1640] For example, if an engineer encounters an error while operating a factory robot, they can use a smartphone app to take a photo of the problem and send it to the AI ​​via the app. The AI ​​will then analyze the image and provide the engineer with the cause and solution of the problem in real time. This process enables effective operation guidance and problem-solving support.

[1641] Servers and terminals use software such as Python, OpenCV (image processing), TensorFlow (AI model), and SQLite (database management), and data processing and calculation are carried out through image processing and AI analysis.

[1642] Specific prompt examples:

[1643] "An error occurred while operating the factory robot. Please analyze the picture below and tell me the cause and how to solve it."

[1644] (upload photo here)

[1645] Through this system, users can receive individual learning support, and engineers can receive effective operation instruction and problem-solving support.

[1646] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1647] Step 1:

[1648] The user uses a device to create an account and perform initial settings. Input data includes the user's name, email address, password, language, and avatar information. The device sends this information to the server, which stores it in a database. Specifically, the user enters the required information on the device's registration screen and presses the send button.

[1649] Step 2:

[1650] The server generates an individualized learning plan based on the user's age, grade level, and learning goal data. The input data is the user's basic information, and the output is an individualized learning plan. To generate this, the server refers to past learning data and standard curricula to determine the optimal learning content for the user. Specifically, an algorithm within the server analyzes the user's information and generates a plan.

[1651] Step 3:

[1652] The user presses the start button on the device to begin learning. The input data is the user's start request, and the output is the display of learning materials (text, video, VR content, etc.) corresponding to the day's learning content. The device sends a request to the server to display this, and the server provides learning materials based on the day's plan. In concrete terms, the user presses the start button and the learning materials are displayed on the device.

[1653] Step 4:

[1654] If a user encounters a problem during learning, they use the device's camera to take a photo of the problem. The input data is the photo, and the output is an explanation or solution for the problem. The device sends the photo to a server, which then analyzes the problem using an image analysis algorithm (such as OpenCV). Specifically, the user takes a photo and uploads it from the device to the server.

[1655] Step 5:

[1656] The server uses the generative AI model to generate an explanation for the problem based on the image analysis results. The input data is the image analysis results, and the output is the generated explanation. The server provides this to the user in text or audio format. Specifically, the AI ​​model generates an explanation based on the analysis results, and sends it from the server to the device in text or audio format.

[1657] Step 6:

[1658] When the learning is completed, the device sends learning progress information to the server. The input data is the learning progress information, and the output is the adjusted learning plan for the next day. The server analyzes the received progress data and adjusts the learning plan for the next day. Specifically, the device sends the learning progress to the server, and the server generates a newly adjusted learning plan.

[1659] Step 7:

[1660] Engineers working in factories also use terminals to create accounts and perform initial setup. The input data is the engineer's name, skill level, and past work history, and the output is an individual operation training plan. The terminal sends this to the server, which then stores the engineer's information in a database. Specifically, the engineer enters information on the terminal's registration screen and presses the send button.

[1661] Step 8:

[1662] If a problem occurs during operation, the technician takes a photo or video with the device and sends it to the server. The input data is the photo or video, and the output is the cause of the problem and a solution. The server performs image analysis, and a generative AI model generates the cause of the problem and a solution. Specifically, the technician takes a photo or video and uploads it from the device to the server.

[1663] Step 9:

[1664] The server displays the generated explanations and instruction content to the engineer based on the analysis results. The input data is the analysis results, and the output is the generated explanations and instruction content. The server provides this to the engineer in text or audio format. Specifically, the generative AI model generates explanations based on the analysis results, and sends them from the server to the terminal in text or audio format.

[1665] Step 10:

[1666] When the operation is completed, the terminal sends operation progress information to the server. The input data is the operation progress information, and the output is the adjustment result of the next operation instruction content. The server analyzes the received progress data and adjusts the next operation instruction content. Specifically, the terminal sends the operation progress to the server, and the server generates newly adjusted instruction content.

[1667] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1668] The present invention is a tutoring system that uses AI to support users' learning. In particular, it aims to improve learning efficiency and motivation by incorporating an emotion engine that recognizes the user's emotions. This system functions through the interaction between a server, a terminal, and the user. A specific embodiment of this system will be described below.

[1669] User registration and initial settings

[1670] First, a user creates an account using a device (smartphone, tablet, PC, etc.). When creating an account, basic information such as name, email address, and password is entered and submitted, which is then sent to the server. The server stores the received information in a database and creates the account.

[1671] Next, the user selects the language and avatar as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This information is also sent from the device to the server and stored in the database.

[1672] Generate a lesson plan

[1673] The server retrieves information about the user's age, grade level, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[1674] Start learning

[1675] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content and sends them to the device. For example, for the first math class, learning materials for learning number concepts are provided using VR. The device displays this learning material to the user, and learning begins.

[1676] Questions and Explanations

[1677] If a user comes across a question during their study, they can take a photo of the problem with their smartphone. The photo is then sent to a server via their device. The server then uses an image analysis algorithm to analyze the photo and determine the nature of the question. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via their device.

[1678] Track your progress and adjust your study plan

[1679] Once the learning is complete, the device sends a progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[1680] Incorporating an emotion engine

[1681] The system also includes an emotion engine that recognizes the user's emotions. The emotion engine uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. For example, if the user loses concentration or feels stressed while studying, the emotion engine will detect this situation.

[1682] Dynamic adjustment based on emotions

[1683] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data: for example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material, or it can speed up the progress if positive emotions are recognized.

[1684] Modifying materials based on emotions

[1685] Based on the emotions recognized by the emotion engine, the server can also change the content of the displayed learning materials and explanations. For example, if the user is having fun, the server can display learning materials incorporating game elements to further stimulate interest. In this way, the server can provide an optimal learning environment according to the user's emotional state.

[1686] Specific examples

[1687] For example, if 8-year-old User A feels tired, the emotion engine will recognize this and the server will change the learning materials to make them more relaxing. Also, if User A feels stressed during a particular unit, the server will check the user's progress and adjust the difficulty of that unit to be less difficult. This makes the user's learning experience more effective and enjoyable.

[1688] This invention enables a tutoring system that utilizes AI to respond flexibly to the user's emotional state, maximizing learning efficiency and motivation.

[1689] The processing flow will be explained below.

[1690] Step 1:

[1691] The user launches the application and is directed to the account creation screen.

[1692] Step 2:

[1693] The device prompts the user to enter the required information (name, email address, password, etc.).

[1694] Step 3:

[1695] The user enters the information and presses the submit button.

[1696] Step 4:

[1697] The device sends the entered information to the server.

[1698] Step 5:

[1699] The server stores the received information in a database and creates an account.

[1700] Step 6:

[1701] The server sends the user an initial setup page.

[1702] Step 7:

[1703] The device will display an initial setup page, prompting the user to select a language and avatar.

[1704] Step 8:

[1705] Users can choose their preferred language and avatar.

[1706] Step 9:

[1707] The device sends the user's configuration information to the server.

[1708] Step 10:

[1709] The server stores the received configuration information in a database.

[1710] Step 11:

[1711] The server retrieves information about the user's age, grade level, and learning goals from a database.

[1712] Step 12:

[1713] The server generates an individualized learning plan based on the information it obtains.

[1714] Step 13:

[1715] The server generates a learning plan and sends it to the device.

[1716] Step 14:

[1717] The device displays the learning plan to the user.

[1718] Step 15:

[1719] The user presses the start button to begin learning.

[1720] Step 16:

[1721] The device sends a start request to the server.

[1722] Step 17:

[1723] The server prepares the day's learning content based on the user's request.

[1724] Step 18:

[1725] The server sends the corresponding educational materials (text, videos, VR content, etc.) to the device.

[1726] Step 19:

[1727] The device displays the teaching materials to the user.

[1728] Step 20:

[1729] As users continue their studies, if they come across something they don't understand, they can take a photo of it with their smartphone.

[1730] Step 21:

[1731] The device takes a photo and sends it to the server.

[1732] Step 22:

[1733] The server then uses an image analysis algorithm to analyze the photos it receives and determine the nature of the problem.

[1734] Step 23:

[1735] The server generates an explanation for the problem and creates explanatory content in text or audio.

[1736] Step 24:

[1737] The server sends the generated commentary to the device.

[1738] Step 25:

[1739] The device will display the explanation to the user.

[1740] Step 26:

[1741] Users continue to learn.

[1742] Step 27:

[1743] The device sends progress information to the server at the end of the learning process.

[1744] Step 28:

[1745] The server stores the received progress data in a database.

[1746] Step 29:

[1747] The server adjusts the next day's study plan based on the progress data.

[1748] Step 30:

[1749] The system also includes an emotion engine that recognizes the user's emotions. During training, the device sends the user's facial expressions and voice to the emotion engine.

[1750] Step 31:

[1751] The emotion engine uses facial recognition and voice analysis technology to analyze the user's emotions.

[1752] Step 32:

[1753] The emotion engine sends the analysis results to the server.

[1754] Step 33:

[1755] The server dynamically adjusts according to the user's emotions based on data from the emotion engine.

[1756] Step 34:

[1757] For example, if a user is feeling stressed, the server can reduce the difficulty of the study plan or switch to more engaging material.

[1758] Step 35:

[1759] The server changes the content of the teaching materials and explanations displayed based on the data from the emotion engine.

[1760] Step 36:

[1761] For example, if the user is having fun, the server will display educational materials that incorporate game elements on the device.

[1762] Step 37:

[1763] The device displays tailored learning materials and explanations to the user.

[1764] Step 38:

[1765] After the learning session ends, the device sends the emotion engine analysis data and learning progress data to the server.

[1766] Step 39:

[1767] The server stores this data in a database and adjusts the next day's study plan accordingly.

[1768] Step 40:

[1769] The server sends the new study plan to the device, which then notifies the user.

[1770] Step 41:

[1771] Users can review their new study plan and prepare for the next day's study.

[1772] Example 2

[1773] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1774] Conventional learning support systems have difficulty in fully responding to the individual learning needs of users, limiting their ability to improve learning efficiency and motivation. Furthermore, because they are unable to dynamically adjust learning plans or change learning materials taking into account the user's emotional state, they provide a uniform learning process, resulting in a suboptimal learning experience.

[1775] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1776] In this invention, the server includes means for a user to create an account and select a language and avatar in the initial settings, means for generating an individual study plan based on the user's age, grade, and study goals, means for preparing and displaying study materials in response to a user's request at the start of study, means for analyzing photos of unclear points taken by the user and generating explanations, means for displaying the generated explanations and study materials to the user, means for recording the user's study progress and adjusting the study plan for the next day, means for analyzing the user's facial expressions and tone of voice and dynamically adjusting the study plan based on the user's emotional state, and means for changing the content of the study materials and explanations to be displayed based on the user's emotional state. This makes it possible to provide a flexible study process that meets the individual learning needs of users and corresponds to their emotional state.

[1777] "User" refers to a person who uses the tutoring system to study.

[1778] "Account" refers to a user profile that contains authentication information for a user to access and use the System.

[1779] "Initial settings" refers to basic setting operations such as language selection and avatar settings that users perform before using the system.

[1780] "Language" refers to the interface language that the user uses within the system.

[1781] "Avatar" refers to a character or icon that a user selects within a virtual environment.

[1782] A "study plan" refers to a series of study schedules and content created based on a user's age, grade level, and learning goals.

[1783] "Teaching materials" refers to educational materials such as textbooks, videos, and VR content used by users to study.

[1784] "Explanation" refers to text or audio materials that explain questions and learning content in a way that is easy for users to understand.

[1785] "Study progress" refers to the progress and learning status achieved by the user in the course of their studies.

[1786] An "emotion engine" refers to technology that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[1787] "Database" refers to a system for centrally managing and storing data such as user settings and study plans.

[1788] "Generative AI model" refers to artificial intelligence technology for automatically generating study plans based on user information.

[1789] A "prompt sentence" refers to an input sentence that gives specific instructions to a generative AI model.

[1790] MODE FOR CARRYING OUT THE INVENTION

[1791] This invention relates to a tutoring system that uses AI to support users' learning. In particular, it aims to improve learning efficiency and motivation by incorporating an emotion engine that recognizes users' emotions. This system functions through the interaction of a server, a terminal, and a user.

[1792] User registration and initial settings

[1793] First, a user creates an account using a device such as a smartphone, tablet, or PC. When creating an account, basic information such as name, email address, and password is entered and submitted, which is then sent to the server. The server stores the received information in a database and creates the account.

[1794] Next, the user selects the language and avatar as the initial settings. For example, they may choose Japanese and set a popular anime character as their avatar. This information is also sent from the device to the server and stored in the database.

[1795] Generate a lesson plan

[1796] The server retrieves information about the user's age, grade level, and learning goals from the database and uses a generative AI model to generate a personalized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[1797] Start learning

[1798] The user presses the start button on the device to begin learning. This request is sent to the server, which prepares the learning materials (text, video, VR content, etc.) corresponding to the day's learning content and sends them to the device. For example, for the first math class, learning materials for learning number concepts are provided using VR. The device displays this learning material to the user, and learning begins.

[1799] Questions and Explanations

[1800] If a user comes across a question during their study, they can take a photo of the problem with their smartphone. The photo is then sent to a server via their device. The server then uses an image analysis algorithm to analyze the photo and determine the nature of the question. The server then generates an explanation for the problem and creates explanatory content in text and audio. For example, a step-by-step solution to a difficult math problem can be provided. This explanation is then displayed to the user via their device.

[1801] Track your progress and adjust your study plan

[1802] Once the learning is complete, the device sends a progress report to the server. The server stores the received progress data in a database and adjusts the learning plan for the next day. For example, if there is a delay in a particular unit, new learning materials will be added to reinforce that part. This new learning plan is sent to the device and the user is notified.

[1803] Incorporating an emotion engine

[1804] The system also includes an emotion engine that recognizes the user's emotions. The emotion engine uses facial recognition and voice analysis technologies to analyze the user's emotional state from their facial expressions and tone of voice. For example, if the user loses concentration or feels stressed while studying, the emotion engine will detect this situation.

[1805] Dynamic adjustment based on emotions

[1806] The server receives data from the emotion engine and dynamically adjusts the learning plan based on that data: for example, if the user is feeling stressed, it can lower the difficulty level or switch to more engaging material, or it can speed up the progress if positive emotions are recognized.

[1807] Modifying materials based on emotions

[1808] Based on the emotions recognized by the emotion engine, the server can also change the content of the displayed learning materials and explanations. For example, if the user is having fun, the server can display learning materials incorporating game elements to further stimulate interest. In this way, the server can provide an optimal learning environment according to the user's emotional state.

[1809] Specific examples

[1810] For example, if 8-year-old User A feels tired, the emotion engine will recognize this and the server will change the learning materials to make them more relaxing. Also, if User A feels stressed during a particular unit, the server will check the user's progress and adjust the difficulty of that unit to be less difficult. This makes the user's learning experience more effective and enjoyable.

[1811] Sample prompt sentence

[1812] Example user profile:

[1813] Age: 8

[1814] Grade: 3rd grade

[1815] Learning Objectives: Learning basic mathematical operations and basic science concepts

[1816] Prompt sentence to input to the generative AI model:

[1817] "Create a daily lesson plan for third graders that includes basic math operations (e.g., addition, subtraction) and basic science concepts (e.g., familiar natural phenomena)."

[1818] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1819] Step 1:

[1820] A user creates an account using a terminal. As input, the user enters basic information such as name, email address, and password into the terminal and presses the "Submit" button. The terminal sends this information to the server. The output is that the user is presented with a message confirming successful account registration.

[1821] Step 2:

[1822] The server stores the received user information in a database. The input is user information sent from the terminal, such as name, email address, and password. The database adds this information as a new record. The output is that a new user record is created and saved in the database.

[1823] Step 3:

[1824] The user uses the device to perform initial setup. As input, the user selects the desired language and avatar and saves the settings. The device sends this information to the server. The output is that the initial setup information is sent to the server and stored in a database.

[1825] Step 4:

[1826] The server retrieves information about the user's age, grade, and learning goals from the database. The input is the user ID. The server uses the user ID as a key to retrieve related information from the database. The output is the user's age, grade, and learning goals.

[1827] Step 5:

[1828] The server uses the generative AI model to create an individualized learning plan. The input is the information about the user's age, grade, and learning goals obtained in step 4. The generative AI model creates prompt sentences based on this information and automatically generates a learning plan. The output is the generation of an individualized learning plan.

[1829] Step 6:

[1830] The server sends the generated learning plan to the device. The input is the learning plan output from the generative AI model. The server sends this to the user's device. The output is the learning plan displayed on the device.

[1831] Step 7:

[1832] The user presses the start button on the terminal to start learning. The input is the user's operation. The terminal notifies the server of this operation. The output is that a request to start learning is sent to the server.

[1833] Step 8:

[1834] The server prepares learning materials corresponding to the day's learning content. The input is a request to start learning. The server selects appropriate learning materials from the database and sends them to the terminal. The output is the learning materials displayed on the terminal.

[1835] Step 9:

[1836] If a user has a question while studying, they can use the camera to take a photo of the question. The input is the visual information of the question. The user takes a photo using the camera and saves it on the device. The output is the photographed photo data.

[1837] Step 10:

[1838] The device sends the captured photo data to the server. The input is a photo taken by the user. The device uploads this photo to the server. The output is the photo data stored on the server.

[1839] Step 11:

[1840] The server analyzes the photo using an image analysis algorithm. The input is the photo data sent from the device. The server analyzes the image and identifies the problem content. The output is the analyzed problem content.

[1841] Step 12:

[1842] The server generates an explanation based on the analysis results. The input is the problem content obtained by image analysis. The server creates an explanation using the explanation generation model. The output is the generated explanation.

[1843] Step 13:

[1844] The server sends the generated comment to the terminal. The input is the generated comment. The server sends this information to the terminal. The output is the comment displayed on the terminal.

[1845] Step 14:

[1846] The terminal displays the explanatory text to the user. The input is the explanatory text sent by the server. The terminal displays it in an appropriate format for the user. The output is the explanatory text that is presented to the user.

[1847] Step 15:

[1848] When the learning is completed, the terminal sends the learning progress data to the server. The input is the user's learning progress information. The terminal uploads it to the server. The output is the learning progress data stored on the server.

[1849] Step 16:

[1850] The server stores the received learning progress data in a database. The input is the progress data sent from the terminal. The server stores this in a database. The output is the learning progress data recorded in the database.

[1851] Step 17:

[1852] The server adjusts the learning plan for the next day. The input is the saved learning progress data. The server creates a new learning plan using a generative AI model based on the progress data. The output is the adjusted learning plan.

[1853] Step 18:

[1854] The server sends the new lesson plan to the device. The input is the adjusted lesson plan. The server sends this to the device. The output is the new lesson plan displayed on the device.

[1855] Step 19:

[1856] The device analyzes the user's facial expressions and tone of voice using an emotion engine. The input is the user's facial expressions and voice information. The device captures this data with a camera and microphone and analyzes it with the emotion engine. The output is the analyzed emotion data.

[1857] Step 20:

[1858] The emotion engine sends the analysis results to the server. The input is the analyzed emotion data. The emotion engine sends this data to the server. The output is the emotion data stored on the server.

[1859] Step 21:

[1860] The server dynamically adjusts the learning plan based on the analysis results. The input is the emotion data sent from the emotion engine. The server dynamically changes the learning plan using a generative AI model based on this. The output is the adjusted learning plan.

[1861] Step 22:

[1862] The server sends the adjusted lesson plan to the device. The input is the new lesson plan. The server sends it to the device. The output is the adjusted lesson plan displayed on the user's device.

[1863] Step 23:

[1864] The server changes the content of the learning materials and explanations displayed based on the results of the emotion engine. The input is the adjusted learning plan and emotional data. The server takes the emotional state into consideration and selects the most appropriate learning materials and explanations, which it then sends to the device. The output is the updated learning materials and explanations that are displayed to the user.

[1865] (Application example 2)

[1866] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1867] Conventional learning support systems provide fixed learning plans without considering the user's emotional state, making it difficult to maximize the user's learning efficiency and motivation. Furthermore, when the user is stressed or fatigued, they do not provide appropriate support, which reduces the effectiveness of learning.

[1868] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to create an account and select a language and avatar in the initial settings, means for generating an individual study plan based on the user's age, grade, and learning goals, means for analyzing photos of unclear parts taken by the user and generating explanations, means for displaying the generated explanations and learning materials to the user, means for recording the user's study progress and adjusting the study plan for the next day, means for recognizing the user's emotions using facial recognition technology and voice analysis technology, means for dynamically adjusting the study content based on the recognized emotional data, and means for providing feedback according to the user's emotional state. This makes it possible to maximize the user's learning efficiency and motivation and flexibly adjust the study plan according to the user's individual emotional state.

[1869] The "user account creation means" is a means for a user to create an account for using the system and select a language and avatar as initial settings.

[1870] A "study plan generator" is a means for generating an individualized study plan based on the user's age, grade level, and learning goals.

[1871] The "photo analysis means" is a means for analyzing a photo of an unknown location taken by a user and generating an explanation based on the analysis.

[1872] The "teaching material display means" is a means for displaying the generated explanations and teaching materials to the user.

[1873] The "progress recording means" is a means for recording the user's learning progress and adjusting the next day's learning plan based on that progress.

[1874] The "emotion recognition means" is a means for recognizing the user's emotions using face recognition technology or voice analysis technology.

[1875] The "dynamic adjustment means" is a means for dynamically adjusting the learning content based on the recognized emotion data.

[1876] A "feedback providing means" is a means for providing feedback to a user according to their emotional state.

[1877] The "database storage means" is a means for storing user setting information, study plans, and emotional data in a database.

[1878] The "audio playback means" is a means for playing back audio of the generated commentary or teaching material.

[1879] This invention is a tutoring system that uses AI to support users' learning, and in particular, improves learning efficiency and motivation by incorporating an emotion engine that recognizes users' emotions. This system functions through the interaction of a server, terminals, and users.

[1880] Specifically, this is carried out as follows.

[1881] User account creation method

[1882] The server provides a means for users to create an account using a device (smartphone, tablet, PC, etc.). When creating an account, the user enters basic information such as name, email address, and password and sends it to the server. The server stores the received information in a database and creates the account. The user then selects the language and avatar as initial settings. For example, the user might select Japanese and set a popular anime character as their avatar.

[1883] Learning plan generation tool

[1884] The server retrieves information about the user's age, grade, and learning goals from the database and generates an individualized learning plan. For example, an 8-year-old user might be scheduled to learn basic math problems and basic science concepts each day. This generated learning plan is sent to the device and displayed to the user.

[1885] Photo analysis tools

[1886] If a user encounters a problem while studying, they can use their device to...

Claims

1. A way for users to create an account and select their initial language and avatar; means for generating a personalized learning plan based on the user's age, grade level, and learning goals; A means for analyzing a photograph of an unknown location taken by a user and generating an explanation; a means for displaying the generated explanations and teaching materials to a user; A system that includes a means for recording a user's learning progress and adjusting their learning plan for the next day.

2. 2. The system according to claim 1, further comprising means for storing user setting information and study plans in a database.

3. 2. The system of claim 1, further comprising audio playback means for the commentary and educational materials.

Images