system

Abstract

The system provides children with smooth support when they encounter learning difficulties, allowing them to solve problems while maintaining the enjoyment of learning. [Solution] A system comprising: means for the user to take an image of the learning content they are having trouble with and input a descriptive text; means for the terminal to send the captured image and descriptive text to a server; means for the server to extract text data from the image using OCR technology; means for the server to analyze the extracted text data and classify it into an appropriate learning category; means for the server to generate appropriate answers and explanations based on the classified learning categories; means for the server to send the generated answers and explanations to the terminal; means for the terminal to display the received answers and explanations to the user; and means for the terminal to send user feedback to the server and store it in a database.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Parents have a strong desire to watch over their children's learning, but when they try to teach, they tend to get emotional, and there is a problem that the fun of the children's learning is impaired. Furthermore, when children feel learning difficulties, they may not be able to receive appropriate support, which may reduce their learning motivation. Such a situation may also have an adverse effect on the relationship between parents and children.

Means for Solving the Problems

[0005] The present invention provides a system that includes means for taking an image of the learning content the user is having trouble with and inputting a descriptive text; means for the terminal to send the captured image and descriptive text to a server; means for the server to extract text data from the image using OCR technology; means for the server to analyze the extracted text data and classify it into an appropriate learning category; means for the server to generate appropriate answers and explanations based on the classified learning categories; means for the server to send the generated answers and explanations to the terminal; means for the terminal to display the received answers and explanations to the user; and means for the terminal to send user feedback to the server and store it in a database. This system allows children to receive support smoothly when they encounter learning difficulties and to solve problems while maintaining the enjoyment of learning. It also facilitates parent-child relationships and allows parents to watch over their children's learning with peace of mind.

[0006] The term "user" refers to any person who uses the system, and is a concept that includes children who need learning support and the parents who provide that support.

[0007] A "device" refers to a device used by a user, such as a smartphone, tablet, or computer.

[0008] A "server" is a remote computer system responsible for processing and storing data for the entire system.

[0009] "Images" refer to visual data that users capture with their devices and send to the server.

[0010] "Description" refers to the text information that users enter to request learning support.

[0011] "OCR technology" is an abbreviation for optical character recognition, and it is a technology for extracting character information from image data.

[0012] "Text data" refers to character information extracted from an image using OCR technology.

[0013] A "learning category" refers to an educational field (e.g., mathematics, science, history, etc.) into which the content of the questions is classified based on the extracted text data.

[0014] "Answers and explanations" refer to solutions and explanations that the server generates based on problems received from users.

[0015] "Feedback" refers to information that users input about their experience using the system and their satisfaction level, and send to the server.

[0016] A "database" is a system for organizing and storing information on a server, and includes user information, feedback, and training data. [Brief explanation of the drawing]

[0017] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] It shows an emotion map on which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0021] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] The present invention relates to a learning support system and is designed to allow users to receive prompt and appropriate support when they encounter difficulties in learning. The embodiments for carrying out the present invention are described in detail below.

[0039] System Configuration

[0040] This system consists of users, devices, and a server. Users include children who need learning support and their parents who support them. Devices are mainly smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage.

[0041] Basic program processing

[0042] This system's program has the following functions:

[0043] 1. User Registration

[0044] Users (parents and children) register with the system using a device. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[0045] 2. Learning support request

[0046] When a user (child) encounters a problem with homework or learning material, they input text or images on their device and send them to the server. For example, if a child doesn't know how to solve a math problem, they can take a picture of an equation written in their notebook, add a simple explanation like "Please tell me how to solve it," and send it.

[0047] 3. Image and text analysis

[0048] The server analyzes the received image using OCR technology and extracts text data. The extracted text data is further analyzed to determine which learning category (e.g., mathematics, science, history) the question belongs to.

[0049] 4. Generating answers and explanations

[0050] The server generates appropriate answers and explanations based on the categorized learning categories. For example, if asked how to solve a quadratic equation, it will create a step-by-step explanation using the quadratic formula.

[0051] 5. Providing a response

[0052] The server sends the generated answers and explanations to the device. The device displays the received information to the user (child). The child can then solve the problem by following these explanations.

[0053] 6. Obtaining and saving feedback

[0054] The user (child) inputs their satisfaction level with the answers and explanations on their device and sends it to the server. The server stores this feedback in a database and uses it to improve future support.

[0055] Specific example

[0056] For example, consider a case where a user (child) doesn't know how to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. The user takes a picture of the equation with their device, enters a description saying "Please tell me how to solve it," and sends it to the server. The server uses OCR technology to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Next, the server generates a step-by-step explanation using the quadratic formula and sends it to the device. The device displays this explanation to the user (child), and the child solves the problem following the explanation. Finally, the child inputs their satisfaction level with the explanation and sends the feedback to the server. The server stores this feedback in a database and uses it to improve the accuracy of future learning support.

[0057] The above describes a specific embodiment for carrying out the present invention. This system allows parents to support their children at the appropriate time, and enables children to solve problems while maintaining the enjoyment of learning.

[0058] The following describes the processing flow.

[0059] Step 1: The user registers using their device. The user enters basic information such as their name, email address, and password.

[0060] Step 2: The terminal sends the entered user information to the server. The server receives this information.

[0061] Step 3: The server saves the received user information to the database and assigns a unique ID to each user.

[0062] Step 4: The server sends a notification to the terminal that user registration is complete. The terminal displays a registration completion message to the user.

[0063] Step 5: When the user (child) encounters a learning difficulty, they can use the device to take a picture of the problem and enter a brief description (e.g., "Please tell me how to solve this problem").

[0064] Step 6: The device sends the photo and description to the server. The server receives this data.

[0065] Step 7: The server uses OCR technology to extract text data from the received photo data. This converts the text information in the photo into digital data.

[0066] Step 8: The server analyzes the extracted text data to determine which learning category the problem belongs to (e.g., mathematics, science, history, etc.).

[0067] Step 9: The server generates appropriate answers and explanations based on the identified learning category. For example, if the topic is quadratic equations in mathematics, an explanation using the quadratic formula will be generated.

[0068] Step 10: The server sends the generated answers and explanations to the user's (child's) device. The device receives this data and displays it to the user.

[0069] Step 11: The user (child) solves the problem by following the answers and explanations displayed on the device. If there are any unclear points during the problem-solving process, they can send further questions.

[0070] Step 12: After the user (child) solves the problem and expresses their satisfaction with the explanation, they input their feedback into the device and send it to the server. The server receives this feedback data.

[0071] Step 13: The server saves the received feedback data to a database and uses it to improve the accuracy of learning support in the future.

[0072] (Example 1)

[0073] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0074] Traditionally, there has been a lack of means to receive quick and accurate support when encountering particularly difficult problems in learning. The manual searching and referencing of reference books were time-consuming and slowed down the learning process. Furthermore, providing efficient support based on appropriate feedback was difficult.

[0075] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0076] In this invention, the server includes means for taking an image of the learning content that the user is having trouble with and inputting a descriptive text; means for the terminal to send the captured image and descriptive text to the server; means for the server to extract text data from the image using optical character recognition technology; means for the server to analyze the extracted text data and classify it into an appropriate learning category; means for the server to use a generative AI model to generate appropriate answers and explanations based on the classified learning categories; means for the server to send the generated answers and explanations to the terminal; means for the terminal to display the received answers and explanations to the user; and means for the terminal to send user feedback to the server and store it in a database. This makes it possible for users to receive quick and appropriate support when they encounter difficulties in their learning.

[0077] "Users" refer to people who use the learning support system, and this includes children who have difficulty learning and the parents who support them.

[0078] A "device" refers to a device used by a user, and primarily includes smartphones, tablets, or personal computers.

[0079] A "server" refers to a cloud-based system, specifically a device responsible for processing and storing data.

[0080] Optical Character Recognition (OCR) is a technology that extracts text data from images.

[0081] A "generative AI model" refers to an artificial intelligence model designed to generate appropriate answers and explanations, and includes, for example, conversational generative models.

[0082] A "learning category" refers to a type of classification used to categorize learning content, and includes academic fields such as mathematics, science, and history.

[0083] "Feedback" refers to evaluations and opinions on answers and explanations provided by users, and this data is used to improve future support.

[0084] A "database" refers to a collection of information stored on a server, including user information and feedback data.

[0085] The present invention relates to a learning support system and is designed to allow users to receive prompt and appropriate support when they encounter difficulties in learning. The embodiments for carrying out the present invention are described in detail below.

[0086] System Configuration

[0087] This system consists of users, devices, and a server. Users include children who need learning support and their parents who support them. Devices are primarily smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage.

[0088] Basic program processing

[0089] This system's program has the following functions:

[0090] 1. User Registration

[0091] Users (parents and children) register with the system using a device. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[0092] 2. Learning support request

[0093] When a user (child) encounters a problem with their homework or learning material, they input text or images on their device and send them to the server. For example, if a child doesn't know how to solve a math problem, they might take a picture of an equation written in their notebook, add a simple explanation like "Please tell me how to solve it," and send it.

[0094] 3. Image and text analysis

[0095] The server analyzes the received images using OCR technology and extracts text data. The primary software used is optical character recognition technology (e.g., Google® Cloud Vision API). The extracted text data is further analyzed to determine which learning category (e.g., mathematics, science, history) the question belongs to.

[0096] 4. Generating answers and explanations

[0097] The server generates appropriate answers and explanations based on the classified learning categories. For example, if asked how to solve a quadratic equation, it will create a step-by-step explanation using the quadratic formula. For this purpose, it utilizes a generative AI model (e.g., OpenAI® GPT-3®).

[0098] 5. Providing a response

[0099] The server sends the generated answers and explanations to the terminal. The terminal displays the received information to the user (child). The user (child) can then solve the problem by following these explanations.

[0100] 6. Obtaining and saving feedback

[0101] The user (child) inputs their satisfaction level with the answers and explanations on their device and sends it to the server. The server stores this feedback in a database and uses it to improve future support.

[0102] Specific example

[0103] For example, consider a case where a user (a child) doesn't know how to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. The system will process it in the following steps:

[0104] 1. The user (child) takes a picture of this equation with their device, enters the description "Tell me how to solve it," and sends it to the server.

[0105] 2. The server uses OCR technology to extract the text data "2x² + 3x - 5 = 0" from the image and identifies the problem as a quadratic equation.

[0106] 3. The server generates a step-by-step explanation using the solution formula and sends it to the terminal.

[0107] 4. The device displays this explanation to the user (child), and the child solves the problem according to the explanation.

[0108] 5. The user (child) enters their satisfaction level with the explanation and sends feedback to the server.

[0109] 6. The server saves the feedback to a database and uses it to improve future learning support.

[0110] Example of a prompt

[0111] Assume the user will input the following prompt into the generated AI model:

[0112] "Please explain how to solve this quadratic equation: 2x² + 3x - 5 = 0."

[0113] Based on this prompt, the server generates a detailed explanation and provides it to the user through the terminal.

[0114] The above describes a specific embodiment for carrying out the present invention. This process makes it possible to provide learning support quickly and effectively.

[0115] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0116] Step 1: User Registration

[0117] 1. Input: Users (parents and children) use the device to enter basic information such as name, email address, and password.

[0118] 2. Specific actions:

[0119] The user launches a web browser or dedicated app and opens the user registration screen.

[0120] The user clicks the "Register" button.

[0121] 3. Data Processing and Output: The terminal sends the input information to the server. The server stores the received information in a database and generates and records a unique user ID. The server sends a success message to the terminal, and the terminal displays a registration completion message to the user.

[0122] Step 2: Request learning support

[0123] 1. Input: The user (child) enters the text of the problem or a photo of the notebook containing the problem, along with a brief description, on the device.

[0124] 2. Specific actions:

[0125] The user (child) taps the "Request Support" button within the app.

[0126] The user (child) takes a picture of the problem with a note that says, "Please tell me how to solve it."

[0127] 3. Data Processing and Output: The terminal sends the input image, description, and user ID data to the server. The server passes the received data to the analysis engine.

[0128] Step 3: Image and text analysis

[0129] 1. Input: The server receives image and description data from the terminal.

[0130] 2. Specific actions:

[0131] The server analyzes the image using optical character recognition technology (e.g., Google Cloud Vision API) and extracts the text data.

[0132] 3. Data Processing and Output: The server analyzes the extracted text data and uses a software engine to determine which learning category (e.g., mathematics, science, history) the problem belongs to. The category information is saved and passed on to the next step.

[0133] Step 4: Generating answers and explanations

[0134] 1. Input: The server sends a prompt to the generative AI model (e.g., OpenAI GPT-3) based on the identified learning categories and extracted text data.

[0135] 2. Specific actions:

[0136] The server sends the prompt message: "Please tell me how to solve the quadratic equation 2x² + 3x - 5 = 0." to the generating AI model.

[0137] 3. Data Processing and Output: The generative AI model generates appropriate answers and explanations in response to prompts. The server evaluates the validity of the generated answers and explanations, formats them appropriately, and then sends them to the terminal.

[0138] Step 5: Providing a response

[0139] 1. Input: The terminal receives answers and explanations sent from the server.

[0140] 2. Specific actions:

[0141] The terminal displays the received information on the user interface.

[0142] The user (child) reviews and understands the displayed explanation.

[0143] 3. Data processing and output: The user (child) solves the problem based on the explanation.

[0144] Step 6: Obtain and save feedback

[0145] 1. Input: The user (child) inputs their satisfaction level with the explanation using the terminal.

[0146] 2. Specific actions:

[0147] The user (child) chooses one option from choices such as "very satisfied," "satisfied," or "dissatisfied."

[0148] The user (child) clicks the "Send" button.

[0149] 3. Data Processing and Output: The terminal sends the input feedback to the server. The server receives the feedback and stores it in a database. The server uses this feedback to improve the accuracy of future learning support.

[0150] (Application Example 1)

[0151] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0152] Traditional learning support systems offer limited means for users to receive prompt and appropriate support when they encounter difficulties. Furthermore, there is a lack of effective learning support methods in physical stores, resulting in insufficient support for customers to effectively utilize learning materials. There is also a need for improved support accuracy based on feedback and the provision of interactive learning experiences for in-store use.

[0153] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0154] In this invention, the server includes means for taking an image of the learning content that the user is having trouble with and inputting a descriptive text; means for the terminal to send the captured image and descriptive text to the server; means for the server to extract text data from the image using OCR technology; means for the server to analyze the extracted text data and classify it into an appropriate learning category; means for the server to generate appropriate answers and explanations based on the classified learning categories; means for the server to send the generated answers and explanations to the terminal; means for the terminal to display the received answers and explanations to the user; means for the terminal to send user feedback to the server and store it in a database; means for the user to receive learning support through a robot or smart glasses placed in a physical store; and means for acquiring and registering product information and accessing learning content. This enables interactive learning support in physical stores, allowing users to effectively solve problems and improve the accuracy of support by utilizing feedback information.

[0155] "Users" refer to children who receive learning support using the learning support system, and their guardians who provide that support.

[0156] "Device" refers to electronic devices used by users, such as smartphones, tablets, and personal computers.

[0157] A "server" refers to a cloud-based system, specifically a device that processes and stores data sent by users.

[0158] "OCR technology" refers to optical character recognition technology that automatically extracts text data from images.

[0159] "Robot" refers to a device placed in a physical store that provides learning support while interacting with users.

[0160] "Smart glasses" refer to devices that users wear to display information from learning materials or to acquire information using the user's gaze.

[0161] "Product information" refers to detailed information about learning materials and products, and is obtained through methods such as QR codes (registered trademark).

[0162] "Learning content" refers to information such as textbooks, workbooks, and explanatory materials that users refer to as they progress through their studies.

[0163] "Feedback" refers to users' opinions on the learning support provided, such as their satisfaction level and suggestions for improvement.

[0164] "Interactive learning support" refers to conversational support that allows users to receive learning assistance in real time.

[0165] This invention relates to a learning support system designed to provide users with quick and appropriate support when they encounter learning difficulties. The system consists of a user, a terminal, a server, and devices (robots, smart glasses) for providing learning support in physical stores.

[0166] System Configuration

[0167] 1. User

[0168] Users refer to children who need learning support and their guardians who support them.

[0169] 2. Terminal

[0170] The device primarily uses smartphones, tablets, or personal computers, and provides technology that allows users to take pictures of learning content and input explanatory text.

[0171] 3. Server

[0172] The server is a cloud-based system and has the following features:

[0173] Text data is extracted from images using OCR (Optical Character Recognition) technology.

[0174] The extracted text data is analyzed and classified into appropriate learning categories.

[0175] It generates appropriate answers and explanations based on the categorized learning categories.

[0176] The generated answers and explanations are sent to the device.

[0177] User feedback is stored in a database to improve the accuracy of future learning support.

[0178] 4. Learning support devices in physical stores

[0179] The robots and smart glasses placed within physical stores are devices that provide users with learning support. Through these devices, users can obtain and register product information and access learning content.

[0180] Basic program processing

[0181] 1. User Registration

[0182] Users register with the system using a terminal, and this information is sent to the server. The server stores the user information in a database and assigns a unique ID to each user.

[0183] 2. Learning support request

[0184] When a user encounters a problem with their learning material, they input text or images on their device and send them to the server. For example, if they can't solve a math problem, they might take a picture of an equation written in their notebook and send it with a simple explanation like, "Please tell me how to solve this."

[0185] 3. Image and text analysis

[0186] The server analyzes the received image using OCR technology and extracts text data. It then analyzes the extracted text data to determine which learning category the problem belongs to.

[0187] 4. Generating answers and explanations

[0188] The server generates appropriate answers and explanations based on the categorized learning category. For example, if asked how to solve a quadratic equation, it will create a step-by-step explanation using the quadratic formula.

[0189] 5. Providing a response

[0190] The server sends the generated answers and explanations to the terminal, and the terminal displays the received information to the user.

[0191] 6. Obtaining and saving feedback

[0192] Users input their satisfaction level with the provided answers and explanations and send this information to the server. The server stores this feedback in a database and uses the feedback information to improve the accuracy of future support.

[0193] 7. Learning support at physical stores

[0194] In physical stores selling educational materials, users can receive learning support using terminals, robots placed in the store, or smart glasses. Users obtain information related to the materials in the store via QR codes, etc., and receive learning support based on that information.

[0195] Specific example

[0196] For example, if a user doesn't know how to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework, they can take a picture of the equation with their device, type a description like "Please tell me how to solve it," and send it to the server. The server uses OCR technology to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Next, the server generates a step-by-step explanation using the quadratic formula and sends it to the device. The device displays this explanation to the user, who then solves the problem following the explanation. Furthermore, the user inputs their satisfaction level with the explanation and sends the feedback to the server. In physical stores, learning support can be more interactive by using robots or smart glasses.

[0197] Example of a prompt

[0198] "Please scan the QR code."

[0199] "Please briefly describe the problem you are experiencing (exit to finish)."

[0200] "Please enter the image path for the teaching material."

[0201] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0202] Step 1:

[0203] Users take pictures of learning materials using their smartphone or tablet camera and enter descriptive text. This input consists of images related to the learning material and text that describes their details. This data is prepared by the device.

[0204] Step 2:

[0205] The device sends the captured image and accompanying text to the server. The input image data and text data are uploaded to a cloud-based server via the internet. The server receives this data and proceeds to the next step.

[0206] Step 3:

[0207] The server extracts text data from the received image using OCR technology. Specifically, the server uses OCR software (for example, Tesseract OCR) to recognize character information within the image and extract it as text data. The input is image data, and the output is the text data within the image.

[0208] Step 4:

[0209] The server analyzes the extracted text data and classifies it into appropriate learning categories. The server uses a natural language processing (NLP) model to analyze the text data and classify it into specific learning categories (e.g., mathematics, science, history, etc.). The input is the text data after OCR processing, and the output is the learning categories.

[0210] Step 5:

[0211] The server generates appropriate answers and explanations based on the classified learning categories. A generative AI model (e.g., GPT-3) is used to generate explanations for questions and problems related to the classified categories. The input is detailed text data of the learning categories and problems, and the output is explanations and answers to the problems.

[0212] Step 6:

[0213] The server sends the generated answers and explanations to the terminal. The text data of the explanations and answers is then sent back to the user's terminal via the internet. The input is the generated explanation data, and the output is the explanation data displayed on the terminal.

[0214] Step 7:

[0215] The terminal displays the received answers and explanations to the user. The content of the explanations and answers is displayed on the terminal screen, and the user views it. The input is the explanation data sent from the server, and the output is the explanation data displayed to the user.

[0216] Step 8:

[0217] The user inputs feedback on the answers and explanations, and sends that feedback from the device to the server. The input is the user's feedback data, which the device sends to the server. The output is the transmission of the feedback data.

[0218] Step 9:

[0219] The server stores user feedback in a database and uses it to improve the accuracy of learning support. The input is feedback data, which the server stores in the database. The output is the stored feedback, which serves as data to improve the quality of learning support in the future.

[0220] Step 10:

[0221] Users receive learning support through robots or smart glasses placed in physical stores. In the store, users obtain product information via QR codes and request learning support from a server. This enables interactive learning support. Input is product information via QR codes, and output is learning support related to a specific product.

[0222] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0223] The present invention relates to a learning support system that allows users to receive prompt and appropriate support when they experience difficulties in learning, and further recognizes the user's emotions and provides support accordingly. The embodiments for carrying out the present invention will be described in detail below.

[0224] System Configuration

[0225] This system consists of users, devices, a server, and an emotion engine. Users include children who need learning support and their parents who support them. Devices are primarily smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage. The emotion engine is used to recognize the emotional state of the users.

[0226] Basic program processing

[0227] This system's program has the following functions:

[0228] 1. User Registration

[0229] The user registers with the system using a terminal. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[0230] 2. Learning support request

[0231] If a user (child) encounters difficulties with their studies, they can use the device to take a picture of the problem and enter a brief explanation. For example, if they don't know how to solve a math problem, they can take a picture of the equation written in their notebook and send it with the explanation, "Please tell me how to solve it."

[0232] 3. Image and text analysis

[0233] The server analyzes the received image using OCR technology and extracts text data. The extracted text data is then analyzed to determine which learning category the problem belongs to.

[0234] 4. Emotion recognition

[0235] The device analyzes the user's facial recognition and voice using an emotion engine to determine the user's emotional state. For example, it recognizes if the user is anxious or confused.

[0236] 5. Generating answers and explanations

[0237] The server generates appropriate answers and explanations based on the classified learning categories and the user's emotional state recognized by the emotion engine. For example, if the user is confused, a more detailed and helpful explanation will be provided.

[0238] 6. Providing a response

[0239] The server sends the generated answers and explanations to the device. The device displays this information to the user, and the user (child) solves the problem according to the explanation.

[0240] 7. Obtaining and saving feedback

[0241] The user (child) inputs their satisfaction level with the answers and explanations and sends the feedback to their device. The server receives this feedback, stores it in a database, and uses it to improve future learning support.

[0242] 8. Saving emotional data

[0243] The emotion data recognized by the emotion engine is also sent to the server and stored in the database. This allows for personalized support in subsequent sessions.

[0244] Specific example

[0245] For example, consider a situation where a user (child) is struggling to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. The user takes a picture of the equation with their device, enters a caption saying "Please tell me how to solve it," and sends it. The server uses OCR technology to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Simultaneously, an emotion engine analyzes the user's face and voice and recognizes that the user is confused. Based on this information, the server generates a step-by-step explanation using the quadratic formula, providing a detailed explanation with particularly clear language and concrete examples. The server sends the generated explanation to the device, which displays it to the user. The user (child) solves the problem according to the displayed explanation, and after solving the problem, enters their satisfaction level and sends it. The server stores the feedback and emotion data in a database and uses it for future support.

[0246] The above describes a specific embodiment for carrying out the present invention. This system allows parents to support their children at the appropriate time, and enables children to solve problems while maintaining the enjoyment of learning. Furthermore, the introduction of an emotion engine provides personalized support according to the user's emotional state, resulting in more effective learning support.

[0247] The following describes the processing flow.

[0248] Step 1: The user registers using their device. The user enters basic information such as their name, email address, and password.

[0249] Step 2: The terminal sends the entered user information to the server. The server receives this information.

[0250] Step 3: The server saves the received user information to the database and assigns a unique ID to each user.

[0251] Step 4: The server sends a notification to the terminal that user registration is complete. The terminal displays a registration completion message to the user.

[0252] Step 5: When the user (child) encounters a learning difficulty, they can use the device to take a picture of the problem and enter a brief description (e.g., "Please tell me how to solve this problem").

[0253] Step 6: The device sends the photo and description to the server. The server receives this data.

[0254] Step 7: The server uses OCR technology to extract text data from the received photo data. This converts the text information in the photo into digital data.

[0255] Step 8: The server analyzes the extracted text data to determine which learning category the problem belongs to (e.g., mathematics, science, history, etc.).

[0256] Step 9: The device sends the user's facial recognition and voice input data to the emotion engine. The emotion engine analyzes this data and determines the user's emotional state.

[0257] Step 10: The server generates appropriate answers and explanations based on the identified learning categories and the emotional state recognized by the emotion engine. For example, if the user is confused, a more detailed and helpful explanation will be provided.

[0258] Step 11: The server sends the generated answers and explanations to the user's (child's) device. The device receives this data and displays it to the user.

[0259] Step 12: The user (child) solves the problem by following the answers and explanations displayed on the device. If there are any questions during the problem-solving process, they can send another question.

[0260] Step 13: After the user (child) solves the problem and expresses their satisfaction with the explanation, they input their feedback into the device and send it to the server. The server receives this feedback data.

[0261] Step 14: The server saves the received feedback data and the emotion data recognized by the emotion engine to a database, which is then used to improve the accuracy of future learning support.

[0262] (Example 2)

[0263] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0264] Conventional learning support systems have a problem in that it is difficult for users to receive prompt and appropriate support when they encounter difficulties during learning, and in particular, they cannot provide individualized support that addresses the user's emotional state. Furthermore, they lack a means to effectively utilize user feedback on the support provided and improve the accuracy of future learning support.

[0265] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0266] In this invention, the server includes means for extracting text data from an image using optical character recognition technology, means for analyzing the extracted text data and classifying it into an appropriate learning category, and means for analyzing facial images and voice data and recognizing the user's emotional state. This makes it possible to quickly classify difficulties experienced by the user during learning into a specific learning category and to provide personalized support tailored to the user's emotional state. Furthermore, by utilizing user feedback and emotional data, the accuracy of future learning support can be improved.

[0267] "User" refers to anyone who uses the system to receive learning support, and primarily includes children who need learning support and the parents who support them.

[0268] A "device" is a device used by a user to interact with a system, and includes smartphones, tablets, and personal computers.

[0269] A "server" refers to a cloud-based system that is responsible for processing and storing data.

[0270] Optical Character Recognition (OCR) refers to a technology that analyzes and extracts text data from images.

[0271] A "learning category" refers to the specific field or topic to which the analyzed problem belongs, such as mathematics, science, or history.

[0272] "Emotional state" refers to the psychological and emotional state recognized from the user's facial image and voice data, and includes anxiety, confusion, joy, etc.

[0273] A "generative AI model" refers to a model that uses artificial intelligence technology to automatically generate answers and explanations to user questions.

[0274] "Feedback" refers to the evaluation and opinions of users regarding the support and explanations provided.

[0275] "Face image" refers to visual data obtained from a photograph or video capture of the user's face.

[0276] "Voice data" refers to data recorded from the user's speech or voice.

[0277] A "database" refers to a system for managing and storing data such as user information, feedback, and sentiment data.

[0278] This invention is a learning support system that provides prompt and appropriate support when a user experiences difficulties during learning, and further recognizes the user's emotional state and provides individualized support accordingly. This system consists of a user, a terminal, and a server.

[0279] System Configuration

[0280] This system consists of a user, a terminal, a server, and an emotion engine. The user mainly includes children who need learning support and parents who assist those children. Smartphones, tablets, or personal computers are mainly used as the terminal. The server is a cloud-based system responsible for data processing and storage. The emotion engine is used to recognize the user's emotional state.

[0281] Basic Processing of the Program

[0282] The program of this system has the following main functions:

[0283] 1. User Registration

[0284] The user registers with the system using the terminal. The user information is sent to the server and a unique ID is assigned. The server saves this information in the database.

[0285] 2. Acceptance of Learning Support Requests

[0286] When the user (child) encounters difficulties with the learning content, they use the terminal to take a photo of the problem and enter a brief explanation. For example, when they don't know how to solve a math problem, they take a photo of the equation written in the notebook and send it with an explanation like "Please teach me how to solve it".

[0287] 3. Analysis of Images and Texts

[0288] The server analyzes the received image using optical character recognition (OCR) technology and extracts text data. For example, Tesseract OCR is used. The extracted text data is then analyzed, and a natural language processing library (such as NLTK) is used to determine which learning category the problem belongs to.

[0289] 4. Emotion recognition

[0290] The device analyzes the user's facial image and voice data using an emotion engine to determine the user's emotional state. For example, it uses facial recognition technology (such as Microsoft® Emotion API) to recognize emotions such as anxiety or confusion.

[0291] 5. Generating answers and explanations

[0292] The server uses a generative AI model to generate appropriate answers and explanations based on classified learning categories and the user's emotional state recognized by the emotion engine. For example, GPT-4® is used as one of the generative AI models. A confused user will be provided with a more detailed and helpful explanation.

[0293] 6. Providing a response

[0294] The server sends the generated answers and explanations to the terminal. The terminal displays the received information to the user, and the user (child) solves the problem according to the explanation.

[0295] 7. Obtaining and saving feedback

[0296] The user (child) inputs their satisfaction level with the answers and explanations and sends the feedback to their device. The server receives this feedback, stores it in a database, and uses it to improve future learning support.

[0297] 8. Saving emotional data

[0298] The emotion data recognized by the emotion engine is also sent to the server and stored in the database. This allows for personalized support in subsequent sessions.

[0299] Specific example

[0300] For example, consider a situation where a user (child) is struggling to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. They take a picture of the equation with their device, type a message saying "Please explain how to solve it," and send it. The server uses OCR technology (Tesseract OCR) to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Simultaneously, the emotion engine (Microsoft Emotion API) analyzes the user's face and voice, recognizing their confusion. Based on this information, the server uses a generative AI model (GPT-4) to generate a step-by-step explanation using the quadratic formula, providing detailed explanations with clear language and concrete examples. The server sends the generated explanation to the device, which displays it to the user. The user (child) solves the problem following the displayed explanation and, after solving it, inputs their satisfaction level and sends it. The server stores the feedback and emotion data in a database for future support.

[0301] For example, here is an example of a prompt statement for a generative AI model (GPT-4):

[0302] A child is having trouble solving the quadratic equation "2x² + 3x - 5 = 0". Please provide a step-by-step, detailed explanation using easy-to-understand language.

[0303] The user's emotional state is one of confusion.

[0304] With this system, parents can support their children at the appropriate timing, and children can also solve problems while maintaining the joy of learning. In addition, the introduction of the emotion engine provides personalized support according to the user's emotional state, realizing more effective learning support.

[0305] The flow of the specific process in Example 2 will be described using FIG. 13.

[0306] Step 1:

[0307] User registration

[0308] Input: User information (name, email address, age, etc.)

[0309] Operation: The user uses the terminal to register with the learning support system. The terminal sends the input information to the server. The server saves the received user information in the database and generates a unique user ID. The generated user ID is sent back to the terminal, and the terminal notifies the user that "registration is complete".

[0310] Output: A unique user ID

[0311] Step 2:

[0312] Receiving a learning support request

[0313] Input: Problem image of the learning content, a simple description of the problem

[0314] Operation: When the user (child) faces difficulties in the learning content, the user uses the terminal to take a photo of the problem and enters a description such as "Please teach me how to solve it". The terminal sends the taken image and the description to the server.

[0315] Output: The problem image and the description are sent to the server.

[0316] Step 3:

[0317] Image and text analysis

[0318] Input: Problem image, description

[0319] Operation: The server analyzes the received image using optical character recognition (OCR) technology and extracts text data. Tesseract OCR is used as the OCR technology. The extracted text data is analyzed using a natural language processing library (such as NLTK) to determine which learning category the problem belongs to.

[0320] Output: Text data, training categories

[0321] Step 4:

[0322] emotion recognition

[0323] Input: User's face image, voice data

[0324] Operation: The device captures a picture of the user's face with its camera and records their voice with its microphone. The device sends this data to the server. The server uses facial recognition technology (such as Microsoft Emotion API) to analyze the user's emotional state. The server recognizes specific emotions, such as anxiety or confusion.

[0325] Output: User's emotional state

[0326] Step 5:

[0327] Generating answers and explanations

[0328] Input: Text data, learning categories, sentiment state

[0329] Operation: The server uses a generative AI model (such as GPT-4) to generate appropriate answers and explanations based on the analyzed text data and sentiment data. For example, if the problem involves a quadratic equation, it will generate a detailed explanation using the quadratic formula.

[0330] Output: Answers and explanations

[0331] Step 6:

[0332] Providing a response

[0333] Input: Answer or explanation

[0334] Operation: The server sends the generated answers and explanations to the terminal. The terminal displays the received information to the user. The user solves the problem according to the displayed explanations.

[0335] Output: Explanation text displayed to the user

[0336] Step 7:

[0337] Obtaining and saving feedback

[0338] Input: User feedback (satisfaction ratings and comments)

[0339] Operation: After the user resolves a problem, the device displays a feedback form. The user enters a satisfaction rating and comments, and sends them to the server via the device. The server stores the received feedback in its database.

[0340] Output: Saved feedback data

[0341] Step 8:

[0342] Storage of emotional data

[0343] Input: User's emotional state data

[0344] Operation: The server stores the user's emotional data, recognized by the emotion engine, in a database. This allows for personalized support in subsequent learning support sessions.

[0345] Output: Saved sentiment data

[0346] (Application Example 2)

[0347] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0348] Conventional learning support systems have limited means of providing users with quick and appropriate support when they encounter difficulties. Furthermore, providing individualized support tailored to the user's emotional state is challenging. Additionally, in physical stores, there is a lack of means to provide appropriate information when customers have difficulty choosing products. The time spent manually searching for information contributes to decreased customer satisfaction. To address these challenges, a system is needed that provides quick and appropriate support when users encounter difficulties, and furthermore, individualized support tailored to their emotional state. Moreover, efficient information provision utilizing emotion recognition is required for product selection in physical stores.

[0349] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0350] In this invention, the server includes means for acquiring product information using a smart device, means for transmitting the product information acquired by the terminal and customer requests to the server, and means for the server to analyze the product information and customer requests and determine the emotional state using an emotion recognition engine. This enables rapid and appropriate learning support and personalized product recommendations in physical stores.

[0351] A "user" refers to an individual who needs learning support or assistance with product selection.

[0352] "Terminal" refers to devices such as smartphones, tablets, and personal computers that are operated by the user.

[0353] A "server" refers to a cloud-based system that handles data processing and storage, and generates appropriate support for users.

[0354] "Images" refers to photographic data of learning materials or products taken by the user using their device.

[0355] "Description" refers to the text data that users input regarding learning content, product issues, or requests.

[0356] "OCR technology" refers to optical character recognition technology used to extract text data from images.

[0357] "Learning categories" refer to educational fields or subjects used to classify the text data analyzed by the server.

[0358] "Answers and explanations" refers to specific explanations and suggestions regarding learning support and product information generated by the server.

[0359] "Feedback" refers to the evaluation and opinions of users regarding the support and suggestions they receive.

[0360] A "database" refers to a system used to store data such as user information and feedback.

[0361] A "smart device" refers to an internet-connected device operated by a user, such as a smartphone or smart glasses.

[0362] "Product information" refers to data about the features and details of a product.

[0363] "Customer requirements" refers to the preferences and questions that customers enter when selecting a product.

[0364] An "emotion recognition engine" refers to software that analyzes a user's emotional state from their facial expressions and voice.

[0365] "Emotional state" refers to the user's psychological state as determined by the emotion recognition engine.

[0366] "Product suggestions" refer to product suggestions generated by the server based on customer requests and emotional states.

[0367] The system of the present invention uses smart devices, servers, and an emotion recognition engine to provide learning support and in-store shopping support. The embodiments for carrying out the present invention are described in detail below.

[0368] System Configuration

[0369] This system primarily consists of a user, a smart device, a server, and an emotion engine. The user is an individual who utilizes the system for learning support or product selection. Smart devices include smartphones and smart glasses. The server is a cloud-based system responsible for data processing and storage. The emotion engine is responsible for recognizing the user's emotional state.

[0370] Basic program processing

[0371] This system's program has the following functions:

[0372] 1. User Registration

[0373] Users register with the system using their smart devices. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[0374] 2. Product support request

[0375] If a user has trouble choosing a product, they can use their smart device to scan the product's barcode or QR code and enter a simple request. For example, they might enter, "I want to know more details about this product."

[0376] 3. Image and text analysis

[0377] The server uses OCR technology to analyze the received product information and extract text data. The extracted text data is then analyzed to understand the characteristics and requirements of the requested product.

[0378] 4. Emotion recognition

[0379] Smart devices use an emotion engine to analyze the user's facial expressions and voice to determine the user's emotional state. For example, they can determine whether the user is confused or distressed.

[0380] 5. Generating answers and suggestions

[0381] The server generates appropriate responses and suggestions based on product information and the user's emotional state, as recognized by the emotion engine. For example, if the user is confused, it will provide detailed product information and suggest related products.

[0382] 6. Providing a response

[0383] The server sends generated answers and suggestions to the smart device and displays them to the user. The user can then use this information to select products.

[0384] 7. Obtaining and saving feedback

[0385] Users input their satisfaction level with the support and suggestions they receive as feedback and send it to the server from their smart device. The server stores this feedback in a database and uses it to improve future support.

[0386] Hardware and software to be used

[0387] The following hardware and software are used to implement this system:

[0388] Hardware: Smart devices (smartphones, smart glasses), cloud servers

[0389] Software: Emotion recognition engine (e.g., Microsoft Azure® Emotion API), OCR technology (e.g., Google Vision API), database system (e.g., Amazon RDS)

[0390] Specific example

[0391] For example, consider a scenario where a user is having trouble choosing a product in a physical store. Using smart glasses, the user scans the product's barcode and enters a request saying, "I want to know more details about this product." The server extracts the product's text information from the barcode using OCR technology and recognizes the user's confusion from their facial expressions and voice using an emotion engine. Based on this information, the server generates a detailed product description and related product suggestions, which are then sent to the smart glasses for the user to see. The user selects a product based on the displayed information and enters their final satisfaction level as feedback. The server saves this feedback and uses it to improve the accuracy of future support.

[0392] Example of a prompt

[0393] Prompt text to input to the generative AI model:

[0394] "If a customer appears confused, provide a detailed explanation of how to use the product and related products. For example, include product features and reviews from other customers in your suggestions."

[0395] As described above, the system of the present invention provides users with prompt and appropriate support in both learning support and in-store shopping experiences, and further enables personalized responses tailored to their emotional state.

[0396] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0397] Step 1: User Registration

[0398] Users register with the system using a smart device (e.g., a smartphone).

[0399] Input: User's basic information (name, email address, etc.)

[0400] Output: Assign a unique ID to the user and send the information to the server.

[0401] Specific operation: The user enters information into the application form and presses the "Submit" button. The terminal encrypts the entered information and sends it to the server. The server stores the received information in a database and generates a unique user ID.

[0402] Step 2: Product Support Request

[0403] When users have trouble choosing a product, they can use their smart device to scan the product's barcode or QR code and enter their request.

[0404] Input: Product barcode or QR code, request text (e.g., "I want to know more details about this product")

[0405] Output: Sends product information and requests to the server.

[0406] Specific operation: The user scans a barcode using the camera on their smart device and enters the request. The device packages the image data and text and sends it to the server.

[0407] Step 3: Image and text analysis

[0408] The server analyzes the received product information using OCR technology and extracts text data.

[0409] Input: Barcode image, text request

[0410] Output: Extracted text data

[0411] Specific operation: The server uses OCR technology such as the Google Vision API to extract text from barcode images. It then combines this with the text request and performs analysis.

[0412] Step 4: Emotion Recognition

[0413] The system analyzes the user's facial expressions and voice captured by a smart device to determine their emotional state.

[0414] Input: User's facial expression image, audio data

[0415] Output: Data on emotional state (e.g., confusion, anxiety, excitement, etc.)

[0416] Specific operation: The smart device's camera and microphone are used to collect the user's facial expressions and voice, and these are sent to an emotion recognition engine (e.g., Microsoft Azure Emotion API). The server analyzes the acquired data and determines the emotional state.

[0417] Step 5: Generating answers and suggestions

[0418] The server generates appropriate responses and suggestions based on emotional states and product information.

[0419] Input: Emotional state data, product information

[0420] Output: Text data of appropriate answers and suggestions

[0421] Specific operation: Based on emotional states and product information, the server uses pre-configured rules and generative AI models to generate responses and suggestions in a format that is easy for the user to understand.

[0422] Step 6: Providing a response

[0423] The server sends the generated answers and suggestions to the smart device and displays them to the user.

[0424] Input: Text data of generated responses and suggestions

[0425] Output: Responses and suggestions displayed on the user's smart device.

[0426] Specific operation: The server sends the generated answers and suggestions to the terminal, and the terminal displays the received data to the user. The user then selects a product based on this information.

[0427] Step 7: Obtain and save feedback

[0428] Users input feedback on the support and suggestions they receive and send it from their device to the server.

[0429] Input: Text data of ratings and opinions entered by the user.

[0430] Output: Save the feedback data to the server's database.

[0431] Specific operation: The user uses a smart device to enter a rating for support or suggestions and presses the "Submit" button. The device sends the entered feedback data to the server, which stores it in a database.

[0432] By following the steps outlined above, the program's processing is executed, allowing users to receive prompt and accurate support in learning assistance and product selection.

[0433] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0434] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0435] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0436] [Second Embodiment]

[0437] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0438] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0439] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0440] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0441] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0442] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0443] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0444] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0445] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0446] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0447] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0448] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0449] The present invention relates to a learning support system and is designed to allow users to receive prompt and appropriate support when they encounter difficulties in learning. The embodiments for carrying out the present invention are described in detail below.

[0450] System Configuration

[0451] This system consists of users, devices, and a server. Users include children who need learning support and their parents who support them. Devices are mainly smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage.

[0452] Basic program processing

[0453] This system's program has the following functions:

[0454] 1. User Registration

[0455] Users (parents and children) register with the system using a device. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[0456] 2. Learning support request

[0457] When a user (child) encounters a problem with homework or learning material, they input text or images on their device and send them to the server. For example, if a child doesn't know how to solve a math problem, they can take a picture of an equation written in their notebook, add a simple explanation like "Please tell me how to solve it," and send it.

[0458] 3. Image and text analysis

[0459] The server analyzes the received image using OCR technology and extracts text data. The extracted text data is further analyzed to determine which learning category (e.g., mathematics, science, history) the question belongs to.

[0460] 4. Generating answers and explanations

[0461] The server generates appropriate answers and explanations based on the categorized learning categories. For example, if asked how to solve a quadratic equation, it will create a step-by-step explanation using the quadratic formula.

[0462] 5. Providing a response

[0463] The server sends the generated answers and explanations to the device. The device displays the received information to the user (child). The child can then solve the problem by following these explanations.

[0464] 6. Obtaining and saving feedback

[0465] The user (child) inputs their satisfaction level with the answers and explanations on their device and sends it to the server. The server stores this feedback in a database and uses it to improve future support.

[0466] Specific example

[0467] For example, consider a case where a user (child) doesn't know how to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. The user takes a picture of the equation with their device, enters a description saying "Please tell me how to solve it," and sends it to the server. The server uses OCR technology to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Next, the server generates a step-by-step explanation using the quadratic formula and sends it to the device. The device displays this explanation to the user (child), and the child solves the problem following the explanation. Finally, the child inputs their satisfaction level with the explanation and sends the feedback to the server. The server stores this feedback in a database and uses it to improve the accuracy of future learning support.

[0468] The above describes a specific embodiment for carrying out the present invention. This system allows parents to support their children at the appropriate time, and enables children to solve problems while maintaining the enjoyment of learning.

[0469] The following describes the processing flow.

[0470] Step 1: The user registers using their device. The user enters basic information such as their name, email address, and password.

[0471] Step 2: The terminal sends the entered user information to the server. The server receives this information.

[0472] Step 3: The server saves the received user information to the database and assigns a unique ID to each user.

[0473] Step 4: The server sends a notification to the terminal that user registration is complete. The terminal displays a registration completion message to the user.

[0474] Step 5: When the user (child) encounters a learning difficulty, they can use the device to take a picture of the problem and enter a brief description (e.g., "Please tell me how to solve this problem").

[0475] Step 6: The device sends the photo and description to the server. The server receives this data.

[0476] Step 7: The server uses OCR technology to extract text data from the received photo data. This converts the text information in the photo into digital data.

[0477] Step 8: The server analyzes the extracted text data to determine which learning category the problem belongs to (e.g., mathematics, science, history, etc.).

[0478] Step 9: The server generates appropriate answers and explanations based on the identified learning category. For example, if the topic is quadratic equations in mathematics, an explanation using the quadratic formula will be generated.

[0479] Step 10: The server sends the generated answers and explanations to the user's (child's) device. The device receives this data and displays it to the user.

[0480] Step 11: The user (child) solves the problem by following the answers and explanations displayed on the device. If there are any unclear points during the problem-solving process, they can send further questions.

[0481] Step 12: After the user (child) solves the problem and expresses their satisfaction with the explanation, they input their feedback into the device and send it to the server. The server receives this feedback data.

[0482] Step 13: The server saves the received feedback data to a database and uses it to improve the accuracy of learning support in the future.

[0483] (Example 1)

[0484] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0485] Traditionally, there has been a lack of means to receive quick and accurate support when encountering particularly difficult problems in learning. The manual searching and referencing of reference books were time-consuming and slowed down the learning process. Furthermore, providing efficient support based on appropriate feedback was difficult.

[0486] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0487] In this invention, the server includes means for taking an image of the learning content that the user is having trouble with and inputting a descriptive text; means for the terminal to send the captured image and descriptive text to the server; means for the server to extract text data from the image using optical character recognition technology; means for the server to analyze the extracted text data and classify it into an appropriate learning category; means for the server to use a generative AI model to generate appropriate answers and explanations based on the classified learning categories; means for the server to send the generated answers and explanations to the terminal; means for the terminal to display the received answers and explanations to the user; and means for the terminal to send user feedback to the server and store it in a database. This makes it possible for users to receive quick and appropriate support when they encounter difficulties in their learning.

[0488] "Users" refer to people who use the learning support system, and this includes children who have difficulty learning and the parents who support them.

[0489] A "device" refers to a device used by a user, and primarily includes smartphones, tablets, or personal computers.

[0490] A "server" refers to a cloud-based system, specifically a device responsible for processing and storing data.

[0491] Optical Character Recognition (OCR) is a technology that extracts text data from images.

[0492] A "generative AI model" refers to an artificial intelligence model designed to generate appropriate answers and explanations, and includes, for example, conversational generative models.

[0493] A "learning category" refers to a type of classification used to categorize learning content, and includes academic fields such as mathematics, science, and history.

[0494] "Feedback" refers to evaluations and opinions on answers and explanations provided by users, and this data is used to improve future support.

[0495] A "database" refers to a collection of information stored on a server, including user information and feedback data.

[0496] The present invention relates to a learning support system and is designed to allow users to receive prompt and appropriate support when they encounter difficulties in learning. The embodiments for carrying out the present invention are described in detail below.

[0497] System Configuration

[0498] This system consists of users, devices, and a server. Users include children who need learning support and their parents who support them. Devices are primarily smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage.

[0499] Basic program processing

[0500] This system's program has the following functions:

[0501] 1. User Registration

[0502] Users (parents and children) register with the system using a device. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[0503] 2. Learning support request

[0504] When a user (child) encounters a problem with their homework or learning material, they input text or images on their device and send them to the server. For example, if a child doesn't know how to solve a math problem, they might take a picture of an equation written in their notebook, add a simple explanation like "Please tell me how to solve it," and send it.

[0505] 3. Image and text analysis

[0506] The server analyzes the received images using OCR technology and extracts text data. The primary software used is optical character recognition (e.g., Google Cloud Vision API). The extracted text data is further analyzed to determine which learning category (e.g., mathematics, science, history) the question belongs to.

[0507] 4. Generating answers and explanations

[0508] The server generates appropriate answers and explanations based on the classified learning categories. For example, if asked how to solve a quadratic equation, it will create a step-by-step explanation using the quadratic formula. For this purpose, it utilizes a generative AI model (e.g., OpenAI GPT-3).

[0509] 5. Providing a response

[0510] The server sends the generated answers and explanations to the terminal. The terminal displays the received information to the user (child). The user (child) can then solve the problem by following these explanations.

[0511] 6. Obtaining and saving feedback

[0512] The user (child) inputs their satisfaction level with the answers and explanations on their device and sends it to the server. The server stores this feedback in a database and uses it to improve future support.

[0513] Specific example

[0514] For example, consider a case where a user (a child) doesn't know how to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. The system will process it in the following steps:

[0515] 1. The user (child) takes a picture of this equation with their device, enters the description "Tell me how to solve it," and sends it to the server.

[0516] 2. The server uses OCR technology to extract the text data "2x² + 3x - 5 = 0" from the image and identifies the problem as a quadratic equation.

[0517] 3. The server generates a step-by-step explanation using the solution formula and sends it to the terminal.

[0518] 4. The device displays this explanation to the user (child), and the child solves the problem according to the explanation.

[0519] 5. The user (child) enters their satisfaction level with the explanation and sends feedback to the server.

[0520] 6. The server saves the feedback to a database and uses it to improve future learning support.

[0521] Example of a prompt

[0522] Assume the user will input the following prompt into the generated AI model:

[0523] "Please explain how to solve this quadratic equation: 2x² + 3x - 5 = 0."

[0524] Based on this prompt, the server generates a detailed explanation and provides it to the user through the terminal.

[0525] The above describes a specific embodiment for carrying out the present invention. This process makes it possible to provide learning support quickly and effectively.

[0526] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0527] Step 1: User Registration

[0528] 1. Input: Users (parents and children) use the device to enter basic information such as name, email address, and password.

[0529] 2. Specific actions:

[0530] The user launches a web browser or dedicated app and opens the user registration screen.

[0531] The user clicks the "Register" button.

[0532] 3. Data Processing and Output: The terminal sends the input information to the server. The server stores the received information in a database and generates and records a unique user ID. The server sends a success message to the terminal, and the terminal displays a registration completion message to the user.

[0533] Step 2: Request learning support

[0534] 1. Input: The user (child) enters the text of the problem or a photo of the notebook containing the problem, along with a brief description, on the device.

[0535] 2. Specific actions:

[0536] The user (child) taps the "Request Support" button within the app.

[0537] The user (child) takes a picture of the problem with a note that says, "Please tell me how to solve it."

[0538] 3. Data Processing and Output: The terminal sends the input image, description, and user ID data to the server. The server passes the received data to the analysis engine.

[0539] Step 3: Image and text analysis

[0540] 1. Input: The server receives image and description data from the terminal.

[0541] 2. Specific actions:

[0542] The server analyzes the image using optical character recognition technology (e.g., Google Cloud Vision API) and extracts the text data.

[0543] 3. Data Processing and Output: The server analyzes the extracted text data and uses a software engine to determine which learning category (e.g., mathematics, science, history) the problem belongs to. The category information is saved and passed on to the next step.

[0544] Step 4: Generating answers and explanations

[0545] 1. Input: The server sends a prompt to the generative AI model (e.g., OpenAI GPT-3) based on the identified learning categories and extracted text data.

[0546] 2. Specific actions:

[0547] The server sends the prompt message: "Please tell me how to solve the quadratic equation 2x² + 3x - 5 = 0." to the generating AI model.

[0548] 3. Data Processing and Output: The generative AI model generates appropriate answers and explanations in response to prompts. The server evaluates the validity of the generated answers and explanations, formats them appropriately, and then sends them to the terminal.

[0549] Step 5: Providing a response

[0550] 1. Input: The terminal receives answers and explanations sent from the server.

[0551] 2. Specific actions:

[0552] The terminal displays the received information on the user interface.

[0553] The user (child) reviews and understands the displayed explanation.

[0554] 3. Data processing and output: The user (child) solves the problem based on the explanation.

[0555] Step 6: Obtain and save feedback

[0556] 1. Input: The user (child) inputs their satisfaction level with the explanation using the terminal.

[0557] 2. Specific actions:

[0558] The user (child) chooses one option from choices such as "very satisfied," "satisfied," or "dissatisfied."

[0559] The user (child) clicks the "Send" button.

[0560] 3. Data Processing and Output: The terminal sends the input feedback to the server. The server receives the feedback and stores it in a database. The server uses this feedback to improve the accuracy of future learning support.

[0561] (Application Example 1)

[0562] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0563] Traditional learning support systems offer limited means for users to receive prompt and appropriate support when they encounter difficulties. Furthermore, there is a lack of effective learning support methods in physical stores, resulting in insufficient support for customers to effectively utilize learning materials. There is also a need for improved support accuracy based on feedback and the provision of interactive learning experiences for in-store use.

[0564] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0565] In this invention, the server includes means for taking an image of the learning content that the user is having trouble with and inputting a descriptive text; means for the terminal to send the captured image and descriptive text to the server; means for the server to extract text data from the image using OCR technology; means for the server to analyze the extracted text data and classify it into an appropriate learning category; means for the server to generate appropriate answers and explanations based on the classified learning categories; means for the server to send the generated answers and explanations to the terminal; means for the terminal to display the received answers and explanations to the user; means for the terminal to send user feedback to the server and store it in a database; means for the user to receive learning support through a robot or smart glasses placed in a physical store; and means for acquiring and registering product information and accessing learning content. This enables interactive learning support in physical stores, allowing users to effectively solve problems and improve the accuracy of support by utilizing feedback information.

[0566] "Users" refer to children who receive learning support using the learning support system, and their guardians who provide that support.

[0567] "Device" refers to electronic devices used by users, such as smartphones, tablets, and personal computers.

[0568] A "server" refers to a cloud-based system, specifically a device that processes and stores data sent by users.

[0569] "OCR technology" refers to optical character recognition technology that automatically extracts text data from images.

[0570] "Robot" refers to a device placed in a physical store that provides learning support while interacting with users.

[0571] "Smart glasses" refer to devices that users wear to display information from learning materials or to acquire information using the user's gaze.

[0572] "Product information" refers to detailed information about learning materials and products, and is obtained through methods such as QR codes.

[0573] "Learning content" refers to information such as textbooks, workbooks, and explanatory materials that users refer to as they progress through their studies.

[0574] "Feedback" refers to users' opinions on the learning support provided, such as their satisfaction level and suggestions for improvement.

[0575] "Interactive learning support" refers to conversational support that allows users to receive learning assistance in real time.

[0576] This invention relates to a learning support system designed to provide users with quick and appropriate support when they encounter learning difficulties. The system consists of a user, a terminal, a server, and devices (robots, smart glasses) for providing learning support in physical stores.

[0577] System Configuration

[0578] 1. User

[0579] Users refer to children who need learning support and their guardians who support them.

[0580] 2. Terminal

[0581] The device primarily uses smartphones, tablets, or personal computers, and provides technology that allows users to take pictures of learning content and input explanatory text.

[0582] 3. Server

[0583] The server is a cloud-based system and has the following features:

[0584] Text data is extracted from images using OCR (Optical Character Recognition) technology.

[0585] The extracted text data is analyzed and classified into appropriate learning categories.

[0586] It generates appropriate answers and explanations based on the categorized learning categories.

[0587] The generated answers and explanations are sent to the device.

[0588] User feedback is stored in a database to improve the accuracy of future learning support.

[0589] 4. Learning support devices in physical stores

[0590] The robots and smart glasses placed within physical stores are devices that provide users with learning support. Through these devices, users can obtain and register product information and access learning content.

[0591] Basic program processing

[0592] 1. User Registration

[0593] Users register with the system using a terminal, and this information is sent to the server. The server stores the user information in a database and assigns a unique ID to each user.

[0594] 2. Learning support request

[0595] When a user encounters a problem with their learning material, they input text or images on their device and send them to the server. For example, if they can't solve a math problem, they might take a picture of an equation written in their notebook and send it with a simple explanation like, "Please tell me how to solve this."

[0596] 3. Image and text analysis

[0597] The server analyzes the received image using OCR technology and extracts text data. It then analyzes the extracted text data to determine which learning category the problem belongs to.

[0598] 4. Generating answers and explanations

[0599] The server generates appropriate answers and explanations based on the categorized learning category. For example, if asked how to solve a quadratic equation, it will create a step-by-step explanation using the quadratic formula.

[0600] 5. Providing a response

[0601] The server sends the generated answers and explanations to the terminal, and the terminal displays the received information to the user.

[0602] 6. Obtaining and saving feedback

[0603] Users input their satisfaction level with the provided answers and explanations and send this information to the server. The server stores this feedback in a database and uses the feedback information to improve the accuracy of future support.

[0604] 7. Learning support at physical stores

[0605] In physical stores selling educational materials, users can receive learning support using terminals, robots placed in the store, or smart glasses. Users obtain information related to the materials in the store via QR codes, etc., and receive learning support based on that information.

[0606] Specific example

[0607] For example, if a user doesn't know how to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework, they can take a picture of the equation with their device, type a description like "Please tell me how to solve it," and send it to the server. The server uses OCR technology to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Next, the server generates a step-by-step explanation using the quadratic formula and sends it to the device. The device displays this explanation to the user, who then solves the problem following the explanation. Furthermore, the user inputs their satisfaction level with the explanation and sends the feedback to the server. In physical stores, learning support can be more interactive by using robots or smart glasses.

[0608] Example of a prompt

[0609] "Please scan the QR code."

[0610] "Please briefly describe the problem you are experiencing (exit to finish)."

[0611] "Please enter the image path for the teaching material."

[0612] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0613] Step 1:

[0614] Users take pictures of learning materials using their smartphone or tablet camera and enter descriptive text. This input consists of images related to the learning material and text that describes their details. This data is prepared by the device.

[0615] Step 2:

[0616] The device sends the captured image and accompanying text to the server. The input image data and text data are uploaded to a cloud-based server via the internet. The server receives this data and proceeds to the next step.

[0617] Step 3:

[0618] The server extracts text data from the received image using OCR technology. Specifically, the server uses OCR software (for example, Tesseract OCR) to recognize character information within the image and extract it as text data. The input is image data, and the output is the text data within the image.

[0619] Step 4:

[0620] The server analyzes the extracted text data and classifies it into appropriate learning categories. The server uses a natural language processing (NLP) model to analyze the text data and classify it into specific learning categories (e.g., mathematics, science, history, etc.). The input is the text data after OCR processing, and the output is the learning categories.

[0621] Step 5:

[0622] The server generates appropriate answers and explanations based on the classified learning categories. A generative AI model (e.g., GPT-3) is used to generate explanations for questions and problems related to the classified categories. The input is detailed text data of the learning categories and problems, and the output is explanations and answers to the problems.

[0623] Step 6:

[0624] The server sends the generated answers and explanations to the terminal. The text data of the explanations and answers is then sent back to the user's terminal via the internet. The input is the generated explanation data, and the output is the explanation data displayed on the terminal.

[0625] Step 7:

[0626] The terminal displays the received answers and explanations to the user. The content of the explanations and answers is displayed on the terminal screen, and the user views it. The input is the explanation data sent from the server, and the output is the explanation data displayed to the user.

[0627] Step 8:

[0628] The user inputs feedback on the answers and explanations, and sends that feedback from the device to the server. The input is the user's feedback data, which the device sends to the server. The output is the transmission of the feedback data.

[0629] Step 9:

[0630] The server stores user feedback in a database and uses it to improve the accuracy of learning support. The input is feedback data, which the server stores in the database. The output is the stored feedback, which serves as data to improve the quality of learning support in the future.

[0631] Step 10:

[0632] Users receive learning support through robots or smart glasses placed in physical stores. In the store, users obtain product information via QR codes and request learning support from a server. This enables interactive learning support. Input is product information via QR codes, and output is learning support related to a specific product.

[0633] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0634] The present invention relates to a learning support system that allows users to receive prompt and appropriate support when they experience difficulties in learning, and further recognizes the user's emotions and provides support accordingly. The embodiments for carrying out the present invention will be described in detail below.

[0635] System Configuration

[0636] This system consists of users, devices, a server, and an emotion engine. Users include children who need learning support and their parents who support them. Devices are primarily smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage. The emotion engine is used to recognize the emotional state of the users.

[0637] Basic program processing

[0638] This system's program has the following functions:

[0639] 1. User Registration

[0640] The user registers with the system using a terminal. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[0641] 2. Learning support request

[0642] If a user (child) encounters difficulties with their studies, they can use the device to take a picture of the problem and enter a brief explanation. For example, if they don't know how to solve a math problem, they can take a picture of the equation written in their notebook and send it with the explanation, "Please tell me how to solve it."

[0643] 3. Image and text analysis

[0644] The server analyzes the received image using OCR technology and extracts text data. The extracted text data is then analyzed to determine which learning category the problem belongs to.

[0645] 4. Emotion recognition

[0646] The device analyzes the user's facial recognition and voice using an emotion engine to determine the user's emotional state. For example, it recognizes if the user is anxious or confused.

[0647] 5. Generating answers and explanations

[0648] The server generates appropriate answers and explanations based on the classified learning categories and the user's emotional state recognized by the emotion engine. For example, if the user is confused, a more detailed and helpful explanation will be provided.

[0649] 6. Providing a response

[0650] The server sends the generated answers and explanations to the device. The device displays this information to the user, and the user (child) solves the problem according to the explanation.

[0651] 7. Obtaining and saving feedback

[0652] The user (child) inputs their satisfaction level with the answers and explanations and sends the feedback to their device. The server receives this feedback, stores it in a database, and uses it to improve future learning support.

[0653] 8. Saving emotional data

[0654] The emotion data recognized by the emotion engine is also sent to the server and stored in the database. This allows for personalized support in subsequent sessions.

[0655] Specific example

[0656] For example, consider a situation where a user (child) is struggling to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. The user takes a picture of the equation with their device, enters a caption saying "Please tell me how to solve it," and sends it. The server uses OCR technology to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Simultaneously, an emotion engine analyzes the user's face and voice and recognizes that the user is confused. Based on this information, the server generates a step-by-step explanation using the quadratic formula, providing a detailed explanation with particularly clear language and concrete examples. The server sends the generated explanation to the device, which displays it to the user. The user (child) solves the problem according to the displayed explanation, and after solving the problem, enters their satisfaction level and sends it. The server stores the feedback and emotion data in a database and uses it for future support.

[0657] The above describes a specific embodiment for carrying out the present invention. This system allows parents to support their children at the appropriate time, and enables children to solve problems while maintaining the enjoyment of learning. Furthermore, the introduction of an emotion engine provides personalized support according to the user's emotional state, resulting in more effective learning support.

[0658] The following describes the processing flow.

[0659] Step 1: The user registers using their device. The user enters basic information such as their name, email address, and password.

[0660] Step 2: The terminal sends the entered user information to the server. The server receives this information.

[0661] Step 3: The server saves the received user information to the database and assigns a unique ID to each user.

[0662] Step 4: The server sends a notification to the terminal that user registration is complete. The terminal displays a registration completion message to the user.

[0663] Step 5: When the user (child) encounters a learning difficulty, they can use the device to take a picture of the problem and enter a brief description (e.g., "Please tell me how to solve this problem").

[0664] Step 6: The device sends the photo and description to the server. The server receives this data.

[0665] Step 7: The server uses OCR technology to extract text data from the received photo data. This converts the text information in the photo into digital data.

[0666] Step 8: The server analyzes the extracted text data to determine which learning category the problem belongs to (e.g., mathematics, science, history, etc.).

[0667] Step 9: The device sends the user's facial recognition and voice input data to the emotion engine. The emotion engine analyzes this data and determines the user's emotional state.

[0668] Step 10: The server generates appropriate answers and explanations based on the identified learning categories and the emotional state recognized by the emotion engine. For example, if the user is confused, a more detailed and helpful explanation will be provided.

[0669] Step 11: The server sends the generated answers and explanations to the user's (child's) device. The device receives this data and displays it to the user.

[0670] Step 12: The user (child) solves the problem by following the answers and explanations displayed on the device. If there are any questions during the problem-solving process, they can send another question.

[0671] Step 13: After the user (child) solves the problem and expresses their satisfaction with the explanation, they input their feedback into the device and send it to the server. The server receives this feedback data.

[0672] Step 14: The server saves the received feedback data and the emotion data recognized by the emotion engine to a database, which is then used to improve the accuracy of future learning support.

[0673] (Example 2)

[0674] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0675] Conventional learning support systems have a problem in that it is difficult for users to receive prompt and appropriate support when they encounter difficulties during learning, and in particular, they cannot provide individualized support that addresses the user's emotional state. Furthermore, they lack a means to effectively utilize user feedback on the support provided and improve the accuracy of future learning support.

[0676] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0677] In this invention, the server includes means for extracting text data from an image using optical character recognition technology, means for analyzing the extracted text data and classifying it into an appropriate learning category, and means for analyzing facial images and voice data and recognizing the user's emotional state. This makes it possible to quickly classify difficulties experienced by the user during learning into a specific learning category and to provide personalized support tailored to the user's emotional state. Furthermore, by utilizing user feedback and emotional data, the accuracy of future learning support can be improved.

[0678] "User" refers to anyone who uses the system to receive learning support, and primarily includes children who need learning support and the parents who support them.

[0679] A "device" is a device used by a user to interact with a system, and includes smartphones, tablets, and personal computers.

[0680] A "server" refers to a cloud-based system that is responsible for processing and storing data.

[0681] Optical Character Recognition (OCR) refers to a technology that analyzes and extracts text data from images.

[0682] A "learning category" refers to the specific field or topic to which the analyzed problem belongs, such as mathematics, science, or history.

[0683] "Emotional state" refers to the psychological and emotional state recognized from the user's facial image and voice data, and includes anxiety, confusion, joy, etc.

[0684] A "generative AI model" refers to a model that uses artificial intelligence technology to automatically generate answers and explanations to user questions.

[0685] "Feedback" refers to the evaluation and opinions of users regarding the support and explanations provided.

[0686] "Face image" refers to visual data obtained from a photograph or video capture of the user's face.

[0687] "Voice data" refers to data recorded from the user's speech or voice.

[0688] A "database" refers to a system for managing and storing data such as user information, feedback, and sentiment data.

[0689] This invention is a learning support system that provides prompt and appropriate support when a user encounters difficulties during learning, and further recognizes the user's emotional state and provides personalized support accordingly. This system consists of a user, a terminal, and a server.

[0690] System Configuration

[0691] This system consists of users, devices, a server, and an emotion engine. Users primarily include children who need learning support and their parents who support them. Devices mainly consist of smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage. The emotion engine is used to recognize the emotional state of the users.

[0692] Basic program processing

[0693] This system's program has the following main functions:

[0694] 1. User Registration

[0695] Users register with the system using a terminal. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[0696] 2. Acceptance of learning support requests

[0697] If a user (child) encounters difficulties with their studies, they can use the device to take a picture of the problem and enter a brief explanation. For example, if they don't know how to solve a math problem, they can take a picture of the equation written in their notebook and send it with the explanation, "Please tell me how to solve it."

[0698] 3. Image and text analysis

[0699] The server analyzes the received image using optical character recognition (OCR) technology and extracts text data. For example, Tesseract OCR is used. The extracted text data is then analyzed, and a natural language processing library (such as NLTK) is used to determine which learning category the problem belongs to.

[0700] 4. Emotion recognition

[0701] The device analyzes the user's facial image and voice data using an emotion engine to determine the user's emotional state. For example, it uses facial recognition technology (such as the Microsoft Emotion API) to recognize emotions such as anxiety or confusion.

[0702] 5. Generating answers and explanations

[0703] The server uses a generative AI model to generate appropriate answers and explanations based on classified learning categories and the user's emotional state recognized by the emotion engine. For example, GPT-4 is used as one of the generative AI models. A confused user will be provided with a more detailed and helpful explanation.

[0704] 6. Providing a response

[0705] The server sends the generated answers and explanations to the terminal. The terminal displays the received information to the user, and the user (child) solves the problem according to the explanation.

[0706] 7. Obtaining and saving feedback

[0707] The user (child) inputs their satisfaction level with the answers and explanations and sends the feedback to their device. The server receives this feedback, stores it in a database, and uses it to improve future learning support.

[0708] 8. Saving emotional data

[0709] The emotion data recognized by the emotion engine is also sent to the server and stored in the database. This allows for personalized support in subsequent sessions.

[0710] Specific example

[0711] For example, consider a situation where a user (child) is struggling to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. They take a picture of the equation with their device, type a message saying "Please explain how to solve it," and send it. The server uses OCR technology (Tesseract OCR) to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Simultaneously, the emotion engine (Microsoft Emotion API) analyzes the user's face and voice, recognizing their confusion. Based on this information, the server uses a generative AI model (GPT-4) to generate a step-by-step explanation using the quadratic formula, providing detailed explanations with clear language and concrete examples. The server sends the generated explanation to the device, which displays it to the user. The user (child) solves the problem following the displayed explanation and, after solving it, inputs their satisfaction level and sends it. The server stores the feedback and emotion data in a database for future support.

[0712] For example, here is an example of a prompt statement for a generative AI model (GPT-4):

[0713] A child is having trouble solving the quadratic equation "2x² + 3x - 5 = 0". Please provide a step-by-step, detailed explanation using easy-to-understand language.

[0714] The user's emotional state is one of confusion.

[0715] This system allows parents to support their children at the right time, and enables children to solve problems while maintaining the enjoyment of learning. Furthermore, the introduction of an emotion engine provides personalized support tailored to the user's emotional state, resulting in more effective learning assistance.

[0716] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0717] Step 1:

[0718] User Registration

[0719] Input: User information (name, email address, age, etc.)

[0720] Operation: The user registers with the learning support system using a terminal. The terminal sends the entered information to the server. The server stores the received user information in a database and generates a unique user ID. The generated user ID is sent back to the terminal, and the terminal notifies the user that "registration is complete."

[0721] Output: Unique User ID

[0722] Step 2:

[0723] Acceptance of learning support request

[0724] Input: Image of the learning material problem, a brief description of the problem.

[0725] Operation: When the user (child) encounters difficulty with the learning material, they use the device to take a picture of the problem and enter a description such as "Please tell me how to solve it." The device then sends the captured image and description to the server.

[0726] Output: The problem image and description are sent to the server.

[0727] Step 3:

[0728] Image and text analysis

[0729] Input: Problem image, description

[0730] Operation: The server analyzes the received image using optical character recognition (OCR) technology and extracts text data. Tesseract OCR is used as the OCR technology. The extracted text data is analyzed using a natural language processing library (such as NLTK) to determine which learning category the problem belongs to.

[0731] Output: Text data, training categories

[0732] Step 4:

[0733] emotion recognition

[0734] Input: User's face image, voice data

[0735] Operation: The device captures a picture of the user's face with its camera and records their voice with its microphone. The device sends this data to the server. The server uses facial recognition technology (such as Microsoft Emotion API) to analyze the user's emotional state. The server recognizes specific emotions, such as anxiety or confusion.

[0736] Output: User's emotional state

[0737] Step 5:

[0738] Generating answers and explanations

[0739] Input: Text data, learning categories, sentiment state

[0740] Operation: The server uses a generative AI model (such as GPT-4) to generate appropriate answers and explanations based on the analyzed text data and sentiment data. For example, if the problem involves a quadratic equation, it will generate a detailed explanation using the quadratic formula.

[0741] Output: Answers and explanations

[0742] Step 6:

[0743] Providing a response

[0744] Input: Answer or explanation

[0745] Operation: The server sends the generated answers and explanations to the terminal. The terminal displays the received information to the user. The user solves the problem according to the displayed explanations.

[0746] Output: Explanation text displayed to the user

[0747] Step 7:

[0748] Obtaining and saving feedback

[0749] Input: User feedback (satisfaction ratings and comments)

[0750] Operation: After the user resolves a problem, the device displays a feedback form. The user enters a satisfaction rating and comments, and sends them to the server via the device. The server stores the received feedback in its database.

[0751] Output: Saved feedback data

[0752] Step 8:

[0753] Storage of emotional data

[0754] Input: User's emotional state data

[0755] Operation: The server stores the user's emotional data, recognized by the emotion engine, in a database. This allows for personalized support in subsequent learning support sessions.

[0756] Output: Saved sentiment data

[0757] (Application Example 2)

[0758] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0759] Conventional learning support systems have limited means of providing users with quick and appropriate support when they encounter difficulties. Furthermore, providing individualized support tailored to the user's emotional state is challenging. Additionally, in physical stores, there is a lack of means to provide appropriate information when customers have difficulty choosing products. The time spent manually searching for information contributes to decreased customer satisfaction. To address these challenges, a system is needed that provides quick and appropriate support when users encounter difficulties, and furthermore, individualized support tailored to their emotional state. Moreover, efficient information provision utilizing emotion recognition is required for product selection in physical stores.

[0760] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0761] In this invention, the server includes means for acquiring product information using a smart device, means for transmitting the product information acquired by the terminal and customer requests to the server, and means for the server to analyze the product information and customer requests and determine the emotional state using an emotion recognition engine. This enables rapid and appropriate learning support and personalized product recommendations in physical stores.

[0762] A "user" refers to an individual who needs learning support or assistance with product selection.

[0763] "Terminal" refers to devices such as smartphones, tablets, and personal computers that are operated by the user.

[0764] A "server" refers to a cloud-based system that handles data processing and storage, and generates appropriate support for users.

[0765] "Images" refers to photographic data of learning materials or products taken by the user using their device.

[0766] "Description" refers to the text data that users input regarding learning content, product issues, or requests.

[0767] "OCR technology" refers to optical character recognition technology used to extract text data from images.

[0768] "Learning categories" refer to educational fields or subjects used to classify the text data analyzed by the server.

[0769] "Answers and explanations" refers to specific explanations and suggestions regarding learning support and product information generated by the server.

[0770] "Feedback" refers to the evaluation and opinions of users regarding the support and suggestions they receive.

[0771] A "database" refers to a system used to store data such as user information and feedback.

[0772] A "smart device" refers to an internet-connected device operated by a user, such as a smartphone or smart glasses.

[0773] "Product information" refers to data about the features and details of a product.

[0774] "Customer requirements" refers to the preferences and questions that customers enter when selecting a product.

[0775] An "emotion recognition engine" refers to software that analyzes a user's emotional state from their facial expressions and voice.

[0776] "Emotional state" refers to the user's psychological state as determined by the emotion recognition engine.

[0777] "Product suggestions" refer to product suggestions generated by the server based on customer requests and emotional states.

[0778] The system of the present invention uses smart devices, servers, and an emotion recognition engine to provide learning support and in-store shopping support. The embodiments for carrying out the present invention are described in detail below.

[0779] System Configuration

[0780] This system primarily consists of a user, a smart device, a server, and an emotion engine. The user is an individual who utilizes the system for learning support or product selection. Smart devices include smartphones and smart glasses. The server is a cloud-based system responsible for data processing and storage. The emotion engine is responsible for recognizing the user's emotional state.

[0781] Basic program processing

[0782] This system's program has the following functions:

[0783] 1. User Registration

[0784] Users register with the system using their smart devices. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[0785] 2. Product support request

[0786] If a user has trouble choosing a product, they can use their smart device to scan the product's barcode or QR code and enter a simple request. For example, they might enter, "I want to know more details about this product."

[0787] 3. Image and text analysis

[0788] The server uses OCR technology to analyze the received product information and extract text data. The extracted text data is then analyzed to understand the characteristics and requirements of the requested product.

[0789] 4. Emotion recognition

[0790] Smart devices use an emotion engine to analyze the user's facial expressions and voice to determine the user's emotional state. For example, they can determine whether the user is confused or distressed.

[0791] 5. Generating answers and suggestions

[0792] The server generates appropriate responses and suggestions based on product information and the user's emotional state, as recognized by the emotion engine. For example, if the user is confused, it will provide detailed product information and suggest related products.

[0793] 6. Providing a response

[0794] The server sends generated answers and suggestions to the smart device and displays them to the user. The user can then use this information to select products.

[0795] 7. Obtaining and saving feedback

[0796] Users input their satisfaction level with the support and suggestions they receive as feedback and send it to the server from their smart device. The server stores this feedback in a database and uses it to improve future support.

[0797] Hardware and software to be used

[0798] The following hardware and software are used to implement this system:

[0799] Hardware: Smart devices (smartphones, smart glasses), cloud servers

[0800] Software: Emotion recognition engine (e.g., Microsoft Azure Emotion API), OCR technology (e.g., Google Vision API), database system (e.g., Amazon RDS)

[0801] Specific example

[0802] For example, consider a scenario where a user is having trouble choosing a product in a physical store. Using smart glasses, the user scans the product's barcode and enters a request saying, "I want to know more details about this product." The server extracts the product's text information from the barcode using OCR technology and recognizes the user's confusion from their facial expressions and voice using an emotion engine. Based on this information, the server generates a detailed product description and related product suggestions, which are then sent to the smart glasses for the user to see. The user selects a product based on the displayed information and enters their final satisfaction level as feedback. The server saves this feedback and uses it to improve the accuracy of future support.

[0803] Example of a prompt

[0804] Prompt text to input to the generative AI model:

[0805] "If a customer appears confused, provide a detailed explanation of how to use the product and related products. For example, include product features and reviews from other customers in your suggestions."

[0806] As described above, the system of the present invention provides users with prompt and appropriate support in both learning support and in-store shopping experiences, and further enables personalized responses tailored to their emotional state.

[0807] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0808] Step 1: User Registration

[0809] Users register with the system using a smart device (e.g., a smartphone).

[0810] Input: User's basic information (name, email address, etc.)

[0811] Output: Assign a unique ID to the user and send the information to the server.

[0812] Specific operation: The user enters information into the application form and presses the "Submit" button. The terminal encrypts the entered information and sends it to the server. The server stores the received information in a database and generates a unique user ID.

[0813] Step 2: Product Support Request

[0814] When users have trouble choosing a product, they can use their smart device to scan the product's barcode or QR code and enter their request.

[0815] Input: Product barcode or QR code, request text (e.g., "I want to know more details about this product")

[0816] Output: Sends product information and requests to the server.

[0817] Specific operation: The user scans a barcode using the camera on their smart device and enters the request. The device packages the image data and text and sends it to the server.

[0818] Step 3: Image and text analysis

[0819] The server analyzes the received product information using OCR technology and extracts text data.

[0820] Input: Barcode image, text request

[0821] Output: Extracted text data

[0822] Specific operation: The server uses OCR technology such as the Google Vision API to extract text from barcode images. It then combines this with the text request and performs analysis.

[0823] Step 4: Emotion Recognition

[0824] The system analyzes the user's facial expressions and voice captured by a smart device to determine their emotional state.

[0825] Input: User's facial expression image, audio data

[0826] Output: Data on emotional state (e.g., confusion, anxiety, excitement, etc.)

[0827] Specific operation: The smart device's camera and microphone are used to collect the user's facial expressions and voice, and these are sent to an emotion recognition engine (e.g., Microsoft Azure Emotion API). The server analyzes the acquired data and determines the emotional state.

[0828] Step 5: Generating answers and suggestions

[0829] The server generates appropriate responses and suggestions based on emotional states and product information.

[0830] Input: Emotional state data, product information

[0831] Output: Text data of appropriate answers and suggestions

[0832] Specific operation: Based on emotional states and product information, the server uses pre-configured rules and generative AI models to generate responses and suggestions in a format that is easy for the user to understand.

[0833] Step 6: Providing a response

[0834] The server sends the generated answers and suggestions to the smart device and displays them to the user.

[0835] Input: Text data of generated responses and suggestions

[0836] Output: Responses and suggestions displayed on the user's smart device.

[0837] Specific operation: The server sends the generated answers and suggestions to the terminal, and the terminal displays the received data to the user. The user then selects a product based on this information.

[0838] Step 7: Obtain and save feedback

[0839] Users input feedback on the support and suggestions they receive and send it from their device to the server.

[0840] Input: Text data of ratings and opinions entered by the user.

[0841] Output: Save the feedback data to the server's database.

[0842] Specific operation: The user uses a smart device to enter a rating for support or suggestions and presses the "Submit" button. The device sends the entered feedback data to the server, which stores it in a database.

[0843] By following the steps outlined above, the program's processing is executed, allowing users to receive prompt and accurate support in learning assistance and product selection.

[0844] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0845] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0846] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0847] [Third Embodiment]

[0848] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0849] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0850] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0851] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0852] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0853] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0854] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0855] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0856] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0857] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0858] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0859] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0860] The present invention relates to a learning support system and is designed to allow users to receive prompt and appropriate support when they encounter difficulties in learning. The embodiments for carrying out the present invention are described in detail below.

[0861] System Configuration

[0862] This system consists of users, devices, and a server. Users include children who need learning support and their parents who support them. Devices are mainly smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage.

[0863] Basic program processing

[0864] This system's program has the following functions:

[0865] 1. User Registration

[0866] Users (parents and children) register with the system using a device. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[0867] 2. Learning support request

[0868] When a user (child) encounters a problem with homework or learning material, they input text or images on their device and send them to the server. For example, if a child doesn't know how to solve a math problem, they can take a picture of an equation written in their notebook, add a simple explanation like "Please tell me how to solve it," and send it.

[0869] 3. Image and text analysis

[0870] The server analyzes the received image using OCR technology and extracts text data. The extracted text data is further analyzed to determine which learning category (e.g., mathematics, science, history) the question belongs to.

[0871] 4. Generating answers and explanations

[0872] The server generates appropriate answers and explanations based on the categorized learning categories. For example, if asked how to solve a quadratic equation, it will create a step-by-step explanation using the quadratic formula.

[0873] 5. Providing a response

[0874] The server sends the generated answers and explanations to the device. The device displays the received information to the user (child). The child can then solve the problem by following these explanations.

[0875] 6. Obtaining and saving feedback

[0876] The user (child) inputs their satisfaction level with the answers and explanations on their device and sends it to the server. The server stores this feedback in a database and uses it to improve future support.

[0877] Specific example

[0878] For example, consider a case where a user (child) doesn't know how to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. The user takes a picture of the equation with their device, enters a description saying "Please tell me how to solve it," and sends it to the server. The server uses OCR technology to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Next, the server generates a step-by-step explanation using the quadratic formula and sends it to the device. The device displays this explanation to the user (child), and the child solves the problem following the explanation. Finally, the child inputs their satisfaction level with the explanation and sends the feedback to the server. The server stores this feedback in a database and uses it to improve the accuracy of future learning support.

[0879] The above describes a specific embodiment for carrying out the present invention. This system allows parents to support their children at the appropriate time, and enables children to solve problems while maintaining the enjoyment of learning.

[0880] The following describes the processing flow.

[0881] Step 1: The user registers using their device. The user enters basic information such as their name, email address, and password.

[0882] Step 2: The terminal sends the entered user information to the server. The server receives this information.

[0883] Step 3: The server saves the received user information to the database and assigns a unique ID to each user.

[0884] Step 4: The server sends a notification to the terminal that user registration is complete. The terminal displays a registration completion message to the user.

[0885] Step 5: When the user (child) encounters a learning difficulty, they can use the device to take a picture of the problem and enter a brief description (e.g., "Please tell me how to solve this problem").

[0886] Step 6: The device sends the photo and description to the server. The server receives this data.

[0887] Step 7: The server uses OCR technology to extract text data from the received photo data. This converts the text information in the photo into digital data.

[0888] Step 8: The server analyzes the extracted text data to determine which learning category the problem belongs to (e.g., mathematics, science, history, etc.).

[0889] Step 9: The server generates appropriate answers and explanations based on the identified learning category. For example, if the topic is quadratic equations in mathematics, an explanation using the quadratic formula will be generated.

[0890] Step 10: The server sends the generated answers and explanations to the user's (child's) device. The device receives this data and displays it to the user.

[0891] Step 11: The user (child) solves the problem by following the answers and explanations displayed on the device. If there are any unclear points during the problem-solving process, they can send further questions.

[0892] Step 12: After the user (child) solves the problem and expresses their satisfaction with the explanation, they input their feedback into the device and send it to the server. The server receives this feedback data.

[0893] Step 13: The server saves the received feedback data to a database and uses it to improve the accuracy of learning support in the future.

[0894] (Example 1)

[0895] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0896] Traditionally, there has been a lack of means to receive quick and accurate support when encountering particularly difficult problems in learning. The manual searching and referencing of reference books were time-consuming and slowed down the learning process. Furthermore, providing efficient support based on appropriate feedback was difficult.

[0897] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0898] In this invention, the server includes means for taking an image of the learning content that the user is having trouble with and inputting a descriptive text; means for the terminal to send the captured image and descriptive text to the server; means for the server to extract text data from the image using optical character recognition technology; means for the server to analyze the extracted text data and classify it into an appropriate learning category; means for the server to use a generative AI model to generate appropriate answers and explanations based on the classified learning categories; means for the server to send the generated answers and explanations to the terminal; means for the terminal to display the received answers and explanations to the user; and means for the terminal to send user feedback to the server and store it in a database. This makes it possible for users to receive quick and appropriate support when they encounter difficulties in their learning.

[0899] "Users" refer to people who use the learning support system, and this includes children who have difficulty learning and the parents who support them.

[0900] A "device" refers to a device used by a user, and primarily includes smartphones, tablets, or personal computers.

[0901] A "server" refers to a cloud-based system, specifically a device responsible for processing and storing data.

[0902] Optical Character Recognition (OCR) is a technology that extracts text data from images.

[0903] A "generative AI model" refers to an artificial intelligence model designed to generate appropriate answers and explanations, and includes, for example, conversational generative models.

[0904] A "learning category" refers to a type of classification used to categorize learning content, and includes academic fields such as mathematics, science, and history.

[0905] "Feedback" refers to evaluations and opinions on answers and explanations provided by users, and this data is used to improve future support.

[0906] A "database" refers to a collection of information stored on a server, including user information and feedback data.

[0907] The present invention relates to a learning support system and is designed to allow users to receive prompt and appropriate support when they encounter difficulties in learning. The embodiments for carrying out the present invention are described in detail below.

[0908] System Configuration

[0909] This system consists of users, devices, and a server. Users include children who need learning support and their parents who support them. Devices are primarily smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage.

[0910] Basic program processing

[0911] This system's program has the following functions:

[0912] 1. User Registration

[0913] Users (parents and children) register with the system using a device. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[0914] 2. Learning support request

[0915] When a user (child) encounters a problem with their homework or learning material, they input text or images on their device and send them to the server. For example, if a child doesn't know how to solve a math problem, they might take a picture of an equation written in their notebook, add a simple explanation like "Please tell me how to solve it," and send it.

[0916] 3. Image and text analysis

[0917] The server analyzes the received images using OCR technology and extracts text data. The primary software used is optical character recognition (e.g., Google Cloud Vision API). The extracted text data is further analyzed to determine which learning category (e.g., mathematics, science, history) the question belongs to.

[0918] 4. Generating answers and explanations

[0919] The server generates appropriate answers and explanations based on the classified learning categories. For example, if asked how to solve a quadratic equation, it will create a step-by-step explanation using the quadratic formula. For this purpose, it utilizes a generative AI model (e.g., OpenAI GPT-3).

[0920] 5. Providing a response

[0921] The server sends the generated answers and explanations to the terminal. The terminal displays the received information to the user (child). The user (child) can then solve the problem by following these explanations.

[0922] 6. Obtaining and saving feedback

[0923] The user (child) inputs their satisfaction level with the answers and explanations on their device and sends it to the server. The server stores this feedback in a database and uses it to improve future support.

[0924] Specific example

[0925] For example, consider a case where a user (a child) doesn't know how to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. The system will process it in the following steps:

[0926] 1. The user (child) takes a picture of this equation with their device, enters the description "Tell me how to solve it," and sends it to the server.

[0927] 2. The server uses OCR technology to extract the text data "2x² + 3x - 5 = 0" from the image and identifies the problem as a quadratic equation.

[0928] 3. The server generates a step-by-step explanation using the solution formula and sends it to the terminal.

[0929] 4. The device displays this explanation to the user (child), and the child solves the problem according to the explanation.

[0930] 5. The user (child) enters their satisfaction level with the explanation and sends feedback to the server.

[0931] 6. The server saves the feedback to a database and uses it to improve future learning support.

[0932] Example of a prompt

[0933] Assume the user will input the following prompt into the generated AI model:

[0934] "Please explain how to solve this quadratic equation: 2x² + 3x - 5 = 0."

[0935] Based on this prompt, the server generates a detailed explanation and provides it to the user through the terminal.

[0936] The above describes a specific embodiment for carrying out the present invention. This process makes it possible to provide learning support quickly and effectively.

[0937] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0938] Step 1: User Registration

[0939] 1. Input: Users (parents and children) use the device to enter basic information such as name, email address, and password.

[0940] 2. Specific actions:

[0941] The user launches a web browser or dedicated app and opens the user registration screen.

[0942] The user clicks the "Register" button.

[0943] 3. Data Processing and Output: The terminal sends the input information to the server. The server stores the received information in a database and generates and records a unique user ID. The server sends a success message to the terminal, and the terminal displays a registration completion message to the user.

[0944] Step 2: Request learning support

[0945] 1. Input: The user (child) enters the text of the problem or a photo of the notebook containing the problem, along with a brief description, on the device.

[0946] 2. Specific actions:

[0947] The user (child) taps the "Request Support" button within the app.

[0948] The user (child) takes a picture of the problem with a note that says, "Please tell me how to solve it."

[0949] 3. Data Processing and Output: The terminal sends the input image, description, and user ID data to the server. The server passes the received data to the analysis engine.

[0950] Step 3: Image and text analysis

[0951] 1. Input: The server receives image and description data from the terminal.

[0952] 2. Specific actions:

[0953] The server analyzes the image using optical character recognition technology (e.g., Google Cloud Vision API) and extracts the text data.

[0954] 3. Data Processing and Output: The server analyzes the extracted text data and uses a software engine to determine which learning category (e.g., mathematics, science, history) the problem belongs to. The category information is saved and passed on to the next step.

[0955] Step 4: Generating answers and explanations

[0956] 1. Input: The server sends a prompt to the generative AI model (e.g., OpenAI GPT-3) based on the identified learning categories and extracted text data.

[0957] 2. Specific actions:

[0958] The server sends the prompt message: "Please tell me how to solve the quadratic equation 2x² + 3x - 5 = 0." to the generating AI model.

[0959] 3. Data Processing and Output: The generative AI model generates appropriate answers and explanations in response to prompts. The server evaluates the validity of the generated answers and explanations, formats them appropriately, and then sends them to the terminal.

[0960] Step 5: Providing a response

[0961] 1. Input: The terminal receives answers and explanations sent from the server.

[0962] 2. Specific actions:

[0963] The terminal displays the received information on the user interface.

[0964] The user (child) reviews and understands the displayed explanation.

[0965] 3. Data processing and output: The user (child) solves the problem based on the explanation.

[0966] Step 6: Obtain and save feedback

[0967] 1. Input: The user (child) inputs their satisfaction level with the explanation using the terminal.

[0968] 2. Specific actions:

[0969] The user (child) chooses one option from choices such as "very satisfied," "satisfied," or "dissatisfied."

[0970] The user (child) clicks the "Send" button.

[0971] 3. Data Processing and Output: The terminal sends the input feedback to the server. The server receives the feedback and stores it in a database. The server uses this feedback to improve the accuracy of future learning support.

[0972] (Application Example 1)

[0973] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0974] Traditional learning support systems offer limited means for users to receive prompt and appropriate support when they encounter difficulties. Furthermore, there is a lack of effective learning support methods in physical stores, resulting in insufficient support for customers to effectively utilize learning materials. There is also a need for improved support accuracy based on feedback and the provision of interactive learning experiences for in-store use.

[0975] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0976] In this invention, the server includes means for taking an image of the learning content that the user is having trouble with and inputting a descriptive text; means for the terminal to send the captured image and descriptive text to the server; means for the server to extract text data from the image using OCR technology; means for the server to analyze the extracted text data and classify it into an appropriate learning category; means for the server to generate appropriate answers and explanations based on the classified learning categories; means for the server to send the generated answers and explanations to the terminal; means for the terminal to display the received answers and explanations to the user; means for the terminal to send user feedback to the server and store it in a database; means for the user to receive learning support through a robot or smart glasses placed in a physical store; and means for acquiring and registering product information and accessing learning content. This enables interactive learning support in physical stores, allowing users to effectively solve problems and improve the accuracy of support by utilizing feedback information.

[0977] "Users" refer to children who receive learning support using the learning support system, and their guardians who provide that support.

[0978] "Device" refers to electronic devices used by users, such as smartphones, tablets, and personal computers.

[0979] A "server" refers to a cloud-based system, specifically a device that processes and stores data sent by users.

[0980] "OCR technology" refers to optical character recognition technology that automatically extracts text data from images.

[0981] "Robot" refers to a device placed in a physical store that provides learning support while interacting with users.

[0982] "Smart glasses" refer to devices that users wear to display information from learning materials or to acquire information using the user's gaze.

[0983] "Product information" refers to detailed information about learning materials and products, and is obtained through methods such as QR codes.

[0984] "Learning content" refers to information such as textbooks, workbooks, and explanatory materials that users refer to as they progress through their studies.

[0985] "Feedback" refers to users' opinions on the learning support provided, such as their satisfaction level and suggestions for improvement.

[0986] "Interactive learning support" refers to conversational support that allows users to receive learning assistance in real time.

[0987] This invention relates to a learning support system designed to provide users with quick and appropriate support when they encounter learning difficulties. The system consists of a user, a terminal, a server, and devices (robots, smart glasses) for providing learning support in physical stores.

[0988] System Configuration

[0989] 1. User

[0990] Users refer to children who need learning support and their guardians who support them.

[0991] 2. Terminal

[0992] The device primarily uses smartphones, tablets, or personal computers, and provides technology that allows users to take pictures of learning content and input explanatory text.

[0993] 3. Server

[0994] The server is a cloud-based system and has the following features:

[0995] Text data is extracted from images using OCR (Optical Character Recognition) technology.

[0996] The extracted text data is analyzed and classified into appropriate learning categories.

[0997] It generates appropriate answers and explanations based on the categorized learning categories.

[0998] The generated answers and explanations are sent to the device.

[0999] User feedback is stored in a database to improve the accuracy of future learning support.

[1000] 4. Learning support devices in physical stores

[1001] The robots and smart glasses placed within physical stores are devices that provide users with learning support. Through these devices, users can obtain and register product information and access learning content.

[1002] Basic program processing

[1003] 1. User Registration

[1004] Users register with the system using a terminal, and this information is sent to the server. The server stores the user information in a database and assigns a unique ID to each user.

[1005] 2. Learning support request

[1006] When a user encounters a problem with their learning material, they input text or images on their device and send them to the server. For example, if they can't solve a math problem, they might take a picture of an equation written in their notebook and send it with a simple explanation like, "Please tell me how to solve this."

[1007] 3. Image and text analysis

[1008] The server analyzes the received image using OCR technology and extracts text data. It then analyzes the extracted text data to determine which learning category the problem belongs to.

[1009] 4. Generating answers and explanations

[1010] The server generates appropriate answers and explanations based on the categorized learning category. For example, if asked how to solve a quadratic equation, it will create a step-by-step explanation using the quadratic formula.

[1011] 5. Providing a response

[1012] The server sends the generated answers and explanations to the terminal, and the terminal displays the received information to the user.

[1013] 6. Obtaining and saving feedback

[1014] Users input their satisfaction level with the provided answers and explanations and send this information to the server. The server stores this feedback in a database and uses the feedback information to improve the accuracy of future support.

[1015] 7. Learning support at physical stores

[1016] In physical stores selling educational materials, users can receive learning support using terminals, robots placed in the store, or smart glasses. Users obtain information related to the materials in the store via QR codes, etc., and receive learning support based on that information.

[1017] Specific example

[1018] For example, if a user doesn't know how to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework, they can take a picture of the equation with their device, type a description like "Please tell me how to solve it," and send it to the server. The server uses OCR technology to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Next, the server generates a step-by-step explanation using the quadratic formula and sends it to the device. The device displays this explanation to the user, who then solves the problem following the explanation. Furthermore, the user inputs their satisfaction level with the explanation and sends the feedback to the server. In physical stores, learning support can be more interactive by using robots or smart glasses.

[1019] Example of a prompt

[1020] "Please scan the QR code."

[1021] "Please briefly describe the problem you are experiencing (exit to finish)."

[1022] "Please enter the image path for the teaching material."

[1023] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1024] Step 1:

[1025] Users take pictures of learning materials using their smartphone or tablet camera and enter descriptive text. This input consists of images related to the learning material and text that describes their details. This data is prepared by the device.

[1026] Step 2:

[1027] The device sends the captured image and accompanying text to the server. The input image data and text data are uploaded to a cloud-based server via the internet. The server receives this data and proceeds to the next step.

[1028] Step 3:

[1029] The server extracts text data from the received image using OCR technology. Specifically, the server uses OCR software (for example, Tesseract OCR) to recognize character information within the image and extract it as text data. The input is image data, and the output is the text data within the image.

[1030] Step 4:

[1031] The server analyzes the extracted text data and classifies it into appropriate learning categories. The server uses a natural language processing (NLP) model to analyze the text data and classify it into specific learning categories (e.g., mathematics, science, history, etc.). The input is the text data after OCR processing, and the output is the learning categories.

[1032] Step 5:

[1033] The server generates appropriate answers and explanations based on the classified learning categories. A generative AI model (e.g., GPT-3) is used to generate explanations for questions and problems related to the classified categories. The input is detailed text data of the learning categories and problems, and the output is explanations and answers to the problems.

[1034] Step 6:

[1035] The server sends the generated answers and explanations to the terminal. The text data of the explanations and answers is then sent back to the user's terminal via the internet. The input is the generated explanation data, and the output is the explanation data displayed on the terminal.

[1036] Step 7:

[1037] The terminal displays the received answers and explanations to the user. The content of the explanations and answers is displayed on the terminal screen, and the user views it. The input is the explanation data sent from the server, and the output is the explanation data displayed to the user.

[1038] Step 8:

[1039] The user inputs feedback on the answers and explanations, and sends that feedback from the device to the server. The input is the user's feedback data, which the device sends to the server. The output is the transmission of the feedback data.

[1040] Step 9:

[1041] The server stores user feedback in a database and uses it to improve the accuracy of learning support. The input is feedback data, which the server stores in the database. The output is the stored feedback, which serves as data to improve the quality of learning support in the future.

[1042] Step 10:

[1043] Users receive learning support through robots or smart glasses placed in physical stores. In the store, users obtain product information via QR codes and request learning support from a server. This enables interactive learning support. Input is product information via QR codes, and output is learning support related to a specific product.

[1044] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1045] The present invention relates to a learning support system that allows users to receive prompt and appropriate support when they experience difficulties in learning, and further recognizes the user's emotions and provides support accordingly. The embodiments for carrying out the present invention will be described in detail below.

[1046] System Configuration

[1047] This system consists of users, devices, a server, and an emotion engine. Users include children who need learning support and their parents who support them. Devices are primarily smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage. The emotion engine is used to recognize the emotional state of the users.

[1048] Basic program processing

[1049] This system's program has the following functions:

[1050] 1. User Registration

[1051] The user registers with the system using a terminal. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[1052] 2. Learning support request

[1053] If a user (child) encounters difficulties with their studies, they can use the device to take a picture of the problem and enter a brief explanation. For example, if they don't know how to solve a math problem, they can take a picture of the equation written in their notebook and send it with the explanation, "Please tell me how to solve it."

[1054] 3. Image and text analysis

[1055] The server analyzes the received image using OCR technology and extracts text data. The extracted text data is then analyzed to determine which learning category the problem belongs to.

[1056] 4. Emotion recognition

[1057] The device analyzes the user's facial recognition and voice using an emotion engine to determine the user's emotional state. For example, it recognizes if the user is anxious or confused.

[1058] 5. Generating answers and explanations

[1059] The server generates appropriate answers and explanations based on the classified learning categories and the user's emotional state recognized by the emotion engine. For example, if the user is confused, a more detailed and helpful explanation will be provided.

[1060] 6. Providing a response

[1061] The server sends the generated answers and explanations to the device. The device displays this information to the user, and the user (child) solves the problem according to the explanation.

[1062] 7. Obtaining and saving feedback

[1063] The user (child) inputs their satisfaction level with the answers and explanations and sends the feedback to their device. The server receives this feedback, stores it in a database, and uses it to improve future learning support.

[1064] 8. Saving emotional data

[1065] The emotion data recognized by the emotion engine is also sent to the server and stored in the database. This allows for personalized support in subsequent sessions.

[1066] Specific example

[1067] For example, consider a situation where a user (child) is struggling to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. The user takes a picture of the equation with their device, enters a caption saying "Please tell me how to solve it," and sends it. The server uses OCR technology to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Simultaneously, an emotion engine analyzes the user's face and voice and recognizes that the user is confused. Based on this information, the server generates a step-by-step explanation using the quadratic formula, providing a detailed explanation with particularly clear language and concrete examples. The server sends the generated explanation to the device, which displays it to the user. The user (child) solves the problem according to the displayed explanation, and after solving the problem, enters their satisfaction level and sends it. The server stores the feedback and emotion data in a database and uses it for future support.

[1068] The above describes a specific embodiment for carrying out the present invention. This system allows parents to support their children at the appropriate time, and enables children to solve problems while maintaining the enjoyment of learning. Furthermore, the introduction of an emotion engine provides personalized support according to the user's emotional state, resulting in more effective learning support.

[1069] The following describes the processing flow.

[1070] Step 1: The user registers using their device. The user enters basic information such as their name, email address, and password.

[1071] Step 2: The terminal sends the entered user information to the server. The server receives this information.

[1072] Step 3: The server saves the received user information to the database and assigns a unique ID to each user.

[1073] Step 4: The server sends a notification to the terminal that user registration is complete. The terminal displays a registration completion message to the user.

[1074] Step 5: When the user (child) encounters a learning difficulty, they can use the device to take a picture of the problem and enter a brief description (e.g., "Please tell me how to solve this problem").

[1075] Step 6: The device sends the photo and description to the server. The server receives this data.

[1076] Step 7: The server uses OCR technology to extract text data from the received photo data. This converts the text information in the photo into digital data.

[1077] Step 8: The server analyzes the extracted text data to determine which learning category the problem belongs to (e.g., mathematics, science, history, etc.).

[1078] Step 9: The device sends the user's facial recognition and voice input data to the emotion engine. The emotion engine analyzes this data and determines the user's emotional state.

[1079] Step 10: The server generates appropriate answers and explanations based on the identified learning categories and the emotional state recognized by the emotion engine. For example, if the user is confused, a more detailed and helpful explanation will be provided.

[1080] Step 11: The server sends the generated answers and explanations to the user's (child's) device. The device receives this data and displays it to the user.

[1081] Step 12: The user (child) solves the problem by following the answers and explanations displayed on the device. If there are any questions during the problem-solving process, they can send another question.

[1082] Step 13: After the user (child) solves the problem and expresses their satisfaction with the explanation, they input their feedback into the device and send it to the server. The server receives this feedback data.

[1083] Step 14: The server saves the received feedback data and the emotion data recognized by the emotion engine to a database, which is then used to improve the accuracy of future learning support.

[1084] (Example 2)

[1085] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1086] Conventional learning support systems have a problem in that it is difficult for users to receive prompt and appropriate support when they encounter difficulties during learning, and in particular, they cannot provide individualized support that addresses the user's emotional state. Furthermore, they lack a means to effectively utilize user feedback on the support provided and improve the accuracy of future learning support.

[1087] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1088] In this invention, the server includes means for extracting text data from an image using optical character recognition technology, means for analyzing the extracted text data and classifying it into an appropriate learning category, and means for analyzing facial images and voice data and recognizing the user's emotional state. This makes it possible to quickly classify difficulties experienced by the user during learning into a specific learning category and to provide personalized support tailored to the user's emotional state. Furthermore, by utilizing user feedback and emotional data, the accuracy of future learning support can be improved.

[1089] "User" refers to anyone who uses the system to receive learning support, and primarily includes children who need learning support and the parents who support them.

[1090] A "device" is a device used by a user to interact with a system, and includes smartphones, tablets, and personal computers.

[1091] A "server" refers to a cloud-based system that is responsible for processing and storing data.

[1092] Optical Character Recognition (OCR) refers to a technology that analyzes and extracts text data from images.

[1093] A "learning category" refers to the specific field or topic to which the analyzed problem belongs, such as mathematics, science, or history.

[1094] "Emotional state" refers to the psychological and emotional state recognized from the user's facial image and voice data, and includes anxiety, confusion, joy, etc.

[1095] A "generative AI model" refers to a model that uses artificial intelligence technology to automatically generate answers and explanations to user questions.

[1096] "Feedback" refers to the evaluation and opinions of users regarding the support and explanations provided.

[1097] "Face image" refers to visual data obtained from a photograph or video capture of the user's face.

[1098] "Voice data" refers to data recorded from the user's speech or voice.

[1099] A "database" refers to a system for managing and storing data such as user information, feedback, and sentiment data.

[1100] This invention is a learning support system that provides prompt and appropriate support when a user encounters difficulties during learning, and further recognizes the user's emotional state and provides personalized support accordingly. This system consists of a user, a terminal, and a server.

[1101] System Configuration

[1102] This system consists of users, devices, a server, and an emotion engine. Users primarily include children who need learning support and their parents who support them. Devices mainly consist of smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage. The emotion engine is used to recognize the emotional state of the users.

[1103] Basic program processing

[1104] This system's program has the following main functions:

[1105] 1. User Registration

[1106] Users register with the system using a terminal. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[1107] 2. Acceptance of learning support requests

[1108] If a user (child) encounters difficulties with their studies, they can use the device to take a picture of the problem and enter a brief explanation. For example, if they don't know how to solve a math problem, they can take a picture of the equation written in their notebook and send it with the explanation, "Please tell me how to solve it."

[1109] 3. Image and text analysis

[1110] The server analyzes the received image using optical character recognition (OCR) technology and extracts text data. For example, Tesseract OCR is used. The extracted text data is then analyzed, and a natural language processing library (such as NLTK) is used to determine which learning category the problem belongs to.

[1111] 4. Emotion recognition

[1112] The device analyzes the user's facial image and voice data using an emotion engine to determine the user's emotional state. For example, it uses facial recognition technology (such as the Microsoft Emotion API) to recognize emotions such as anxiety or confusion.

[1113] 5. Generating answers and explanations

[1114] The server uses a generative AI model to generate appropriate answers and explanations based on classified learning categories and the user's emotional state recognized by the emotion engine. For example, GPT-4 is used as one of the generative AI models. A confused user will be provided with a more detailed and helpful explanation.

[1115] 6. Providing a response

[1116] The server sends the generated answers and explanations to the terminal. The terminal displays the received information to the user, and the user (child) solves the problem according to the explanation.

[1117] 7. Obtaining and saving feedback

[1118] The user (child) inputs their satisfaction level with the answers and explanations and sends the feedback to their device. The server receives this feedback, stores it in a database, and uses it to improve future learning support.

[1119] 8. Saving emotional data

[1120] The emotion data recognized by the emotion engine is also sent to the server and stored in the database. This allows for personalized support in subsequent sessions.

[1121] Specific example

[1122] For example, consider a situation where a user (child) is struggling to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. They take a picture of the equation with their device, type a message saying "Please explain how to solve it," and send it. The server uses OCR technology (Tesseract OCR) to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Simultaneously, the emotion engine (Microsoft Emotion API) analyzes the user's face and voice, recognizing their confusion. Based on this information, the server uses a generative AI model (GPT-4) to generate a step-by-step explanation using the quadratic formula, providing detailed explanations with clear language and concrete examples. The server sends the generated explanation to the device, which displays it to the user. The user (child) solves the problem following the displayed explanation and, after solving it, inputs their satisfaction level and sends it. The server stores the feedback and emotion data in a database for future support.

[1123] For example, here is an example of a prompt statement for a generative AI model (GPT-4):

[1124] A child is having trouble solving the quadratic equation "2x² + 3x - 5 = 0". Please provide a step-by-step, detailed explanation using easy-to-understand language.

[1125] The user's emotional state is one of confusion.

[1126] This system allows parents to support their children at the right time, and enables children to solve problems while maintaining the enjoyment of learning. Furthermore, the introduction of an emotion engine provides personalized support tailored to the user's emotional state, resulting in more effective learning assistance.

[1127] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1128] Step 1:

[1129] User Registration

[1130] Input: User information (name, email address, age, etc.)

[1131] Operation: The user registers with the learning support system using a terminal. The terminal sends the entered information to the server. The server stores the received user information in a database and generates a unique user ID. The generated user ID is sent back to the terminal, and the terminal notifies the user that "registration is complete."

[1132] Output: Unique User ID

[1133] Step 2:

[1134] Acceptance of learning support request

[1135] Input: Image of the learning material problem, a brief description of the problem.

[1136] Operation: When the user (child) encounters difficulty with the learning material, they use the device to take a picture of the problem and enter a description such as "Please tell me how to solve it." The device then sends the captured image and description to the server.

[1137] Output: The problem image and description are sent to the server.

[1138] Step 3:

[1139] Image and text analysis

[1140] Input: Problem image, description

[1141] Operation: The server analyzes the received image using optical character recognition (OCR) technology and extracts text data. Tesseract OCR is used as the OCR technology. The extracted text data is analyzed using a natural language processing library (such as NLTK) to determine which learning category the problem belongs to.

[1142] Output: Text data, training categories

[1143] Step 4:

[1144] emotion recognition

[1145] Input: User's face image, voice data

[1146] Operation: The device captures a picture of the user's face with its camera and records their voice with its microphone. The device sends this data to the server. The server uses facial recognition technology (such as Microsoft Emotion API) to analyze the user's emotional state. The server recognizes specific emotions, such as anxiety or confusion.

[1147] Output: User's emotional state

[1148] Step 5:

[1149] Generating answers and explanations

[1150] Input: Text data, learning categories, sentiment state

[1151] Operation: The server uses a generative AI model (such as GPT-4) to generate appropriate answers and explanations based on the analyzed text data and sentiment data. For example, if the problem involves a quadratic equation, it will generate a detailed explanation using the quadratic formula.

[1152] Output: Answers and explanations

[1153] Step 6:

[1154] Providing a response

[1155] Input: Answer or explanation

[1156] Operation: The server sends the generated answers and explanations to the terminal. The terminal displays the received information to the user. The user solves the problem according to the displayed explanations.

[1157] Output: Explanation text displayed to the user

[1158] Step 7:

[1159] Obtaining and saving feedback

[1160] Input: User feedback (satisfaction ratings and comments)

[1161] Operation: After the user resolves a problem, the device displays a feedback form. The user enters a satisfaction rating and comments, and sends them to the server via the device. The server stores the received feedback in its database.

[1162] Output: Saved feedback data

[1163] Step 8:

[1164] Storage of emotional data

[1165] Input: User's emotional state data

[1166] Operation: The server stores the user's emotional data, recognized by the emotion engine, in a database. This allows for personalized support in subsequent learning support sessions.

[1167] Output: Saved sentiment data

[1168] (Application Example 2)

[1169] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1170] Conventional learning support systems have limited means of providing users with quick and appropriate support when they encounter difficulties. Furthermore, providing individualized support tailored to the user's emotional state is challenging. Additionally, in physical stores, there is a lack of means to provide appropriate information when customers have difficulty choosing products. The time spent manually searching for information contributes to decreased customer satisfaction. To address these challenges, a system is needed that provides quick and appropriate support when users encounter difficulties, and furthermore, individualized support tailored to their emotional state. Moreover, efficient information provision utilizing emotion recognition is required for product selection in physical stores.

[1171] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1172] In this invention, the server includes means for acquiring product information using a smart device, means for transmitting the product information acquired by the terminal and customer requests to the server, and means for the server to analyze the product information and customer requests and determine the emotional state using an emotion recognition engine. This enables rapid and appropriate learning support and personalized product recommendations in physical stores.

[1173] A "user" refers to an individual who needs learning support or assistance with product selection.

[1174] "Terminal" refers to devices such as smartphones, tablets, and personal computers that are operated by the user.

[1175] A "server" refers to a cloud-based system that handles data processing and storage, and generates appropriate support for users.

[1176] "Images" refers to photographic data of learning materials or products taken by the user using their device.

[1177] "Description" refers to the text data that users input regarding learning content, product issues, or requests.

[1178] "OCR technology" refers to optical character recognition technology used to extract text data from images.

[1179] "Learning categories" refer to educational fields or subjects used to classify the text data analyzed by the server.

[1180] "Answers and explanations" refers to specific explanations and suggestions regarding learning support and product information generated by the server.

[1181] "Feedback" refers to the evaluation and opinions of users regarding the support and suggestions they receive.

[1182] A "database" refers to a system used to store data such as user information and feedback.

[1183] A "smart device" refers to an internet-connected device operated by a user, such as a smartphone or smart glasses.

[1184] "Product information" refers to data about the features and details of a product.

[1185] "Customer requirements" refers to the preferences and questions that customers enter when selecting a product.

[1186] An "emotion recognition engine" refers to software that analyzes a user's emotional state from their facial expressions and voice.

[1187] "Emotional state" refers to the user's psychological state as determined by the emotion recognition engine.

[1188] "Product suggestions" refer to product suggestions generated by the server based on customer requests and emotional states.

[1189] The system of the present invention uses smart devices, servers, and an emotion recognition engine to provide learning support and in-store shopping support. The embodiments for carrying out the present invention are described in detail below.

[1190] System Configuration

[1191] This system primarily consists of a user, a smart device, a server, and an emotion engine. The user is an individual who utilizes the system for learning support or product selection. Smart devices include smartphones and smart glasses. The server is a cloud-based system responsible for data processing and storage. The emotion engine is responsible for recognizing the user's emotional state.

[1192] Basic program processing

[1193] This system's program has the following functions:

[1194] 1. User Registration

[1195] Users register with the system using their smart devices. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[1196] 2. Product support request

[1197] If a user has trouble choosing a product, they can use their smart device to scan the product's barcode or QR code and enter a simple request. For example, they might enter, "I want to know more details about this product."

[1198] 3. Image and text analysis

[1199] The server uses OCR technology to analyze the received product information and extract text data. The extracted text data is then analyzed to understand the characteristics and requirements of the requested product.

[1200] 4. Emotion recognition

[1201] Smart devices use an emotion engine to analyze the user's facial expressions and voice to determine the user's emotional state. For example, they can determine whether the user is confused or distressed.

[1202] 5. Generating answers and suggestions

[1203] The server generates appropriate responses and suggestions based on product information and the user's emotional state, as recognized by the emotion engine. For example, if the user is confused, it will provide detailed product information and suggest related products.

[1204] 6. Providing a response

[1205] The server sends generated answers and suggestions to the smart device and displays them to the user. The user can then use this information to select products.

[1206] 7. Obtaining and saving feedback

[1207] Users input their satisfaction level with the support and suggestions they receive as feedback and send it to the server from their smart device. The server stores this feedback in a database and uses it to improve future support.

[1208] Hardware and software to be used

[1209] The following hardware and software are used to implement this system:

[1210] Hardware: Smart devices (smartphones, smart glasses), cloud servers

[1211] Software: Emotion recognition engine (e.g., Microsoft Azure Emotion API), OCR technology (e.g., Google Vision API), database system (e.g., Amazon RDS)

[1212] Specific example

[1213] For example, consider a scenario where a user is having trouble choosing a product in a physical store. Using smart glasses, the user scans the product's barcode and enters a request saying, "I want to know more details about this product." The server extracts the product's text information from the barcode using OCR technology and recognizes the user's confusion from their facial expressions and voice using an emotion engine. Based on this information, the server generates a detailed product description and related product suggestions, which are then sent to the smart glasses for the user to see. The user selects a product based on the displayed information and enters their final satisfaction level as feedback. The server saves this feedback and uses it to improve the accuracy of future support.

[1214] Example of a prompt

[1215] Prompt text to input to the generative AI model:

[1216] "If a customer appears confused, provide a detailed explanation of how to use the product and related products. For example, include product features and reviews from other customers in your suggestions."

[1217] As described above, the system of the present invention provides users with prompt and appropriate support in both learning support and in-store shopping experiences, and further enables personalized responses tailored to their emotional state.

[1218] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1219] Step 1: User Registration

[1220] Users register with the system using a smart device (e.g., a smartphone).

[1221] Input: User's basic information (name, email address, etc.)

[1222] Output: Assign a unique ID to the user and send the information to the server.

[1223] Specific operation: The user enters information into the application form and presses the "Submit" button. The terminal encrypts the entered information and sends it to the server. The server stores the received information in a database and generates a unique user ID.

[1224] Step 2: Product Support Request

[1225] When users have trouble choosing a product, they can use their smart device to scan the product's barcode or QR code and enter their request.

[1226] Input: Product barcode or QR code, request text (e.g., "I want to know more details about this product")

[1227] Output: Sends product information and requests to the server.

[1228] Specific operation: The user scans a barcode using the camera on their smart device and enters the request. The device packages the image data and text and sends it to the server.

[1229] Step 3: Image and text analysis

[1230] The server analyzes the received product information using OCR technology and extracts text data.

[1231] Input: Barcode image, text request

[1232] Output: Extracted text data

[1233] Specific operation: The server uses OCR technology such as the Google Vision API to extract text from barcode images. It then combines this with the text request and performs analysis.

[1234] Step 4: Emotion Recognition

[1235] The system analyzes the user's facial expressions and voice captured by a smart device to determine their emotional state.

[1236] Input: User's facial expression image, audio data

[1237] Output: Data on emotional state (e.g., confusion, anxiety, excitement, etc.)

[1238] Specific operation: The smart device's camera and microphone are used to collect the user's facial expressions and voice, and these are sent to an emotion recognition engine (e.g., Microsoft Azure Emotion API). The server analyzes the acquired data and determines the emotional state.

[1239] Step 5: Generating answers and suggestions

[1240] The server generates appropriate responses and suggestions based on emotional states and product information.

[1241] Input: Emotional state data, product information

[1242] Output: Text data of appropriate answers and suggestions

[1243] Specific operation: Based on emotional states and product information, the server uses pre-configured rules and generative AI models to generate responses and suggestions in a format that is easy for the user to understand.

[1244] Step 6: Providing a response

[1245] The server sends the generated answers and suggestions to the smart device and displays them to the user.

[1246] Input: Text data of generated responses and suggestions

[1247] Output: Responses and suggestions displayed on the user's smart device.

[1248] Specific operation: The server sends the generated answers and suggestions to the terminal, and the terminal displays the received data to the user. The user then selects a product based on this information.

[1249] Step 7: Obtain and save feedback

[1250] Users input feedback on the support and suggestions they receive and send it from their device to the server.

[1251] Input: Text data of ratings and opinions entered by the user.

[1252] Output: Save the feedback data to the server's database.

[1253] Specific operation: The user uses a smart device to enter a rating for support or suggestions and presses the "Submit" button. The device sends the entered feedback data to the server, which stores it in a database.

[1254] By following the steps outlined above, the program's processing is executed, allowing users to receive prompt and accurate support in learning assistance and product selection.

[1255] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1256] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1257] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1258] [Fourth Embodiment]

[1259] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1260] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1261] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1262] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1263] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1264] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1265] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1266] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1267] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1268] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1269] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1270] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1271] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1272] The present invention relates to a learning support system and is designed to allow users to receive prompt and appropriate support when they encounter difficulties in learning. The embodiments for carrying out the present invention are described in detail below.

[1273] System Configuration

[1274] This system consists of users, devices, and a server. Users include children who need learning support and their parents who support them. Devices are mainly smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage.

[1275] Basic program processing

[1276] This system's program has the following functions:

[1277] 1. User Registration

[1278] Users (parents and children) register with the system using a device. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[1279] 2. Learning support request

[1280] When a user (child) encounters a problem with homework or learning material, they input text or images on their device and send them to the server. For example, if a child doesn't know how to solve a math problem, they can take a picture of an equation written in their notebook, add a simple explanation like "Please tell me how to solve it," and send it.

[1281] 3. Image and text analysis

[1282] The server analyzes the received image using OCR technology and extracts text data. The extracted text data is further analyzed to determine which learning category (e.g., mathematics, science, history) the question belongs to.

[1283] 4. Generating answers and explanations

[1284] The server generates appropriate answers and explanations based on the categorized learning categories. For example, if asked how to solve a quadratic equation, it will create a step-by-step explanation using the quadratic formula.

[1285] 5. Providing a response

[1286] The server sends the generated answers and explanations to the device. The device displays the received information to the user (child). The child can then solve the problem by following these explanations.

[1287] 6. Obtaining and saving feedback

[1288] The user (child) inputs their satisfaction level with the answers and explanations on their device and sends it to the server. The server stores this feedback in a database and uses it to improve future support.

[1289] Specific example

[1290] For example, consider a case where a user (child) doesn't know how to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. The user takes a picture of the equation with their device, enters a description saying "Please tell me how to solve it," and sends it to the server. The server uses OCR technology to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Next, the server generates a step-by-step explanation using the quadratic formula and sends it to the device. The device displays this explanation to the user (child), and the child solves the problem following the explanation. Finally, the child inputs their satisfaction level with the explanation and sends the feedback to the server. The server stores this feedback in a database and uses it to improve the accuracy of future learning support.

[1291] The above describes a specific embodiment for carrying out the present invention. This system allows parents to support their children at the appropriate time, and enables children to solve problems while maintaining the enjoyment of learning.

[1292] The following describes the processing flow.

[1293] Step 1: The user registers using their device. The user enters basic information such as their name, email address, and password.

[1294] Step 2: The terminal sends the entered user information to the server. The server receives this information.

[1295] Step 3: The server saves the received user information to the database and assigns a unique ID to each user.

[1296] Step 4: The server sends a notification to the terminal that user registration is complete. The terminal displays a registration completion message to the user.

[1297] Step 5: When the user (child) encounters a learning difficulty, they can use the device to take a picture of the problem and enter a brief description (e.g., "Please tell me how to solve this problem").

[1298] Step 6: The device sends the photo and description to the server. The server receives this data.

[1299] Step 7: The server uses OCR technology to extract text data from the received photo data. This converts the text information in the photo into digital data.

[1300] Step 8: The server analyzes the extracted text data to determine which learning category the problem belongs to (e.g., mathematics, science, history, etc.).

[1301] Step 9: The server generates appropriate answers and explanations based on the identified learning category. For example, if the topic is quadratic equations in mathematics, an explanation using the quadratic formula will be generated.

[1302] Step 10: The server sends the generated answers and explanations to the user's (child's) device. The device receives this data and displays it to the user.

[1303] Step 11: The user (child) solves the problem by following the answers and explanations displayed on the device. If there are any unclear points during the problem-solving process, they can send further questions.

[1304] Step 12: After the user (child) solves the problem and expresses their satisfaction with the explanation, they input their feedback into the device and send it to the server. The server receives this feedback data.

[1305] Step 13: The server saves the received feedback data to a database and uses it to improve the accuracy of learning support in the future.

[1306] (Example 1)

[1307] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1308] Traditionally, there has been a lack of means to receive quick and accurate support when encountering particularly difficult problems in learning. The manual searching and referencing of reference books were time-consuming and slowed down the learning process. Furthermore, providing efficient support based on appropriate feedback was difficult.

[1309] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1310] In this invention, the server includes means for taking an image of the learning content that the user is having trouble with and inputting a descriptive text; means for the terminal to send the captured image and descriptive text to the server; means for the server to extract text data from the image using optical character recognition technology; means for the server to analyze the extracted text data and classify it into an appropriate learning category; means for the server to use a generative AI model to generate appropriate answers and explanations based on the classified learning categories; means for the server to send the generated answers and explanations to the terminal; means for the terminal to display the received answers and explanations to the user; and means for the terminal to send user feedback to the server and store it in a database. This makes it possible for users to receive quick and appropriate support when they encounter difficulties in their learning.

[1311] "Users" refer to people who use the learning support system, and this includes children who have difficulty learning and the parents who support them.

[1312] A "device" refers to a device used by a user, and primarily includes smartphones, tablets, or personal computers.

[1313] A "server" refers to a cloud-based system, specifically a device responsible for processing and storing data.

[1314] Optical Character Recognition (OCR) is a technology that extracts text data from images.

[1315] A "generative AI model" refers to an artificial intelligence model designed to generate appropriate answers and explanations, and includes, for example, conversational generative models.

[1316] A "learning category" refers to a type of classification used to categorize learning content, and includes academic fields such as mathematics, science, and history.

[1317] "Feedback" refers to evaluations and opinions on answers and explanations provided by users, and this data is used to improve future support.

[1318] A "database" refers to a collection of information stored on a server, including user information and feedback data.

[1319] The present invention relates to a learning support system and is designed to allow users to receive prompt and appropriate support when they encounter difficulties in learning. The embodiments for carrying out the present invention are described in detail below.

[1320] System Configuration

[1321] This system consists of users, devices, and a server. Users include children who need learning support and their parents who support them. Devices are primarily smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage.

[1322] Basic program processing

[1323] This system's program has the following functions:

[1324] 1. User Registration

[1325] Users (parents and children) register with the system using a device. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[1326] 2. Learning support request

[1327] When a user (child) encounters a problem with their homework or learning material, they input text or images on their device and send them to the server. For example, if a child doesn't know how to solve a math problem, they might take a picture of an equation written in their notebook, add a simple explanation like "Please tell me how to solve it," and send it.

[1328] 3. Image and text analysis

[1329] The server analyzes the received images using OCR technology and extracts text data. The primary software used is optical character recognition (e.g., Google Cloud Vision API). The extracted text data is further analyzed to determine which learning category (e.g., mathematics, science, history) the question belongs to.

[1330] 4. Generating answers and explanations

[1331] The server generates appropriate answers and explanations based on the classified learning categories. For example, if asked how to solve a quadratic equation, it will create a step-by-step explanation using the quadratic formula. For this purpose, it utilizes a generative AI model (e.g., OpenAI GPT-3).

[1332] 5. Providing a response

[1333] The server sends the generated answers and explanations to the terminal. The terminal displays the received information to the user (child). The user (child) can then solve the problem by following these explanations.

[1334] 6. Obtaining and saving feedback

[1335] The user (child) inputs their satisfaction level with the answers and explanations on their device and sends it to the server. The server stores this feedback in a database and uses it to improve future support.

[1336] Specific example

[1337] For example, consider a case where a user (a child) doesn't know how to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. The system will process it in the following steps:

[1338] 1. The user (child) takes a picture of this equation with their device, enters the description "Tell me how to solve it," and sends it to the server.

[1339] 2. The server uses OCR technology to extract the text data "2x² + 3x - 5 = 0" from the image and identifies the problem as a quadratic equation.

[1340] 3. The server generates a step-by-step explanation using the solution formula and sends it to the terminal.

[1341] 4. The device displays this explanation to the user (child), and the child solves the problem according to the explanation.

[1342] 5. The user (child) enters their satisfaction level with the explanation and sends feedback to the server.

[1343] 6. The server saves the feedback to a database and uses it to improve future learning support.

[1344] Example of a prompt

[1345] Assume the user will input the following prompt into the generated AI model:

[1346] "Please explain how to solve this quadratic equation: 2x² + 3x - 5 = 0."

[1347] Based on this prompt, the server generates a detailed explanation and provides it to the user through the terminal.

[1348] The above describes a specific embodiment for carrying out the present invention. This process makes it possible to provide learning support quickly and effectively.

[1349] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1350] Step 1: User Registration

[1351] 1. Input: Users (parents and children) use the device to enter basic information such as name, email address, and password.

[1352] 2. Specific actions:

[1353] The user launches a web browser or dedicated app and opens the user registration screen.

[1354] The user clicks the "Register" button.

[1355] 3. Data Processing and Output: The terminal sends the input information to the server. The server stores the received information in a database and generates and records a unique user ID. The server sends a success message to the terminal, and the terminal displays a registration completion message to the user.

[1356] Step 2: Request learning support

[1357] 1. Input: The user (child) enters the text of the problem or a photo of the notebook containing the problem, along with a brief description, on the device.

[1358] 2. Specific actions:

[1359] The user (child) taps the "Request Support" button within the app.

[1360] The user (child) takes a picture of the problem with a note that says, "Please tell me how to solve it."

[1361] 3. Data Processing and Output: The terminal sends the input image, description, and user ID data to the server. The server passes the received data to the analysis engine.

[1362] Step 3: Image and text analysis

[1363] 1. Input: The server receives image and description data from the terminal.

[1364] 2. Specific actions:

[1365] The server analyzes the image using optical character recognition technology (e.g., Google Cloud Vision API) and extracts the text data.

[1366] 3. Data Processing and Output: The server analyzes the extracted text data and uses a software engine to determine which learning category (e.g., mathematics, science, history) the problem belongs to. The category information is saved and passed on to the next step.

[1367] Step 4: Generating answers and explanations

[1368] 1. Input: The server sends a prompt to the generative AI model (e.g., OpenAI GPT-3) based on the identified learning categories and extracted text data.

[1369] 2. Specific actions:

[1370] The server sends the prompt message: "Please tell me how to solve the quadratic equation 2x² + 3x - 5 = 0." to the generating AI model.

[1371] 3. Data Processing and Output: The generative AI model generates appropriate answers and explanations in response to prompts. The server evaluates the validity of the generated answers and explanations, formats them appropriately, and then sends them to the terminal.

[1372] Step 5: Providing a response

[1373] 1. Input: The terminal receives answers and explanations sent from the server.

[1374] 2. Specific actions:

[1375] The terminal displays the received information on the user interface.

[1376] The user (child) reviews and understands the displayed explanation.

[1377] 3. Data processing and output: The user (child) solves the problem based on the explanation.

[1378] Step 6: Obtain and save feedback

[1379] 1. Input: The user (child) inputs their satisfaction level with the explanation using the terminal.

[1380] 2. Specific actions:

[1381] The user (child) chooses one option from choices such as "very satisfied," "satisfied," or "dissatisfied."

[1382] The user (child) clicks the "Send" button.

[1383] 3. Data Processing and Output: The terminal sends the input feedback to the server. The server receives the feedback and stores it in a database. The server uses this feedback to improve the accuracy of future learning support.

[1384] (Application Example 1)

[1385] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1386] Traditional learning support systems offer limited means for users to receive prompt and appropriate support when they encounter difficulties. Furthermore, there is a lack of effective learning support methods in physical stores, resulting in insufficient support for customers to effectively utilize learning materials. There is also a need for improved support accuracy based on feedback and the provision of interactive learning experiences for in-store use.

[1387] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1388] In this invention, the server includes means for taking an image of the learning content that the user is having trouble with and inputting a descriptive text; means for the terminal to send the captured image and descriptive text to the server; means for the server to extract text data from the image using OCR technology; means for the server to analyze the extracted text data and classify it into an appropriate learning category; means for the server to generate appropriate answers and explanations based on the classified learning categories; means for the server to send the generated answers and explanations to the terminal; means for the terminal to display the received answers and explanations to the user; means for the terminal to send user feedback to the server and store it in a database; means for the user to receive learning support through a robot or smart glasses placed in a physical store; and means for acquiring and registering product information and accessing learning content. This enables interactive learning support in physical stores, allowing users to effectively solve problems and improve the accuracy of support by utilizing feedback information.

[1389] "Users" refer to children who receive learning support using the learning support system, and their guardians who provide that support.

[1390] "Device" refers to electronic devices used by users, such as smartphones, tablets, and personal computers.

[1391] A "server" refers to a cloud-based system, specifically a device that processes and stores data sent by users.

[1392] "OCR technology" refers to optical character recognition technology that automatically extracts text data from images.

[1393] "Robot" refers to a device placed in a physical store that provides learning support while interacting with users.

[1394] "Smart glasses" refer to devices that users wear to display information from learning materials or to acquire information using the user's gaze.

[1395] "Product information" refers to detailed information about learning materials and products, and is obtained through methods such as QR codes.

[1396] "Learning content" refers to information such as textbooks, workbooks, and explanatory materials that users refer to as they progress through their studies.

[1397] "Feedback" refers to users' opinions on the learning support provided, such as their satisfaction level and suggestions for improvement.

[1398] "Interactive learning support" refers to conversational support that allows users to receive learning assistance in real time.

[1399] This invention relates to a learning support system designed to provide users with quick and appropriate support when they encounter learning difficulties. The system consists of a user, a terminal, a server, and devices (robots, smart glasses) for providing learning support in physical stores.

[1400] System Configuration

[1401] 1. User

[1402] Users refer to children who need learning support and their guardians who support them.

[1403] 2. Terminal

[1404] The device primarily uses smartphones, tablets, or personal computers, and provides technology that allows users to take pictures of learning content and input explanatory text.

[1405] 3. Server

[1406] The server is a cloud-based system and has the following features:

[1407] Text data is extracted from images using OCR (Optical Character Recognition) technology.

[1408] The extracted text data is analyzed and classified into appropriate learning categories.

[1409] It generates appropriate answers and explanations based on the categorized learning categories.

[1410] The generated answers and explanations are sent to the device.

[1411] User feedback is stored in a database to improve the accuracy of future learning support.

[1412] 4. Learning support devices in physical stores

[1413] The robots and smart glasses placed within physical stores are devices that provide users with learning support. Through these devices, users can obtain and register product information and access learning content.

[1414] Basic program processing

[1415] 1. User Registration

[1416] Users register with the system using a terminal, and this information is sent to the server. The server stores the user information in a database and assigns a unique ID to each user.

[1417] 2. Learning support request

[1418] When a user encounters a problem with their learning material, they input text or images on their device and send them to the server. For example, if they can't solve a math problem, they might take a picture of an equation written in their notebook and send it with a simple explanation like, "Please tell me how to solve this."

[1419] 3. Image and text analysis

[1420] The server analyzes the received image using OCR technology and extracts text data. It then analyzes the extracted text data to determine which learning category the problem belongs to.

[1421] 4. Generating answers and explanations

[1422] The server generates appropriate answers and explanations based on the categorized learning category. For example, if asked how to solve a quadratic equation, it will create a step-by-step explanation using the quadratic formula.

[1423] 5. Providing a response

[1424] The server sends the generated answers and explanations to the terminal, and the terminal displays the received information to the user.

[1425] 6. Obtaining and saving feedback

[1426] Users input their satisfaction level with the provided answers and explanations and send this information to the server. The server stores this feedback in a database and uses the feedback information to improve the accuracy of future support.

[1427] 7. Learning support at physical stores

[1428] In physical stores selling educational materials, users can receive learning support using terminals, robots placed in the store, or smart glasses. Users obtain information related to the materials in the store via QR codes, etc., and receive learning support based on that information.

[1429] Specific example

[1430] For example, if a user doesn't know how to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework, they can take a picture of the equation with their device, type a description like "Please tell me how to solve it," and send it to the server. The server uses OCR technology to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Next, the server generates a step-by-step explanation using the quadratic formula and sends it to the device. The device displays this explanation to the user, who then solves the problem following the explanation. Furthermore, the user inputs their satisfaction level with the explanation and sends the feedback to the server. In physical stores, learning support can be more interactive by using robots or smart glasses.

[1431] Example of a prompt

[1432] "Please scan the QR code."

[1433] "Please briefly describe the problem you are experiencing (exit to finish)."

[1434] "Please enter the image path for the teaching material."

[1435] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1436] Step 1:

[1437] Users take pictures of learning materials using their smartphone or tablet camera and enter descriptive text. This input consists of images related to the learning material and text that describes their details. This data is prepared by the device.

[1438] Step 2:

[1439] The device sends the captured image and accompanying text to the server. The input image data and text data are uploaded to a cloud-based server via the internet. The server receives this data and proceeds to the next step.

[1440] Step 3:

[1441] The server extracts text data from the received image using OCR technology. Specifically, the server uses OCR software (for example, Tesseract OCR) to recognize character information within the image and extract it as text data. The input is image data, and the output is the text data within the image.

[1442] Step 4:

[1443] The server analyzes the extracted text data and classifies it into appropriate learning categories. The server uses a natural language processing (NLP) model to analyze the text data and classify it into specific learning categories (e.g., mathematics, science, history, etc.). The input is the text data after OCR processing, and the output is the learning categories.

[1444] Step 5:

[1445] The server generates appropriate answers and explanations based on the classified learning categories. A generative AI model (e.g., GPT-3) is used to generate explanations for questions and problems related to the classified categories. The input is detailed text data of the learning categories and problems, and the output is explanations and answers to the problems.

[1446] Step 6:

[1447] The server sends the generated answers and explanations to the terminal. The text data of the explanations and answers is then sent back to the user's terminal via the internet. The input is the generated explanation data, and the output is the explanation data displayed on the terminal.

[1448] Step 7:

[1449] The terminal displays the received answers and explanations to the user. The content of the explanations and answers is displayed on the terminal screen, and the user views it. The input is the explanation data sent from the server, and the output is the explanation data displayed to the user.

[1450] Step 8:

[1451] The user inputs feedback on the answers and explanations, and sends that feedback from the device to the server. The input is the user's feedback data, which the device sends to the server. The output is the transmission of the feedback data.

[1452] Step 9:

[1453] The server stores user feedback in a database and uses it to improve the accuracy of learning support. The input is feedback data, which the server stores in the database. The output is the stored feedback, which serves as data to improve the quality of learning support in the future.

[1454] Step 10:

[1455] Users receive learning support through robots or smart glasses placed in physical stores. In the store, users obtain product information via QR codes and request learning support from a server. This enables interactive learning support. Input is product information via QR codes, and output is learning support related to a specific product.

[1456] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1457] The present invention relates to a learning support system that allows users to receive prompt and appropriate support when they experience difficulties in learning, and further recognizes the user's emotions and provides support accordingly. The embodiments for carrying out the present invention will be described in detail below.

[1458] System Configuration

[1459] This system consists of users, devices, a server, and an emotion engine. Users include children who need learning support and their parents who support them. Devices are primarily smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage. The emotion engine is used to recognize the emotional state of the users.

[1460] Basic program processing

[1461] This system's program has the following functions:

[1462] 1. User Registration

[1463] The user registers with the system using a terminal. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[1464] 2. Learning support request

[1465] If a user (child) encounters difficulties with their studies, they can use the device to take a picture of the problem and enter a brief explanation. For example, if they don't know how to solve a math problem, they can take a picture of the equation written in their notebook and send it with the explanation, "Please tell me how to solve it."

[1466] 3. Image and text analysis

[1467] The server analyzes the received image using OCR technology and extracts text data. The extracted text data is then analyzed to determine which learning category the problem belongs to.

[1468] 4. Emotion recognition

[1469] The device analyzes the user's facial recognition and voice using an emotion engine to determine the user's emotional state. For example, it recognizes if the user is anxious or confused.

[1470] 5. Generating answers and explanations

[1471] The server generates appropriate answers and explanations based on the classified learning categories and the user's emotional state recognized by the emotion engine. For example, if the user is confused, a more detailed and helpful explanation will be provided.

[1472] 6. Providing a response

[1473] The server sends the generated answers and explanations to the device. The device displays this information to the user, and the user (child) solves the problem according to the explanation.

[1474] 7. Obtaining and saving feedback

[1475] The user (child) inputs their satisfaction level with the answers and explanations and sends the feedback to their device. The server receives this feedback, stores it in a database, and uses it to improve future learning support.

[1476] 8. Saving emotional data

[1477] The emotion data recognized by the emotion engine is also sent to the server and stored in the database. This allows for personalized support in subsequent sessions.

[1478] Specific example

[1479] For example, consider a situation where a user (child) is struggling to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. The user takes a picture of the equation with their device, enters a caption saying "Please tell me how to solve it," and sends it. The server uses OCR technology to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Simultaneously, an emotion engine analyzes the user's face and voice and recognizes that the user is confused. Based on this information, the server generates a step-by-step explanation using the quadratic formula, providing a detailed explanation with particularly clear language and concrete examples. The server sends the generated explanation to the device, which displays it to the user. The user (child) solves the problem according to the displayed explanation, and after solving the problem, enters their satisfaction level and sends it. The server stores the feedback and emotion data in a database and uses it for future support.

[1480] The above describes a specific embodiment for carrying out the present invention. This system allows parents to support their children at the appropriate time, and enables children to solve problems while maintaining the enjoyment of learning. Furthermore, the introduction of an emotion engine provides personalized support according to the user's emotional state, resulting in more effective learning support.

[1481] The following describes the processing flow.

[1482] Step 1: The user registers using their device. The user enters basic information such as their name, email address, and password.

[1483] Step 2: The terminal sends the entered user information to the server. The server receives this information.

[1484] Step 3: The server saves the received user information to the database and assigns a unique ID to each user.

[1485] Step 4: The server sends a notification to the terminal that user registration is complete. The terminal displays a registration completion message to the user.

[1486] Step 5: When the user (child) encounters a learning difficulty, they can use the device to take a picture of the problem and enter a brief description (e.g., "Please tell me how to solve this problem").

[1487] Step 6: The device sends the photo and description to the server. The server receives this data.

[1488] Step 7: The server uses OCR technology to extract text data from the received photo data. This converts the text information in the photo into digital data.

[1489] Step 8: The server analyzes the extracted text data to determine which learning category the problem belongs to (e.g., mathematics, science, history, etc.).

[1490] Step 9: The device sends the user's facial recognition and voice input data to the emotion engine. The emotion engine analyzes this data and determines the user's emotional state.

[1491] Step 10: The server generates appropriate answers and explanations based on the identified learning categories and the emotional state recognized by the emotion engine. For example, if the user is confused, a more detailed and helpful explanation will be provided.

[1492] Step 11: The server sends the generated answers and explanations to the user's (child's) device. The device receives this data and displays it to the user.

[1493] Step 12: The user (child) solves the problem by following the answers and explanations displayed on the device. If there are any questions during the problem-solving process, they can send another question.

[1494] Step 13: After the user (child) solves the problem and expresses their satisfaction with the explanation, they input their feedback into the device and send it to the server. The server receives this feedback data.

[1495] Step 14: The server saves the received feedback data and the emotion data recognized by the emotion engine to a database, which is then used to improve the accuracy of future learning support.

[1496] (Example 2)

[1497] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1498] Conventional learning support systems have a problem in that it is difficult for users to receive prompt and appropriate support when they encounter difficulties during learning, and in particular, they cannot provide individualized support that addresses the user's emotional state. Furthermore, they lack a means to effectively utilize user feedback on the support provided and improve the accuracy of future learning support.

[1499] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1500] In this invention, the server includes means for extracting text data from an image using optical character recognition technology, means for analyzing the extracted text data and classifying it into an appropriate learning category, and means for analyzing facial images and voice data and recognizing the user's emotional state. This makes it possible to quickly classify difficulties experienced by the user during learning into a specific learning category and to provide personalized support tailored to the user's emotional state. Furthermore, by utilizing user feedback and emotional data, the accuracy of future learning support can be improved.

[1501] "User" refers to anyone who uses the system to receive learning support, and primarily includes children who need learning support and the parents who support them.

[1502] A "device" is a device used by a user to interact with a system, and includes smartphones, tablets, and personal computers.

[1503] A "server" refers to a cloud-based system that is responsible for processing and storing data.

[1504] Optical Character Recognition (OCR) refers to a technology that analyzes and extracts text data from images.

[1505] A "learning category" refers to the specific field or topic to which the analyzed problem belongs, such as mathematics, science, or history.

[1506] "Emotional state" refers to the psychological and emotional state recognized from the user's facial image and voice data, and includes anxiety, confusion, joy, etc.

[1507] A "generative AI model" refers to a model that uses artificial intelligence technology to automatically generate answers and explanations to user questions.

[1508] "Feedback" refers to the evaluation and opinions of users regarding the support and explanations provided.

[1509] "Face image" refers to visual data obtained from a photograph or video capture of the user's face.

[1510] "Voice data" refers to data recorded from the user's speech or voice.

[1511] A "database" refers to a system for managing and storing data such as user information, feedback, and sentiment data.

[1512] This invention is a learning support system that provides prompt and appropriate support when a user encounters difficulties during learning, and further recognizes the user's emotional state and provides personalized support accordingly. This system consists of a user, a terminal, and a server.

[1513] System Configuration

[1514] This system consists of users, devices, a server, and an emotion engine. Users primarily include children who need learning support and their parents who support them. Devices mainly consist of smartphones, tablets, or personal computers. The server is a cloud-based system responsible for data processing and storage. The emotion engine is used to recognize the emotional state of the users.

[1515] Basic program processing

[1516] This system's program has the following main functions:

[1517] 1. User Registration

[1518] Users register with the system using a terminal. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[1519] 2. Acceptance of learning support requests

[1520] If a user (child) encounters difficulties with their studies, they can use the device to take a picture of the problem and enter a brief explanation. For example, if they don't know how to solve a math problem, they can take a picture of the equation written in their notebook and send it with the explanation, "Please tell me how to solve it."

[1521] 3. Image and text analysis

[1522] The server analyzes the received image using optical character recognition (OCR) technology and extracts text data. For example, Tesseract OCR is used. The extracted text data is then analyzed, and a natural language processing library (such as NLTK) is used to determine which learning category the problem belongs to.

[1523] 4. Emotion recognition

[1524] The device analyzes the user's facial image and voice data using an emotion engine to determine the user's emotional state. For example, it uses facial recognition technology (such as the Microsoft Emotion API) to recognize emotions such as anxiety or confusion.

[1525] 5. Generating answers and explanations

[1526] The server uses a generative AI model to generate appropriate answers and explanations based on classified learning categories and the user's emotional state recognized by the emotion engine. For example, GPT-4 is used as one of the generative AI models. A confused user will be provided with a more detailed and helpful explanation.

[1527] 6. Providing a response

[1528] The server sends the generated answers and explanations to the terminal. The terminal displays the received information to the user, and the user (child) solves the problem according to the explanation.

[1529] 7. Obtaining and saving feedback

[1530] The user (child) inputs their satisfaction level with the answers and explanations and sends the feedback to their device. The server receives this feedback, stores it in a database, and uses it to improve future learning support.

[1531] 8. Saving emotional data

[1532] The emotion data recognized by the emotion engine is also sent to the server and stored in the database. This allows for personalized support in subsequent sessions.

[1533] Specific example

[1534] For example, consider a situation where a user (child) is struggling to solve the quadratic equation "2x² + 3x - 5 = 0" for their math homework. They take a picture of the equation with their device, type a message saying "Please explain how to solve it," and send it. The server uses OCR technology (Tesseract OCR) to extract the text "2x² + 3x - 5 = 0" from the received image and identifies it as a quadratic equation problem. Simultaneously, the emotion engine (Microsoft Emotion API) analyzes the user's face and voice, recognizing their confusion. Based on this information, the server uses a generative AI model (GPT-4) to generate a step-by-step explanation using the quadratic formula, providing detailed explanations with clear language and concrete examples. The server sends the generated explanation to the device, which displays it to the user. The user (child) solves the problem following the displayed explanation and, after solving it, inputs their satisfaction level and sends it. The server stores the feedback and emotion data in a database for future support.

[1535] For example, here is an example of a prompt statement for a generative AI model (GPT-4):

[1536] A child is having trouble solving the quadratic equation "2x² + 3x - 5 = 0". Please provide a step-by-step, detailed explanation using easy-to-understand language.

[1537] The user's emotional state is one of confusion.

[1538] This system allows parents to support their children at the right time, and enables children to solve problems while maintaining the enjoyment of learning. Furthermore, the introduction of an emotion engine provides personalized support tailored to the user's emotional state, resulting in more effective learning assistance.

[1539] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1540] Step 1:

[1541] User Registration

[1542] Input: User information (name, email address, age, etc.)

[1543] Operation: The user registers with the learning support system using a terminal. The terminal sends the entered information to the server. The server stores the received user information in a database and generates a unique user ID. The generated user ID is sent back to the terminal, and the terminal notifies the user that "registration is complete."

[1544] Output: Unique User ID

[1545] Step 2:

[1546] Acceptance of learning support request

[1547] Input: Image of the learning material problem, a brief description of the problem.

[1548] Operation: When the user (child) encounters difficulty with the learning material, they use the device to take a picture of the problem and enter a description such as "Please tell me how to solve it." The device then sends the captured image and description to the server.

[1549] Output: The problem image and description are sent to the server.

[1550] Step 3:

[1551] Image and text analysis

[1552] Input: Problem image, description

[1553] Operation: The server analyzes the received image using optical character recognition (OCR) technology and extracts text data. Tesseract OCR is used as the OCR technology. The extracted text data is analyzed using a natural language processing library (such as NLTK) to determine which learning category the problem belongs to.

[1554] Output: Text data, training categories

[1555] Step 4:

[1556] emotion recognition

[1557] Input: User's face image, voice data

[1558] Operation: The device captures a picture of the user's face with its camera and records their voice with its microphone. The device sends this data to the server. The server uses facial recognition technology (such as Microsoft Emotion API) to analyze the user's emotional state. The server recognizes specific emotions, such as anxiety or confusion.

[1559] Output: User's emotional state

[1560] Step 5:

[1561] Generating answers and explanations

[1562] Input: Text data, learning categories, sentiment state

[1563] Operation: The server uses a generative AI model (such as GPT-4) to generate appropriate answers and explanations based on the analyzed text data and sentiment data. For example, if the problem involves a quadratic equation, it will generate a detailed explanation using the quadratic formula.

[1564] Output: Answers and explanations

[1565] Step 6:

[1566] Providing a response

[1567] Input: Answer or explanation

[1568] Operation: The server sends the generated answers and explanations to the terminal. The terminal displays the received information to the user. The user solves the problem according to the displayed explanations.

[1569] Output: Explanation text displayed to the user

[1570] Step 7:

[1571] Obtaining and saving feedback

[1572] Input: User feedback (satisfaction ratings and comments)

[1573] Operation: After the user resolves a problem, the device displays a feedback form. The user enters a satisfaction rating and comments, and sends them to the server via the device. The server stores the received feedback in its database.

[1574] Output: Saved feedback data

[1575] Step 8:

[1576] Storage of emotional data

[1577] Input: User's emotional state data

[1578] Operation: The server stores the user's emotional data, recognized by the emotion engine, in a database. This allows for personalized support in subsequent learning support sessions.

[1579] Output: Saved sentiment data

[1580] (Application Example 2)

[1581] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1582] Conventional learning support systems have limited means of providing users with quick and appropriate support when they encounter difficulties. Furthermore, providing individualized support tailored to the user's emotional state is challenging. Additionally, in physical stores, there is a lack of means to provide appropriate information when customers have difficulty choosing products. The time spent manually searching for information contributes to decreased customer satisfaction. To address these challenges, a system is needed that provides quick and appropriate support when users encounter difficulties, and furthermore, individualized support tailored to their emotional state. Moreover, efficient information provision utilizing emotion recognition is required for product selection in physical stores.

[1583] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1584] In this invention, the server includes means for acquiring product information using a smart device, means for transmitting the product information acquired by the terminal and customer requests to the server, and means for the server to analyze the product information and customer requests and determine the emotional state using an emotion recognition engine. This enables rapid and appropriate learning support and personalized product recommendations in physical stores.

[1585] A "user" refers to an individual who needs learning support or assistance with product selection.

[1586] "Terminal" refers to devices such as smartphones, tablets, and personal computers that are operated by the user.

[1587] A "server" refers to a cloud-based system that handles data processing and storage, and generates appropriate support for users.

[1588] "Images" refers to photographic data of learning materials or products taken by the user using their device.

[1589] "Description" refers to the text data that users input regarding learning content, product issues, or requests.

[1590] "OCR technology" refers to optical character recognition technology used to extract text data from images.

[1591] "Learning categories" refer to educational fields or subjects used to classify the text data analyzed by the server.

[1592] "Answers and explanations" refers to specific explanations and suggestions regarding learning support and product information generated by the server.

[1593] "Feedback" refers to the evaluation and opinions of users regarding the support and suggestions they receive.

[1594] A "database" refers to a system used to store data such as user information and feedback.

[1595] A "smart device" refers to an internet-connected device operated by a user, such as a smartphone or smart glasses.

[1596] "Product information" refers to data about the features and details of a product.

[1597] "Customer requirements" refers to the preferences and questions that customers enter when selecting a product.

[1598] An "emotion recognition engine" refers to software that analyzes a user's emotional state from their facial expressions and voice.

[1599] "Emotional state" refers to the user's psychological state as determined by the emotion recognition engine.

[1600] "Product suggestions" refer to product suggestions generated by the server based on customer requests and emotional states.

[1601] The system of the present invention uses smart devices, servers, and an emotion recognition engine to provide learning support and in-store shopping support. The embodiments for carrying out the present invention are described in detail below.

[1602] System Configuration

[1603] This system primarily consists of a user, a smart device, a server, and an emotion engine. The user is an individual who utilizes the system for learning support or product selection. Smart devices include smartphones and smart glasses. The server is a cloud-based system responsible for data processing and storage. The emotion engine is responsible for recognizing the user's emotional state.

[1604] Basic program processing

[1605] This system's program has the following functions:

[1606] 1. User Registration

[1607] Users register with the system using their smart devices. User information is sent to the server, and a unique ID is assigned. The server stores this information in a database.

[1608] 2. Product support request

[1609] If a user has trouble choosing a product, they can use their smart device to scan the product's barcode or QR code and enter a simple request. For example, they might enter, "I want to know more details about this product."

[1610] 3. Image and text analysis

[1611] The server uses OCR technology to analyze the received product information and extract text data. The extracted text data is then analyzed to understand the characteristics and requirements of the requested product.

[1612] 4. Emotion recognition

[1613] Smart devices use an emotion engine to analyze the user's facial expressions and voice to determine the user's emotional state. For example, they can determine whether the user is confused or distressed.

[1614] 5. Generating answers and suggestions

[1615] The server generates appropriate responses and suggestions based on product information and the user's emotional state, as recognized by the emotion engine. For example, if the user is confused, it will provide detailed product information and suggest related products.

[1616] 6. Providing a response

[1617] The server sends generated answers and suggestions to the smart device and displays them to the user. The user can then use this information to select products.

[1618] 7. Obtaining and saving feedback

[1619] Users input their satisfaction level with the support and suggestions they receive as feedback and send it to the server from their smart device. The server stores this feedback in a database and uses it to improve future support.

[1620] Hardware and software to be used

[1621] The following hardware and software are used to implement this system:

[1622] Hardware: Smart devices (smartphones, smart glasses), cloud servers

[1623] Software: Emotion recognition engine (e.g., Microsoft Azure Emotion API), OCR technology (e.g., Google Vision API), database system (e.g., Amazon RDS)

[1624] Specific example

[1625] For example, consider a scenario where a user is having trouble choosing a product in a physical store. Using smart glasses, the user scans the product's barcode and enters a request saying, "I want to know more details about this product." The server extracts the product's text information from the barcode using OCR technology and recognizes the user's confusion from their facial expressions and voice using an emotion engine. Based on this information, the server generates a detailed product description and related product suggestions, which are then sent to the smart glasses for the user to see. The user selects a product based on the displayed information and enters their final satisfaction level as feedback. The server saves this feedback and uses it to improve the accuracy of future support.

[1626] Example of a prompt

[1627] Prompt text to input to the generative AI model:

[1628] "If a customer appears confused, provide a detailed explanation of how to use the product and related products. For example, include product features and reviews from other customers in your suggestions."

[1629] As described above, the system of the present invention provides users with prompt and appropriate support in both learning support and in-store shopping experiences, and further enables personalized responses tailored to their emotional state.

[1630] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1631] Step 1: User Registration

[1632] Users register with the system using a smart device (e.g., a smartphone).

[1633] Input: User's basic information (name, email address, etc.)

[1634] Output: Assign a unique ID to the user and send the information to the server.

[1635] Specific operation: The user enters information into the application form and presses the "Submit" button. The terminal encrypts the entered information and sends it to the server. The server stores the received information in a database and generates a unique user ID.

[1636] Step 2: Product Support Request

[1637] When users have trouble choosing a product, they can use their smart device to scan the product's barcode or QR code and enter their request.

[1638] Input: Product barcode or QR code, request text (e.g., "I want to know more details about this product")

[1639] Output: Sends product information and requests to the server.

[1640] Specific operation: The user scans a barcode using the camera on their smart device and enters the request. The device packages the image data and text and sends it to the server.

[1641] Step 3: Image and text analysis

[1642] The server analyzes the received product information using OCR technology and extracts text data.

[1643] Input: Barcode image, text request

[1644] Output: Extracted text data

[1645] Specific operation: The server uses OCR technology such as the Google Vision API to extract text from barcode images. It then combines this with the text request and performs analysis.

[1646] Step 4: Emotion Recognition

[1647] The system analyzes the user's facial expressions and voice captured by a smart device to determine their emotional state.

[1648] Input: User's facial expression image, audio data

[1649] Output: Data on emotional state (e.g., confusion, anxiety, excitement, etc.)

[1650] Specific operation: The smart device's camera and microphone are used to collect the user's facial expressions and voice, and these are sent to an emotion recognition engine (e.g., Microsoft Azure Emotion API). The server analyzes the acquired data and determines the emotional state.

[1651] Step 5: Generating answers and suggestions

[1652] The server generates appropriate responses and suggestions based on emotional states and product information.

[1653] Input: Emotional state data, product information

[1654] Output: Text data of appropriate answers and suggestions

[1655] Specific operation: Based on emotional states and product information, the server uses pre-configured rules and generative AI models to generate responses and suggestions in a format that is easy for the user to understand.

[1656] Step 6: Providing a response

[1657] The server sends the generated answers and suggestions to the smart device and displays them to the user.

[1658] Input: Text data of generated responses and suggestions

[1659] Output: Responses and suggestions displayed on the user's smart device.

[1660] Specific operation: The server sends the generated answers and suggestions to the terminal, and the terminal displays the received data to the user. The user then selects a product based on this information.

[1661] Step 7: Obtain and save feedback

[1662] Users input feedback on the support and suggestions they receive and send it from their device to the server.

[1663] Input: Text data of ratings and opinions entered by the user.

[1664] Output: Save the feedback data to the server's database.

[1665] Specific operation: The user uses a smart device to enter a rating for support or suggestions and presses the "Submit" button. The device sends the entered feedback data to the server, which stores it in a database.

[1666] By following the steps outlined above, the program's processing is executed, allowing users to receive prompt and accurate support in learning assistance and product selection.

[1667] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1668] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1669] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1670] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1671] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1672] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1673] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1674] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1675] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1676] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1677] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1678] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1679] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1680] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1681] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1682] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1683] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1684] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1685] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1686] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1687] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1688] The following is further disclosed regarding the embodiments described above.

[1689] (Claim 1)

[1690] A method for users to take a picture of the learning content they are having trouble with and input a description,

[1691] A means for the device to send images and descriptions to a server,

[1692] A server is a means of extracting text data from an image using OCR technology,

[1693] The server analyzes the extracted text data and classifies it into appropriate learning categories,

[1694] A means by which the server generates appropriate answers and explanations based on classified learning categories,

[1695] A means by which the server sends the generated answers and explanations to the terminal,

[1696] A means of displaying the answers and explanations received by the terminal to the user,

[1697] A means of sending user feedback from the device to the server and storing it in a database,

[1698] A system that includes this.

[1699] (Claim 2)

[1700] The system according to claim 1, further comprising means for the server to format the generated answers and explanations into a format that is easy for the user to understand.

[1701] (Claim 3)

[1702] The system according to claim 1, further comprising means by which the server utilizes user feedback to improve the accuracy of future learning support.

[1703] (Claim 4)

[1704] The system according to claim 1, further comprising means for a terminal to input user information and perform user registration.

[1705] (Claim 5)

[1706] The system according to claim 1, further comprising means for the server to store new user information in a database and assign a unique ID.

[1707] "Example 1"

[1708] (Claim 1)

[1709] A method for users to take a picture of the learning content they are having trouble with and input a description,

[1710] A means for the device to send images and descriptions to a server,

[1711] A server extracts text data from an image using optical character recognition technology,

[1712] The server analyzes the extracted text data and classifies it into appropriate learning categories,

[1713] A means of using a generative AI model to generate appropriate answers and explanations based on the classified learning categories of the server,

[1714] A means by which the server sends the generated answers and explanations to the terminal,

[1715] A means of displaying the answers and explanations received by the terminal to the user,

[1716] A means of sending user feedback from the device to the server and storing it in a database,

[1717] A system that includes this.

[1718] (Claim 2)

[1719] The system according to claim 1, further comprising means for the server to format the generated answers and explanations into a format that is easy for the user to understand.

[1720] (Claim 3)

[1721] The system according to claim 1, further comprising means by which the server utilizes user feedback to improve the accuracy of future learning support.

[1722] "Application Example 1"

[1723] Revised claims:

[1724] (Claim 1)

[1725] A method for users to take a picture of the learning content they are having trouble with and input a description,

[1726] A means for the device to send images and descriptions to a server,

[1727] A server is a means of extracting text data from an image using OCR technology,

[1728] The server analyzes the extracted text data and classifies it into appropriate learning categories,

[1729] A means by which the server generates appropriate answers and explanations based on classified learning categories,

[1730] A means by which the server sends the generated answers and explanations to the terminal,

[1731] A means of displaying the answers and explanations received by the terminal to the user,

[1732] A means of sending user feedback from the device to the server and storing it in a database,

[1733] A means for users to receive learning support through robots or smart glasses placed in physical stores,

[1734] A means of obtaining and registering product information and accessing learning content,

[1735] A system that includes this.

[1736] (Claim 2)

[1737] The system according to claim 1, further comprising means for the server to format the generated answers and explanations into a user-friendly format, thereby realizing interactive learning support in the store.

[1738] (Claim 3)

[1739] The system according to claim 1, further comprising means by which the server utilizes user feedback to improve the accuracy of future learning support, and which improves the accuracy of learning content based on the registration and updating of product information.

[1740] "Example 2 of combining an emotion engine"

[1741] (Claim 1)

[1742] A method for users to take a picture of the learning content they are having trouble with and input a description,

[1743] A means for the device to send images and descriptions to a server,

[1744] A server extracts text data from an image using optical character recognition technology,

[1745] The server analyzes the extracted text data and classifies it into appropriate learning categories,

[1746] A means by which the terminal transmits the user's facial image and voice data to a server,

[1747] The server analyzes facial images and voice data to recognize the user's emotional state,

[1748] A means by which a server uses a generative AI model to generate appropriate answers and explanations based on learning categories and emotional states,

[1749] A means by which the server sends the generated answers and explanations to the terminal,

[1750] A means of displaying the answers and explanations received by the terminal to the user,

[1751] A means of sending user feedback from the device to the server and storing it in a database,

[1752] A means by which the server stores emotional data in a database,

[1753] A system that includes this.

[1754] (Claim 2)

[1755] The system according to claim 1, further comprising means for the server to format the generated answers and explanations into a format that is easy for the user to understand.

[1756] (Claim 3)

[1757] The system according to claim 1, further comprising means by which the server utilizes user feedback and sentiment data to improve the accuracy of future learning support.

[1758] "Application example 2 when combining with an emotional engine"

[1759] (Claim 1)

[1760] A method for users to take a picture of the learning content they are having trouble with and input a description,

[1761] A means for the device to send images and descriptions to a server,

[1762] A server is a means of extracting text data from an image using OCR technology,

[1763] The server analyzes the extracted text data and classifies it into appropriate learning categories,

[1764] A means by which the server generates appropriate answers and explanations based on classified learning categories,

[1765] A means by which the server sends the generated answers and explanations to the terminal,

[1766] A means of displaying the answers and explanations received by the terminal to the user,

[1767] A means of sending user feedback from the device to the server and storing it in a database,

[1768] Methods for obtaining product information using smart devices,

[1769] A means for transmitting product information acquired by the terminal and customer requests to a server,

[1770] A server analyzes product information and customer requests, and uses an emotion recognition engine to determine the emotional state.

[1771] A server provides a means for generating appropriate product suggestions and descriptions based on emotional states,

[1772] A system that includes this.

[1773] (Claim 2)

[1774] The system according to claim 1, further comprising means for the server to format the generated answers and explanations into a format that is easy for the user to understand.

[1775] (Claim 3)

[1776] The system according to claim 1, further comprising means by which the server utilizes user feedback to improve the accuracy of future learning support and product recommendations. [Explanation of Symbols]

[1777] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A method for users to take a picture of the learning content they are having trouble with and input a description, A means for the device to send images and descriptions to a server, A server is a means of extracting text data from an image using OCR technology, The server analyzes the extracted text data and classifies it into appropriate learning categories, A means by which the server generates appropriate answers and explanations based on classified learning categories, A means by which the server sends the generated answers and explanations to the terminal, A means of displaying the answers and explanations received by the terminal to the user, A means of sending user feedback from the device to the server and storing it in a database, A system that includes this.

2. The system according to claim 1, further comprising means for the server to format the generated answers and explanations into a format that is easy for the user to understand.

3. The system according to claim 1, further comprising means by which the server utilizes user feedback to improve the accuracy of future learning support.

4. The system according to claim 1, further comprising means for a terminal to input user information and perform user registration.

5. The system according to claim 1, further comprising means for the server to store new user information in a database and assign a unique ID.

Images