System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving homework information answered by a user; means for analyzing the received homework information and converting the answers to textual data; means for sending the textual data to a production AI to obtain correct answers and feedback; means for sending feedback from the production AI to the user; means for recording the user's answer history and feedback and tracking progress; and means for analyzing the user's understanding of learning and providing self-learning tips and materials.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As children's learning load increases, it is becoming more difficult for parents to provide adequate support. This has resulted in a decline in children's learning efficiency and motivation to learn. Furthermore, it is difficult to provide effective feedback and support that is tailored to each child's level of understanding and progress. Therefore, there is a need to provide a system that can solve these issues, increase children's motivation to learn, and help them learn efficiently. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for receiving information about homework completed by a user, a means for analyzing the received homework information and converting the answers into text data, a means for sending the text data to a generation AI to determine whether the answers are correct and receive feedback, a means for sending the feedback obtained from the generation AI to the user, a means for recording the user's answer history and feedback and tracking progress, and a means for analyzing the user's level of understanding and providing self-study hints and learning materials. This system allows children to send their homework to the AI ​​and receive instant feedback. Furthermore, by tracking and analyzing the user's level of understanding and progress, optimal learning support tailored to individual needs is possible, improving motivation and efficiency in learning.

[0006] "User" is a term used to refer to children and their parents who use the system to answer homework questions and receive feedback.

[0007] "Homework information" is a term that indicates the content of homework, such as the questions that children have answered, images of the questions, and text data.

[0008] "Means" is a term that refers to the functions, processes, and techniques that a system uses to perform a specific process.

[0009] "Receiving" is a term that describes the process by which a system obtains data sent by a user.

[0010] "Analysis" is a term that refers to the process of breaking down and interpreting received homework information and converting it into desired information.

[0011] "Text data" is a term that refers to information in text format, such as character string data or numerical data obtained by analysis.

[0012] "Generative AI" is a term used to describe algorithms or models that use artificial intelligence techniques to determine answers to problems and generate feedback.

[0013] "Correctness determination" is a term that refers to the process of determining whether the answer submitted by the user is correct.

[0014] "Feedback" is a term that refers to detailed explanations and advice about the results of correct answers, as well as information that guides the next learning step.

[0015] "Transmission" is the term used to describe the process by which a system sends data or feedback back to a user.

[0016] "History" is a term that refers to a record of a user's answers and feedback to date.

[0017] "Progress" is a term used to describe data used to track a user's learning progress and understanding.

[0018] "Self-study tips" is a term that refers to advice and methods that can be used as reference when users are studying on their own.

[0019] "Instructional materials" is a term that refers to supplementary materials, resources, video lectures, etc. that support learning. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention is a system for receiving information on homework answers provided by users, analyzing, evaluating, and providing feedback on the answers. This system is realized through cooperation between users, terminals, and a server.

[0042] User operations

[0043] First, the user submits the homework answer using the device. The answer may be submitted as a photo or as text. For example, the user may take a photo of the notebook in which they have completed an arithmetic problem and save it on the device. The image file is then uploaded to the server.

[0044] Device Features

[0045] The device sends photos taken by the user and text entered by the user to the server. When sending, the device also sends identification information such as the user ID. This information is used by the server to process the data. The device also displays feedback from the server to the user. This allows the user to instantly receive the evaluation results for their answers.

[0046] Server Processing

[0047] The server receives the homework information sent from the device. If the received information is photo data, it first uses optical character recognition (OCR) technology to convert the letters and formulas in the photo into text data. The converted text data is then sent to the generation AI, which determines whether the answer is correct.

[0048] The generation AI analyzes the received text data and determines whether the answer is correct or incorrect. If the answer is incorrect, it generates detailed feedback about which part is incorrect and how to derive the correct answer. This feedback is sent back to the server, which then sends it to the user.

[0049] Recording and analyzing learning

[0050] The server stores the user's answer history and feedback in a database. This allows the server to track the questions the user has answered in the past and the feedback they received. Furthermore, by analyzing this data, the server can understand the user's level of understanding and progress and provide appropriate study tips and materials.

[0051] Example

[0052] For example, consider the case where a user solves the math problem "6 x 7 = ?" and answers "44."

[0053] 1. The user writes this answer in a notebook and takes a photo of the notebook.

[0054] 2. The device sends the photo it took to the server.

[0055] 3. The server uses OCR technology to convert the photo into text data ("6 x 7 = 44").

[0056] 4. The server sends this text data to the generation AI.

[0057] 5. The generation AI determines that "44" is an incorrect answer and generates feedback saying, "The correct answer is 42. Let's move on to the next step."

[0058] 6. The server receives this feedback and sends it to the user's device.

[0059] 7. The device displays the feedback to the user, who confirms it.

[0060] In this way, the system provides efficient learning support by instantly evaluating the user's homework answers and providing detailed feedback.The system also stores the user's learning history in a database and analyzes their understanding and progress, making it possible to provide hints and learning materials for self-study.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The user prepares the homework answers by writing them in a notebook or by typing them directly into the device.

[0064] Step 2:

[0065] The user takes a photo of the note on the device or inputs text. For example, the user takes a photo of the note using a smartphone or tablet and saves it on the device.

[0066] Step 3:

[0067] The device sends the homework information from the user to the server. The information sent includes photo data, text data, and identification information such as the user ID.

[0068] Step 4:

[0069] The server receives the homework information sent from the device, temporarily stores the received data, and prepares it for processing.

[0070] Step 5:

[0071] The server analyzes the received homework information. If the homework is a photo, it uses optical character recognition (OCR) technology to convert the characters and mathematical formulas in the image into text data.

[0072] Step 6:

[0073] The server sends the text data to the AI ​​that generates it, which then analyzes the received text data and determines the correct answer.

[0074] Step 7:

[0075] The generation AI determines whether the user's answer is correct and generates feedback. Based on the data it receives, the generation AI determines whether the user's answer is correct, and if it is incorrect, generates detailed feedback on how the answer should be answered.

[0076] Step 8:

[0077] The server receives the feedback from the generated AI and prepares it to be sent to the user. The feedback includes correct and incorrect answers and detailed explanations.

[0078] Step 9:

[0079] The device receives feedback from the server and displays it to the user, who can then check the feedback and determine whether their answer was correct or incorrect.

[0080] Step 10:

[0081] The server stores the user's answer history and feedback in a database, which provides data for later analysis of the user's learning progress and understanding.

[0082] Step 11:

[0083] The server analyzes the user's level of understanding and progress, and evaluates the user's stumbling points and level of understanding based on past answer history and feedback.

[0084] Step 12:

[0085] The server provides self-study tips and materials based on the user's learning needs, such as relevant video lectures and additional practice questions.

[0086] In this way, the system not only evaluates users' homework and provides immediate feedback, but also provides long-term learning support. By continuously tracking each user's learning progress and providing appropriate support as needed, the system aims to motivate students to learn more efficiently.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] Traditional learning systems have the problem that it takes time to evaluate and provide feedback on users' homework answers, making it difficult to track users' understanding and progress in real time. This prevents users from receiving appropriate feedback immediately, resulting in reduced learning efficiency. Furthermore, there is a lack of a means to accumulate past answer history and feedback in a database and manage learning progress over the long term, making it difficult to properly evaluate users' understanding and skill gaps.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes: a means for receiving information about homework completed by a user; a means for analyzing the received homework information and converting the answers into text data; a means for sending the text data to a generative AI model to determine whether the answers are correct and receive feedback; a means for sending the feedback obtained from the generative AI model to the user; a means for recording the user's answer history and feedback and tracking progress; and a means for analyzing the user's level of understanding and providing self-study hints and learning materials. This allows the user to receive evaluations and feedback on their completed homework in real time, improving learning efficiency. It also enables the long-term accumulation and analysis of learning data, allowing for an accurate understanding of the user's level of understanding and progress, enabling appropriate learning guidance.

[0092] "User" refers to an individual who uses the system to submit homework answers and receive evaluations and feedback.

[0093] "Homework information" refers to the content of the homework that the user has answered, and is sent to the server as text data or photo data.

[0094] The "receiving means" is a mechanism by which the server receives the homework information sent by the user.

[0095] The "analysis means" is a technology that allows the server to analyze the homework information received and convert it into text data as needed.

[0096] A "generative AI model" is an artificial intelligence model that analyzes received text data and generates correct answers and feedback.

[0097] The "means of transmission" is a mechanism for transmitting feedback obtained from the generative AI model to the user's device.

[0098] "Means for recording and tracking progress" refers to technology for storing a user's answer history and feedback content, and for managing the user's learning progress over the long term.

[0099] "Means for providing self-study hints and learning materials" is a mechanism for analyzing the user's level of understanding and providing appropriate learning advice and learning materials.

[0100] The "means of receiving the photograph and converting it into text data using optical character recognition technology" is a technology that converts the photograph data sent by the user into text data using optical character recognition technology.

[0101] The "database" is a system that serves as the foundation for storing and analyzing users' learning progress and level of understanding.

[0102] A "skill gap" refers to the difference that exists between a user's current level of understanding and their desired level of understanding.

[0103] The present invention is a system for receiving information on homework answers provided by users, analyzing and evaluating the information, and providing feedback. This system functions based on the cooperation of users, terminals, and a server.

[0104] First, the user enters the answer to their homework using their device and sends it to the server. The answer may be sent as a photo or directly entered as text. For example, if a user solves an arithmetic problem and writes the answer in a notebook, they can take a photo of the notebook using a device such as a smartphone or tablet and save the image file on the device. They can then upload the image file to the server using a dedicated application.

[0105] The device sends identification information such as the user ID to the server along with the photo data taken by the user and the text data entered by the user. This allows the server to process the data appropriately. The device also receives evaluation results and feedback from the server and displays them to the user. This allows users to instantly check the evaluation results of their answers and improve the quality of their learning.

[0106] The server first receives the homework information sent from the device. If the received information is photo data, the server uses optical character recognition (OCR) technology to convert the characters and mathematical formulas in the photo into text data. Specifically, OCR software such as Google Cloud Vision API is used. This converted text data is then sent to a generative AI model. A generative AI model is an artificial intelligence that analyzes the text data and generates a correct answer and detailed feedback. A typical generative AI model is OpenAI GPT-4.

[0107] The generative AI model analyzes the accuracy of the answer based on the received text data and determines whether it is correct or incorrect. If the answer is incorrect, the generative AI model generates detailed feedback about which part is incorrect and how to derive the correct answer. This generated feedback is sent back to the server, which then sends it to the user's device. The feedback content is saved in a database and maintained as the user's learning history.

[0108] As a specific example of how it works, suppose a user solves the problem "6 x 7 = ?". The user writes the answer "44" in a notebook, takes a photo of it, and uses their device to send the photo to the server. The device sends the photo data and user ID to the server. The server uses OCR technology to convert the photo data into text data "6 x 7 = 44" and sends it to the generative AI model. The generative AI model determines that "44" is an incorrect answer and generates feedback saying "The correct answer is 42. Let's proceed to the next step," which is sent back to the server. The server sends this feedback to the user's device, which then displays the feedback.

[0109] An example of a prompt for a generative AI model is as follows:

[0110] "The user answered the following question: 6 x 7 = 44. Please analyze and evaluate this answer to generate corrective feedback."

[0111] This system allows users to receive instant, detailed feedback, enabling them to study efficiently. Furthermore, by analyzing the learning history stored in the database, it is possible to accurately grasp the user's level of understanding and progress, and provide appropriate learning guidance and teaching materials as needed.

[0112] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0113] Step 1:

[0114] The user enters the homework answer.

[0115] Input: Homework answered by the user (photo data or text data)

[0116] How it works: A user writes their homework answers in a notebook and takes a photo of it with their smartphone or tablet, or types it in as text.

[0117] Output: Homework answer data (photo or text)

[0118] Step 2:

[0119] The user sends the answer data through the terminal.

[0120] Input: User's homework answer data, user ID

[0121] How it works: A user opens the app, uploads a photo or text of their answer, and presses the submit button, automatically attaching their user ID.

[0122] Output: Homework answer data and user ID sent to the device

[0123] Step 3:

[0124] The device sends the answer data and user ID to the server.

[0125] Input: Answer data, user ID

[0126] Operation: After the send button is pressed, the terminal sends the answer data and user ID to the server.

[0127] Output: Answer data and user ID sent to the server

[0128] Step 4:

[0129] The server receives the answer data and the user ID.

[0130] Input: Answer data and user ID sent from the device

[0131] Operation: The server uses the receiving module to receive the answer data and the user ID.

[0132] Output: Answer data and user ID saved on the server

[0133] Step 5:

[0134] The server analyzes the answer data.

[0135] Input: Received answer data

[0136] How it works: The server analyzes the answer data and converts it into text using optical character recognition (OCR) technology, such as Google Cloud Vision API or similar OCR software.

[0137] Output: Parsed text data

[0138] Step 6:

[0139] The server generates prompts that send text data to the generative AI model.

[0140] Input: Parsed text data

[0141] How it works: The server uses the prompt generation module to create a prompt sentence to send to the generative AI model.

[0142] Output: Generated prompt statement

[0143] Step 7:

[0144] The server sends the prompt to the generative AI model.

[0145] Input: Generated prompt text

[0146] How it works: The server sends a prompt to a generative AI model (e.g., OpenAI GPT-4).

[0147] Output: Feedback generated by the generative AI model

[0148] Step 8:

[0149] A generative AI model generates an evaluation and feedback on the answer.

[0150] Input: prompt statement

[0151] How it works: A generative AI model analyzes prompts, determines whether an answer is correct or incorrect, and generates detailed feedback.

[0152] Output: Generated feedback

[0153] Step 9:

[0154] The server receives the feedback and sends it to the user's device.

[0155] Input: Feedback from the generative AI model, user ID

[0156] Operation: The server uses a feedback receiving module to receive feedback from the generative AI model and sends it to the device with the corresponding user ID.

[0157] Output: Feedback sent to the terminal

[0158] Step 10:

[0159] The terminal displays the received feedback to the user.

[0160] Input: Feedback sent by the server

[0161] How it works: The device receives the feedback and displays it to the user via a dedicated application.

[0162] Output: Displayed feedback

[0163] Step 11:

[0164] The server records and stores the user's answer history and feedback.

[0165] Input: Answer history and feedback received

[0166] How it works: The server stores answer history and feedback in a database.

[0167] Output: Saved answer history and feedback

[0168] Step 12:

[0169] The server analyzes the user's learning progress.

[0170] Input: Saved answer history and feedback

[0171] How it works: The server analyzes the database to assess the user's learning progress and skill gaps, and provides appropriate learning tips and materials.

[0172] Output: Study tips and materials based on the analysis results

[0173] (Application example 1)

[0174] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0175] Conventional homework assessment systems have the problem that it is difficult for users to receive immediate feedback on their homework answers, which reduces learning efficiency. They also lack a mechanism for tracking users' progress and level of understanding in real time and providing appropriate learning hints and learning materials. Furthermore, there is no way to effectively utilize learning history or past feedback. To solve these problems, it is necessary to provide users with real-time feedback and improve learning efficiency.

[0176] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0177] In this invention, the server includes: means for receiving information about homework completed by a user; means for analyzing the received homework information and converting the answers into text data; means for sending the text data to a generation AI to determine whether the answers are correct and receive feedback; means for sending the feedback obtained from the generation AI to the user; means for recording the user's answer history and feedback and tracking progress; means for analyzing the user's level of understanding and providing self-study hints and learning materials; means for uploading images of the homework answers taken by the user to the server; means for the server to convert the image data into text data using OCR technology; means for displaying the determination of correct answers and feedback to the user in real time; and means for saving the user's learning history and analyzing past answers and feedback. This allows the system to analyze the information about homework completed by a user in real time and provide immediate feedback. Furthermore, by saving and analyzing the learning history, the system can provide appropriate learning materials according to the user's progress and level of understanding.

[0178] A "user" is an entity that utilizes the system to submit homework answers and receive feedback.

[0179] "Homework information" is data indicating the content of the homework that the user has answered, and is sent from the terminal as photographs or text data.

[0180] The "receiving means" is a mechanism for importing the homework information sent by the user into the server and processing it.

[0181] The "means for analyzing and converting into text data" is a function for analyzing the received homework information and converting it from photo data into text data using optical character recognition technology.

[0182] "Generative AI" is an artificial intelligence model that determines the correctness of answers based on generated text data and generates feedback.

[0183] "Server" means the computer system that receives, processes, stores, and generates feedback on homework information submitted by users.

[0184] "Feedback" refers to the results of the generative AI's determination of the correct answer and information on how the user can arrive at the correct answer.

[0185] The "means for uploading" is a mechanism for sending the homework answer images taken by the user to the server.

[0186] "OCR technology" is an abbreviation for optical character recognition technology, which converts characters and mathematical formulas in image data into text data.

[0187] The "means for displaying to the user in real time" is a mechanism for instantly presenting the feedback sent from the server to the user.

[0188] "Study history" is data that includes questions that the user has answered in the past and feedback on those questions.

[0189] The "analysis means" is a function that uses saved learning history data to evaluate the user's progress and level of understanding.

[0190] "Teaching materials" refer to study materials and questions provided according to the user's level of understanding.

[0191] This invention provides a system that receives information about homework answers provided by users, analyzes, evaluates, and provides feedback. This system is realized through cooperation between users, terminals, and a server.

[0192] User operations

[0193] First, the user submits the homework answer using the device. The answer may be submitted as a photo or as text. For example, the user may take a photo of the notebook in which they have completed an arithmetic problem and save it on the device. The image file is then uploaded to the server.

[0194] Device Features

[0195] The device sends photos taken by the user and text entered by the user to the server. When sending, the device also sends identification information such as the user ID. This information is used by the server to process the data. The device also displays feedback from the server to the user. This allows the user to instantly receive the evaluation results for their answers.

[0196] Server Processing

[0197] The server receives homework information sent from the device. If the received information is photo data, optical character recognition (OCR) technology is used to convert the characters and mathematical formulas in the photo into text data. This conversion is performed using the pytesseract library. The converted text data is then sent to a generative AI model, which determines whether the answer is correct or incorrect. If the answer is incorrect, detailed feedback is generated about which part is incorrect and how to derive the correct answer.

[0198] View Feedback

[0199] The server then sends this feedback back to the terminal, which then displays it to the user, allowing the user to immediately see detailed feedback on their answers.

[0200] Recording and analyzing learning

[0201] The server stores the user's answer history and feedback in a database. This data is used to track the questions the user has answered in the past and the feedback they received. Furthermore, by analyzing this data, the server can understand the user's progress and level of understanding and provide appropriate study tips and materials. This analysis is performed using machine learning algorithms.

[0202] Specific examples

[0203] For example, consider the case where a user solves the problem "6 x 7 = ?" and answers "44."

[0204] 1. The user writes this answer in a notebook and takes a photo of the notebook.

[0205] 2. The device sends the photo it took to the server.

[0206] 3. The server uses OCR technology to convert the photo into text data ("6 x 7 = 44").

[0207] 4. The server sends this text data to the generation AI.

[0208] 5. The generative AI model determines that "44" is an incorrect answer and generates feedback such as "The correct answer is 42. Let's move on to the next step."

[0209] 6. The server receives this feedback and sends it to the user's device.

[0210] 7. The device displays the feedback to the user, who confirms it.

[0211] Prompt Sentence Examples

[0212] Prompt the generative AI model as follows:

[0213] Check the user's answer.

[0214] Question: "6 x 7 = ?"

[0215] Answer: "44"

[0216] Determine the correct answer and generate the necessary feedback.

[0217] Using this prompt, the generative AI model can generate appropriate feedback and provide it to the user.

[0218] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0219] Step 1:

[0220] A user writes answers to homework in a notebook and takes a photo of the notebook using a device such as a smartphone or tablet.

[0221] Input: Homework answers written in a notebook.

[0222] Output: A photo image file containing the homework answers.

[0223] Step 2:

[0224] The photos taken by the device are uploaded to the server, and are sent together with identification information such as the user ID and answer date and time.

[0225] Input: Photo image file taken, user ID, answer date and time.

[0226] Output: Images and related information sent to the server.

[0227] Step 3:

[0228] The server analyzes the received photo data using optical character recognition (OCR) technology and converts characters and mathematical expressions in the image into text data. This conversion is performed using the pytesseract library.

[0229] Input: Uploaded photo image file.

[0230] Output: Text data converted from characters and mathematical expressions in the image.

[0231] Step 4:

[0232] The server sends the converted text data to the generative AI model, which determines whether the answer is correct and generates feedback.

[0233] Input: Text data converted using OCR technology.

[0234] Output: Generative AI model determines whether the answer is correct or incorrect and provides detailed feedback.

[0235] Step 5:

[0236] The server receives feedback from the generative AI model and sends it to the user's device.

[0237] Input: Feedback from the generative AI model.

[0238] Output: Feedback data sent to the device.

[0239] Step 6:

[0240] The terminal displays the received feedback to the user, who then confirms it.

[0241] Input: Feedback data sent by the server.

[0242] Output: User sees feedback.

[0243] Step 7:

[0244] The server stores the user's answer history and feedback in a database and analyzes their learning progress and level of understanding.

[0245] Input: User's answer history, feedback from the server.

[0246] Output: Answer history and feedback stored in a database, and analysis of the user's understanding.

[0247] Step 8:

[0248] The server provides self-study hints and learning materials based on the user's level of learning comprehension, encouraging improvement in learning from the next time onwards.

[0249] Input: Parsed comprehension data.

[0250] Output: Study tips and materials provided to the user.

[0251] Through these steps, users receive real-time feedback and can learn more efficiently.

[0252] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0253] This system receives information about homework completed by a user, uses a generation AI to determine whether the answer is correct and provides detailed feedback, and also combines an emotion engine to collect and analyze the user's emotion data and provide customized support based on that data. This system operates in cooperation with the user, terminal, server, and emotion engine.

[0254] User operations

[0255] First, the user submits their homework answers using their device. Homework answers can be submitted as photos or as text. Furthermore, when submitting the homework information, the user's emotional state at that time is also transmitted. For example, the device's camera can be used to capture the user's facial expression and analyze the emotion.

[0256] Device Features

[0257] The device sends the user's photos, input text, and emotional data to the server. When sending this data, the device also includes identification information such as the user ID. The device also displays feedback from the server and advice based on the user's emotions. In this way, the user can receive not only their answers but also feedback that takes into account their current emotions.

[0258] Server Processing

[0259] The server receives the homework information and emotion data sent from the device. If the homework information received is photo data, it first uses optical character recognition (OCR) technology to convert the letters and mathematical formulas in the photo into text data. The converted text data is then sent to the generation AI, which determines whether it is the correct answer.

[0260] The generation AI analyzes the received text data and determines whether the answer is correct or incorrect. If the answer is incorrect, it generates detailed feedback along with an explanation. Meanwhile, the emotion engine analyzes the received user's emotional data and understands their emotional state. The server reflects the emotional data analyzed by the emotion engine in the feedback received from the generation AI and provides feedback appropriate to the user's emotional state.

[0261] Recording and analyzing learning

[0262] The server stores the user's answer history, feedback, and emotional data in a database. This allows the server to track the questions the user has answered in the past and their emotional state at the time. Furthermore, by analyzing this data, the server can understand the user's level of understanding and learning progress and provide appropriate learning tips and materials.

[0263] Example

[0264] For example, consider the case where a user solves the math problem "6 x 7 = ?" and answers "44."

[0265] 1. The user writes the answer in a notebook, takes a photo of the notebook, and also takes a photo of their own expression using the device's camera.

[0266] 2. The device takes a photo of the homework and sends the user's facial expression data to the server.

[0267] 3. The server uses OCR technology to convert the homework photo into text data ("6 x 7 = 44"), and at the same time uses an emotion engine to analyze the facial expression data and recognize the child's emotional state (for example, "confused").

[0268] 4. The server sends this text data to the generation AI, which determines that "44" is an incorrect answer and generates feedback saying, "The correct answer is 42. Let's move on to the next step."

[0269] 5. Based on the emotional data of "confused," the emotion engine generates additional advice: "Don't rush, try again. If you don't understand, check the hints."

[0270] 6. The server sends the generated feedback and advice to the user.

[0271] 7. The device receives feedback and advice from the server and displays it to the user.

[0272] 8. The user reviews the feedback and advice to understand whether their answer was correct or incorrect and what their next steps are.

[0273] In this way, the system instantly evaluates the user's homework answers and provides detailed feedback and personalized advice based on their emotions, thereby motivating the user to study more efficiently.

[0274] The processing flow will be explained below.

[0275] Step 1:

[0276] The user prepares the homework answer by writing it in a notebook or by typing it directly into the device.

[0277] Step 2:

[0278] The user takes a photo of the note with the device, and also captures their facial expressions using the device's camera to collect their emotional state.

[0279] Step 3:

[0280] The device takes a photo of the homework and sends it to the server along with facial expression data (emotion data), along with identification information such as the user ID.

[0281] Step 4:

[0282] The server receives the homework information and emotion data sent from the device, temporarily stores the data, and then starts processing it.

[0283] Step 5:

[0284] The server analyzes the image data of the homework and uses OCR technology to convert the letters and mathematical formulas in the photo into text data, which is then sent to the next step of processing.

[0285] Step 6:

[0286] The server uses an emotion engine to analyze the user's emotional state from the received facial expression data. For example, it can recognize emotions such as "confusion" or "happiness" from the user's facial expression.

[0287] Step 7:

[0288] The server sends the text data and emotion data to the generation AI, which uses this data to determine whether the answer is correct and generates detailed feedback.

[0289] Step 8:

[0290] The generation AI judges the answer and generates feedback. For example, for the answer "6 x 7 = 44," it generates feedback such as "That's incorrect. The correct answer is 42."

[0291] Step 9:

[0292] The emotion engine generates additional advice that reflects the emotional data based on feedback from the generation AI. For example, in response to the emotional data of "confused," the engine generates advice such as "Try again without rushing."

[0293] Step 10:

[0294] The server compiles the feedback and advice received from the generative AI and emotion engine and prepares it to be sent to the user.

[0295] Step 11:

[0296] The terminal receives feedback and emotion-based advice from the server and displays it to the user, who can then review it to understand whether their answer was correct or incorrect and what to do next.

[0297] Step 12:

[0298] The server stores the user's answer history and emotional data in a database, which stores data that tracks the user's learning progress, comprehension, and emotional state.

[0299] Step 13:

[0300] The server analyzes the user's learning comprehension, progress, and emotional data, and provides individually customized self-study tips and learning materials. For example, it analyzes past emotional data and provides more detailed explanations for particularly difficult problems.

[0301] In this way, our system not only instantly evaluates users' homework and provides detailed feedback and emotion-based advice, but also provides long-term learning support. By adjusting feedback based on users' emotion data, we aim to increase their motivation to learn and provide effective learning support.

[0302] Example 2

[0303] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0304] Conventional educational support systems have the ability to determine whether a user's answers are correct, but they are unable to provide feedback or advice that takes into account the user's emotional state. This makes it difficult to provide individual support to increase the user's motivation to learn, and they are unable to provide efficient learning support. Furthermore, there is a lack of technology that can comprehensively manage and analyze a user's learning progress and emotional data, and provide customized feedback.

[0305] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0306] In this invention, the server includes means for receiving information about homework answers completed by the user and emotional data about the user, means for analyzing the received homework information and converting the answers into text data, means for sending the text data to a generation AI to determine whether the answers are correct and receive feedback, means for incorporating the results of the emotional analysis into the feedback received from the generation AI to generate feedback and advice appropriate to the user's emotional state, means for sending the generated feedback and advice to the user, means for recording the user's answer history, feedback, and emotional data and tracking progress, and means for analyzing the user's level of understanding of the learning and providing hints and learning materials for self-study. This provides customized feedback that takes the user's emotional state into consideration, thereby improving their motivation to learn and providing efficient learning support.

[0307] A "user" is an individual or entity who uses the system to submit homework answers.

[0308] A "homework assignment" is a problem or task assigned to a user for educational or training purposes.

[0309] "Information" refers to homework answers and related data submitted by users.

[0310] "Emotional data" refers to data about a user's emotional state collected through the device's camera or sensors.

[0311] A "terminal" is a device on which a user enters answers to homework, and includes a camera, smartphone, tablet, PC, etc.

[0312] A "server" is a computer system that receives data sent from a terminal and analyzes and processes it.

[0313] Optical character recognition (OCR) is a technology that analyzes characters and mathematical formulas in photo data and converts them into text data.

[0314] "Generative AI" is an artificial intelligence technology that analyzes text data and generates correct answers and feedback.

[0315] "Feedback" is information that includes an evaluation of the user's homework answers as well as next steps and advice.

[0316] The "emotion engine" is an artificial intelligence technology that analyzes the user's emotional data and reflects the results in feedback.

[0317] A "database" is a system for storing and managing a user's answer history, feedback, and emotional data.

[0318] "Level of understanding of learning" is an index that evaluates how much a user has understood through homework.

[0319] "Self-study tips and materials" are additional learning resources and advice provided to help users further their learning.

[0320] "Customized advice" is specialized feedback or advice provided based on a user's individual answer history and emotional state.

[0321] The present invention is a system that receives homework answers and emotion data submitted by a user, analyzes them, and provides feedback and customized advice. This system operates in cooperation with a user, a terminal, a server, and an emotion engine.

[0322] System Overview

[0323] The system includes the following hardware and software components:

[0324] Devices operated by users (e.g. smartphones, tablets, PCs)

[0325] Server that processes and stores data

[0326] Optical character recognition technology (OCR)

[0327] Generative AI Models

[0328] Emotion Engine

[0329] Database

[0330] Device behavior

[0331] The user submits their homework answers using a device. Answers can be entered as text or photos. The device's camera is used to collect the user's facial expression data, which is then transmitted as emotion data. The device then transmits the user's answer and emotion data to the server as a single data packet.

[0332] Server Processing

[0333] The server receives data sent from the user's device. First, it uses the emotion engine to analyze the emotion data and identify the user's emotional state. If the homework information is in the form of photographs, it uses optical character recognition (OCR) technology to convert the letters and mathematical formulas in the photographs into text data. This conversion process is performed based on an image analysis algorithm.

[0334] Answer validation and feedback generation

[0335] The server sends the converted text data to a generative AI model, which determines whether the answer is correct and generates detailed feedback. The generative AI determines whether the homework answer is correct and provides feedback along with a detailed explanation if the answer is incorrect. Furthermore, by incorporating the analysis results of the emotion engine into the feedback obtained from the generative AI, feedback and advice are generated that reflects the user's emotional state.

[0336] Integrated feedback and advice

[0337] The server integrates the feedback obtained from the generative AI with advice based on emotion data and sends it to the device. The device then displays the feedback and advice received from the server to the user, allowing the user to receive support based on the correctness of their answer and their emotion.

[0338] Data storage and analysis

[0339] The server stores all user data, answer history, feedback, and emotional data in a database. This allows the server to track users' past answer data and emotional state, analyze their learning progress and comprehension, and periodically analyze each user's skill gaps to provide customized advice.

[0340] Specific examples

[0341] For example, if a user solves the math problem "6 x 7 = ?" and answers "44," the user writes the answer in a notebook, takes a photo of the notebook, and also captures their facial expression. The device then sends the photo of the homework and the user's facial expression data to the server. The server then converts the photo of the homework into text data using OCR technology and sends it to the generation AI. The generation AI determines that "44" is an incorrect answer and generates feedback such as "The correct answer is 42. Let's move on to the next step." Based on the emotional data of "confused," the emotion engine generates advice such as "Don't rush, try again. If you don't understand, check the hint." The server then combines the generated feedback and advice and sends it to the device. The device then displays this information to the user.

[0342] Prompt Sentence Examples

[0343] An example of a prompt sentence to be input to the generative AI model is shown below.

[0344] The answer to the homework problem the user solved was "6 x 7 = 44." Check the accuracy of this answer and provide feedback on the correct answer and why, as well as next steps for the user if they got it wrong. Additionally, your data indicates the user's emotional state is "confused." Include advice based on their emotional state.

[0345] As described above, the present invention accurately evaluates a user's homework answers and provides customized feedback based on emotional data, thereby increasing the user's motivation to study and providing efficient learning support.

[0346] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0347] Step 1:

[0348] The user enters their homework answers into the device or takes a photo of their notes. They also capture their facial expressions with the device's camera to obtain emotional data.

[0349] Input: Homework answers (text or photos), facial expression data

[0350] Output: Homework answer data, emotion data

[0351] Specific actions: The user writes a homework answer by hand or digitally and takes a photo of it. The device's camera captures their facial expression and prepares it for transmission to the system.

[0352] Step 2:

[0353] The device sends the homework answer data (text or photo) and emotion data, along with identification information such as the user ID, to the server.

[0354] Input: Homework answer data, emotion data, user ID

[0355] Output: Data packet (homework answer data, emotion data, user ID)

[0356] Specific operation: The device packs the user ID, homework answer data, and emotion data into a single packet and sends it to the server using the HTTPS protocol.

[0357] Step 3:

[0358] The server receives the data sent from the device and first sends it to the emotion engine for emotional analysis. If the homework answers are photo data, they are converted into text data using OCR.

[0359] Input: Data packet (homework answer data, emotion data, user ID)

[0360] Output: Analyzed emotion data, text data (if necessary)

[0361] Specific operation: The server receives the data packet and first sends it to the emotion engine for analysis of the user's emotion data. If the homework answer is a photo, OCR technology is used to convert the characters and formulas in the photo into text data.

[0362] Step 4:

[0363] The server sends the converted text data to the generative AI model, which determines the correct answer and provides detailed feedback.

[0364] Input: Text data

[0365] Output: Feedback data

[0366] How it works: The server sends text data as prompts to the generative AI model, which then determines whether the answer is correct or incorrect and generates feedback including detailed explanations if the answer is incorrect.

[0367] Step 5:

[0368] An emotion engine analyzes the emotional data and generates customized advice.

[0369] Input: Emotion data

[0370] Output: Advice data

[0371] Specific operation: The emotion engine analyzes the user's emotional state (e.g., "confused") and generates additional advice based on the results.

[0372] Step 6:

[0373] The server integrates feedback from the generative AI and advice from the emotion engine and sends it to the device.

[0374] Input: Feedback data, advice data

[0375] Output: Consolidated message

[0376] Specific operation: The server combines the feedback obtained from the generation AI and the advice from the emotion engine into a single integrated message and sends it to the device via HTTPS protocol.

[0377] Step 7:

[0378] The terminal receives the integrated message from the server and displays it to the user.

[0379] Input: Unified message

[0380] Output: Display data

[0381] Specific operation: The device analyzes the received integrated message and displays it to the user in an appropriate format. The user checks the feedback and advice and plans their next action.

[0382] Step 8:

[0383] The server stores the user's answer history, feedback, and emotion data in a database, thereby tracking the user's understanding and learning progress.

[0384] Input: Answer history, feedback, emotional data

[0385] Output: Saved data

[0386] Specific operation: The server stores the user's answer history, feedback, and emotional data in a database, and continuously manages and analyzes them to understand each user's learning progress and level of understanding.

[0387] (Application example 2)

[0388] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0389] Conventional homework feedback systems have struggled to provide users with prompt and detailed feedback on their completed homework. Furthermore, the feedback was not customized to take into account the user's feelings, resulting in problems such as users not receiving appropriate support and reduced learning efficiency. Furthermore, it was difficult to track users' learning history and progress and provide appropriate advice and learning materials.

[0390] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0391] In this invention, the server includes means for receiving information on homework answers answered by the user, means for analyzing the received homework information and converting the answers into text data, means for sending the text data to the generation AI to determine whether the answer was correct and to receive feedback, means for analyzing the user's facial expressions to collect emotional data, means for analyzing the emotional data and providing customized advice based on the feedback from the generation AI, means for sending the user advice based on the feedback and emotional data obtained from the generation AI, means for recording the user's answer history and feedback and tracking progress, and means for analyzing the user's level of understanding of the study and providing self-study hints and learning materials. This allows the user to receive prompt and detailed feedback and advice that takes their emotions into consideration, enabling efficient study.

[0392] A "user" is someone who uses the system to submit homework and receive feedback.

[0393] "Homework information" is data including the content of the homework answered by the user, and is sent to the server in the form of a photograph or text.

[0394] "Text data" refers to data in which homework information has been converted into a string of characters using OCR technology or similar.

[0395] "Generative AI" is an artificial intelligence that analyzes received text data, determines whether the answer is correct, and generates feedback.

[0396] "Emotion data" is data that indicates the emotional state of the user analyzed from facial expressions, etc.

[0397] "Customized advice" is individual advice provided based on feedback from the generative AI and taking into account the user's emotional data.

[0398] The "means for analyzing facial expressions" is a mechanism for detecting facial expressions from images or videos of the user's face and analyzing emotions.

[0399] "Answer history" is a record of the homework answers a user has given and the feedback they have received.

[0400] "Means for tracking progress" refers to a system for tracking and managing learning progress based on the user's answer history and feedback.

[0401] The "means for analyzing the level of understanding" is a mechanism for evaluating the user's past answers and feedback to understand the user's level of understanding and weaknesses.

[0402] "Self-study tips and materials" are study support materials and advice provided according to the user's level of understanding and progress.

[0403] A "database" is a recording device for saving and storing information such as a user's answer history, feedback, and emotional data.

[0404] Optical character recognition (OCR) is a technology that analyzes characters in image data and converts them into text data.

[0405] This system receives information about homework completed by a user and uses a generation AI to determine whether the answer is correct and provide detailed feedback. Furthermore, it uses an emotion engine to collect and analyze the user's emotion data and provide customized support based on that data. This system operates in cooperation with the user, device, server, and emotion engine.

[0406] User operations

[0407] Users submit their homework answers using their devices. Homework answers may be submitted as photos or as text. When submitting homework information, the user's emotional state at that time is also transmitted. For example, the device's camera may be used to capture the user's facial expression and analyze the emotion.

[0408] Device Features

[0409] The device sends the user's photos, input text, and emotional data to the server. When sending this data, the device also includes identification information such as the user ID. The device also displays feedback from the server and advice based on the user's emotions. In this way, the user can receive not only their answers but also feedback that takes into account their current emotions.

[0410] Server Processing

[0411] The server receives the homework information and emotion data sent from the device. If the received homework information is photo data, it first uses optical character recognition (OCR) technology to convert the characters and mathematical formulas in the photo into text data. The OCR library used is "Tesseract OCR." The converted text data is sent to the generation AI, which determines whether it is correct. "GPT-4" is used as the generation AI model. The generation AI analyzes the received text data and determines whether it is correct or incorrect. If the answer is incorrect, it generates detailed feedback along with an explanation.

[0412] Emotional Data Analysis

[0413] The user's emotional data is obtained by analyzing facial expressions. An emotion recognition engine such as Microsoft Azure Face API is used to analyze the facial expression data and recognize the user's emotional state. The emotion engine analyzes the received user's emotional data and understands that emotional state. The server reflects the emotional data analyzed by the emotion engine in the feedback obtained from the generation AI and provides feedback appropriate to the user's emotional state.

[0414] Recording and analyzing learning

[0415] The server stores the user's answer history, feedback, and emotional data in a database. A typical relational database management system (RDBMS) is used as the database. This allows us to track the questions the user has answered in the past and their emotional state at the time. Furthermore, by analyzing this data, we can understand the user's level of understanding and learning progress and provide appropriate learning tips and materials.

[0416] Specific examples

[0417] For example, if a user answers "5" to the math problem "2+2=?", the following scenario is possible: The user takes a photo of their notebook with their camera and enters "2+2=5." At this time, the user's facial expression data is also sent. The server uses OCR technology to convert "2+2=5" into text data, and the generative AI (GPT-4) determines that "5" is an incorrect answer and generates detailed feedback such as "The correct answer is 4." At the same time, the emotion engine analyzes the user's facial expression data as "anxiety." Based on this information, the server adds customized advice such as "Calm down and think about it. Let's try again," and sends the final feedback to the user's device.

[0418] Prompt Sentence Examples

[0419] "The user answers '2 + 2 = 5'. Please rate this answer, tell us the correct answer, provide detailed feedback, and offer additional advice if the user is in a state of uncertainty."

[0420] In this way, the system of the present invention instantly evaluates the homework answers of the user and provides detailed feedback and customized advice based on emotions, thereby helping the user to study more efficiently.

[0421] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0422] Step 1:

[0423] User submits homework

[0424] The user submits their homework answers using a device. At this time, the homework information is photographed and facial expression data is captured at the same time. When the user takes a photo of their homework and presses the send button, the homework information and facial expression data are entered into the device. This generates the homework photo data (JPEG, PNG, etc.) and facial expression data.

[0425] Step 2:

[0426] The device sends the data to the server

[0427] The device sends the homework photo and facial expression data taken by the user to the server using protocols such as HTTP POST requests. The input is the photo data and facial expression data received from the parent, and the output is the data packet sent to the server.

[0428] Step 3:

[0429] The server analyzes the homework information using OCR technology

[0430] The server performs OCR processing on the received homework photo data and converts the characters in the image into text data. This processing uses the Tesseract OCR library. The input is the photo data (JPEG, PNG, etc.) received by the server, and the output is the text data (String) generated by OCR.

[0431] Step 4:

[0432] The server sends the text data to the generation AI and receives feedback and a correct answer.

[0433] The server sends the text data obtained by OCR to the generation AI (GPT-4), determines whether the answer is correct, and generates feedback. It constructs a prompt sentence and sends the text data as input to the generation AI. The input is the text data and the prompt sentence, and the output is the generated feedback sentence.

[0434] Example prompt: "The user answered '2 + 2 = 5'. Please rate this answer, tell us the correct answer, and provide detailed feedback."

[0435] Step 5:

[0436] The server analyzes the facial expression data and obtains emotion data.

[0437] The server sends the facial expression data to an emotion recognition engine (Microsoft Azure Face API) to analyze the emotion data. The input is the facial expression data (image), and the output is the analyzed emotional state (e.g., "anxiety").

[0438] Step 6:

[0439] The server generates feedback as customized advice.

[0440] The server generates customized advice based on the feedback obtained from the generation AI, taking into account the emotional data. It uses the emotional data as input and adds additional advice to the feedback of the generation AI. The input is the feedback from the generation AI and the emotional data, and the output is a customized feedback sentence.

[0441] Step 7:

[0442] The server sends the generated feedback to the user's device.

[0443] The server sends the generated customized feedback to the user's terminal, where the input is the customized feedback sentence and the output is the feedback displayed on the user's terminal.

[0444] Step 8:

[0445] The device displays feedback

[0446] The terminal displays the feedback received from the server to the user. The input is the feedback text received from the server, and the output is the feedback displayed on the terminal's display.

[0447] Step 9:

[0448] The server records answer history and feedback, and tracks progress.

[0449] The server records the user's answer history and feedback in a database and tracks the progress. The input is the user's answer history and feedback data, and the output is the information recorded in the database.

[0450] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0451] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0452] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0453] [Second embodiment]

[0454] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0455] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0456] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0457] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0458] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0459] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0460] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0461] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0462] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0463] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0464] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0465] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0466] The present invention is a system for receiving information on homework answers provided by users, analyzing, evaluating, and providing feedback on the answers. This system is realized through cooperation between users, terminals, and a server.

[0467] User operations

[0468] First, the user submits the homework answer using the device. The answer may be submitted as a photo or as text. For example, the user may take a photo of the notebook in which they have completed an arithmetic problem and save it on the device. The image file is then uploaded to the server.

[0469] Device Features

[0470] The device sends photos taken by the user and text entered by the user to the server. When sending, the device also sends identification information such as the user ID. This information is used by the server to process the data. The device also displays feedback from the server to the user. This allows the user to instantly receive the evaluation results for their answers.

[0471] Server Processing

[0472] The server receives the homework information sent from the device. If the received information is photo data, it first uses optical character recognition (OCR) technology to convert the letters and formulas in the photo into text data. The converted text data is then sent to the generation AI, which determines whether the answer is correct.

[0473] The generation AI analyzes the received text data and determines whether the answer is correct or incorrect. If the answer is incorrect, it generates detailed feedback about which part is incorrect and how to derive the correct answer. This feedback is sent back to the server, which then sends it to the user.

[0474] Recording and analyzing learning

[0475] The server stores the user's answer history and feedback in a database. This allows the server to track the questions the user has answered in the past and the feedback they received. Furthermore, by analyzing this data, the server can understand the user's level of understanding and progress and provide appropriate study tips and materials.

[0476] Example

[0477] For example, consider the case where a user solves the math problem "6 x 7 = ?" and answers "44."

[0478] 1. The user writes this answer in a notebook and takes a photo of the notebook.

[0479] 2. The device sends the photo it took to the server.

[0480] 3. The server uses OCR technology to convert the photo into text data ("6 x 7 = 44").

[0481] 4. The server sends this text data to the generation AI.

[0482] 5. The generation AI determines that "44" is an incorrect answer and generates feedback saying, "The correct answer is 42. Let's move on to the next step."

[0483] 6. The server receives this feedback and sends it to the user's device.

[0484] 7. The device displays the feedback to the user, who confirms it.

[0485] In this way, the system provides efficient learning support by instantly evaluating the user's homework answers and providing detailed feedback.The system also stores the user's learning history in a database and analyzes their understanding and progress, making it possible to provide hints and learning materials for self-study.

[0486] The processing flow will be explained below.

[0487] Step 1:

[0488] The user prepares the homework answers by writing them in a notebook or by typing them directly into the device.

[0489] Step 2:

[0490] The user takes a photo of the note on the device or inputs text. For example, the user takes a photo of the note using a smartphone or tablet and saves it on the device.

[0491] Step 3:

[0492] The device sends the homework information from the user to the server. The information sent includes photo data, text data, and identification information such as the user ID.

[0493] Step 4:

[0494] The server receives the homework information sent from the device, temporarily stores the received data, and prepares it for processing.

[0495] Step 5:

[0496] The server analyzes the received homework information. If the homework is a photo, it uses optical character recognition (OCR) technology to convert the characters and mathematical formulas in the image into text data.

[0497] Step 6:

[0498] The server sends the text data to the AI ​​that generates it, which then analyzes the received text data and determines the correct answer.

[0499] Step 7:

[0500] The generation AI determines whether the user's answer is correct and generates feedback. Based on the data it receives, the generation AI determines whether the user's answer is correct, and if it is incorrect, generates detailed feedback on how the answer should be answered.

[0501] Step 8:

[0502] The server receives the feedback from the generated AI and prepares it to be sent to the user. The feedback includes correct and incorrect answers and detailed explanations.

[0503] Step 9:

[0504] The device receives feedback from the server and displays it to the user, who can then check the feedback and determine whether their answer was correct or incorrect.

[0505] Step 10:

[0506] The server stores the user's answer history and feedback in a database, which provides data for later analysis of the user's learning progress and understanding.

[0507] Step 11:

[0508] The server analyzes the user's level of understanding and progress, and evaluates the user's stumbling points and level of understanding based on past answer history and feedback.

[0509] Step 12:

[0510] The server provides self-study tips and materials based on the user's learning needs, such as relevant video lectures and additional practice questions.

[0511] In this way, the system not only evaluates users' homework and provides immediate feedback, but also provides long-term learning support. By continuously tracking each user's learning progress and providing appropriate support as needed, the system aims to motivate students to learn more efficiently.

[0512] Example 1

[0513] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0514] Traditional learning systems have the problem that it takes time to evaluate and provide feedback on users' homework answers, making it difficult to track users' understanding and progress in real time. This prevents users from receiving appropriate feedback immediately, resulting in reduced learning efficiency. Furthermore, there is a lack of a means to accumulate past answer history and feedback in a database and manage learning progress over the long term, making it difficult to properly evaluate users' understanding and skill gaps.

[0515] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0516] In this invention, the server includes: a means for receiving information about homework completed by a user; a means for analyzing the received homework information and converting the answers into text data; a means for sending the text data to a generative AI model to determine whether the answers are correct and receive feedback; a means for sending the feedback obtained from the generative AI model to the user; a means for recording the user's answer history and feedback and tracking progress; and a means for analyzing the user's level of understanding and providing self-study hints and learning materials. This allows the user to receive evaluations and feedback on their completed homework in real time, improving learning efficiency. It also enables the long-term accumulation and analysis of learning data, allowing for an accurate understanding of the user's level of understanding and progress, enabling appropriate learning guidance.

[0517] "User" refers to an individual who uses the system to submit homework answers and receive evaluations and feedback.

[0518] "Homework information" refers to the content of the homework that the user has answered, and is sent to the server as text data or photo data.

[0519] The "receiving means" is a mechanism by which the server receives the homework information sent by the user.

[0520] The "analysis means" is a technology that allows the server to analyze the homework information received and convert it into text data as needed.

[0521] A "generative AI model" is an artificial intelligence model that analyzes received text data and generates correct answers and feedback.

[0522] The "means of transmission" is a mechanism for transmitting feedback obtained from the generative AI model to the user's device.

[0523] "Means for recording and tracking progress" refers to technology for storing a user's answer history and feedback content, and for managing the user's learning progress over the long term.

[0524] "Means for providing self-study hints and learning materials" is a mechanism for analyzing the user's level of understanding and providing appropriate learning advice and learning materials.

[0525] The "means of receiving the photograph and converting it into text data using optical character recognition technology" is a technology that converts the photograph data sent by the user into text data using optical character recognition technology.

[0526] The "database" is a system that serves as the foundation for storing and analyzing users' learning progress and level of understanding.

[0527] A "skill gap" refers to the difference that exists between a user's current level of understanding and their desired level of understanding.

[0528] The present invention is a system for receiving information on homework answers provided by users, analyzing and evaluating the information, and providing feedback. This system functions based on the cooperation of users, terminals, and a server.

[0529] First, the user enters the answer to their homework using their device and sends it to the server. The answer may be sent as a photo or directly entered as text. For example, if a user solves an arithmetic problem and writes the answer in a notebook, they can take a photo of the notebook using a device such as a smartphone or tablet and save the image file on the device. They can then upload the image file to the server using a dedicated application.

[0530] The device sends identification information such as the user ID to the server along with the photo data taken by the user and the text data entered by the user. This allows the server to process the data appropriately. The device also receives evaluation results and feedback from the server and displays them to the user. This allows users to instantly check the evaluation results of their answers and improve the quality of their learning.

[0531] The server first receives the homework information sent from the device. If the received information is photo data, the server uses optical character recognition (OCR) technology to convert the characters and mathematical formulas in the photo into text data. Specifically, OCR software such as Google Cloud Vision API is used. This converted text data is then sent to a generative AI model. A generative AI model is an artificial intelligence that analyzes the text data and generates a correct answer and detailed feedback. A typical generative AI model is OpenAI GPT-4.

[0532] The generative AI model analyzes the accuracy of the answer based on the received text data and determines whether it is correct or incorrect. If the answer is incorrect, the generative AI model generates detailed feedback about which part is incorrect and how to derive the correct answer. This generated feedback is sent back to the server, which then sends it to the user's device. The feedback content is saved in a database and maintained as the user's learning history.

[0533] As a specific example of how it works, suppose a user solves the problem "6 x 7 = ?". The user writes the answer "44" in a notebook, takes a photo of it, and uses their device to send the photo to the server. The device sends the photo data and user ID to the server. The server uses OCR technology to convert the photo data into text data "6 x 7 = 44" and sends it to the generative AI model. The generative AI model determines that "44" is an incorrect answer and generates feedback saying "The correct answer is 42. Let's proceed to the next step," which is sent back to the server. The server sends this feedback to the user's device, which then displays the feedback.

[0534] An example of a prompt for a generative AI model is as follows:

[0535] "The user answered the following question: 6 x 7 = 44. Please analyze and evaluate this answer to generate corrective feedback."

[0536] This system allows users to receive instant, detailed feedback, enabling them to study efficiently. Furthermore, by analyzing the learning history stored in the database, it is possible to accurately grasp the user's level of understanding and progress, and provide appropriate learning guidance and teaching materials as needed.

[0537] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0538] Step 1:

[0539] The user enters the homework answer.

[0540] Input: Homework answered by the user (photo data or text data)

[0541] How it works: A user writes their homework answers in a notebook and takes a photo of it with their smartphone or tablet, or types it in as text.

[0542] Output: Homework answer data (photo or text)

[0543] Step 2:

[0544] The user sends the answer data through the terminal.

[0545] Input: User's homework answer data, user ID

[0546] How it works: A user opens the app, uploads a photo or text of their answer, and presses the submit button, automatically attaching their user ID.

[0547] Output: Homework answer data and user ID sent to the device

[0548] Step 3:

[0549] The device sends the answer data and user ID to the server.

[0550] Input: Answer data, user ID

[0551] Operation: After the send button is pressed, the terminal sends the answer data and user ID to the server.

[0552] Output: Answer data and user ID sent to the server

[0553] Step 4:

[0554] The server receives the answer data and the user ID.

[0555] Input: Answer data and user ID sent from the device

[0556] Operation: The server uses the receiving module to receive the answer data and the user ID.

[0557] Output: Answer data and user ID saved on the server

[0558] Step 5:

[0559] The server analyzes the answer data.

[0560] Input: Received answer data

[0561] How it works: The server analyzes the answer data and converts it into text using optical character recognition (OCR) technology, such as Google Cloud Vision API or similar OCR software.

[0562] Output: Parsed text data

[0563] Step 6:

[0564] The server generates prompts that send text data to the generative AI model.

[0565] Input: Parsed text data

[0566] How it works: The server uses the prompt generation module to create a prompt sentence to send to the generative AI model.

[0567] Output: Generated prompt statement

[0568] Step 7:

[0569] The server sends the prompt to the generative AI model.

[0570] Input: Generated prompt text

[0571] How it works: The server sends a prompt to a generative AI model (e.g., OpenAI GPT-4).

[0572] Output: Feedback generated by the generative AI model

[0573] Step 8:

[0574] A generative AI model generates an evaluation and feedback on the answer.

[0575] Input: prompt statement

[0576] How it works: A generative AI model analyzes prompts, determines whether an answer is correct or incorrect, and generates detailed feedback.

[0577] Output: Generated feedback

[0578] Step 9:

[0579] The server receives the feedback and sends it to the user's device.

[0580] Input: Feedback from the generative AI model, user ID

[0581] Operation: The server uses a feedback receiving module to receive feedback from the generative AI model and sends it to the device with the corresponding user ID.

[0582] Output: Feedback sent to the terminal

[0583] Step 10:

[0584] The terminal displays the received feedback to the user.

[0585] Input: Feedback sent by the server

[0586] How it works: The device receives the feedback and displays it to the user via a dedicated application.

[0587] Output: Displayed feedback

[0588] Step 11:

[0589] The server records and stores the user's answer history and feedback.

[0590] Input: Answer history and feedback received

[0591] How it works: The server stores answer history and feedback in a database.

[0592] Output: Saved answer history and feedback

[0593] Step 12:

[0594] The server analyzes the user's learning progress.

[0595] Input: Saved answer history and feedback

[0596] How it works: The server analyzes the database to assess the user's learning progress and skill gaps, and provides appropriate learning tips and materials.

[0597] Output: Study tips and materials based on the analysis results

[0598] (Application example 1)

[0599] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0600] Conventional homework assessment systems have the problem that it is difficult for users to receive immediate feedback on their homework answers, which reduces learning efficiency. They also lack a mechanism for tracking users' progress and level of understanding in real time and providing appropriate learning hints and learning materials. Furthermore, there is no way to effectively utilize learning history or past feedback. To solve these problems, it is necessary to provide users with real-time feedback and improve learning efficiency.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0602] In this invention, the server includes: means for receiving information about homework completed by a user; means for analyzing the received homework information and converting the answers into text data; means for sending the text data to a generation AI to determine whether the answers are correct and receive feedback; means for sending the feedback obtained from the generation AI to the user; means for recording the user's answer history and feedback and tracking progress; means for analyzing the user's level of understanding and providing self-study hints and learning materials; means for uploading images of the homework answers taken by the user to the server; means for the server to convert the image data into text data using OCR technology; means for displaying the determination of correct answers and feedback to the user in real time; and means for saving the user's learning history and analyzing past answers and feedback. This allows the system to analyze the information about homework completed by a user in real time and provide immediate feedback. Furthermore, by saving and analyzing the learning history, the system can provide appropriate learning materials according to the user's progress and level of understanding.

[0603] A "user" is an entity that utilizes the system to submit homework answers and receive feedback.

[0604] "Homework information" is data indicating the content of the homework that the user has answered, and is sent from the terminal as photographs or text data.

[0605] The "receiving means" is a mechanism for importing the homework information sent by the user into the server and processing it.

[0606] The "means for analyzing and converting into text data" is a function for analyzing the received homework information and converting it from photo data into text data using optical character recognition technology.

[0607] "Generative AI" is an artificial intelligence model that determines the correctness of answers based on generated text data and generates feedback.

[0608] "Server" means the computer system that receives, processes, stores, and generates feedback on homework information submitted by users.

[0609] "Feedback" refers to the results of the generative AI's determination of the correct answer and information on how the user can arrive at the correct answer.

[0610] The "means for uploading" is a mechanism for sending the homework answer images taken by the user to the server.

[0611] "OCR technology" is an abbreviation for optical character recognition technology, which converts characters and mathematical formulas in image data into text data.

[0612] The "means for displaying to the user in real time" is a mechanism for instantly presenting the feedback sent from the server to the user.

[0613] "Study history" is data that includes questions that the user has answered in the past and feedback on those questions.

[0614] The "analysis means" is a function that uses saved learning history data to evaluate the user's progress and level of understanding.

[0615] "Teaching materials" refer to study materials and questions provided according to the user's level of understanding.

[0616] This invention provides a system that receives information about homework answers provided by users, analyzes, evaluates, and provides feedback. This system is realized through cooperation between users, terminals, and a server.

[0617] User operations

[0618] First, the user submits the homework answer using the device. The answer may be submitted as a photo or as text. For example, the user may take a photo of the notebook in which they have completed an arithmetic problem and save it on the device. The image file is then uploaded to the server.

[0619] Device Features

[0620] The device sends photos taken by the user and text entered by the user to the server. When sending, the device also sends identification information such as the user ID. This information is used by the server to process the data. The device also displays feedback from the server to the user. This allows the user to instantly receive the evaluation results for their answers.

[0621] Server Processing

[0622] The server receives homework information sent from the device. If the received information is photo data, optical character recognition (OCR) technology is used to convert the characters and mathematical formulas in the photo into text data. This conversion is performed using the pytesseract library. The converted text data is then sent to a generative AI model, which determines whether the answer is correct or incorrect. If the answer is incorrect, detailed feedback is generated about which part is incorrect and how to derive the correct answer.

[0623] View Feedback

[0624] The server then sends this feedback back to the terminal, which then displays it to the user, allowing the user to immediately see detailed feedback on their answers.

[0625] Recording and analyzing learning

[0626] The server stores the user's answer history and feedback in a database. This data is used to track the questions the user has answered in the past and the feedback they received. Furthermore, by analyzing this data, the server can understand the user's progress and level of understanding and provide appropriate study tips and materials. This analysis is performed using machine learning algorithms.

[0627] Specific examples

[0628] For example, consider the case where a user solves the problem "6 x 7 = ?" and answers "44."

[0629] 1. The user writes this answer in a notebook and takes a photo of the notebook.

[0630] 2. The device sends the photo it took to the server.

[0631] 3. The server uses OCR technology to convert the photo into text data ("6 x 7 = 44").

[0632] 4. The server sends this text data to the generation AI.

[0633] 5. The generative AI model determines that "44" is an incorrect answer and generates feedback such as "The correct answer is 42. Let's move on to the next step."

[0634] 6. The server receives this feedback and sends it to the user's device.

[0635] 7. The device displays the feedback to the user, who confirms it.

[0636] Prompt Sentence Examples

[0637] Prompt the generative AI model as follows:

[0638] Check the user's answer.

[0639] Question: "6 x 7 = ?"

[0640] Answer: "44"

[0641] Determine the correct answer and generate the necessary feedback.

[0642] Using this prompt, the generative AI model can generate appropriate feedback and provide it to the user.

[0643] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0644] Step 1:

[0645] A user writes answers to homework in a notebook and takes a photo of the notebook using a device such as a smartphone or tablet.

[0646] Input: Homework answers written in a notebook.

[0647] Output: A photo image file containing the homework answers.

[0648] Step 2:

[0649] The photos taken by the device are uploaded to the server, and are sent together with identification information such as the user ID and answer date and time.

[0650] Input: Photo image file taken, user ID, answer date and time.

[0651] Output: Images and related information sent to the server.

[0652] Step 3:

[0653] The server analyzes the received photo data using optical character recognition (OCR) technology and converts characters and mathematical expressions in the image into text data. This conversion is performed using the pytesseract library.

[0654] Input: Uploaded photo image file.

[0655] Output: Text data converted from characters and mathematical expressions in the image.

[0656] Step 4:

[0657] The server sends the converted text data to the generative AI model, which determines whether the answer is correct and generates feedback.

[0658] Input: Text data converted using OCR technology.

[0659] Output: Generative AI model determines whether the answer is correct or incorrect and provides detailed feedback.

[0660] Step 5:

[0661] The server receives feedback from the generative AI model and sends it to the user's device.

[0662] Input: Feedback from the generative AI model.

[0663] Output: Feedback data sent to the device.

[0664] Step 6:

[0665] The terminal displays the received feedback to the user, who then confirms it.

[0666] Input: Feedback data sent by the server.

[0667] Output: User sees feedback.

[0668] Step 7:

[0669] The server stores the user's answer history and feedback in a database and analyzes their learning progress and level of understanding.

[0670] Input: User's answer history, feedback from the server.

[0671] Output: Answer history and feedback stored in a database, and analysis of the user's understanding.

[0672] Step 8:

[0673] The server provides self-study hints and learning materials based on the user's level of learning comprehension, encouraging improvement in learning from the next time onwards.

[0674] Input: Parsed comprehension data.

[0675] Output: Study tips and materials provided to the user.

[0676] Through these steps, users receive real-time feedback and can learn more efficiently.

[0677] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0678] This system receives information about homework completed by a user, uses a generation AI to determine whether the answer is correct and provides detailed feedback, and also combines an emotion engine to collect and analyze the user's emotion data and provide customized support based on that data. This system operates in cooperation with the user, terminal, server, and emotion engine.

[0679] User operations

[0680] First, the user submits their homework answers using their device. Homework answers can be submitted as photos or as text. Furthermore, when submitting the homework information, the user's emotional state at that time is also transmitted. For example, the device's camera can be used to capture the user's facial expression and analyze the emotion.

[0681] Device Features

[0682] The device sends the user's photos, input text, and emotional data to the server. When sending this data, the device also includes identification information such as the user ID. The device also displays feedback from the server and advice based on the user's emotions. In this way, the user can receive not only their answers but also feedback that takes into account their current emotions.

[0683] Server Processing

[0684] The server receives the homework information and emotion data sent from the device. If the homework information received is photo data, it first uses optical character recognition (OCR) technology to convert the letters and mathematical formulas in the photo into text data. The converted text data is then sent to the generation AI, which determines whether it is the correct answer.

[0685] The generation AI analyzes the received text data and determines whether the answer is correct or incorrect. If the answer is incorrect, it generates detailed feedback along with an explanation. Meanwhile, the emotion engine analyzes the received user's emotional data and understands their emotional state. The server reflects the emotional data analyzed by the emotion engine in the feedback received from the generation AI and provides feedback appropriate to the user's emotional state.

[0686] Recording and analyzing learning

[0687] The server stores the user's answer history, feedback, and emotional data in a database. This allows the server to track the questions the user has answered in the past and their emotional state at the time. Furthermore, by analyzing this data, the server can understand the user's level of understanding and learning progress and provide appropriate learning tips and materials.

[0688] Example

[0689] For example, consider the case where a user solves the math problem "6 x 7 = ?" and answers "44."

[0690] 1. The user writes the answer in a notebook, takes a photo of the notebook, and also takes a photo of their own expression using the device's camera.

[0691] 2. The device takes a photo of the homework and sends the user's facial expression data to the server.

[0692] 3. The server uses OCR technology to convert the homework photo into text data ("6 x 7 = 44"), and at the same time uses an emotion engine to analyze the facial expression data and recognize the child's emotional state (for example, "confused").

[0693] 4. The server sends this text data to the generation AI, which determines that "44" is an incorrect answer and generates feedback saying, "The correct answer is 42. Let's move on to the next step."

[0694] 5. Based on the emotional data of "confused," the emotion engine generates additional advice: "Don't rush, try again. If you don't understand, check the hints."

[0695] 6. The server sends the generated feedback and advice to the user.

[0696] 7. The device receives feedback and advice from the server and displays it to the user.

[0697] 8. The user reviews the feedback and advice to understand whether their answer was correct or incorrect and what their next steps are.

[0698] In this way, the system instantly evaluates the user's homework answers and provides detailed feedback and personalized advice based on their emotions, thereby motivating the user to study more efficiently.

[0699] The processing flow will be explained below.

[0700] Step 1:

[0701] The user prepares the homework answer by writing it in a notebook or by typing it directly into the device.

[0702] Step 2:

[0703] The user takes a photo of the note with the device, and also captures their facial expressions using the device's camera to collect their emotional state.

[0704] Step 3:

[0705] The device takes a photo of the homework and sends it to the server along with facial expression data (emotion data), along with identification information such as the user ID.

[0706] Step 4:

[0707] The server receives the homework information and emotion data sent from the device, temporarily stores the data, and then starts processing it.

[0708] Step 5:

[0709] The server analyzes the image data of the homework and uses OCR technology to convert the letters and mathematical formulas in the photo into text data, which is then sent to the next step of processing.

[0710] Step 6:

[0711] The server uses an emotion engine to analyze the user's emotional state from the received facial expression data. For example, it can recognize emotions such as "confusion" or "happiness" from the user's facial expression.

[0712] Step 7:

[0713] The server sends the text data and emotion data to the generation AI, which uses this data to determine whether the answer is correct and generates detailed feedback.

[0714] Step 8:

[0715] The generation AI judges the answer and generates feedback. For example, for the answer "6 x 7 = 44," it generates feedback such as "That's incorrect. The correct answer is 42."

[0716] Step 9:

[0717] The emotion engine generates additional advice that reflects the emotional data based on feedback from the generation AI. For example, in response to the emotional data of "confused," the engine generates advice such as "Try again without rushing."

[0718] Step 10:

[0719] The server compiles the feedback and advice received from the generative AI and emotion engine and prepares it to be sent to the user.

[0720] Step 11:

[0721] The terminal receives feedback and emotion-based advice from the server and displays it to the user, who can then review it to understand whether their answer was correct or incorrect and what to do next.

[0722] Step 12:

[0723] The server stores the user's answer history and emotional data in a database, which stores data that tracks the user's learning progress, comprehension, and emotional state.

[0724] Step 13:

[0725] The server analyzes the user's learning comprehension, progress, and emotional data, and provides individually customized self-study tips and learning materials. For example, it analyzes past emotional data and provides more detailed explanations for particularly difficult problems.

[0726] In this way, our system not only instantly evaluates users' homework and provides detailed feedback and emotion-based advice, but also provides long-term learning support. By adjusting feedback based on users' emotion data, we aim to increase their motivation to learn and provide effective learning support.

[0727] Example 2

[0728] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0729] Conventional educational support systems have the ability to determine whether a user's answers are correct, but they are unable to provide feedback or advice that takes into account the user's emotional state. This makes it difficult to provide individual support to increase the user's motivation to learn, and they are unable to provide efficient learning support. Furthermore, there is a lack of technology that can comprehensively manage and analyze a user's learning progress and emotional data, and provide customized feedback.

[0730] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0731] In this invention, the server includes means for receiving information about homework answers completed by the user and emotional data about the user, means for analyzing the received homework information and converting the answers into text data, means for sending the text data to a generation AI to determine whether the answers are correct and receive feedback, means for incorporating the results of the emotional analysis into the feedback received from the generation AI to generate feedback and advice appropriate to the user's emotional state, means for sending the generated feedback and advice to the user, means for recording the user's answer history, feedback, and emotional data and tracking progress, and means for analyzing the user's level of understanding of the learning and providing hints and learning materials for self-study. This provides customized feedback that takes the user's emotional state into consideration, thereby improving their motivation to learn and providing efficient learning support.

[0732] A "user" is an individual or entity who uses the system to submit homework answers.

[0733] A "homework assignment" is a problem or task assigned to a user for educational or training purposes.

[0734] "Information" refers to homework answers and related data submitted by users.

[0735] "Emotional data" refers to data about a user's emotional state collected through the device's camera or sensors.

[0736] A "terminal" is a device on which a user enters answers to homework, and includes a camera, smartphone, tablet, PC, etc.

[0737] A "server" is a computer system that receives data sent from a terminal and analyzes and processes it.

[0738] Optical character recognition (OCR) is a technology that analyzes characters and mathematical formulas in photo data and converts them into text data.

[0739] "Generative AI" is an artificial intelligence technology that analyzes text data and generates correct answers and feedback.

[0740] "Feedback" is information that includes an evaluation of the user's homework answers as well as next steps and advice.

[0741] The "emotion engine" is an artificial intelligence technology that analyzes the user's emotional data and reflects the results in feedback.

[0742] A "database" is a system for storing and managing a user's answer history, feedback, and emotional data.

[0743] "Level of understanding of learning" is an index that evaluates how much a user has understood through homework.

[0744] "Self-study tips and materials" are additional learning resources and advice provided to help users further their learning.

[0745] "Customized advice" is specialized feedback or advice provided based on a user's individual answer history and emotional state.

[0746] The present invention is a system that receives homework answers and emotion data submitted by a user, analyzes them, and provides feedback and customized advice. This system operates in cooperation with a user, a terminal, a server, and an emotion engine.

[0747] System Overview

[0748] The system includes the following hardware and software components:

[0749] Devices operated by users (e.g. smartphones, tablets, PCs)

[0750] Server that processes and stores data

[0751] Optical character recognition technology (OCR)

[0752] Generative AI Models

[0753] Emotion Engine

[0754] Database

[0755] Device behavior

[0756] The user submits their homework answers using a device. Answers can be entered as text or photos. The device's camera is used to collect the user's facial expression data, which is then transmitted as emotion data. The device then transmits the user's answer and emotion data to the server as a single data packet.

[0757] Server Processing

[0758] The server receives data sent from the user's device. First, it uses the emotion engine to analyze the emotion data and identify the user's emotional state. If the homework information is in the form of photographs, it uses optical character recognition (OCR) technology to convert the letters and mathematical formulas in the photographs into text data. This conversion process is performed based on an image analysis algorithm.

[0759] Answer validation and feedback generation

[0760] The server sends the converted text data to a generative AI model, which determines whether the answer is correct and generates detailed feedback. The generative AI determines whether the homework answer is correct and provides feedback along with a detailed explanation if the answer is incorrect. Furthermore, by incorporating the analysis results of the emotion engine into the feedback obtained from the generative AI, feedback and advice are generated that reflects the user's emotional state.

[0761] Integrated feedback and advice

[0762] The server integrates the feedback obtained from the generative AI with advice based on emotion data and sends it to the device. The device then displays the feedback and advice received from the server to the user, allowing the user to receive support based on the correctness of their answer and their emotion.

[0763] Data storage and analysis

[0764] The server stores all user data, answer history, feedback, and emotional data in a database. This allows the server to track users' past answer data and emotional state, analyze their learning progress and comprehension, and periodically analyze each user's skill gaps to provide customized advice.

[0765] Specific examples

[0766] For example, if a user solves the math problem "6 x 7 = ?" and answers "44," the user writes the answer in a notebook, takes a photo of the notebook, and also captures their facial expression. The device then sends the photo of the homework and the user's facial expression data to the server. The server then converts the photo of the homework into text data using OCR technology and sends it to the generation AI. The generation AI determines that "44" is an incorrect answer and generates feedback such as "The correct answer is 42. Let's move on to the next step." Based on the emotional data of "confused," the emotion engine generates advice such as "Don't rush, try again. If you don't understand, check the hint." The server then combines the generated feedback and advice and sends it to the device. The device then displays this information to the user.

[0767] Prompt Sentence Examples

[0768] An example of a prompt sentence to be input to the generative AI model is shown below.

[0769] The answer to the homework problem the user solved was "6 x 7 = 44." Check the accuracy of this answer and provide feedback on the correct answer and why, as well as next steps for the user if they got it wrong. Additionally, your data indicates the user's emotional state is "confused." Include advice based on their emotional state.

[0770] As described above, the present invention accurately evaluates a user's homework answers and provides customized feedback based on emotional data, thereby increasing the user's motivation to study and providing efficient learning support.

[0771] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0772] Step 1:

[0773] The user enters their homework answers into the device or takes a photo of their notes. They also capture their facial expressions with the device's camera to obtain emotional data.

[0774] Input: Homework answers (text or photos), facial expression data

[0775] Output: Homework answer data, emotion data

[0776] Specific actions: The user writes a homework answer by hand or digitally and takes a photo of it. The device's camera captures their facial expression and prepares it for transmission to the system.

[0777] Step 2:

[0778] The device sends the homework answer data (text or photo) and emotion data, along with identification information such as the user ID, to the server.

[0779] Input: Homework answer data, emotion data, user ID

[0780] Output: Data packet (homework answer data, emotion data, user ID)

[0781] Specific operation: The device packs the user ID, homework answer data, and emotion data into a single packet and sends it to the server using the HTTPS protocol.

[0782] Step 3:

[0783] The server receives the data sent from the device and first sends it to the emotion engine for emotional analysis. If the homework answers are photo data, they are converted into text data using OCR.

[0784] Input: Data packet (homework answer data, emotion data, user ID)

[0785] Output: Analyzed emotion data, text data (if necessary)

[0786] Specific operation: The server receives the data packet and first sends it to the emotion engine for analysis of the user's emotion data. If the homework answer is a photo, OCR technology is used to convert the characters and formulas in the photo into text data.

[0787] Step 4:

[0788] The server sends the converted text data to the generative AI model, which determines the correct answer and provides detailed feedback.

[0789] Input: Text data

[0790] Output: Feedback data

[0791] How it works: The server sends text data as prompts to the generative AI model, which then determines whether the answer is correct or incorrect and generates feedback including detailed explanations if the answer is incorrect.

[0792] Step 5:

[0793] An emotion engine analyzes the emotional data and generates customized advice.

[0794] Input: Emotion data

[0795] Output: Advice data

[0796] Specific operation: The emotion engine analyzes the user's emotional state (e.g., "confused") and generates additional advice based on the results.

[0797] Step 6:

[0798] The server integrates feedback from the generative AI and advice from the emotion engine and sends it to the device.

[0799] Input: Feedback data, advice data

[0800] Output: Consolidated message

[0801] Specific operation: The server combines the feedback obtained from the generation AI and the advice from the emotion engine into a single integrated message and sends it to the device via HTTPS protocol.

[0802] Step 7:

[0803] The terminal receives the integrated message from the server and displays it to the user.

[0804] Input: Unified message

[0805] Output: Display data

[0806] Specific operation: The device analyzes the received integrated message and displays it to the user in an appropriate format. The user checks the feedback and advice and plans their next action.

[0807] Step 8:

[0808] The server stores the user's answer history, feedback, and emotion data in a database, thereby tracking the user's understanding and learning progress.

[0809] Input: Answer history, feedback, emotional data

[0810] Output: Saved data

[0811] Specific operation: The server stores the user's answer history, feedback, and emotional data in a database, and continuously manages and analyzes them to understand each user's learning progress and level of understanding.

[0812] (Application example 2)

[0813] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0814] Conventional homework feedback systems have struggled to provide users with prompt and detailed feedback on their completed homework. Furthermore, the feedback was not customized to take into account the user's feelings, resulting in problems such as users not receiving appropriate support and reduced learning efficiency. Furthermore, it was difficult to track users' learning history and progress and provide appropriate advice and learning materials.

[0815] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0816] In this invention, the server includes means for receiving information on homework answers answered by the user, means for analyzing the received homework information and converting the answers into text data, means for sending the text data to the generation AI to determine whether the answer was correct and to receive feedback, means for analyzing the user's facial expressions to collect emotional data, means for analyzing the emotional data and providing customized advice based on the feedback from the generation AI, means for sending the user advice based on the feedback and emotional data obtained from the generation AI, means for recording the user's answer history and feedback and tracking progress, and means for analyzing the user's level of understanding of the study and providing self-study hints and learning materials. This allows the user to receive prompt and detailed feedback and advice that takes their emotions into consideration, enabling efficient study.

[0817] A "user" is someone who uses the system to submit homework and receive feedback.

[0818] "Homework information" is data including the content of the homework answered by the user, and is sent to the server in the form of a photograph or text.

[0819] "Text data" refers to data in which homework information has been converted into a string of characters using OCR technology or similar.

[0820] "Generative AI" is an artificial intelligence that analyzes received text data, determines whether the answer is correct, and generates feedback.

[0821] "Emotion data" is data that indicates the emotional state of the user analyzed from facial expressions, etc.

[0822] "Customized advice" is individual advice provided based on feedback from the generative AI and taking into account the user's emotional data.

[0823] The "means for analyzing facial expressions" is a mechanism for detecting facial expressions from images or videos of the user's face and analyzing emotions.

[0824] "Answer history" is a record of the homework answers a user has given and the feedback they have received.

[0825] "Means for tracking progress" refers to a system for tracking and managing learning progress based on the user's answer history and feedback.

[0826] The "means for analyzing the level of understanding" is a mechanism for evaluating the user's past answers and feedback to understand the user's level of understanding and weaknesses.

[0827] "Self-study tips and materials" are study support materials and advice provided according to the user's level of understanding and progress.

[0828] A "database" is a recording device for saving and storing information such as a user's answer history, feedback, and emotional data.

[0829] Optical character recognition (OCR) is a technology that analyzes characters in image data and converts them into text data.

[0830] This system receives information about homework completed by a user and uses a generation AI to determine whether the answer is correct and provide detailed feedback. Furthermore, it uses an emotion engine to collect and analyze the user's emotion data and provide customized support based on that data. This system operates in cooperation with the user, device, server, and emotion engine.

[0831] User operations

[0832] Users submit their homework answers using their devices. Homework answers may be submitted as photos or as text. When submitting homework information, the user's emotional state at that time is also transmitted. For example, the device's camera may be used to capture the user's facial expression and analyze the emotion.

[0833] Device Features

[0834] The device sends the user's photos, input text, and emotional data to the server. When sending this data, the device also includes identification information such as the user ID. The device also displays feedback from the server and advice based on the user's emotions. In this way, the user can receive not only their answers but also feedback that takes into account their current emotions.

[0835] Server Processing

[0836] The server receives the homework information and emotion data sent from the device. If the received homework information is photo data, it first uses optical character recognition (OCR) technology to convert the characters and mathematical formulas in the photo into text data. The OCR library used is "Tesseract OCR." The converted text data is sent to the generation AI, which determines whether it is correct. "GPT-4" is used as the generation AI model. The generation AI analyzes the received text data and determines whether it is correct or incorrect. If the answer is incorrect, it generates detailed feedback along with an explanation.

[0837] Emotional Data Analysis

[0838] The user's emotional data is obtained by analyzing facial expressions. An emotion recognition engine such as Microsoft Azure Face API is used to analyze the facial expression data and recognize the user's emotional state. The emotion engine analyzes the received user's emotional data and understands that emotional state. The server reflects the emotional data analyzed by the emotion engine in the feedback obtained from the generation AI and provides feedback appropriate to the user's emotional state.

[0839] Recording and analyzing learning

[0840] The server stores the user's answer history, feedback, and emotional data in a database. A typical relational database management system (RDBMS) is used as the database. This allows us to track the questions the user has answered in the past and their emotional state at the time. Furthermore, by analyzing this data, we can understand the user's level of understanding and learning progress and provide appropriate learning tips and materials.

[0841] Specific examples

[0842] For example, if a user answers "5" to the math problem "2+2=?", the following scenario is possible: The user takes a photo of their notebook with their camera and enters "2+2=5." At this time, the user's facial expression data is also sent. The server uses OCR technology to convert "2+2=5" into text data, and the generative AI (GPT-4) determines that "5" is an incorrect answer and generates detailed feedback such as "The correct answer is 4." At the same time, the emotion engine analyzes the user's facial expression data as "anxiety." Based on this information, the server adds customized advice such as "Calm down and think about it. Let's try again," and sends the final feedback to the user's device.

[0843] Prompt Sentence Examples

[0844] "The user answers '2 + 2 = 5'. Please rate this answer, tell us the correct answer, provide detailed feedback, and offer additional advice if the user is in a state of uncertainty."

[0845] In this way, the system of the present invention instantly evaluates the homework answers of the user and provides detailed feedback and customized advice based on emotions, thereby helping the user to study more efficiently.

[0846] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0847] Step 1:

[0848] User submits homework

[0849] The user submits their homework answers using a device. At this time, the homework information is photographed and facial expression data is captured at the same time. When the user takes a photo of their homework and presses the send button, the homework information and facial expression data are entered into the device. This generates the homework photo data (JPEG, PNG, etc.) and facial expression data.

[0850] Step 2:

[0851] The device sends the data to the server

[0852] The device sends the homework photo and facial expression data taken by the user to the server using protocols such as HTTP POST requests. The input is the photo data and facial expression data received from the parent, and the output is the data packet sent to the server.

[0853] Step 3:

[0854] The server analyzes the homework information using OCR technology

[0855] The server performs OCR processing on the received homework photo data and converts the characters in the image into text data. This processing uses the Tesseract OCR library. The input is the photo data (JPEG, PNG, etc.) received by the server, and the output is the text data (String) generated by OCR.

[0856] Step 4:

[0857] The server sends the text data to the generation AI and receives feedback and a correct answer.

[0858] The server sends the text data obtained by OCR to the generation AI (GPT-4), determines whether the answer is correct, and generates feedback. It constructs a prompt sentence and sends the text data as input to the generation AI. The input is the text data and the prompt sentence, and the output is the generated feedback sentence.

[0859] Example prompt: "The user answered '2 + 2 = 5'. Please rate this answer, tell us the correct answer, and provide detailed feedback."

[0860] Step 5:

[0861] The server analyzes the facial expression data and obtains emotion data.

[0862] The server sends the facial expression data to an emotion recognition engine (Microsoft Azure Face API) to analyze the emotion data. The input is the facial expression data (image), and the output is the analyzed emotional state (e.g., "anxiety").

[0863] Step 6:

[0864] The server generates feedback as customized advice.

[0865] The server generates customized advice based on the feedback obtained from the generation AI, taking into account the emotional data. It uses the emotional data as input and adds additional advice to the feedback of the generation AI. The input is the feedback from the generation AI and the emotional data, and the output is a customized feedback sentence.

[0866] Step 7:

[0867] The server sends the generated feedback to the user's device.

[0868] The server sends the generated customized feedback to the user's terminal, where the input is the customized feedback sentence and the output is the feedback displayed on the user's terminal.

[0869] Step 8:

[0870] The device displays feedback

[0871] The terminal displays the feedback received from the server to the user. The input is the feedback text received from the server, and the output is the feedback displayed on the terminal's display.

[0872] Step 9:

[0873] The server records answer history and feedback, and tracks progress.

[0874] The server records the user's answer history and feedback in a database and tracks the progress. The input is the user's answer history and feedback data, and the output is the information recorded in the database.

[0875] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0876] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0877] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0878] [Third embodiment]

[0879] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0880] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0881] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0882] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0883] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0884] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0885] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0886] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0887] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0888] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0889] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0890] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0891] The present invention is a system for receiving information on homework answers provided by users, analyzing, evaluating, and providing feedback on the answers. This system is realized through cooperation between users, terminals, and a server.

[0892] User operations

[0893] First, the user submits the homework answer using the device. The answer may be submitted as a photo or as text. For example, the user may take a photo of the notebook in which they have completed an arithmetic problem and save it on the device. The image file is then uploaded to the server.

[0894] Device Features

[0895] The device sends photos taken by the user and text entered by the user to the server. When sending, the device also sends identification information such as the user ID. This information is used by the server to process the data. The device also displays feedback from the server to the user. This allows the user to instantly receive the evaluation results for their answers.

[0896] Server Processing

[0897] The server receives the homework information sent from the device. If the received information is photo data, it first uses optical character recognition (OCR) technology to convert the letters and formulas in the photo into text data. The converted text data is then sent to the generation AI, which determines whether the answer is correct.

[0898] The generation AI analyzes the received text data and determines whether the answer is correct or incorrect. If the answer is incorrect, it generates detailed feedback about which part is incorrect and how to derive the correct answer. This feedback is sent back to the server, which then sends it to the user.

[0899] Recording and analyzing learning

[0900] The server stores the user's answer history and feedback in a database. This allows the server to track the questions the user has answered in the past and the feedback they received. Furthermore, by analyzing this data, the server can understand the user's level of understanding and progress and provide appropriate study tips and materials.

[0901] Example

[0902] For example, consider the case where a user solves the math problem "6 x 7 = ?" and answers "44."

[0903] 1. The user writes this answer in a notebook and takes a photo of the notebook.

[0904] 2. The device sends the photo it took to the server.

[0905] 3. The server uses OCR technology to convert the photo into text data ("6 x 7 = 44").

[0906] 4. The server sends this text data to the generation AI.

[0907] 5. The generation AI determines that "44" is an incorrect answer and generates feedback saying, "The correct answer is 42. Let's move on to the next step."

[0908] 6. The server receives this feedback and sends it to the user's device.

[0909] 7. The device displays the feedback to the user, who confirms it.

[0910] In this way, the system provides efficient learning support by instantly evaluating the user's homework answers and providing detailed feedback.The system also stores the user's learning history in a database and analyzes their understanding and progress, making it possible to provide hints and learning materials for self-study.

[0911] The processing flow will be explained below.

[0912] Step 1:

[0913] The user prepares the homework answers by writing them in a notebook or by typing them directly into the device.

[0914] Step 2:

[0915] The user takes a photo of the note on the device or inputs text. For example, the user takes a photo of the note using a smartphone or tablet and saves it on the device.

[0916] Step 3:

[0917] The device sends the homework information from the user to the server. The information sent includes photo data, text data, and identification information such as the user ID.

[0918] Step 4:

[0919] The server receives the homework information sent from the device, temporarily stores the received data, and prepares it for processing.

[0920] Step 5:

[0921] The server analyzes the received homework information. If the homework is a photo, it uses optical character recognition (OCR) technology to convert the characters and mathematical formulas in the image into text data.

[0922] Step 6:

[0923] The server sends the text data to the AI ​​that generates it, which then analyzes the received text data and determines the correct answer.

[0924] Step 7:

[0925] The generation AI determines whether the user's answer is correct and generates feedback. Based on the data it receives, the generation AI determines whether the user's answer is correct, and if it is incorrect, generates detailed feedback on how the answer should be answered.

[0926] Step 8:

[0927] The server receives the feedback from the generated AI and prepares it to be sent to the user. The feedback includes correct and incorrect answers and detailed explanations.

[0928] Step 9:

[0929] The device receives feedback from the server and displays it to the user, who can then check the feedback and determine whether their answer was correct or incorrect.

[0930] Step 10:

[0931] The server stores the user's answer history and feedback in a database, which provides data for later analysis of the user's learning progress and understanding.

[0932] Step 11:

[0933] The server analyzes the user's level of understanding and progress, and evaluates the user's stumbling points and level of understanding based on past answer history and feedback.

[0934] Step 12:

[0935] The server provides self-study tips and materials based on the user's learning needs, such as relevant video lectures and additional practice questions.

[0936] In this way, the system not only evaluates users' homework and provides immediate feedback, but also provides long-term learning support. By continuously tracking each user's learning progress and providing appropriate support as needed, the system aims to motivate students to learn more efficiently.

[0937] Example 1

[0938] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0939] Traditional learning systems have the problem that it takes time to evaluate and provide feedback on users' homework answers, making it difficult to track users' understanding and progress in real time. This prevents users from receiving appropriate feedback immediately, resulting in reduced learning efficiency. Furthermore, there is a lack of a means to accumulate past answer history and feedback in a database and manage learning progress over the long term, making it difficult to properly evaluate users' understanding and skill gaps.

[0940] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0941] In this invention, the server includes: a means for receiving information about homework completed by a user; a means for analyzing the received homework information and converting the answers into text data; a means for sending the text data to a generative AI model to determine whether the answers are correct and receive feedback; a means for sending the feedback obtained from the generative AI model to the user; a means for recording the user's answer history and feedback and tracking progress; and a means for analyzing the user's level of understanding and providing self-study hints and learning materials. This allows the user to receive evaluations and feedback on their completed homework in real time, improving learning efficiency. It also enables the long-term accumulation and analysis of learning data, allowing for an accurate understanding of the user's level of understanding and progress, enabling appropriate learning guidance.

[0942] "User" refers to an individual who uses the system to submit homework answers and receive evaluations and feedback.

[0943] "Homework information" refers to the content of the homework that the user has answered, and is sent to the server as text data or photo data.

[0944] The "receiving means" is a mechanism by which the server receives the homework information sent by the user.

[0945] The "analysis means" is a technology that allows the server to analyze the homework information received and convert it into text data as needed.

[0946] A "generative AI model" is an artificial intelligence model that analyzes received text data and generates correct answers and feedback.

[0947] The "means of transmission" is a mechanism for transmitting feedback obtained from the generative AI model to the user's device.

[0948] "Means for recording and tracking progress" refers to technology for storing a user's answer history and feedback content, and for managing the user's learning progress over the long term.

[0949] "Means for providing self-study hints and learning materials" is a mechanism for analyzing the user's level of understanding and providing appropriate learning advice and learning materials.

[0950] The "means of receiving the photograph and converting it into text data using optical character recognition technology" is a technology that converts the photograph data sent by the user into text data using optical character recognition technology.

[0951] The "database" is a system that serves as the foundation for storing and analyzing users' learning progress and level of understanding.

[0952] A "skill gap" refers to the difference that exists between a user's current level of understanding and their desired level of understanding.

[0953] The present invention is a system for receiving information on homework answers provided by users, analyzing and evaluating the information, and providing feedback. This system functions based on the cooperation of users, terminals, and a server.

[0954] First, the user enters the answer to their homework using their device and sends it to the server. The answer may be sent as a photo or directly entered as text. For example, if a user solves an arithmetic problem and writes the answer in a notebook, they can take a photo of the notebook using a device such as a smartphone or tablet and save the image file on the device. They can then upload the image file to the server using a dedicated application.

[0955] The device sends identification information such as the user ID to the server along with the photo data taken by the user and the text data entered by the user. This allows the server to process the data appropriately. The device also receives evaluation results and feedback from the server and displays them to the user. This allows users to instantly check the evaluation results of their answers and improve the quality of their learning.

[0956] The server first receives the homework information sent from the device. If the received information is photo data, the server uses optical character recognition (OCR) technology to convert the characters and mathematical formulas in the photo into text data. Specifically, OCR software such as Google Cloud Vision API is used. This converted text data is then sent to a generative AI model. A generative AI model is an artificial intelligence that analyzes the text data and generates a correct answer and detailed feedback. A typical generative AI model is OpenAI GPT-4.

[0957] The generative AI model analyzes the accuracy of the answer based on the received text data and determines whether it is correct or incorrect. If the answer is incorrect, the generative AI model generates detailed feedback about which part is incorrect and how to derive the correct answer. This generated feedback is sent back to the server, which then sends it to the user's device. The feedback content is saved in a database and maintained as the user's learning history.

[0958] As a specific example of how it works, suppose a user solves the problem "6 x 7 = ?". The user writes the answer "44" in a notebook, takes a photo of it, and uses their device to send the photo to the server. The device sends the photo data and user ID to the server. The server uses OCR technology to convert the photo data into text data "6 x 7 = 44" and sends it to the generative AI model. The generative AI model determines that "44" is an incorrect answer and generates feedback saying "The correct answer is 42. Let's proceed to the next step," which is sent back to the server. The server sends this feedback to the user's device, which then displays the feedback.

[0959] An example of a prompt for a generative AI model is as follows:

[0960] "The user answered the following question: 6 x 7 = 44. Please analyze and evaluate this answer to generate corrective feedback."

[0961] This system allows users to receive instant, detailed feedback, enabling them to study efficiently. Furthermore, by analyzing the learning history stored in the database, it is possible to accurately grasp the user's level of understanding and progress, and provide appropriate learning guidance and teaching materials as needed.

[0962] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0963] Step 1:

[0964] The user enters the homework answer.

[0965] Input: Homework answered by the user (photo data or text data)

[0966] How it works: A user writes their homework answers in a notebook and takes a photo of it with their smartphone or tablet, or types it in as text.

[0967] Output: Homework answer data (photo or text)

[0968] Step 2:

[0969] The user sends the answer data through the terminal.

[0970] Input: User's homework answer data, user ID

[0971] How it works: A user opens the app, uploads a photo or text of their answer, and presses the submit button, automatically attaching their user ID.

[0972] Output: Homework answer data and user ID sent to the device

[0973] Step 3:

[0974] The device sends the answer data and user ID to the server.

[0975] Input: Answer data, user ID

[0976] Operation: After the send button is pressed, the terminal sends the answer data and user ID to the server.

[0977] Output: Answer data and user ID sent to the server

[0978] Step 4:

[0979] The server receives the answer data and the user ID.

[0980] Input: Answer data and user ID sent from the device

[0981] Operation: The server uses the receiving module to receive the answer data and the user ID.

[0982] Output: Answer data and user ID saved on the server

[0983] Step 5:

[0984] The server analyzes the answer data.

[0985] Input: Received answer data

[0986] How it works: The server analyzes the answer data and converts it into text using optical character recognition (OCR) technology, such as Google Cloud Vision API or similar OCR software.

[0987] Output: Parsed text data

[0988] Step 6:

[0989] The server generates prompts that send text data to the generative AI model.

[0990] Input: Parsed text data

[0991] How it works: The server uses the prompt generation module to create a prompt sentence to send to the generative AI model.

[0992] Output: Generated prompt statement

[0993] Step 7:

[0994] The server sends the prompt to the generative AI model.

[0995] Input: Generated prompt text

[0996] How it works: The server sends a prompt to a generative AI model (e.g., OpenAI GPT-4).

[0997] Output: Feedback generated by the generative AI model

[0998] Step 8:

[0999] A generative AI model generates an evaluation and feedback on the answer.

[1000] Input: prompt statement

[1001] How it works: A generative AI model analyzes prompts, determines whether an answer is correct or incorrect, and generates detailed feedback.

[1002] Output: Generated feedback

[1003] Step 9:

[1004] The server receives the feedback and sends it to the user's device.

[1005] Input: Feedback from the generative AI model, user ID

[1006] Operation: The server uses a feedback receiving module to receive feedback from the generative AI model and sends it to the device with the corresponding user ID.

[1007] Output: Feedback sent to the terminal

[1008] Step 10:

[1009] The terminal displays the received feedback to the user.

[1010] Input: Feedback sent by the server

[1011] How it works: The device receives the feedback and displays it to the user via a dedicated application.

[1012] Output: Displayed feedback

[1013] Step 11:

[1014] The server records and stores the user's answer history and feedback.

[1015] Input: Answer history and feedback received

[1016] How it works: The server stores answer history and feedback in a database.

[1017] Output: Saved answer history and feedback

[1018] Step 12:

[1019] The server analyzes the user's learning progress.

[1020] Input: Saved answer history and feedback

[1021] How it works: The server analyzes the database to assess the user's learning progress and skill gaps, and provides appropriate learning tips and materials.

[1022] Output: Study tips and materials based on the analysis results

[1023] (Application example 1)

[1024] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1025] Conventional homework assessment systems have the problem that it is difficult for users to receive immediate feedback on their homework answers, which reduces learning efficiency. They also lack a mechanism for tracking users' progress and level of understanding in real time and providing appropriate learning hints and learning materials. Furthermore, there is no way to effectively utilize learning history or past feedback. To solve these problems, it is necessary to provide users with real-time feedback and improve learning efficiency.

[1026] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1027] In this invention, the server includes: means for receiving information about homework completed by a user; means for analyzing the received homework information and converting the answers into text data; means for sending the text data to a generation AI to determine whether the answers are correct and receive feedback; means for sending the feedback obtained from the generation AI to the user; means for recording the user's answer history and feedback and tracking progress; means for analyzing the user's level of understanding and providing self-study hints and learning materials; means for uploading images of the homework answers taken by the user to the server; means for the server to convert the image data into text data using OCR technology; means for displaying the determination of correct answers and feedback to the user in real time; and means for saving the user's learning history and analyzing past answers and feedback. This allows the system to analyze the information about homework completed by a user in real time and provide immediate feedback. Furthermore, by saving and analyzing the learning history, the system can provide appropriate learning materials according to the user's progress and level of understanding.

[1028] A "user" is an entity that utilizes the system to submit homework answers and receive feedback.

[1029] "Homework information" is data indicating the content of the homework that the user has answered, and is sent from the terminal as photographs or text data.

[1030] The "receiving means" is a mechanism for importing the homework information sent by the user into the server and processing it.

[1031] The "means for analyzing and converting into text data" is a function for analyzing the received homework information and converting it from photo data into text data using optical character recognition technology.

[1032] "Generative AI" is an artificial intelligence model that determines the correctness of answers based on generated text data and generates feedback.

[1033] "Server" means the computer system that receives, processes, stores, and generates feedback on homework information submitted by users.

[1034] "Feedback" refers to the results of the generative AI's determination of the correct answer and information on how the user can arrive at the correct answer.

[1035] The "means for uploading" is a mechanism for sending the homework answer images taken by the user to the server.

[1036] "OCR technology" is an abbreviation for optical character recognition technology, which converts characters and mathematical formulas in image data into text data.

[1037] The "means for displaying to the user in real time" is a mechanism for instantly presenting the feedback sent from the server to the user.

[1038] "Study history" is data that includes questions that the user has answered in the past and feedback on those questions.

[1039] The "analysis means" is a function that uses saved learning history data to evaluate the user's progress and level of understanding.

[1040] "Teaching materials" refer to study materials and questions provided according to the user's level of understanding.

[1041] This invention provides a system that receives information about homework answers provided by users, analyzes, evaluates, and provides feedback. This system is realized through cooperation between users, terminals, and a server.

[1042] User operations

[1043] First, the user submits the homework answer using the device. The answer may be submitted as a photo or as text. For example, the user may take a photo of the notebook in which they have completed an arithmetic problem and save it on the device. The image file is then uploaded to the server.

[1044] Device Features

[1045] The device sends photos taken by the user and text entered by the user to the server. When sending, the device also sends identification information such as the user ID. This information is used by the server to process the data. The device also displays feedback from the server to the user. This allows the user to instantly receive the evaluation results for their answers.

[1046] Server Processing

[1047] The server receives homework information sent from the device. If the received information is photo data, optical character recognition (OCR) technology is used to convert the characters and mathematical formulas in the photo into text data. This conversion is performed using the pytesseract library. The converted text data is then sent to a generative AI model, which determines whether the answer is correct or incorrect. If the answer is incorrect, detailed feedback is generated about which part is incorrect and how to derive the correct answer.

[1048] View Feedback

[1049] The server then sends this feedback back to the terminal, which then displays it to the user, allowing the user to immediately see detailed feedback on their answers.

[1050] Recording and analyzing learning

[1051] The server stores the user's answer history and feedback in a database. This data is used to track the questions the user has answered in the past and the feedback they received. Furthermore, by analyzing this data, the server can understand the user's progress and level of understanding and provide appropriate study tips and materials. This analysis is performed using machine learning algorithms.

[1052] Specific examples

[1053] For example, consider the case where a user solves the problem "6 x 7 = ?" and answers "44."

[1054] 1. The user writes this answer in a notebook and takes a photo of the notebook.

[1055] 2. The device sends the photo it took to the server.

[1056] 3. The server uses OCR technology to convert the photo into text data ("6 x 7 = 44").

[1057] 4. The server sends this text data to the generation AI.

[1058] 5. The generative AI model determines that "44" is an incorrect answer and generates feedback such as "The correct answer is 42. Let's move on to the next step."

[1059] 6. The server receives this feedback and sends it to the user's device.

[1060] 7. The device displays the feedback to the user, who confirms it.

[1061] Prompt Sentence Examples

[1062] Prompt the generative AI model as follows:

[1063] Check the user's answer.

[1064] Question: "6 x 7 = ?"

[1065] Answer: "44"

[1066] Determine the correct answer and generate the necessary feedback.

[1067] Using this prompt, the generative AI model can generate appropriate feedback and provide it to the user.

[1068] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1069] Step 1:

[1070] A user writes answers to homework in a notebook and takes a photo of the notebook using a device such as a smartphone or tablet.

[1071] Input: Homework answers written in a notebook.

[1072] Output: A photo image file containing the homework answers.

[1073] Step 2:

[1074] The photos taken by the device are uploaded to the server, and are sent together with identification information such as the user ID and answer date and time.

[1075] Input: Photo image file taken, user ID, answer date and time.

[1076] Output: Images and related information sent to the server.

[1077] Step 3:

[1078] The server analyzes the received photo data using optical character recognition (OCR) technology and converts characters and mathematical expressions in the image into text data. This conversion is performed using the pytesseract library.

[1079] Input: Uploaded photo image file.

[1080] Output: Text data converted from characters and mathematical expressions in the image.

[1081] Step 4:

[1082] The server sends the converted text data to the generative AI model, which determines whether the answer is correct and generates feedback.

[1083] Input: Text data converted using OCR technology.

[1084] Output: Generative AI model determines whether the answer is correct or incorrect and provides detailed feedback.

[1085] Step 5:

[1086] The server receives feedback from the generative AI model and sends it to the user's device.

[1087] Input: Feedback from the generative AI model.

[1088] Output: Feedback data sent to the device.

[1089] Step 6:

[1090] The terminal displays the received feedback to the user, who then confirms it.

[1091] Input: Feedback data sent by the server.

[1092] Output: User sees feedback.

[1093] Step 7:

[1094] The server stores the user's answer history and feedback in a database and analyzes their learning progress and level of understanding.

[1095] Input: User's answer history, feedback from the server.

[1096] Output: Answer history and feedback stored in a database, and analysis of the user's understanding.

[1097] Step 8:

[1098] The server provides self-study hints and learning materials based on the user's level of learning comprehension, encouraging improvement in learning from the next time onwards.

[1099] Input: Parsed comprehension data.

[1100] Output: Study tips and materials provided to the user.

[1101] Through these steps, users receive real-time feedback and can learn more efficiently.

[1102] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1103] This system receives information about homework completed by a user, uses a generation AI to determine whether the answer is correct and provides detailed feedback, and also combines an emotion engine to collect and analyze the user's emotion data and provide customized support based on that data. This system operates in cooperation with the user, terminal, server, and emotion engine.

[1104] User operations

[1105] First, the user submits their homework answers using their device. Homework answers can be submitted as photos or as text. Furthermore, when submitting the homework information, the user's emotional state at that time is also transmitted. For example, the device's camera can be used to capture the user's facial expression and analyze the emotion.

[1106] Device Features

[1107] The device sends the user's photos, input text, and emotional data to the server. When sending this data, the device also includes identification information such as the user ID. The device also displays feedback from the server and advice based on the user's emotions. In this way, the user can receive not only their answers but also feedback that takes into account their current emotions.

[1108] Server Processing

[1109] The server receives the homework information and emotion data sent from the device. If the homework information received is photo data, it first uses optical character recognition (OCR) technology to convert the letters and mathematical formulas in the photo into text data. The converted text data is then sent to the generation AI, which determines whether it is the correct answer.

[1110] The generation AI analyzes the received text data and determines whether the answer is correct or incorrect. If the answer is incorrect, it generates detailed feedback along with an explanation. Meanwhile, the emotion engine analyzes the received user's emotional data and understands their emotional state. The server reflects the emotional data analyzed by the emotion engine in the feedback received from the generation AI and provides feedback appropriate to the user's emotional state.

[1111] Recording and analyzing learning

[1112] The server stores the user's answer history, feedback, and emotional data in a database. This allows the server to track the questions the user has answered in the past and their emotional state at the time. Furthermore, by analyzing this data, the server can understand the user's level of understanding and learning progress and provide appropriate learning tips and materials.

[1113] Example

[1114] For example, consider the case where a user solves the math problem "6 x 7 = ?" and answers "44."

[1115] 1. The user writes the answer in a notebook, takes a photo of the notebook, and also takes a photo of their own expression using the device's camera.

[1116] 2. The device takes a photo of the homework and sends the user's facial expression data to the server.

[1117] 3. The server uses OCR technology to convert the homework photo into text data ("6 x 7 = 44"), and at the same time uses an emotion engine to analyze the facial expression data and recognize the child's emotional state (for example, "confused").

[1118] 4. The server sends this text data to the generation AI, which determines that "44" is an incorrect answer and generates feedback saying, "The correct answer is 42. Let's move on to the next step."

[1119] 5. Based on the emotional data of "confused," the emotion engine generates additional advice: "Don't rush, try again. If you don't understand, check the hints."

[1120] 6. The server sends the generated feedback and advice to the user.

[1121] 7. The device receives feedback and advice from the server and displays it to the user.

[1122] 8. The user reviews the feedback and advice to understand whether their answer was correct or incorrect and what their next steps are.

[1123] In this way, the system instantly evaluates the user's homework answers and provides detailed feedback and personalized advice based on their emotions, thereby motivating the user to study more efficiently.

[1124] The processing flow will be explained below.

[1125] Step 1:

[1126] The user prepares the homework answer by writing it in a notebook or by typing it directly into the device.

[1127] Step 2:

[1128] The user takes a photo of the note with the device, and also captures their facial expressions using the device's camera to collect their emotional state.

[1129] Step 3:

[1130] The device takes a photo of the homework and sends it to the server along with facial expression data (emotion data), along with identification information such as the user ID.

[1131] Step 4:

[1132] The server receives the homework information and emotion data sent from the device, temporarily stores the data, and then starts processing it.

[1133] Step 5:

[1134] The server analyzes the image data of the homework and uses OCR technology to convert the letters and mathematical formulas in the photo into text data, which is then sent to the next step of processing.

[1135] Step 6:

[1136] The server uses an emotion engine to analyze the user's emotional state from the received facial expression data. For example, it can recognize emotions such as "confusion" or "happiness" from the user's facial expression.

[1137] Step 7:

[1138] The server sends the text data and emotion data to the generation AI, which uses this data to determine whether the answer is correct and generates detailed feedback.

[1139] Step 8:

[1140] The generation AI judges the answer and generates feedback. For example, for the answer "6 x 7 = 44," it generates feedback such as "That's incorrect. The correct answer is 42."

[1141] Step 9:

[1142] The emotion engine generates additional advice that reflects the emotional data based on feedback from the generation AI. For example, in response to the emotional data of "confused," the engine generates advice such as "Try again without rushing."

[1143] Step 10:

[1144] The server compiles the feedback and advice received from the generative AI and emotion engine and prepares it to be sent to the user.

[1145] Step 11:

[1146] The terminal receives feedback and emotion-based advice from the server and displays it to the user, who can then review it to understand whether their answer was correct or incorrect and what to do next.

[1147] Step 12:

[1148] The server stores the user's answer history and emotional data in a database, which stores data that tracks the user's learning progress, comprehension, and emotional state.

[1149] Step 13:

[1150] The server analyzes the user's learning comprehension, progress, and emotional data, and provides individually customized self-study tips and learning materials. For example, it analyzes past emotional data and provides more detailed explanations for particularly difficult problems.

[1151] In this way, our system not only instantly evaluates users' homework and provides detailed feedback and emotion-based advice, but also provides long-term learning support. By adjusting feedback based on users' emotion data, we aim to increase their motivation to learn and provide effective learning support.

[1152] Example 2

[1153] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1154] Conventional educational support systems have the ability to determine whether a user's answers are correct, but they are unable to provide feedback or advice that takes into account the user's emotional state. This makes it difficult to provide individual support to increase the user's motivation to learn, and they are unable to provide efficient learning support. Furthermore, there is a lack of technology that can comprehensively manage and analyze a user's learning progress and emotional data, and provide customized feedback.

[1155] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1156] In this invention, the server includes means for receiving information about homework answers completed by the user and emotional data about the user, means for analyzing the received homework information and converting the answers into text data, means for sending the text data to a generation AI to determine whether the answers are correct and receive feedback, means for incorporating the results of the emotional analysis into the feedback received from the generation AI to generate feedback and advice appropriate to the user's emotional state, means for sending the generated feedback and advice to the user, means for recording the user's answer history, feedback, and emotional data and tracking progress, and means for analyzing the user's level of understanding of the learning and providing hints and learning materials for self-study. This provides customized feedback that takes the user's emotional state into consideration, thereby improving their motivation to learn and providing efficient learning support.

[1157] A "user" is an individual or entity who uses the system to submit homework answers.

[1158] A "homework assignment" is a problem or task assigned to a user for educational or training purposes.

[1159] "Information" refers to homework answers and related data submitted by users.

[1160] "Emotional data" refers to data about a user's emotional state collected through the device's camera or sensors.

[1161] A "terminal" is a device on which a user enters answers to homework, and includes a camera, smartphone, tablet, PC, etc.

[1162] A "server" is a computer system that receives data sent from a terminal and analyzes and processes it.

[1163] Optical character recognition (OCR) is a technology that analyzes characters and mathematical formulas in photo data and converts them into text data.

[1164] "Generative AI" is an artificial intelligence technology that analyzes text data and generates correct answers and feedback.

[1165] "Feedback" is information that includes an evaluation of the user's homework answers as well as next steps and advice.

[1166] The "emotion engine" is an artificial intelligence technology that analyzes the user's emotional data and reflects the results in feedback.

[1167] A "database" is a system for storing and managing a user's answer history, feedback, and emotional data.

[1168] "Level of understanding of learning" is an index that evaluates how much a user has understood through homework.

[1169] "Self-study tips and materials" are additional learning resources and advice provided to help users further their learning.

[1170] "Customized advice" is specialized feedback or advice provided based on a user's individual answer history and emotional state.

[1171] The present invention is a system that receives homework answers and emotion data submitted by a user, analyzes them, and provides feedback and customized advice. This system operates in cooperation with a user, a terminal, a server, and an emotion engine.

[1172] System Overview

[1173] The system includes the following hardware and software components:

[1174] Devices operated by users (e.g. smartphones, tablets, PCs)

[1175] Server that processes and stores data

[1176] Optical character recognition technology (OCR)

[1177] Generative AI Models

[1178] Emotion Engine

[1179] Database

[1180] Device behavior

[1181] The user submits their homework answers using a device. Answers can be entered as text or photos. The device's camera is used to collect the user's facial expression data, which is then transmitted as emotion data. The device then transmits the user's answer and emotion data to the server as a single data packet.

[1182] Server Processing

[1183] The server receives data sent from the user's device. First, it uses the emotion engine to analyze the emotion data and identify the user's emotional state. If the homework information is in the form of photographs, it uses optical character recognition (OCR) technology to convert the letters and mathematical formulas in the photographs into text data. This conversion process is performed based on an image analysis algorithm.

[1184] Answer validation and feedback generation

[1185] The server sends the converted text data to a generative AI model, which determines whether the answer is correct and generates detailed feedback. The generative AI determines whether the homework answer is correct and provides feedback along with a detailed explanation if the answer is incorrect. Furthermore, by incorporating the analysis results of the emotion engine into the feedback obtained from the generative AI, feedback and advice are generated that reflects the user's emotional state.

[1186] Integrated feedback and advice

[1187] The server integrates the feedback obtained from the generative AI with advice based on emotion data and sends it to the device. The device then displays the feedback and advice received from the server to the user, allowing the user to receive support based on the correctness of their answer and their emotion.

[1188] Data storage and analysis

[1189] The server stores all user data, answer history, feedback, and emotional data in a database. This allows the server to track users' past answer data and emotional state, analyze their learning progress and comprehension, and periodically analyze each user's skill gaps to provide customized advice.

[1190] Specific examples

[1191] For example, if a user solves the math problem "6 x 7 = ?" and answers "44," the user writes the answer in a notebook, takes a photo of the notebook, and also captures their facial expression. The device then sends the photo of the homework and the user's facial expression data to the server. The server then converts the photo of the homework into text data using OCR technology and sends it to the generation AI. The generation AI determines that "44" is an incorrect answer and generates feedback such as "The correct answer is 42. Let's move on to the next step." Based on the emotional data of "confused," the emotion engine generates advice such as "Don't rush, try again. If you don't understand, check the hint." The server then combines the generated feedback and advice and sends it to the device. The device then displays this information to the user.

[1192] Prompt Sentence Examples

[1193] An example of a prompt sentence to be input to the generative AI model is shown below.

[1194] The answer to the homework problem the user solved was "6 x 7 = 44." Check the accuracy of this answer and provide feedback on the correct answer and why, as well as next steps for the user if they got it wrong. Additionally, your data indicates the user's emotional state is "confused." Include advice based on their emotional state.

[1195] As described above, the present invention accurately evaluates a user's homework answers and provides customized feedback based on emotional data, thereby increasing the user's motivation to study and providing efficient learning support.

[1196] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1197] Step 1:

[1198] The user enters their homework answers into the device or takes a photo of their notes. They also capture their facial expressions with the device's camera to obtain emotional data.

[1199] Input: Homework answers (text or photos), facial expression data

[1200] Output: Homework answer data, emotion data

[1201] Specific actions: The user writes a homework answer by hand or digitally and takes a photo of it. The device's camera captures their facial expression and prepares it for transmission to the system.

[1202] Step 2:

[1203] The device sends the homework answer data (text or photo) and emotion data, along with identification information such as the user ID, to the server.

[1204] Input: Homework answer data, emotion data, user ID

[1205] Output: Data packet (homework answer data, emotion data, user ID)

[1206] Specific operation: The device packs the user ID, homework answer data, and emotion data into a single packet and sends it to the server using the HTTPS protocol.

[1207] Step 3:

[1208] The server receives the data sent from the device and first sends it to the emotion engine for emotional analysis. If the homework answers are photo data, they are converted into text data using OCR.

[1209] Input: Data packet (homework answer data, emotion data, user ID)

[1210] Output: Analyzed emotion data, text data (if necessary)

[1211] Specific operation: The server receives the data packet and first sends it to the emotion engine for analysis of the user's emotion data. If the homework answer is a photo, OCR technology is used to convert the characters and formulas in the photo into text data.

[1212] Step 4:

[1213] The server sends the converted text data to the generative AI model, which determines the correct answer and provides detailed feedback.

[1214] Input: Text data

[1215] Output: Feedback data

[1216] How it works: The server sends text data as prompts to the generative AI model, which then determines whether the answer is correct or incorrect and generates feedback including detailed explanations if the answer is incorrect.

[1217] Step 5:

[1218] An emotion engine analyzes the emotional data and generates customized advice.

[1219] Input: Emotion data

[1220] Output: Advice data

[1221] Specific operation: The emotion engine analyzes the user's emotional state (e.g., "confused") and generates additional advice based on the results.

[1222] Step 6:

[1223] The server integrates feedback from the generative AI and advice from the emotion engine and sends it to the device.

[1224] Input: Feedback data, advice data

[1225] Output: Consolidated message

[1226] Specific operation: The server combines the feedback obtained from the generation AI and the advice from the emotion engine into a single integrated message and sends it to the device via HTTPS protocol.

[1227] Step 7:

[1228] The terminal receives the integrated message from the server and displays it to the user.

[1229] Input: Unified message

[1230] Output: Display data

[1231] Specific operation: The device analyzes the received integrated message and displays it to the user in an appropriate format. The user checks the feedback and advice and plans their next action.

[1232] Step 8:

[1233] The server stores the user's answer history, feedback, and emotion data in a database, thereby tracking the user's understanding and learning progress.

[1234] Input: Answer history, feedback, emotional data

[1235] Output: Saved data

[1236] Specific operation: The server stores the user's answer history, feedback, and emotional data in a database, and continuously manages and analyzes them to understand each user's learning progress and level of understanding.

[1237] (Application example 2)

[1238] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1239] Conventional homework feedback systems have struggled to provide users with prompt and detailed feedback on their completed homework. Furthermore, the feedback was not customized to take into account the user's feelings, resulting in problems such as users not receiving appropriate support and reduced learning efficiency. Furthermore, it was difficult to track users' learning history and progress and provide appropriate advice and learning materials.

[1240] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1241] In this invention, the server includes means for receiving information on homework answers answered by the user, means for analyzing the received homework information and converting the answers into text data, means for sending the text data to the generation AI to determine whether the answer was correct and to receive feedback, means for analyzing the user's facial expressions to collect emotional data, means for analyzing the emotional data and providing customized advice based on the feedback from the generation AI, means for sending the user advice based on the feedback and emotional data obtained from the generation AI, means for recording the user's answer history and feedback and tracking progress, and means for analyzing the user's level of understanding of the study and providing self-study hints and learning materials. This allows the user to receive prompt and detailed feedback and advice that takes their emotions into consideration, enabling efficient study.

[1242] A "user" is someone who uses the system to submit homework and receive feedback.

[1243] "Homework information" is data including the content of the homework answered by the user, and is sent to the server in the form of a photograph or text.

[1244] "Text data" refers to data in which homework information has been converted into a string of characters using OCR technology or similar.

[1245] "Generative AI" is an artificial intelligence that analyzes received text data, determines whether the answer is correct, and generates feedback.

[1246] "Emotion data" is data that indicates the emotional state of the user analyzed from facial expressions, etc.

[1247] "Customized advice" is individual advice provided based on feedback from the generative AI and taking into account the user's emotional data.

[1248] The "means for analyzing facial expressions" is a mechanism for detecting facial expressions from images or videos of the user's face and analyzing emotions.

[1249] "Answer history" is a record of the homework answers a user has given and the feedback they have received.

[1250] "Means for tracking progress" refers to a system for tracking and managing learning progress based on the user's answer history and feedback.

[1251] The "means for analyzing the level of understanding" is a mechanism for evaluating the user's past answers and feedback to understand the user's level of understanding and weaknesses.

[1252] "Self-study tips and materials" are study support materials and advice provided according to the user's level of understanding and progress.

[1253] A "database" is a recording device for saving and storing information such as a user's answer history, feedback, and emotional data.

[1254] Optical character recognition (OCR) is a technology that analyzes characters in image data and converts them into text data.

[1255] This system receives information about homework completed by a user and uses a generation AI to determine whether the answer is correct and provide detailed feedback. Furthermore, it uses an emotion engine to collect and analyze the user's emotion data and provide customized support based on that data. This system operates in cooperation with the user, device, server, and emotion engine.

[1256] User operations

[1257] Users submit their homework answers using their devices. Homework answers may be submitted as photos or as text. When submitting homework information, the user's emotional state at that time is also transmitted. For example, the device's camera may be used to capture the user's facial expression and analyze the emotion.

[1258] Device Features

[1259] The device sends the user's photos, input text, and emotional data to the server. When sending this data, the device also includes identification information such as the user ID. The device also displays feedback from the server and advice based on the user's emotions. In this way, the user can receive not only their answers but also feedback that takes into account their current emotions.

[1260] Server Processing

[1261] The server receives the homework information and emotion data sent from the device. If the received homework information is photo data, it first uses optical character recognition (OCR) technology to convert the characters and mathematical formulas in the photo into text data. The OCR library used is "Tesseract OCR." The converted text data is sent to the generation AI, which determines whether it is correct. "GPT-4" is used as the generation AI model. The generation AI analyzes the received text data and determines whether it is correct or incorrect. If the answer is incorrect, it generates detailed feedback along with an explanation.

[1262] Emotional Data Analysis

[1263] The user's emotional data is obtained by analyzing facial expressions. An emotion recognition engine such as Microsoft Azure Face API is used to analyze the facial expression data and recognize the user's emotional state. The emotion engine analyzes the received user's emotional data and understands that emotional state. The server reflects the emotional data analyzed by the emotion engine in the feedback obtained from the generation AI and provides feedback appropriate to the user's emotional state.

[1264] Recording and analyzing learning

[1265] The server stores the user's answer history, feedback, and emotional data in a database. A typical relational database management system (RDBMS) is used as the database. This allows us to track the questions the user has answered in the past and their emotional state at the time. Furthermore, by analyzing this data, we can understand the user's level of understanding and learning progress and provide appropriate learning tips and materials.

[1266] Specific examples

[1267] For example, if a user answers "5" to the math problem "2+2=?", the following scenario is possible: The user takes a photo of their notebook with their camera and enters "2+2=5." At this time, the user's facial expression data is also sent. The server uses OCR technology to convert "2+2=5" into text data, and the generative AI (GPT-4) determines that "5" is an incorrect answer and generates detailed feedback such as "The correct answer is 4." At the same time, the emotion engine analyzes the user's facial expression data as "anxiety." Based on this information, the server adds customized advice such as "Calm down and think about it. Let's try again," and sends the final feedback to the user's device.

[1268] Prompt Sentence Examples

[1269] "The user answers '2 + 2 = 5'. Please rate this answer, tell us the correct answer, provide detailed feedback, and offer additional advice if the user is in a state of uncertainty."

[1270] In this way, the system of the present invention instantly evaluates the homework answers of the user and provides detailed feedback and customized advice based on emotions, thereby helping the user to study more efficiently.

[1271] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1272] Step 1:

[1273] User submits homework

[1274] The user submits their homework answers using a device. At this time, the homework information is photographed and facial expression data is captured at the same time. When the user takes a photo of their homework and presses the send button, the homework information and facial expression data are entered into the device. This generates the homework photo data (JPEG, PNG, etc.) and facial expression data.

[1275] Step 2:

[1276] The device sends the data to the server

[1277] The device sends the homework photo and facial expression data taken by the user to the server using protocols such as HTTP POST requests. The input is the photo data and facial expression data received from the parent, and the output is the data packet sent to the server.

[1278] Step 3:

[1279] The server analyzes the homework information using OCR technology

[1280] The server performs OCR processing on the received homework photo data and converts the characters in the image into text data. This processing uses the Tesseract OCR library. The input is the photo data (JPEG, PNG, etc.) received by the server, and the output is the text data (String) generated by OCR.

[1281] Step 4:

[1282] The server sends the text data to the generation AI and receives feedback and a correct answer.

[1283] The server sends the text data obtained by OCR to the generation AI (GPT-4), determines whether the answer is correct, and generates feedback. It constructs a prompt sentence and sends the text data as input to the generation AI. The input is the text data and the prompt sentence, and the output is the generated feedback sentence.

[1284] Example prompt: "The user answered '2 + 2 = 5'. Please rate this answer, tell us the correct answer, and provide detailed feedback."

[1285] Step 5:

[1286] The server analyzes the facial expression data and obtains emotion data.

[1287] The server sends the facial expression data to an emotion recognition engine (Microsoft Azure Face API) to analyze the emotion data. The input is the facial expression data (image), and the output is the analyzed emotional state (e.g., "anxiety").

[1288] Step 6:

[1289] The server generates feedback as customized advice.

[1290] The server generates customized advice based on the feedback obtained from the generation AI, taking into account the emotional data. It uses the emotional data as input and adds additional advice to the feedback of the generation AI. The input is the feedback from the generation AI and the emotional data, and the output is a customized feedback sentence.

[1291] Step 7:

[1292] The server sends the generated feedback to the user's device.

[1293] The server sends the generated customized feedback to the user's terminal, where the input is the customized feedback sentence and the output is the feedback displayed on the user's terminal.

[1294] Step 8:

[1295] The device displays feedback

[1296] The terminal displays the feedback received from the server to the user. The input is the feedback text received from the server, and the output is the feedback displayed on the terminal's display.

[1297] Step 9:

[1298] The server records answer history and feedback, and tracks progress.

[1299] The server records the user's answer history and feedback in a database and tracks the progress. The input is the user's answer history and feedback data, and the output is the information recorded in the database.

[1300] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1301] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1302] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1303] [Fourth embodiment]

[1304] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1305] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1306] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1307] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1308] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1309] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1310] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1311] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1312] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1313] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1314] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1315] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1316] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1317] The present invention is a system for receiving information on homework answers provided by users, analyzing, evaluating, and providing feedback on the answers. This system is realized through cooperation between users, terminals, and a server.

[1318] User operations

[1319] First, the user submits the homework answer using the device. The answer may be submitted as a photo or as text. For example, the user may take a photo of the notebook in which they have completed an arithmetic problem and save it on the device. The image file is then uploaded to the server.

[1320] Device Features

[1321] The device sends photos taken by the user and text entered by the user to the server. When sending, the device also sends identification information such as the user ID. This information is used by the server to process the data. The device also displays feedback from the server to the user. This allows the user to instantly receive the evaluation results for their answers.

[1322] Server Processing

[1323] The server receives the homework information sent from the device. If the received information is photo data, it first uses optical character recognition (OCR) technology to convert the letters and formulas in the photo into text data. The converted text data is then sent to the generation AI, which determines whether the answer is correct.

[1324] The generation AI analyzes the received text data and determines whether the answer is correct or incorrect. If the answer is incorrect, it generates detailed feedback about which part is incorrect and how to derive the correct answer. This feedback is sent back to the server, which then sends it to the user.

[1325] Recording and analyzing learning

[1326] The server stores the user's answer history and feedback in a database. This allows the server to track the questions the user has answered in the past and the feedback they received. Furthermore, by analyzing this data, the server can understand the user's level of understanding and progress and provide appropriate study tips and materials.

[1327] Example

[1328] For example, consider the case where a user solves the math problem "6 x 7 = ?" and answers "44."

[1329] 1. The user writes this answer in a notebook and takes a photo of the notebook.

[1330] 2. The device sends the photo it took to the server.

[1331] 3. The server uses OCR technology to convert the photo into text data ("6 x 7 = 44").

[1332] 4. The server sends this text data to the generation AI.

[1333] 5. The generation AI determines that "44" is an incorrect answer and generates feedback saying, "The correct answer is 42. Let's move on to the next step."

[1334] 6. The server receives this feedback and sends it to the user's device.

[1335] 7. The device displays the feedback to the user, who confirms it.

[1336] In this way, the system provides efficient learning support by instantly evaluating the user's homework answers and providing detailed feedback.The system also stores the user's learning history in a database and analyzes their understanding and progress, making it possible to provide hints and learning materials for self-study.

[1337] The processing flow will be explained below.

[1338] Step 1:

[1339] The user prepares the homework answers by writing them in a notebook or by typing them directly into the device.

[1340] Step 2:

[1341] The user takes a photo of the note on the device or inputs text. For example, the user takes a photo of the note using a smartphone or tablet and saves it on the device.

[1342] Step 3:

[1343] The device sends the homework information from the user to the server. The information sent includes photo data, text data, and identification information such as the user ID.

[1344] Step 4:

[1345] The server receives the homework information sent from the device, temporarily stores the received data, and prepares it for processing.

[1346] Step 5:

[1347] The server analyzes the received homework information. If the homework is a photo, it uses optical character recognition (OCR) technology to convert the characters and mathematical formulas in the image into text data.

[1348] Step 6:

[1349] The server sends the text data to the AI ​​that generates it, which then analyzes the received text data and determines the correct answer.

[1350] Step 7:

[1351] The generation AI determines whether the user's answer is correct and generates feedback. Based on the data it receives, the generation AI determines whether the user's answer is correct, and if it is incorrect, generates detailed feedback on how the answer should be answered.

[1352] Step 8:

[1353] The server receives the feedback from the generated AI and prepares it to be sent to the user. The feedback includes correct and incorrect answers and detailed explanations.

[1354] Step 9:

[1355] The device receives feedback from the server and displays it to the user, who can then check the feedback and determine whether their answer was correct or incorrect.

[1356] Step 10:

[1357] The server stores the user's answer history and feedback in a database, which provides data for later analysis of the user's learning progress and understanding.

[1358] Step 11:

[1359] The server analyzes the user's level of understanding and progress, and evaluates the user's stumbling points and level of understanding based on past answer history and feedback.

[1360] Step 12:

[1361] The server provides self-study tips and materials based on the user's learning needs, such as relevant video lectures and additional practice questions.

[1362] In this way, the system not only evaluates users' homework and provides immediate feedback, but also provides long-term learning support. By continuously tracking each user's learning progress and providing appropriate support as needed, the system aims to motivate students to learn more efficiently.

[1363] Example 1

[1364] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1365] Traditional learning systems have the problem that it takes time to evaluate and provide feedback on users' homework answers, making it difficult to track users' understanding and progress in real time. This prevents users from receiving appropriate feedback immediately, resulting in reduced learning efficiency. Furthermore, there is a lack of a means to accumulate past answer history and feedback in a database and manage learning progress over the long term, making it difficult to properly evaluate users' understanding and skill gaps.

[1366] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1367] In this invention, the server includes: a means for receiving information about homework completed by a user; a means for analyzing the received homework information and converting the answers into text data; a means for sending the text data to a generative AI model to determine whether the answers are correct and receive feedback; a means for sending the feedback obtained from the generative AI model to the user; a means for recording the user's answer history and feedback and tracking progress; and a means for analyzing the user's level of understanding and providing self-study hints and learning materials. This allows the user to receive evaluations and feedback on their completed homework in real time, improving learning efficiency. It also enables the long-term accumulation and analysis of learning data, allowing for an accurate understanding of the user's level of understanding and progress, enabling appropriate learning guidance.

[1368] "User" refers to an individual who uses the system to submit homework answers and receive evaluations and feedback.

[1369] "Homework information" refers to the content of the homework that the user has answered, and is sent to the server as text data or photo data.

[1370] The "receiving means" is a mechanism by which the server receives the homework information sent by the user.

[1371] The "analysis means" is a technology that allows the server to analyze the homework information received and convert it into text data as needed.

[1372] A "generative AI model" is an artificial intelligence model that analyzes received text data and generates correct answers and feedback.

[1373] The "means of transmission" is a mechanism for transmitting feedback obtained from the generative AI model to the user's device.

[1374] "Means for recording and tracking progress" refers to technology for storing a user's answer history and feedback content, and for managing the user's learning progress over the long term.

[1375] "Means for providing self-study hints and learning materials" is a mechanism for analyzing the user's level of understanding and providing appropriate learning advice and learning materials.

[1376] The "means of receiving the photograph and converting it into text data using optical character recognition technology" is a technology that converts the photograph data sent by the user into text data using optical character recognition technology.

[1377] The "database" is a system that serves as the foundation for storing and analyzing users' learning progress and level of understanding.

[1378] A "skill gap" refers to the difference that exists between a user's current level of understanding and their desired level of understanding.

[1379] The present invention is a system for receiving information on homework answers provided by users, analyzing and evaluating the information, and providing feedback. This system functions based on the cooperation of users, terminals, and a server.

[1380] First, the user enters the answer to their homework using their device and sends it to the server. The answer may be sent as a photo or directly entered as text. For example, if a user solves an arithmetic problem and writes the answer in a notebook, they can take a photo of the notebook using a device such as a smartphone or tablet and save the image file on the device. They can then upload the image file to the server using a dedicated application.

[1381] The device sends identification information such as the user ID to the server along with the photo data taken by the user and the text data entered by the user. This allows the server to process the data appropriately. The device also receives evaluation results and feedback from the server and displays them to the user. This allows users to instantly check the evaluation results of their answers and improve the quality of their learning.

[1382] The server first receives the homework information sent from the device. If the received information is photo data, the server uses optical character recognition (OCR) technology to convert the characters and mathematical formulas in the photo into text data. Specifically, OCR software such as Google Cloud Vision API is used. This converted text data is then sent to a generative AI model. A generative AI model is an artificial intelligence that analyzes the text data and generates a correct answer and detailed feedback. A typical generative AI model is OpenAI GPT-4.

[1383] The generative AI model analyzes the accuracy of the answer based on the received text data and determines whether it is correct or incorrect. If the answer is incorrect, the generative AI model generates detailed feedback about which part is incorrect and how to derive the correct answer. This generated feedback is sent back to the server, which then sends it to the user's device. The feedback content is saved in a database and maintained as the user's learning history.

[1384] As a specific example of how it works, suppose a user solves the problem "6 x 7 = ?". The user writes the answer "44" in a notebook, takes a photo of it, and uses their device to send the photo to the server. The device sends the photo data and user ID to the server. The server uses OCR technology to convert the photo data into text data "6 x 7 = 44" and sends it to the generative AI model. The generative AI model determines that "44" is an incorrect answer and generates feedback saying "The correct answer is 42. Let's proceed to the next step," which is sent back to the server. The server sends this feedback to the user's device, which then displays the feedback.

[1385] An example of a prompt for a generative AI model is as follows:

[1386] "The user answered the following question: 6 x 7 = 44. Please analyze and evaluate this answer to generate corrective feedback."

[1387] This system allows users to receive instant, detailed feedback, enabling them to study efficiently. Furthermore, by analyzing the learning history stored in the database, it is possible to accurately grasp the user's level of understanding and progress, and provide appropriate learning guidance and teaching materials as needed.

[1388] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1389] Step 1:

[1390] The user enters the homework answer.

[1391] Input: Homework answered by the user (photo data or text data)

[1392] How it works: A user writes their homework answers in a notebook and takes a photo of it with their smartphone or tablet, or types it in as text.

[1393] Output: Homework answer data (photo or text)

[1394] Step 2:

[1395] The user sends the answer data through the terminal.

[1396] Input: User's homework answer data, user ID

[1397] How it works: A user opens the app, uploads a photo or text of their answer, and presses the submit button, automatically attaching their user ID.

[1398] Output: Homework answer data and user ID sent to the device

[1399] Step 3:

[1400] The device sends the answer data and user ID to the server.

[1401] Input: Answer data, user ID

[1402] Operation: After the send button is pressed, the terminal sends the answer data and user ID to the server.

[1403] Output: Answer data and user ID sent to the server

[1404] Step 4:

[1405] The server receives the answer data and the user ID.

[1406] Input: Answer data and user ID sent from the device

[1407] Operation: The server uses the receiving module to receive the answer data and the user ID.

[1408] Output: Answer data and user ID saved on the server

[1409] Step 5:

[1410] The server analyzes the answer data.

[1411] Input: Received answer data

[1412] How it works: The server analyzes the answer data and converts it into text using optical character recognition (OCR) technology, such as Google Cloud Vision API or similar OCR software.

[1413] Output: Parsed text data

[1414] Step 6:

[1415] The server generates prompts that send text data to the generative AI model.

[1416] Input: Parsed text data

[1417] How it works: The server uses the prompt generation module to create a prompt sentence to send to the generative AI model.

[1418] Output: Generated prompt statement

[1419] Step 7:

[1420] The server sends the prompt to the generative AI model.

[1421] Input: Generated prompt text

[1422] How it works: The server sends a prompt to a generative AI model (e.g., OpenAI GPT-4).

[1423] Output: Feedback generated by the generative AI model

[1424] Step 8:

[1425] A generative AI model generates an evaluation and feedback on the answer.

[1426] Input: prompt statement

[1427] How it works: A generative AI model analyzes prompts, determines whether an answer is correct or incorrect, and generates detailed feedback.

[1428] Output: Generated feedback

[1429] Step 9:

[1430] The server receives the feedback and sends it to the user's device.

[1431] Input: Feedback from the generative AI model, user ID

[1432] Operation: The server uses a feedback receiving module to receive feedback from the generative AI model and sends it to the device with the corresponding user ID.

[1433] Output: Feedback sent to the terminal

[1434] Step 10:

[1435] The terminal displays the received feedback to the user.

[1436] Input: Feedback sent by the server

[1437] How it works: The device receives the feedback and displays it to the user via a dedicated application.

[1438] Output: Displayed feedback

[1439] Step 11:

[1440] The server records and stores the user's answer history and feedback.

[1441] Input: Answer history and feedback received

[1442] How it works: The server stores answer history and feedback in a database.

[1443] Output: Saved answer history and feedback

[1444] Step 12:

[1445] The server analyzes the user's learning progress.

[1446] Input: Saved answer history and feedback

[1447] How it works: The server analyzes the database to assess the user's learning progress and skill gaps, and provides appropriate learning tips and materials.

[1448] Output: Study tips and materials based on the analysis results

[1449] (Application example 1)

[1450] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1451] Conventional homework assessment systems have the problem that it is difficult for users to receive immediate feedback on their homework answers, which reduces learning efficiency. They also lack a mechanism for tracking users' progress and level of understanding in real time and providing appropriate learning hints and learning materials. Furthermore, there is no way to effectively utilize learning history or past feedback. To solve these problems, it is necessary to provide users with real-time feedback and improve learning efficiency.

[1452] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1453] In this invention, the server includes: means for receiving information about homework completed by a user; means for analyzing the received homework information and converting the answers into text data; means for sending the text data to a generation AI to determine whether the answers are correct and receive feedback; means for sending the feedback obtained from the generation AI to the user; means for recording the user's answer history and feedback and tracking progress; means for analyzing the user's level of understanding and providing self-study hints and learning materials; means for uploading images of the homework answers taken by the user to the server; means for the server to convert the image data into text data using OCR technology; means for displaying the determination of correct answers and feedback to the user in real time; and means for saving the user's learning history and analyzing past answers and feedback. This allows the system to analyze the information about homework completed by a user in real time and provide immediate feedback. Furthermore, by saving and analyzing the learning history, the system can provide appropriate learning materials according to the user's progress and level of understanding.

[1454] A "user" is an entity that utilizes the system to submit homework answers and receive feedback.

[1455] "Homework information" is data indicating the content of the homework that the user has answered, and is sent from the terminal as photographs or text data.

[1456] The "receiving means" is a mechanism for importing the homework information sent by the user into the server and processing it.

[1457] The "means for analyzing and converting into text data" is a function for analyzing the received homework information and converting it from photo data into text data using optical character recognition technology.

[1458] "Generative AI" is an artificial intelligence model that determines the correctness of answers based on generated text data and generates feedback.

[1459] "Server" means the computer system that receives, processes, stores, and generates feedback on homework information submitted by users.

[1460] "Feedback" refers to the results of the generative AI's determination of the correct answer and information on how the user can arrive at the correct answer.

[1461] The "means for uploading" is a mechanism for sending the homework answer images taken by the user to the server.

[1462] "OCR technology" is an abbreviation for optical character recognition technology, which converts characters and mathematical formulas in image data into text data.

[1463] The "means for displaying to the user in real time" is a mechanism for instantly presenting the feedback sent from the server to the user.

[1464] "Study history" is data that includes questions that the user has answered in the past and feedback on those questions.

[1465] The "analysis means" is a function that uses saved learning history data to evaluate the user's progress and level of understanding.

[1466] "Teaching materials" refer to study materials and questions provided according to the user's level of understanding.

[1467] This invention provides a system that receives information about homework answers provided by users, analyzes, evaluates, and provides feedback. This system is realized through cooperation between users, terminals, and a server.

[1468] User operations

[1469] First, the user submits the homework answer using the device. The answer may be submitted as a photo or as text. For example, the user may take a photo of the notebook in which they have completed an arithmetic problem and save it on the device. The image file is then uploaded to the server.

[1470] Device Features

[1471] The device sends photos taken by the user and text entered by the user to the server. When sending, the device also sends identification information such as the user ID. This information is used by the server to process the data. The device also displays feedback from the server to the user. This allows the user to instantly receive the evaluation results for their answers.

[1472] Server Processing

[1473] The server receives homework information sent from the device. If the received information is photo data, optical character recognition (OCR) technology is used to convert the characters and mathematical formulas in the photo into text data. This conversion is performed using the pytesseract library. The converted text data is then sent to a generative AI model, which determines whether the answer is correct or incorrect. If the answer is incorrect, detailed feedback is generated about which part is incorrect and how to derive the correct answer.

[1474] View Feedback

[1475] The server then sends this feedback back to the terminal, which then displays it to the user, allowing the user to immediately see detailed feedback on their answers.

[1476] Recording and analyzing learning

[1477] The server stores the user's answer history and feedback in a database. This data is used to track the questions the user has answered in the past and the feedback they received. Furthermore, by analyzing this data, the server can understand the user's progress and level of understanding and provide appropriate study tips and materials. This analysis is performed using machine learning algorithms.

[1478] Specific examples

[1479] For example, consider the case where a user solves the problem "6 x 7 = ?" and answers "44."

[1480] 1. The user writes this answer in a notebook and takes a photo of the notebook.

[1481] 2. The device sends the photo it took to the server.

[1482] 3. The server uses OCR technology to convert the photo into text data ("6 x 7 = 44").

[1483] 4. The server sends this text data to the generation AI.

[1484] 5. The generative AI model determines that "44" is an incorrect answer and generates feedback such as "The correct answer is 42. Let's move on to the next step."

[1485] 6. The server receives this feedback and sends it to the user's device.

[1486] 7. The device displays the feedback to the user, who confirms it.

[1487] Prompt Sentence Examples

[1488] Prompt the generative AI model as follows:

[1489] Check the user's answer.

[1490] Question: "6 x 7 = ?"

[1491] Answer: "44"

[1492] Determine the correct answer and generate the necessary feedback.

[1493] Using this prompt, the generative AI model can generate appropriate feedback and provide it to the user.

[1494] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1495] Step 1:

[1496] A user writes answers to homework in a notebook and takes a photo of the notebook using a device such as a smartphone or tablet.

[1497] Input: Homework answers written in a notebook.

[1498] Output: A photo image file containing the homework answers.

[1499] Step 2:

[1500] The photos taken by the device are uploaded to the server, and are sent together with identification information such as the user ID and answer date and time.

[1501] Input: Photo image file taken, user ID, answer date and time.

[1502] Output: Images and related information sent to the server.

[1503] Step 3:

[1504] The server analyzes the received photo data using optical character recognition (OCR) technology and converts characters and mathematical expressions in the image into text data. This conversion is performed using the pytesseract library.

[1505] Input: Uploaded photo image file.

[1506] Output: Text data converted from characters and mathematical expressions in the image.

[1507] Step 4:

[1508] The server sends the converted text data to the generative AI model, which determines whether the answer is correct and generates feedback.

[1509] Input: Text data converted using OCR technology.

[1510] Output: Generative AI model determines whether the answer is correct or incorrect and provides detailed feedback.

[1511] Step 5:

[1512] The server receives feedback from the generative AI model and sends it to the user's device.

[1513] Input: Feedback from the generative AI model.

[1514] Output: Feedback data sent to the device.

[1515] Step 6:

[1516] The terminal displays the received feedback to the user, who then confirms it.

[1517] Input: Feedback data sent by the server.

[1518] Output: User sees feedback.

[1519] Step 7:

[1520] The server stores the user's answer history and feedback in a database and analyzes their learning progress and level of understanding.

[1521] Input: User's answer history, feedback from the server.

[1522] Output: Answer history and feedback stored in a database, and analysis of the user's understanding.

[1523] Step 8:

[1524] The server provides self-study hints and learning materials based on the user's level of learning comprehension, encouraging improvement in learning from the next time onwards.

[1525] Input: Parsed comprehension data.

[1526] Output: Study tips and materials provided to the user.

[1527] Through these steps, users receive real-time feedback and can learn more efficiently.

[1528] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1529] This system receives information about homework completed by a user, uses a generation AI to determine whether the answer is correct and provides detailed feedback, and also combines an emotion engine to collect and analyze the user's emotion data and provide customized support based on that data. This system operates in cooperation with the user, terminal, server, and emotion engine.

[1530] User operations

[1531] First, the user submits their homework answers using their device. Homework answers can be submitted as photos or as text. Furthermore, when submitting the homework information, the user's emotional state at that time is also transmitted. For example, the device's camera can be used to capture the user's facial expression and analyze the emotion.

[1532] Device Features

[1533] The device sends the user's photos, input text, and emotional data to the server. When sending this data, the device also includes identification information such as the user ID. The device also displays feedback from the server and advice based on the user's emotions. In this way, the user can receive not only their answers but also feedback that takes into account their current emotions.

[1534] Server Processing

[1535] The server receives the homework information and emotion data sent from the device. If the homework information received is photo data, it first uses optical character recognition (OCR) technology to convert the letters and mathematical formulas in the photo into text data. The converted text data is then sent to the generation AI, which determines whether it is the correct answer.

[1536] The generation AI analyzes the received text data and determines whether the answer is correct or incorrect. If the answer is incorrect, it generates detailed feedback along with an explanation. Meanwhile, the emotion engine analyzes the received user's emotional data and understands their emotional state. The server reflects the emotional data analyzed by the emotion engine in the feedback received from the generation AI and provides feedback appropriate to the user's emotional state.

[1537] Recording and analyzing learning

[1538] The server stores the user's answer history, feedback, and emotional data in a database. This allows the server to track the questions the user has answered in the past and their emotional state at the time. Furthermore, by analyzing this data, the server can understand the user's level of understanding and learning progress and provide appropriate learning tips and materials.

[1539] Example

[1540] For example, consider the case where a user solves the math problem "6 x 7 = ?" and answers "44."

[1541] 1. The user writes the answer in a notebook, takes a photo of the notebook, and also takes a photo of their own expression using the device's camera.

[1542] 2. The device takes a photo of the homework and sends the user's facial expression data to the server.

[1543] 3. The server uses OCR technology to convert the homework photo into text data ("6 x 7 = 44"), and at the same time uses an emotion engine to analyze the facial expression data and recognize the child's emotional state (for example, "confused").

[1544] 4. The server sends this text data to the generation AI, which determines that "44" is an incorrect answer and generates feedback saying, "The correct answer is 42. Let's move on to the next step."

[1545] 5. Based on the emotional data of "confused," the emotion engine generates additional advice: "Don't rush, try again. If you don't understand, check the hints."

[1546] 6. The server sends the generated feedback and advice to the user.

[1547] 7. The device receives feedback and advice from the server and displays it to the user.

[1548] 8. The user reviews the feedback and advice to understand whether their answer was correct or incorrect and what their next steps are.

[1549] In this way, the system instantly evaluates the user's homework answers and provides detailed feedback and personalized advice based on their emotions, thereby motivating the user to study more efficiently.

[1550] The processing flow will be explained below.

[1551] Step 1:

[1552] The user prepares the homework answer by writing it in a notebook or by typing it directly into the device.

[1553] Step 2:

[1554] The user takes a photo of the note with the device, and also captures their facial expressions using the device's camera to collect their emotional state.

[1555] Step 3:

[1556] The device takes a photo of the homework and sends it to the server along with facial expression data (emotion data), along with identification information such as the user ID.

[1557] Step 4:

[1558] The server receives the homework information and emotion data sent from the device, temporarily stores the data, and then starts processing it.

[1559] Step 5:

[1560] The server analyzes the image data of the homework and uses OCR technology to convert the letters and mathematical formulas in the photo into text data, which is then sent to the next step of processing.

[1561] Step 6:

[1562] The server uses an emotion engine to analyze the user's emotional state from the received facial expression data. For example, it can recognize emotions such as "confusion" or "happiness" from the user's facial expression.

[1563] Step 7:

[1564] The server sends the text data and emotion data to the generation AI, which uses this data to determine whether the answer is correct and generates detailed feedback.

[1565] Step 8:

[1566] The generation AI judges the answer and generates feedback. For example, for the answer "6 x 7 = 44," it generates feedback such as "That's incorrect. The correct answer is 42."

[1567] Step 9:

[1568] The emotion engine generates additional advice that reflects the emotional data based on feedback from the generation AI. For example, in response to the emotional data of "confused," the engine generates advice such as "Try again without rushing."

[1569] Step 10:

[1570] The server compiles the feedback and advice received from the generative AI and emotion engine and prepares it to be sent to the user.

[1571] Step 11:

[1572] The terminal receives feedback and emotion-based advice from the server and displays it to the user, who can then review it to understand whether their answer was correct or incorrect and what to do next.

[1573] Step 12:

[1574] The server stores the user's answer history and emotional data in a database, which stores data that tracks the user's learning progress, comprehension, and emotional state.

[1575] Step 13:

[1576] The server analyzes the user's learning comprehension, progress, and emotional data, and provides individually customized self-study tips and learning materials. For example, it analyzes past emotional data and provides more detailed explanations for particularly difficult problems.

[1577] In this way, our system not only instantly evaluates users' homework and provides detailed feedback and emotion-based advice, but also provides long-term learning support. By adjusting feedback based on users' emotion data, we aim to increase their motivation to learn and provide effective learning support.

[1578] Example 2

[1579] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1580] Conventional educational support systems have the ability to determine whether a user's answers are correct, but they are unable to provide feedback or advice that takes into account the user's emotional state. This makes it difficult to provide individual support to increase the user's motivation to learn, and they are unable to provide efficient learning support. Furthermore, there is a lack of technology that can comprehensively manage and analyze a user's learning progress and emotional data, and provide customized feedback.

[1581] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1582] In this invention, the server includes means for receiving information about homework answers completed by the user and emotional data about the user, means for analyzing the received homework information and converting the answers into text data, means for sending the text data to a generation AI to determine whether the answers are correct and receive feedback, means for incorporating the results of the emotional analysis into the feedback received from the generation AI to generate feedback and advice appropriate to the user's emotional state, means for sending the generated feedback and advice to the user, means for recording the user's answer history, feedback, and emotional data and tracking progress, and means for analyzing the user's level of understanding of the learning and providing hints and learning materials for self-study. This provides customized feedback that takes the user's emotional state into consideration, thereby improving their motivation to learn and providing efficient learning support.

[1583] A "user" is an individual or entity who uses the system to submit homework answers.

[1584] A "homework assignment" is a problem or task assigned to a user for educational or training purposes.

[1585] "Information" refers to homework answers and related data submitted by users.

[1586] "Emotional data" refers to data about a user's emotional state collected through the device's camera or sensors.

[1587] A "terminal" is a device on which a user enters answers to homework, and includes a camera, smartphone, tablet, PC, etc.

[1588] A "server" is a computer system that receives data sent from a terminal and analyzes and processes it.

[1589] Optical character recognition (OCR) is a technology that analyzes characters and mathematical formulas in photo data and converts them into text data.

[1590] "Generative AI" is an artificial intelligence technology that analyzes text data and generates correct answers and feedback.

[1591] "Feedback" is information that includes an evaluation of the user's homework answers as well as next steps and advice.

[1592] The "emotion engine" is an artificial intelligence technology that analyzes the user's emotional data and reflects the results in feedback.

[1593] A "database" is a system for storing and managing a user's answer history, feedback, and emotional data.

[1594] "Level of understanding of learning" is an index that evaluates how much a user has understood through homework.

[1595] "Self-study tips and materials" are additional learning resources and advice provided to help users further their learning.

[1596] "Customized advice" is specialized feedback or advice provided based on a user's individual answer history and emotional state.

[1597] The present invention is a system that receives homework answers and emotion data submitted by a user, analyzes them, and provides feedback and customized advice. This system operates in cooperation with a user, a terminal, a server, and an emotion engine.

[1598] System Overview

[1599] The system includes the following hardware and software components:

[1600] Devices operated by users (e.g. smartphones, tablets, PCs)

[1601] Server that processes and stores data

[1602] Optical character recognition technology (OCR)

[1603] Generative AI Models

[1604] Emotion Engine

[1605] Database

[1606] Device behavior

[1607] The user submits their homework answers using a device. Answers can be entered as text or photos. The device's camera is used to collect the user's facial expression data, which is then transmitted as emotion data. The device then transmits the user's answer and emotion data to the server as a single data packet.

[1608] Server Processing

[1609] The server receives data sent from the user's device. First, it uses the emotion engine to analyze the emotion data and identify the user's emotional state. If the homework information is in the form of photographs, it uses optical character recognition (OCR) technology to convert the letters and mathematical formulas in the photographs into text data. This conversion process is performed based on an image analysis algorithm.

[1610] Answer validation and feedback generation

[1611] The server sends the converted text data to a generative AI model, which determines whether the answer is correct and generates detailed feedback. The generative AI determines whether the homework answer is correct and provides feedback along with a detailed explanation if the answer is incorrect. Furthermore, by incorporating the analysis results of the emotion engine into the feedback obtained from the generative AI, feedback and advice are generated that reflects the user's emotional state.

[1612] Integrated feedback and advice

[1613] The server integrates the feedback obtained from the generative AI with advice based on emotion data and sends it to the device. The device then displays the feedback and advice received from the server to the user, allowing the user to receive support based on the correctness of their answer and their emotion.

[1614] Data storage and analysis

[1615] The server stores all user data, answer history, feedback, and emotional data in a database. This allows the server to track users' past answer data and emotional state, analyze their learning progress and comprehension, and periodically analyze each user's skill gaps to provide customized advice.

[1616] Specific examples

[1617] For example, if a user solves the math problem "6 x 7 = ?" and answers "44," the user writes the answer in a notebook, takes a photo of the notebook, and also captures their facial expression. The device then sends the photo of the homework and the user's facial expression data to the server. The server then converts the photo of the homework into text data using OCR technology and sends it to the generation AI. The generation AI determines that "44" is an incorrect answer and generates feedback such as "The correct answer is 42. Let's move on to the next step." Based on the emotional data of "confused," the emotion engine generates advice such as "Don't rush, try again. If you don't understand, check the hint." The server then combines the generated feedback and advice and sends it to the device. The device then displays this information to the user.

[1618] Prompt Sentence Examples

[1619] An example of a prompt sentence to be input to the generative AI model is shown below.

[1620] The answer to the homework problem the user solved was "6 x 7 = 44." Check the accuracy of this answer and provide feedback on the correct answer and why, as well as next steps for the user if they got it wrong. Additionally, your data indicates the user's emotional state is "confused." Include advice based on their emotional state.

[1621] As described above, the present invention accurately evaluates a user's homework answers and provides customized feedback based on emotional data, thereby increasing the user's motivation to study and providing efficient learning support.

[1622] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1623] Step 1:

[1624] The user enters their homework answers into the device or takes a photo of their notes. They also capture their facial expressions with the device's camera to obtain emotional data.

[1625] Input: Homework answers (text or photos), facial expression data

[1626] Output: Homework answer data, emotion data

[1627] Specific actions: The user writes a homework answer by hand or digitally and takes a photo of it. The device's camera captures their facial expression and prepares it for transmission to the system.

[1628] Step 2:

[1629] The device sends the homework answer data (text or photo) and emotion data, along with identification information such as the user ID, to the server.

[1630] Input: Homework answer data, emotion data, user ID

[1631] Output: Data packet (homework answer data, emotion data, user ID)

[1632] Specific operation: The device packs the user ID, homework answer data, and emotion data into a single packet and sends it to the server using the HTTPS protocol.

[1633] Step 3:

[1634] The server receives the data sent from the device and first sends it to the emotion engine for emotional analysis. If the homework answers are photo data, they are converted into text data using OCR.

[1635] Input: Data packet (homework answer data, emotion data, user ID)

[1636] Output: Analyzed emotion data, text data (if necessary)

[1637] Specific operation: The server receives the data packet and first sends it to the emotion engine for analysis of the user's emotion data. If the homework answer is a photo, OCR technology is used to convert the characters and formulas in the photo into text data.

[1638] Step 4:

[1639] The server sends the converted text data to the generative AI model, which determines the correct answer and provides detailed feedback.

[1640] Input: Text data

[1641] Output: Feedback data

[1642] How it works: The server sends text data as prompts to the generative AI model, which then determines whether the answer is correct or incorrect and generates feedback including detailed explanations if the answer is incorrect.

[1643] Step 5:

[1644] An emotion engine analyzes the emotional data and generates customized advice.

[1645] Input: Emotion data

[1646] Output: Advice data

[1647] Specific operation: The emotion engine analyzes the user's emotional state (e.g., "confused") and generates additional advice based on the results.

[1648] Step 6:

[1649] The server integrates feedback from the generative AI and advice from the emotion engine and sends it to the device.

[1650] Input: Feedback data, advice data

[1651] Output: Consolidated message

[1652] Specific operation: The server combines the feedback obtained from the generation AI and the advice from the emotion engine into a single integrated message and sends it to the device via HTTPS protocol.

[1653] Step 7:

[1654] The terminal receives the integrated message from the server and displays it to the user.

[1655] Input: Unified message

[1656] Output: Display data

[1657] Specific operation: The device analyzes the received integrated message and displays it to the user in an appropriate format. The user checks the feedback and advice and plans their next action.

[1658] Step 8:

[1659] The server stores the user's answer history, feedback, and emotion data in a database, thereby tracking the user's understanding and learning progress.

[1660] Input: Answer history, feedback, emotional data

[1661] Output: Saved data

[1662] Specific operation: The server stores the user's answer history, feedback, and emotional data in a database, and continuously manages and analyzes them to understand each user's learning progress and level of understanding.

[1663] (Application example 2)

[1664] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1665] Conventional homework feedback systems have struggled to provide users with prompt and detailed feedback on their completed homework. Furthermore, the feedback was not customized to take into account the user's feelings, resulting in problems such as users not receiving appropriate support and reduced learning efficiency. Furthermore, it was difficult to track users' learning history and progress and provide appropriate advice and learning materials.

[1666] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1667] In this invention, the server includes means for receiving information on homework answers answered by the user, means for analyzing the received homework information and converting the answers into text data, means for sending the text data to the generation AI to determine whether the answer was correct and to receive feedback, means for analyzing the user's facial expressions to collect emotional data, means for analyzing the emotional data and providing customized advice based on the feedback from the generation AI, means for sending the user advice based on the feedback and emotional data obtained from the generation AI, means for recording the user's answer history and feedback and tracking progress, and means for analyzing the user's level of understanding of the study and providing self-study hints and learning materials. This allows the user to receive prompt and detailed feedback and advice that takes their emotions into consideration, enabling efficient study.

[1668] A "user" is someone who uses the system to submit homework and receive feedback.

[1669] "Homework information" is data including the content of the homework answered by the user, and is sent to the server in the form of a photograph or text.

[1670] "Text data" refers to data in which homework information has been converted into a string of characters using OCR technology or similar.

[1671] "Generative AI" is an artificial intelligence that analyzes received text data, determines whether the answer is correct, and generates feedback.

[1672] "Emotion data" is data that indicates the emotional state of the user analyzed from facial expressions, etc.

[1673] "Customized advice" is individual advice provided based on feedback from the generative AI and taking into account the user's emotional data.

[1674] The "means for analyzing facial expressions" is a mechanism for detecting facial expressions from images or videos of the user's face and analyzing emotions.

[1675] "Answer history" is a record of the homework answers a user has given and the feedback they have received.

[1676] "Means for tracking progress" refers to a system for tracking and managing learning progress based on the user's answer history and feedback.

[1677] The "means for analyzing the level of understanding" is a mechanism for evaluating the user's past answers and feedback to understand the user's level of understanding and weaknesses.

[1678] "Self-study tips and materials" are study support materials and advice provided according to the user's level of understanding and progress.

[1679] A "database" is a recording device for saving and storing information such as a user's answer history, feedback, and emotional data.

[1680] Optical character recognition (OCR) is a technology that analyzes characters in image data and converts them into text data.

[1681] This system receives information about homework completed by a user and uses a generation AI to determine whether the answer is correct and provide detailed feedback. Furthermore, it uses an emotion engine to collect and analyze the user's emotion data and provide customized support based on that data. This system operates in cooperation with the user, device, server, and emotion engine.

[1682] User operations

[1683] Users submit their homework answers using their devices. Homework answers may be submitted as photos or as text. When submitting homework information, the user's emotional state at that time is also transmitted. For example, the device's camera may be used to capture the user's facial expression and analyze the emotion.

[1684] Device Features

[1685] The device sends the user's photos, input text, and emotional data to the server. When sending this data, the device also includes identification information such as the user ID. The device also displays feedback from the server and advice based on the user's emotions. In this way, the user can receive not only their answers but also feedback that takes into account their current emotions.

[1686] Server Processing

[1687] The server receives the homework information and emotion data sent from the device. If the received homework information is photo data, it first uses optical character recognition (OCR) technology to convert the characters and mathematical formulas in the photo into text data. The OCR library used is "Tesseract OCR." The converted text data is sent to the generation AI, which determines whether it is correct. "GPT-4" is used as the generation AI model. The generation AI analyzes the received text data and determines whether it is correct or incorrect. If the answer is incorrect, it generates detailed feedback along with an explanation.

[1688] Emotional Data Analysis

[1689] The user's emotional data is obtained by analyzing facial expressions. An emotion recognition engine such as Microsoft Azure Face API is used to analyze the facial expression data and recognize the user's emotional state. The emotion engine analyzes the received user's emotional data and understands that emotional state. The server reflects the emotional data analyzed by the emotion engine in the feedback obtained from the generation AI and provides feedback appropriate to the user's emotional state.

[1690] Recording and analyzing learning

[1691] The server stores the user's answer history, feedback, and emotional data in a database. A typical relational database management system (RDBMS) is used as the database. This allows us to track the questions the user has answered in the past and their emotional state at the time. Furthermore, by analyzing this data, we can understand the user's level of understanding and learning progress and provide appropriate learning tips and materials.

[1692] Specific examples

[1693] For example, if a user answers "5" to the math problem "2+2=?", the following scenario is possible: The user takes a photo of their notebook with their camera and enters "2+2=5." At this time, the user's facial expression data is also sent. The server uses OCR technology to convert "2+2=5" into text data, and the generative AI (GPT-4) determines that "5" is an incorrect answer and generates detailed feedback such as "The correct answer is 4." At the same time, the emotion engine analyzes the user's facial expression data as "anxiety." Based on this information, the server adds customized advice such as "Calm down and think about it. Let's try again," and sends the final feedback to the user's device.

[1694] Prompt Sentence Examples

[1695] "The user answers '2 + 2 = 5'. Please rate this answer, tell us the correct answer, provide detailed feedback, and offer additional advice if the user is in a state of uncertainty."

[1696] In this way, the system of the present invention instantly evaluates the homework answers of the user and provides detailed feedback and customized advice based on emotions, thereby helping the user to study more efficiently.

[1697] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1698] Step 1:

[1699] User submits homework

[1700] The user submits their homework answers using a device. At this time, the homework information is photographed and facial expression data is captured at the same time. When the user takes a photo of their homework and presses the send button, the homework information and facial expression data are entered into the device. This generates the homework photo data (JPEG, PNG, etc.) and facial expression data.

[1701] Step 2:

[1702] The device sends the data to the server

[1703] The device sends the homework photo and facial expression data taken by the user to the server using protocols such as HTTP POST requests. The input is the photo data and facial expression data received from the parent, and the output is the data packet sent to the server.

[1704] Step 3:

[1705] The server analyzes the homework information using OCR technology

[1706] The server performs OCR processing on the received homework photo data and converts the characters in the image into text data. This processing uses the Tesseract OCR library. The input is the photo data (JPEG, PNG, etc.) received by the server, and the output is the text data (String) generated by OCR.

[1707] Step 4:

[1708] The server sends the text data to the generation AI and receives feedback and a correct answer.

[1709] The server sends the text data obtained by OCR to the generation AI (GPT-4), determines whether the answer is correct, and generates feedback. It constructs a prompt sentence and sends the text data as input to the generation AI. The input is the text data and the prompt sentence, and the output is the generated feedback sentence.

[1710] Example prompt: "The user answered '2 + 2 = 5'. Please rate this answer, tell us the correct answer, and provide detailed feedback."

[1711] Step 5:

[1712] The server analyzes the facial expression data and obtains emotion data.

[1713] The server sends the facial expression data to an emotion recognition engine (Microsoft Azure Face API) to analyze the emotion data. The input is the facial expression data (image), and the output is the analyzed emotional state (e.g., "anxiety").

[1714] Step 6:

[1715] The server generates feedback as customized advice.

[1716] The server generates customized advice based on the feedback obtained from the generation AI, taking into account the emotional data. It uses the emotional data as input and adds additional advice to the feedback of the generation AI. The input is the feedback from the generation AI and the emotional data, and the output is a customized feedback sentence.

[1717] Step 7:

[1718] The server sends the generated feedback to the user's device.

[1719] The server sends the generated customized feedback to the user's terminal, where the input is the customized feedback sentence and the output is the feedback displayed on the user's terminal.

[1720] Step 8:

[1721] The device displays feedback

[1722] The terminal displays the feedback received from the server to the user. The input is the feedback text received from the server, and the output is the feedback displayed on the terminal's display.

[1723] Step 9:

[1724] The server records answer history and feedback, and tracks progress.

[1725] The server records the user's answer history and feedback in a database and tracks the progress. The input is the user's answer history and feedback data, and the output is the information recorded in the database.

[1726] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1727] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1728] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1729] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1730] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1731] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1732] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1733] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1734] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1735] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1736] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1737] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1738] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1739] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1740] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1741] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1742] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1743] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1744] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1745] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1746] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1747] The following is further disclosed regarding the above embodiment.

[1748] (Claim 1)

[1749] means for receiving information on homework completed by the user;

[1750] A means for analyzing the received homework information and converting the answers into text data;

[1751] A means for sending text data to the generation AI to obtain a correct answer and feedback;

[1752] A means for transmitting feedback from the generative AI to the user; and

[1753] a means for recording a user's answer history and feedback and tracking progress;

[1754] A means for analyzing the user's learning comprehension and providing self-study tips and learning materials;

[1755] A system including:

[1756] (Claim 2)

[1757] 10. The system of claim 1, further comprising means for receiving the user's completed homework information as a photograph and converting it into text using optical character recognition technology.

[1758] (Claim 3)

[1759] 2. The system according to claim 1, further comprising means for storing the user's learning progress and understanding in a database and periodically analyzing the skill gap for each user.

[1760] "Example 1"

[1761] (Claim 1)

[1762] means for receiving information on homework completed by the user;

[1763] A means for analyzing the received homework information and converting the answers into text data;

[1764] A means for sending text data to a generative AI model to obtain a correct answer and feedback;

[1765] a means for transmitting feedback from the generative AI model to the user; and

[1766] a means for recording a user's answer history and feedback and tracking progress;

[1767] A means for analyzing the user's learning comprehension and providing self-study tips and learning materials;

[1768] A system including:

[1769] (Claim 2)

[1770] 10. The system of claim 1, further comprising means for receiving information of the homework answered by the user as a photograph and converting the photograph into text data using optical character recognition technology.

[1771] (Claim 3)

[1772] 2. The system according to claim 1, further comprising means for storing the user's learning progress and understanding in a database and periodically analyzing the skill gap for each user.

[1773] "Application Example 1"

[1774] (Claim 1)

[1775] means for receiving information on homework completed by the user;

[1776] A means for analyzing the received homework information and converting the answers into text data;

[1777] A means for sending text data to the generation AI to obtain a correct answer and feedback;

[1778] A means for transmitting feedback from the generative AI to the user; and

[1779] a means for recording a user's answer history and feedback and tracking progress;

[1780] A means for analyzing the user's learning comprehension and providing self-study tips and learning materials;

[1781] A means for uploading an image of a homework answer taken by a user to a server;

[1782] A means for the server to convert image data into text data using OCR technology;

[1783] a means for displaying correct answer determination and feedback to the user in real time;

[1784] a means for storing a user's learning history and analyzing past answers and feedback;

[1785] A system including:

[1786] (Claim 2)

[1787] 10. The system of claim 1, further comprising means for receiving the user's completed homework information as a photograph and converting it into text using optical character recognition technology.

[1788] (Claim 3)

[1789] 2. The system according to claim 1, further comprising means for storing the user's learning progress and understanding in a database and periodically analyzing the skill gap for each user.

[1790] "Example 2: Combining Emotion Engines"

[1791] (Claim 1)

[1792] A means for receiving information about homework answered by the user and emotional data of the user;

[1793] A means for analyzing the received homework information and converting the answers into text data;

[1794] A means for sending text data to the generation AI to obtain a correct answer and feedback;

[1795] A means for reflecting the emotion analysis results in the feedback obtained from the generation AI and generating feedback and advice appropriate to the user's emotional state;

[1796] means for transmitting the generated feedback and advice to the user;

[1797] a means for recording a user's answer history, feedback, and sentiment data and for tracking progress;

[1798] A means for analyzing the user's learning comprehension and providing self-study tips and learning materials;

[1799] A system including:

[1800] (Claim 2)

[1801] 10. The system of claim 1, further comprising means for receiving the user's completed homework information as a photograph and converting it into text using optical character recognition technology.

[1802] (Claim 3)

[1803] 2. The system of claim 1, further comprising means for storing users' learning progress and understanding in a database, periodically analyzing skill gaps for each user, and providing customized advice taking into account emotional data.

[1804] "Application example 2 when combining emotion engines"

[1805] (Claim 1)

[1806] means for receiving information on homework completed by the user;

[1807] A means for analyzing the received homework information and converting the answers into text data;

[1808] A means for sending text data to the generation AI to obtain a correct answer and feedback;

[1809] means for analyzing a user's facial expression to collect emotion data;

[1810] A means to analyze emotional data and provide customized advice based on generative AI feedback;

[1811] a means for transmitting advice to the user based on the feedback and emotional data obtained from the generative AI;

[1812] a means for recording a user's answer history and feedback and tracking progress;

[1813] A means for analyzing the user's learning comprehension and providing self-study tips and learning materials;

[1814] A system including:

[1815] (Claim 2)

[1816] 10. The system of claim 1, further comprising means for receiving the user's completed homework information as a photograph and converting it into text using optical character recognition technology.

[1817] (Claim 3)

[1818] 2. The system according to claim 1, further comprising means for storing the user's learning progress and understanding in a database and periodically analyzing the skill gap for each user. [Explanation of symbols]

[1819] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving information on homework completed by the user; A means for analyzing the received homework information and converting the answers into text data; A means for sending text data to the generation AI to obtain a correct answer and feedback; A means for transmitting feedback from the generative AI to the user; and a means for recording a user's answer history and feedback and tracking progress; A means for analyzing the user's learning comprehension and providing self-study tips and learning materials; A system including:

2. 10. The system of claim 1, further comprising means for receiving the user's completed homework information as a photograph and converting it into text using optical character recognition technology.

3. 2. The system according to claim 1, further comprising means for storing the learning progress and understanding of the users in a database and periodically analyzing the skill gap for each user.

Images