System

Abstract

A system is provided.SOLUTION: The system includes a terminal for a learner to photograph an image of a reference book or a textbook, a server for receiving image data photographed by the terminal, an optical character recognition means for extracting a text from the image data in the server, a natural language processing means for analyzing the extracted text, and a means for, based on the analysis result, A system comprising: means for automatically generating questions; a database for storing the generated questions; means for calculating a review timing based on an Ebbinghaus forgetting curve; means for sending a notification to a learner at an optimal review timing; means for receiving a learner's answer; and means for evaluating the answer and generating feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional learning methods have the problem that it is difficult for learners to review at the appropriate time to efficiently solidify their memories. Furthermore, it takes time and effort for learners to create their own questions using their own reference books and textbooks, making it difficult to maintain motivation. This invention aims to enable learners to efficiently solidify their memories by automatically generating questions from images in reference books and textbooks and prompting them to review at times based on Ebbinghaus's forgetting curve. [Means for solving the problem]

[0005] The present invention provides a system that includes a terminal for a learner to take pictures of reference books and textbooks, a server that receives the captured image data, optical character recognition means for extracting text from the image data, and natural language processing means for automatically generating questions based on the analyzed text. It also includes a database that stores the generated questions, means for calculating the timing of review based on Ebbinghaus's forgetting curve, means for sending a notification to the learner when the optimal review timing is reached, and means for accepting and evaluating the learner's answers and generating feedback. This allows the learner to efficiently review and solidify their memory.

[0006] A "terminal" is an electronic device that allows a learner to take pictures of reference books or textbooks and send the image data to a server.

[0007] The "server" is a computer system that receives image data sent from the terminal, processes the images, generates questions, and manages review schedules.

[0008] "Optical character recognition means" is a technology for extracting text from captured image data, and is generally called OCR (Optical Character Recognition).

[0009] "Natural language processing means" is a technology for analyzing extracted text and automatically generating questions from the text.

[0010] A "database" is a storage device for storing information such as generated questions, review schedules, and learners' answer histories.

[0011] The "forgetting curve" is a curve that shows the rate at which memory decays over time, and was proposed by Ebbinghaus.

[0012] The "means for calculating the timing of review" refers to an algorithm or software that calculates the optimal timing for a learner to review based on Ebbinghaus's forgetting curve.

[0013] The "means for sending a notification" is a mechanism for notifying the learner of the need to review based on the calculated review timing.

[0014] "Means for accepting answers" refers to the interface or process for receiving answers entered by learners and processing them as data.

[0015] The "means for generating feedback" is a function for evaluating the learner's answers and providing feedback indicating whether they are correct or incorrect and areas for improvement. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system for helping learners efficiently solidify their memories. In particular, it provides a function that takes pictures of reference books and textbooks and automatically notifies learners when it is time to review them based on the Ebbinghaus forgetting curve.

[0038] System Overview

[0039] Image input of learning materials and question generation

[0040] User

[0041] Learners use the camera on their smartphone or PC to take pictures of pages from reference books or textbooks.

[0042] Terminal

[0043] The captured image data is sent to the server via an application on the terminal.

[0044] server

[0045] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image.

[0046] The extracted text is interpreted and questions and candidate answers are generated using natural language processing (NLP) techniques.

[0047] The generated questions are stored in a database and saved for later access by the user.

[0048] Setting a study schedule

[0049] User

[0050] Learners log in to the application and set their own learning goals and deadlines.

[0051] For example, if you set an exam to take place at the end of the month, that date information will be entered.

[0052] server

[0053] The server applies Ebbinghaus' forgetting curve model based on the input learning goals and deadlines to calculate the optimal timing for review.

[0054] The calculated schedule is stored in a database and saved as user profile information.

[0055] Notification of review timing

[0056] server

[0057] Regularly check learners' schedules based on set review timings.

[0058] When review is required, the server generates a notification and sends it to the learner's device.

[0059] Terminal

[0060] The terminal displays the received notification message to the user.

[0061] For example, you might be notified, "Here's today's review question: What is the speed of light in m / s?"

[0062] Review and feedback

[0063] User

[0064] The learner checks the notification, opens the terminal application, and works on the review questions.

[0065] A question is displayed and the learner enters and submits the answer.

[0066] Terminal

[0067] The user's answer data is sent to the server.

[0068] server

[0069] The server analyzes the received answers and determines whether they are correct or incorrect.

[0070] Generate a feedback message, such as "Correct! We'll review this in a week."

[0071] The user's learning history information is updated and the next review timing is recalculated as necessary.

[0072] Specific examples

[0073] 1. A user takes a photo of a page about the "speed of light" in a physics textbook.

[0074] 2. The device sends the image data to the server.

[0075] 3. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?" and saves it in the database.

[0076] 4. The user sets the exam date one month in the future.

[0077] 5. The server sets the review timing based on Ebbinghaus's forgetting curve.

[0078] 6. When the first review time comes (for example, one day later), the server generates a notification and sends it to the device.

[0079] 7. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?"

[0080] 8. The user enters the answer and submits it.

[0081] 9. The server analyzes the answer and generates feedback to the user saying "That's right!"

[0082] 10. The server recalculates the next review timing and updates the learning history.

[0083] In this way, learners can efficiently review and solidify their memories.

[0084] The processing flow will be explained below.

[0085] Step 1:

[0086] The user launches the application and takes a photo of a page from a reference book or textbook.

[0087] Step 2:

[0088] The terminal transmits the captured image data to the server.

[0089] Step 3:

[0090] The server receives the transmitted image data and uses optical character recognition (OCR) technology to extract text from the image.

[0091] Step 4:

[0092] The server analyzes the extracted text using natural language processing (NLP) technology and generates questions from the text based on the learning content.

[0093] Step 5:

[0094] The server stores the generated questions and answer candidates in a database.

[0095] Step 6:

[0096] Users log in to the application and set their learning goals and deadlines.

[0097] Step 7:

[0098] The terminal transmits the user's setting information to the server.

[0099] Step 8:

[0100] The server applies Ebbinghaus's forgetting curve model to calculate the optimal timing for review based on learning goals and deadlines.

[0101] Step 9:

[0102] The server stores the calculated review timing in a database.

[0103] Step 10:

[0104] The server periodically checks the learner's schedule based on the review timing.

[0105] Step 11:

[0106] The server generates a notification when review is necessary and sends it to the learner's device.

[0107] Step 12:

[0108] The device displays a notification to the learner.

[0109] Step 13:

[0110] The user sees the notification, opens the application and begins working on the review questions.

[0111] Step 14:

[0112] The user enters answers to the review questions and presses the submit button.

[0113] Step 15:

[0114] The terminal transmits the user's answer data to the server.

[0115] Step 16:

[0116] The server receives the answer data, scores it, and determines whether it is correct or incorrect.

[0117] Step 17:

[0118] The server generates and sends a feedback message to the user.

[0119] Step 18:

[0120] The server updates the user's learning history information and recalculates the next review timing as needed.

[0121] Step 19:

[0122] The terminal displays a feedback message to the user.

[0123] This series of steps allows learners to review the material efficiently and solidify their memory.

[0124] Example 1

[0125] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0126] Conventional learning systems lacked efficient methods for learners to solidify their knowledge. Furthermore, they were unable to properly schedule review sessions, preventing the learner from maximizing their learning effectiveness. Furthermore, manually generating and managing questions was time-consuming and burdensome for learners. The purpose of this invention is to provide a system that solves these problems and enables learners to acquire knowledge efficiently.

[0127] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0128] In this invention, the server includes optical character recognition means, natural language processing means, means for automatically generating practice questions, data storage means, means for calculating review timing, means for sending notifications at optimal review timings, means for accepting answers from learners, means for evaluating answers and generating feedback, means for periodically checking the review schedule based on the calculated review timings, means for generating questions and answer candidates using a generative AI model with natural language processing technology, and means for displaying notification messages on the learner's computer device, thereby enabling learners to efficiently review and effectively solidify their memories.

[0129] A "learner" is an individual or group whose purpose is to acquire the learning content.

[0130] "Reference books and textbooks" are printed or electronic media used by learners for the purpose of acquiring knowledge.

[0131] A "computer device" is an electronic device for processing, storing, and communicating data, and includes personal computers, smartphones, and the like.

[0132] An "information processing device" is a server or equivalent system for receiving and processing data over a network.

[0133] "Optical character recognition" is technology or software that extracts text from image data.

[0134] "Natural language processing" is technology or software for analyzing and understanding extracted text.

[0135] "Means for automatically generating practice questions" refers to technology or software that automatically generates questions for learners based on the results of natural language processing analysis.

[0136] The "data storage means" is a recording medium or database for storing generated questions and related data.

[0137] The "means for calculating the timing of review" is a technology or software that calculates the optimal time for review based on Ebbinghaus's forgetting curve.

[0138] The "means for sending notifications" refers to technology or software that sends notifications to learners based on the calculated review timing.

[0139] The "means for accepting learner's answers" is a system or software that receives and processes answer data from learners.

[0140] The "means for evaluating answers and generating feedback" refers to technology or software that analyzes the answers received from learners, evaluates their correctness, and generates appropriate feedback.

[0141] The "means for checking the review schedule" refers to technology or software that periodically checks the learner's schedule based on the calculated review timing.

[0142] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate questions and potential answers.

[0143] A "means for displaying a notification message" is a technique or software that generates and transmits a notification message to be displayed on a learner's computing device.

[0144] This invention relates to a system that helps learners efficiently solidify their memories. In particular, it has the function of taking pictures of reference books and textbooks and automatically notifying learners when it is time to review them based on the Ebbinghaus forgetting curve.

[0145] Image input of learning materials and question generation

[0146] User: The learner takes a photo of a page from a reference book or textbook using the camera on their smartphone or personal computer. This image data is stored on a computer device.

[0147] Terminal: The terminal transmits the captured image data to an information processing device (server) through an application in the computer device. For example, the image data is uploaded to the server using an application on a smartphone.

[0148] Server: The server uses optical character recognition (OCR) technology on the received image data to extract text. OCR technology such as Google Cloud Vision API or Tesseract is used. The extracted text is then analyzed using natural language processing (NLP) technology. A generative AI model (e.g., OpenAI's GPT-3) is used for this analysis, and the question and candidate answers are generated. The generated questions are saved in a data storage device.

[0149] Setting a study schedule

[0150] User: Learners log in to the application and set deadlines such as learning goals and exam dates. For example, learners can "set exam dates one month from now" within the application.

[0151] Server: The server applies Ebbinghaus's forgetting curve model based on the input learning goals and deadlines to calculate the optimal review timing. The calculated schedule is saved in a data storage device and managed as the user's profile information.

[0152] Notification of review timing

[0153] Server: The server periodically checks the learner's schedule based on the set review timing. When a review is required, it generates a notification and sends it to the user's computer device.

[0154] Terminal: The terminal displays the received notification message to the user. For example, using the smartphone's notification function, a message such as "Today's review question: What is the speed of light in m / s?" is displayed.

[0155] Review and feedback

[0156] User: The learner checks the notification, opens the application on their device, and begins working on the review questions. The questions are displayed, and they can enter and submit their answers immediately.

[0157] Terminal: The user's answer data is sent to the server.

[0158] Server: The server analyzes the received answers and determines whether they are correct or incorrect. Based on the analysis results, it generates a feedback message and sends it to the user. It also updates the user's learning history information and recalculates the next review timing if necessary.

[0159] Specific examples

[0160] 1. A user takes a photo of a page about the "speed of light" in a physics textbook.

[0161] 2. The device sends the image data to the server.

[0162] 3. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?" and saves it in the database.

[0163] 4. The user sets the exam date one month in the future.

[0164] 5. The server sets the review timing based on Ebbinghaus's forgetting curve.

[0165] 6. When the first review time comes (for example, one day later), the server generates a notification and sends it to the device.

[0166] 7. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?"

[0167] 8. The user enters the answer and submits it.

[0168] 9. The server analyzes the answer and generates feedback to the user saying "That's right!"

[0169] 10. The server recalculates the next review timing and updates the learning history.

[0170] Prompt Sentence Examples

[0171] 1. Extract text from images

[0172] Extract the text from this image.

[0173] 2. Problem statement generation

[0174] "Generate an appropriate question from this text. Create a question from the text 'The speed of light is 299,792,458 m / s.'"

[0175] 3. Calculating review timing

[0176] "Calculate the next time to review based on Ebbinghaus's forgetting curve. The first time to review is today."

[0177] This invention enables learners to efficiently review and effectively solidify their memories.

[0178] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0179] Step 1:

[0180] The user takes a photo of a page from a reference book or textbook using the camera on their smartphone or personal computer. The input is image data of the learning material, and the output is image data saved on the device.

[0181] Specific action: A student takes a photo of a page about the speed of light in a physics textbook using their smartphone.

[0182] Step 2:

[0183] The terminal sends the captured image data to the server. The input is the image data stored in the terminal, and the output is the image data sent to the server.

[0184] Specific operation: The captured image is uploaded to the server through the application, and the message "Sending to server" is displayed.

[0185] Step 3:

[0186] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image. The input is the image data sent to the server, and the output is the extracted text data. The OCR technology used is Google Cloud Vision API and Tesseract.

[0187] What it does: The text "The speed of light is 299,792,458 m / s" is extracted from the image.

[0188] Step 4:

[0189] The server uses natural language processing (NLP) techniques to analyze the extracted text. The input is the text data from the OCR, and the output is the analysis results. A generative AI model (e.g., OpenAI's GPT-3) is used for this analysis to generate the question and candidate answers.

[0190] Specific operation: From the text "The speed of light is 299,792,458 m / s," the question "What is the speed of light in m / s?" is generated.

[0191] Step 5:

[0192] The server stores the generated questions in a data storage means, and the input is the generated question data, and the output is the question data stored in the database.

[0193] Specific operation: The generated question "What is the speed of light in m / s?" is saved in the database.

[0194] Step 6:

[0195] A user logs in to the application and sets learning goals and deadlines such as exam dates. The input is the goal and deadline data set by the learner, and the output is the setting data sent to the server.

[0196] Specific behavior: The learner "sets the exam date one month in the future" within the application.

[0197] Step 7:

[0198] The server applies the Ebbinghaus forgetting curve model based on the input learning goals and deadlines to calculate the optimal review timing. The input is the learning goals and deadline data, and the output is the calculated review schedule.

[0199] Specific operation: The server calculates the review timing, such as "the next review is one day later, then three days later, then one week later," and records it in the profile.

[0200] Step 8:

[0201] The server periodically checks the learner's schedule based on the calculated review timing. The input is review schedule data, and the output is notification data.

[0202] What happens: The server checks the schedule and generates a notification such as "Today's review questions are here."

[0203] Step 9:

[0204] The terminal displays the received notification message to the user. The input is the notification data sent from the server, and the output is the notification message displayed to the user.

[0205] Specific behavior: A notification will appear on your device saying, "Here's today's review question: What is the speed of light in m / s?"

[0206] Step 10:

[0207] The user checks the notification, opens the application on the device, and works on the review questions. The input is the displayed question text, and the output is the answer data entered by the user.

[0208] Specific operation: The user enters "299,792,458" in response to the question "What is the speed of light in m / s?" and submits it.

[0209] Step 11:

[0210] The terminal transmits the user's answer data to the server. The input is the answer data entered by the user, and the output is the answer data transmitted to the server.

[0211] Specific behavior: The device will display "Sending answers."

[0212] Step 12:

[0213] The server analyzes the received answers and determines whether they are correct or incorrect. The input is the user's answer data, and the output is the evaluation result and a feedback message. The feedback message is generated based on the analysis result.

[0214] Specific operation: The server determines that the user's answer "299,792,458" is correct, generates feedback to the user saying "That's correct! We'll review it in a week," and sends it to the user.

[0215] Step 13:

[0216] The server updates the user's learning history information and recalculates the next review timing as necessary. The input is the updated learning history data, and the output is the recalculated review schedule.

[0217] Specific operation: The server recalculates the next review timing based on the learning history and records it in the profile.

[0218] (Application example 1)

[0219] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0220] Conventional learning support systems lack the ability to notify learners of effective review timing and automatic problem generation functions to help them efficiently solidify their memories. Furthermore, more efficient educational methods are required for supporting learning for workers in practical environments such as factories, but no systems currently exist to address this. Therefore, there is a need for a system that automatically generates questions from images in reference books and textbooks and notifies learners of review timing based on Ebbinghaus's forgetting curve, thereby promoting efficient learning and memory consolidation. It is also necessary to provide similar review support to workers in factories, thereby helping them master work procedures and improve safety.

[0221] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0222] In this invention, the server includes a terminal for a learner to take an image of a reference book or textbook, a means for receiving the image data taken by the terminal, an optical character recognition means for extracting text from the image data within the server, a natural language processing means for analyzing the extracted text, a means for automatically generating questions based on the analysis results, a database for storing the generated questions, a means for calculating the timing of review based on Ebbinghaus's forgetting curve, a means for sending a notification to the learner of the optimal timing for review, a means for accepting the learner's answers, a means for evaluating the answers and generating feedback, smart glasses for a worker to take an image of a work procedure manual, a means for receiving the image data taken by the smart glasses, and a means for calculating the timing of review based on Ebbinghaus's forgetting curve and notifying the worker. This allows learners and workers to efficiently review, solidify their memories, and master work procedures.

[0223] A "learner" is someone who photographs materials from reference books and textbooks and reviews them by solving automatically generated problems, with the aim of effective learning and memory consolidation.

[0224] A "terminal" is a device used by learners to take images of reference books and textbooks, specifically a smartphone or tablet.

[0225] A "server" is a central processing unit that receives image data sent from a terminal, processes and stores the data, calculates review timing, and so on.

[0226] "Optical character recognition means" is a technology that automatically extracts character information from image data, and examples of this include OCR (Optical Character Recognition) engines.

[0227] "Natural language processing means" is a technology that analyzes extracted text information, understands the context, and automatically generates questions. Specifically, NLP (Natural Language Processing) algorithms are used.

[0228] The "database" is a system for efficiently managing and storing generated questions and learners' answer data.

[0229] The "Ebbinghaus forgetting curve" is a theory that shows the decline in memory over time and is used to calculate the optimal timing for review.

[0230] "Notification means" refers to a system that notifies learners when they need to review, and specifically includes push notifications from applications and email notifications.

[0231] "Smart glasses" are a wearable device that allows workers to take images of work procedures and send the image data to a server.

[0232] The "means for evaluating answers and generating feedback" is a system that automatically evaluates answers entered by learners and provides feedback including whether the answers are correct or incorrect and when the next review should be done.

[0233] This invention is a system for enabling learners and workers to efficiently solidify their memories, and in particular provides a function for taking images of reference books, textbooks, and work procedure manuals, and automatically notifying them of the timing for review based on Ebbinghaus's forgetting curve.

[0234] System Overview

[0235] Image input of learning materials and question generation

[0236] 1. User (learner or worker)

[0237] Learners use smartphones or tablets, while workers use smart glasses to take photos of pages from reference books, textbooks, and work instructions.

[0238] 2. Terminal

[0239] Image data captured by learners is sent to a server via an application on the device (smartphone, tablet, smart glasses).

[0240] 3. Server

[0241] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image.

[0242] uses "pytesseract" to extract text information from image data using OCR.

[0243] The extracted text is then used to generate question statements and candidate answers using natural language processing (NLP) techniques, utilizing NLP algorithms such as "scikit-learn."

[0244] The generated questions are stored in a database and saved for later access by the user.

[0245] Setting a study schedule

[0246] 1. Users

[0247] Learners log in to the application and set their own learning goals and deadlines.

[0248] For example, if you set an exam to take place at the end of the month, that date information will be entered.

[0249] 2. Server

[0250] The server applies Ebbinghaus' forgetting curve model based on the input learning goals and deadlines to calculate the optimal timing for review.

[0251] The calculated schedule is stored in a database and saved as user profile information.

[0252] Notification of review timing

[0253] 1. Server

[0254] Regularly check the schedules of learners and workers based on the set review timings.

[0255] When review is required, the server generates a notification and sends it to the learner's or worker's terminal.

[0256] 2. Terminal

[0257] The terminal displays the received notification message to the user.

[0258] For example, you might be notified, "Here's today's review question: What is the speed of light in m / s?"

[0259] Review and feedback

[0260] 1. Users

[0261] The learner or worker checks the notification, opens the terminal application, and works on the review questions.

[0262] A question is displayed and the user enters and submits the answer.

[0263] 2. Terminal

[0264] The user's answer data is sent to the server.

[0265] 3. Server

[0266] The server analyzes the received answers and determines whether they are correct or incorrect.

[0267] Generate a feedback message, such as "Correct! We'll review this in a week."

[0268] The user's learning history information is updated and the next review timing is recalculated as necessary.

[0269] Specific Examples

[0270] For example, when a learner takes a photo of a page about the "speed of light" in a physics textbook, the device sends the image data to the server. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?", and stores it in a database. If the user sets the exam date one month in the future, the server sets a review timing based on Ebbinghaus's forgetting curve. When the first review timing arrives (for example, one day later), the server generates a notification and sends it to the device. The device notifies the user, "Here is today's review question: What is the speed of light in m / s?" When the user enters and submits the answer, the server analyzes the answer and generates feedback saying "That's correct!" and sends it to the user. The next review timing is recalculated, and the user's learning history is updated.

[0271] Prompt Sentence Examples

[0272] For example, use the following prompt for a generative AI model:

[0273] plaintext

[0274] We have scanned specific pages of the manual using OCR technology. Please generate questions based on the following text to help students review effectively:

[0275] The machine operation procedure is as follows: 1. Turn on the power. 2. Perform the initial setup. 3. Select the required operation on the operation panel.

[0276] In this way, the system helps learners and workers to efficiently review, solidify their memories, and master work procedures.

[0277] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0278] Step 1:

[0279] A learner or worker uses a device (smartphone, tablet, smart glasses) to take a photo of a page from a reference book, textbook, or work procedure manual.

[0280] Input: Images of reference books, textbooks, and work instructions

[0281] Output: Image data is saved on the device

[0282] Step 2:

[0283] The terminal transmits the captured image data to the server.

[0284] Input: Image data

[0285] Output: The server receives the image data.

[0286] Step 3:

[0287] The server uses optical character recognition (OCR) technology on the image data it receives to extract text from the image.

[0288] Input: Image data

[0289] Output: Extracted text data

[0290] Specific operation: The server uses "pytesseract" to analyze and extract text information from the image.

[0291] Step 4:

[0292] The server analyzes the extracted text data using natural language processing (NLP) methods to generate question statements and candidate answers.

[0293] Input: Text data

[0294] Output: Generated question and answer candidates

[0295] What it does: The server uses NLP algorithms such as scikit-learn to analyze the text and generate appropriate questions.

[0296] Step 5:

[0297] The server stores the generated questions and answer candidates in a database.

[0298] Input: Generated question and answer candidates

[0299] Output: Questions and answer candidates stored in the database

[0300] Step 6:

[0301] Learners and workers log in to the application and set learning goals and deadlines.

[0302] Input: Learning objectives and due dates

[0303] Output: Objectives and due dates data sent to the server

[0304] Step 7:

[0305] The server applies Ebbinghaus' forgetting curve based on the learning goals and deadlines entered and calculates the optimal timing for review.

[0306] Input: Learning objectives and due dates

[0307] Output: Calculated review timing

[0308] Specific operation: The server creates a review schedule using Ebbinghaus's forgetting curve model.

[0309] Step 8:

[0310] When it is time to review, the server generates a notification and sends it to the learner's or worker's terminal.

[0311] Input:Review timing

[0312] Output: Notification message sent to the terminal

[0313] Specific operation: The server sends a notification using a push notification system or email notification system.

[0314] Step 9:

[0315] The learner or worker checks the notification, opens the terminal application and works on the review questions.

[0316] Input: Notification message

[0317] Output: Answered questions

[0318] Specific behavior: The user receives a notification on their device, opens the application, and answers the question.

[0319] Step 10:

[0320] The terminal transmits the user's answer data to the server.

[0321] Input: Answer data

[0322] Output: Answer data sent to the server

[0323] Step 11:

[0324] The server analyzes the received answers and determines whether they are correct or incorrect.

[0325] Input: Answer data

[0326] Output: Analysis results and feedback messages

[0327] Specific behavior: The server determines whether the answer is correct or incorrect and generates feedback.

[0328] Step 12:

[0329] The server sends the generated feedback to the user's terminal, recalculates the next review timing, and updates the learning history.

[0330] Input: Feedback message

[0331] Output: Feedback sent to the user and an updated review schedule

[0332] Specific operation: The server creates a feedback message and sends it to the device. It also recalculates the next review timing based on the user's learning history.

[0333] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0334] This invention is a system for helping learners efficiently solidify their memories, and aims to improve the learning experience by incorporating an emotion engine. It takes images of reference books and textbooks, automatically generates questions based on their contents, and provides a function to notify users when to review based on Ebbinghaus's forgetting curve. It can also recognize the learner's emotions and adjust the learning content and feedback based on those emotions.

[0335] System Overview

[0336] Image input of learning materials and question generation

[0337] User

[0338] Learners use the camera on their smartphone or PC to take pictures of pages from reference books or textbooks.

[0339] Terminal

[0340] The captured image data is sent to the server via an application on the terminal.

[0341] server

[0342] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image.

[0343] The extracted text is analyzed using natural language processing (NLP) technology to generate question statements and candidate answers based on the learning content.

[0344] The generated questions are stored in a database for later access by the user.

[0345] emotion recognition

[0346] User

[0347] While learners are solving problems, emotional data is collected using facial recognition cameras and microphones.

[0348] Terminal

[0349] The emotion data is processed by the terminal and transmitted to the server.

[0350] server

[0351] The server uses an emotion engine to recognize the learner's emotion from the received data.

[0352] Based on the emotion recognition results, the difficulty level and timing of questions are adjusted, and feedback messages are also customized according to the emotion.

[0353] Setting a study schedule

[0354] User

[0355] The application allows users to set learning goals and deadlines, for example, entering the date of an upcoming exam.

[0356] Terminal

[0357] The entered learning objectives and deadline information are sent to the server.

[0358] server

[0359] The server applies Ebbinghaus' forgetting curve model to calculate the optimal timing for review.

[0360] The calculated schedule is stored in a database and managed as user profile information.

[0361] Notification of review timing

[0362] server

[0363] Regularly check learners' schedules based on set review timings.

[0364] When review is necessary, a notification is generated and sent to the learner's device.

[0365] Terminal

[0366] The device will then display the received notification message to the user, for example, "Here's today's review question: What is the speed of light in m / s?"

[0367] Review and feedback

[0368] User

[0369] The learner sees the notification, opens the application and begins working on the review questions.

[0370] A question is displayed and the learner enters and submits the answer.

[0371] Terminal

[0372] The user's answer data is sent to the server.

[0373] server

[0374] The server analyzes the answer data, scores it, and determines whether it is correct or incorrect.

[0375] Generate a feedback message saying "Correct! We'll review this in a week."

[0376] Updates the user's learning history information and recalculates the next review timing as needed.

[0377] It also uses an emotion engine to generate feedback based on the user's emotions, sending encouraging messages such as "You look good today, keep it up next time!"

[0378] Specific examples

[0379] 1. A user takes a photo of a page about the "speed of light" in a physics textbook.

[0380] 2. The device sends the image data to the server.

[0381] 3. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?" and saves it in the database.

[0382] 4. The user sets the exam date one month in the future.

[0383] 5. The server sets the review timing based on Ebbinghaus's forgetting curve.

[0384] 6. The server generates a notification each time the review time arrives and sends it to the device.

[0385] 7. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?"

[0386] 8. The user enters the answer and submits it.

[0387] 9. The server analyzes the answer and generates feedback to the user saying "That's right!"

[0388] 10. The server also uses an emotion engine to provide feedback based on the user's emotions, such as "Great! You look motivated today."

[0389] 11. The server updates the learning history and recalculates the next review timing.

[0390] In this way, by incorporating emotion recognition technology, it is possible to maintain learners' motivation and maximize the effectiveness of review.

[0391] The processing flow will be explained below.

[0392] Step 1:

[0393] The user launches the application and takes a photo of a page from a reference book or textbook using the camera on their smartphone or PC.

[0394] Step 2:

[0395] The terminal transmits the captured image data to the server.

[0396] Step 3:

[0397] The server receives the transmitted image data and uses optical character recognition (OCR) technology to extract text from the image.

[0398] Step 4:

[0399] The server analyzes the extracted text using natural language processing (NLP) technology and generates questions and candidate answers based on the learning content from the text.

[0400] Step 5:

[0401] The server stores the generated questions and answer candidates in a database.

[0402] Step 6:

[0403] A user logs into the application and sets learning goals and deadlines, for example, entering an exam date.

[0404] Step 7:

[0405] The device sends the learning goals and deadline information set to the server.

[0406] Step 8:

[0407] The server applies Ebbinghaus' forgetting curve model to calculate the optimal time to review based on the set deadline.

[0408] Step 9:

[0409] The server stores the calculated review timing in a database.

[0410] Step 10:

[0411] While the user is solving the problem, emotional data is collected using a camera and microphone.

[0412] Step 11:

[0413] The device processes the emotion data and sends it to the server.

[0414] Step 12:

[0415] The server uses an emotion engine to recognize the learner's emotion from the received emotion data.

[0416] Step 13:

[0417] The server adjusts the difficulty level and timing of questions based on the emotion recognition results.

[0418] Step 14:

[0419] The server generates a notification according to the review timing and sends it to the learner's terminal.

[0420] Step 15:

[0421] The device displays a notification to the learner, for example, "Here's today's review question: What is the speed of light in m / s?"

[0422] Step 16:

[0423] The user sees the notification, opens the app, and begins reviewing the questions.

[0424] Step 17:

[0425] The user enters answers to the review questions and presses the submit button.

[0426] Step 18:

[0427] The terminal transmits the user's answer data to the server.

[0428] Step 19:

[0429] The server receives the answer data, scores the answers, and determines whether they are correct or incorrect.

[0430] Step 20:

[0431] The server generates a feedback message and creates feedback based on the learner's emotions along with the correct or incorrect result.

[0432] Step 21:

[0433] The server sends a feedback message to the learner's terminal.

[0434] Step 22:

[0435] The device displays feedback messages to the learner, such as "Correct! We'll review this in a week" or "Great! You look good today."

[0436] Step 23:

[0437] The server updates the learning history information and recalculates the next review timing as needed.

[0438] This detailed processing flow allows learners to utilize emotion recognition functions to review at the optimal time, helping to solidify their memories.

[0439] Example 2

[0440] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0441] Conventional learning support systems have the problem of reduced learning efficiency because they do not optimize the timing of learners' memory consolidation or review. Furthermore, they do not take learners' emotions into consideration when improving the learning experience, making it difficult to maintain their motivation. This makes it difficult to achieve long-term learning outcomes.

[0442] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0443] In this invention, the server includes optical character recognition means, natural language processing means, means for automatically generating questions, a database for storing the generated questions, means for collecting and transmitting emotional data of learners, means for analyzing the emotional data and adjusting the learning content and feedback, means for accepting answers from learners, and means for evaluating the answers and generating feedback, thereby enabling learners to review at the optimal time and to receive appropriate feedback according to their emotions.

[0444] "Terminal" means a device used by a learner to take images of learning materials and process and transmit other data.

[0445] "Server" is a computer system that receives data sent from a terminal and processes and analyzes it.

[0446] "Optical character recognition means" refers to a technical means for extracting text information from image data.

[0447] "Natural language processing means" refers to technical means for analyzing extracted text and understanding its meaning and structure.

[0448] "Means for automatically generating questions" refers to technical means for automatically generating questions for learners based on the analysis results.

[0449] "Database" refers to the information system that stores and manages generated questions and other learning-related data.

[0450] The "Ebbinghaus forgetting curve" is a theory that describes how human memories are forgotten over time.

[0451] The "means for calculating the timing of review" is a technical means for calculating the optimal timing of review based on Ebbinghaus's forgetting curve.

[0452] The "means for sending notifications" refers to the technical means for sending notifications to learners when it is time to review.

[0453] The "means for collecting and transmitting learner's emotional data" refers to a technical means for detecting the learner's emotions during learning and transmitting the data to a server.

[0454] "Means for analyzing emotional data and adjusting learning content and feedback" refers to technical means for analyzing collected emotional data and adjusting learning content and feedback based on the results.

[0455] "Means for accepting learner's answers" refers to an interface that allows learners to input answers and the technical means for receiving that data.

[0456] The "means for evaluating answers and generating feedback" refers to a technical means for evaluating a learner's answers and generating appropriate feedback based on the results.

[0457] This invention is a system that helps learners efficiently consolidate their memories, and aims to improve the learning experience by combining it with an emotion engine. The system is composed mainly of users (learners), terminals, and servers.

[0458] Image input of learning materials and question generation

[0459] User

[0460] A user can use a camera on a smartphone or computer to take a picture of a page from a reference book or textbook. For example, they can take a picture of a page about the "speed of light" that is part of a physics textbook.

[0461] Terminal

[0462] The captured image data is sent to the server via the device's application, where it is compressed and encrypted.

[0463] server

[0464] The server then uses optical character recognition (OCR) technology on the received image data to extract text from the image, using OCR tools such as Tesseract as a concrete example of the software used for this process.

[0465] The extracted text is analyzed using natural language processing (NLP) techniques, such as spaCy and NLTK, to generate a question and answer candidate based on the learning content.

[0466] The generated problems are stored in a database, and users can solve them later. The database uses a common database system (such as MySQL or PostgreSQL).

[0467] emotion recognition

[0468] User

[0469] While the user is solving the problem, emotion data is collected using facial recognition cameras and microphones, for example by analyzing the user's facial expressions and tone of voice.

[0470] Terminal

[0471] The emotion data is processed by the device and sent to a server, where the collected data is processed in real time or in batches.

[0472] server

[0473] The server uses an emotion engine to recognize the user's emotions from the received data, using Microsoft Azure's Emotion API and Amazon Rekognition.

[0474] The difficulty level and timing of questions are adjusted based on the emotion recognition results. Feedback messages are also customized based on the emotion. For example, a message such as "You seem to be doing well today. Let's keep it up next time!" is generated.

[0475] Setting a study schedule

[0476] User

[0477] The application allows users to set learning goals and deadlines, for example, entering the date of an upcoming exam.

[0478] Terminal

[0479] The entered learning objectives and deadline information are sent to the server.

[0480] server

[0481] Once the server receives the user's input, it applies the Ebbinghaus forgetting curve model to calculate the optimal timing for review. This calculation uses Python libraries such as Scikit-learn and SciPy.

[0482] The calculated schedule is stored in a database and managed as user profile information.

[0483] Notification of review timing

[0484] server

[0485] Based on the set review timing, the server periodically checks the database and generates a notification to the user when the time for review has come.

[0486] The generated notification is sent to the device, for example, "Here's a review question for today: What is the speed of light in m / s?"

[0487] Terminal

[0488] The device will display the received notification message to the user. The push notification function is used for the notification, and when the user taps the notification, the application will open and the user will be taken to the review screen.

[0489] Review and feedback

[0490] User

[0491] The user acknowledges the notification, opens the application, and completes the review questions, such as "What is the speed of light in m / s?", answering "299,792,458 m / s."

[0492] Terminal

[0493] The user's answer data is sent to the server, where it is formatted and encrypted before being sent.

[0494] server

[0495] The server analyzes the received answer data and determines whether it is correct or incorrect, using rule-based algorithms or simple string matching.

[0496] A feedback message is generated and sent to the user, such as "Correct! We'll review it again in a week."

[0497] It also uses an emotion engine to generate feedback based on the user's emotions, such as "Great! You look motivated today."

[0498] Example prompts for generative AI models

[0499] "From an image of the text in a physics textbook, extract the information that "the speed of light is 299,792,458 m / s" and generate questions based on this information."

[0500] "Based on the user's target date for the next exam, create an optimal review schedule based on the Ebbinghaus forgetting curve model."

[0501] "Use facial recognition data from the user to recognize emotions and generate feedback based on that."

[0502] This allows users to study efficiently and maximize their learning effectiveness by receiving personalized feedback based on their emotions.

[0503] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0504] Step 1:

[0505] The user takes a photo of the page of the educational material using the device's camera.

[0506] Specific operation: The user takes a photo of a page in a textbook or reference book using the camera on their smartphone or PC, making sure to focus the camera properly to capture a clear image.

[0507] Input: Teaching material page

[0508] Output: Captured image data

[0509] Step 2:

[0510] The terminal transmits the captured image data to the server.

[0511] Specific operation: An application installed on the device receives image data, compresses and encrypts it, and then sends it to a server using a secure communication protocol (e.g., HTTPS).

[0512] Input: Photographed image data

[0513] Output: Image data sent to the server

[0514] Step 3:

[0515] The server uses OCR technology to extract text from the image data.

[0516] Specific operation: The server uses an OCR tool such as Tesseract to extract text information from the received image data.

[0517] Input: Image data

[0518] Output: Extracted text data

[0519] Step 4:

[0520] The server uses NLP technology to analyze the extracted text and generate question statements and candidate answers.

[0521] Specific operation: The server uses NLP libraries such as spaCy and NLTK to analyze text data. Based on the analysis results, it generates questions and multiple answer candidates appropriate for the learning content.

[0522] Input: Extracted text data

[0523] Output: Generated question and answer candidates

[0524] Step 5:

[0525] The server stores the generated questions in a database.

[0526] Specific operation: The generated question statements and answer candidates are stored in a database system (e.g., MySQL, PostgreSQL) so that the user can access them later.

[0527] Input: Generated question and answer candidates

[0528] Output: Problem data stored in a database

[0529] Step 6:

[0530] Users set learning goals and deadlines in the application.

[0531] Specific operations: The user inputs the next exam date and study goals through the application interface.

[0532] Input: learning goals and exam dates

[0533] Output: Input learning objectives and test date data

[0534] Step 7:

[0535] The terminal transmits the input data to the server.

[0536] Specific operation: Collected learning objectives and test date data are sent to the server. At this time, the data is verified and sanitized to ensure secure transmission.

[0537] Input: Learning objectives and test date data

[0538] Output: Learning objectives and test date data sent to the server

[0539] Step 8:

[0540] The server calculates the timing of review using Ebbinghaus' forgetting curve model.

[0541] Specific operation: The server uses Python's Scikit-learn and SciPy to calculate the optimal timing for review based on Ebbinghaus' forgetting curve.

[0542] Input: Learning objectives and test date data

[0543] Output: Calculated review schedule

[0544] Step 9:

[0545] The server stores the calculated schedule in a database.

[0546] Specific operation: The calculated review schedule is saved in a database and managed as user profile information.

[0547] Input: Calculated review schedule

[0548] Output: Review schedule saved in the database

[0549] Step 10:

[0550] When the time for review arrives, the server generates a notification and sends it to the terminal.

[0551] Specific operation: The server periodically checks the user's study schedule, and when it is time to review, it generates a notification message and sends it to the terminal.

[0552] Input:Review Schedule

[0553] Output: The generated notification message

[0554] Step 11:

[0555] The terminal displays the received notification message to the user.

[0556] Specific operation: The device displays the received notification message to the user and notifies the user as a push notification or in-app notification.

[0557] Input: Notification message

[0558] Output: Notification message displayed to the user

[0559] Step 12:

[0560] The user checks the notification and works through the review questions.

[0561] Specific behavior: The user confirms the notification, opens the application, and answers the review questions. For example, the user answers "299,792,458 m / s" to the question "What is the speed of light in m / s?"

[0562] Input: Notification message

[0563] Output: User's answer data

[0564] Step 13:

[0565] The terminal transmits the user's answer data to the server.

[0566] Specific operation: After preprocessing the user's answer data, it is securely sent to the server.

[0567] Input: User's answer data

[0568] Output: Answer data sent to the server

[0569] Step 14:

[0570] The server analyzes the answer data, determines whether it is correct or incorrect, and generates feedback.

[0571] Specific operation: The server analyzes the received answer data and determines whether it is correct or incorrect. Based on the result of the analysis, it generates a feedback message and sends it to the user. For example, it could give feedback such as "That's right! We'll review it in a week." It can also generate emotion-based feedback using an emotion engine. For example, it could give feedback such as "Great! You look motivated today."

[0572] Input: User's answer data

[0573] Output: The generated feedback message

[0574] (Application example 2)

[0575] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0576] Conventional learning management systems do not take into account the individual emotions and motivation of learners. As a result, if a learner is presented with a difficult question when they are tired or have low concentration, their learning effectiveness and motivation decrease. Furthermore, while there are systems that determine the optimal review timing to maximize learning efficiency and provide notifications at that timing, there are few systems that adjust feedback based on emotion recognition. The purpose of this invention is to solve these problems and provide a learning experience optimized for each individual learner.

[0577] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting emotional data of the learner, means for adjusting the difficulty of questions and the timing of questions based on the collected emotional data, and means for generating feedback messages based on the emotions. This makes it possible to provide an optimal learning experience according to the learner's emotional state.

[0578] A "terminal" is a device used by learners to take images of reference books and textbooks, and includes smartphones, tablets, etc.

[0579] The "server" is a central processing unit that receives and processes image data captured by the terminal.

[0580] "Optical character recognition means" refers to a technology for extracting text from image data, and includes OCR (Optical Character Recognition) technology.

[0581] "Natural language processing means" refers to technology for analyzing extracted text and understanding its meaning, and includes NLP (Natural Language Processing) technology.

[0582] "Means for automatically generating questions" refers to technology that automatically generates questions for learners based on the analysis results.

[0583] A "database" is a digital storage system for storing and managing generated questions.

[0584] The "means for calculating review timing" is a technology that calculates the optimal review timing based on Ebbinghaus's forgetting curve.

[0585] The "means for sending notifications" refers to a technique for sending notifications to learners based on the optimal timing for review.

[0586] The "means for accepting answers" is an interface for receiving and processing the learner's answers as input.

[0587] "Means for generating feedback" are techniques for evaluating learners' answers and providing results and advice.

[0588] "Means for collecting emotional data" refers to technology that recognizes learners' emotions and extracts them as data, and includes facial recognition cameras and voice input devices.

[0589] "Means for adjusting the difficulty of questions and the timing of questions" refers to technology that individually adjusts the difficulty of questions and the timing of questions based on collected emotional data.

[0590] The "means for generating a feedback message" is a technique for creating a customized feedback message based on the learner's emotions.

[0591] This invention is a system that helps learners efficiently solidify their memories. It aims to improve the learning experience by incorporating an emotion engine. Learners can take pictures of textbooks and reference books, automatically generate questions based on the content, and provide reminders for review based on Ebbinghaus's forgetting curve. It can also recognize learners' emotions and adjust learning content and feedback based on those emotions.

[0592] System Overview

[0593] Image input of learning materials and question generation

[0594] Users use the camera on their smartphones, tablets, or other devices to take a photo of a page from a reference book or textbook. The captured image data is sent to a server via an application on the device. The server then uses optical character recognition (OCR) technology to extract text from the image. The extracted text is then analyzed using natural language processing (NLP) technology to generate questions and candidate answers based on the learning content. The generated questions are stored in a database so that users can access them later.

[0595] emotion recognition

[0596] While the user is solving the problems, emotional data is collected using a facial recognition camera and microphone. The emotional data is processed by the device and sent to the server. The server uses an emotion engine to recognize the learner's emotions from the received data. Based on the emotion recognition results, the difficulty of the questions and the timing of questions are adjusted. Feedback messages are also customized according to the emotions.

[0597] Setting a study schedule

[0598] The user sets learning goals and deadlines on the application. For example, they input the date of the next exam. The entered learning goals and deadline information is sent to the server. The server applies Ebbinghaus' forgetting curve model to calculate the optimal review timing. The calculated schedule is saved in a database and managed as user profile information.

[0599] Notification of review timing

[0600] The server periodically checks the learner's schedule based on the set review timing. When a review is required, it generates a notification and sends it to the learner's device. The device then displays the received notification message to the user. For example, the notification may say, "Here is today's review question: What is the speed of light in m / s?"

[0601] Review and feedback

[0602] The user confirms the notification, opens the application, and begins working on the review questions. The questions are displayed, and the learner enters and submits their answers. The user's answer data is sent to the server. The server analyzes the answer data, scores it, and determines whether it is correct or incorrect. It generates a feedback message, displaying "That's right!" The server also uses an emotion engine to provide feedback based on the user's emotions. It may also send encouraging messages such as "You look like you're doing well today, keep it up next time!" The server updates the learning history and recalculates the next review timing as necessary.

[0603] Specific examples

[0604] A user takes a photo of a page in a physics textbook about the "speed of light." The image data of this page is sent to the server. The server uses OCR technology to extract the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?", and saves it in a database. The user sets the exam date one month in the future. The server calculates the timing for review based on Ebbinghaus's forgetting curve. Each time the timing for review arrives, the server generates a notification and sends it to the device. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?" When the user enters and submits the correct answer, the server analyzes it and provides feedback such as "That's correct!" It also provides emotion-based feedback such as "Great! You seem motivated today." The server updates the learning history and recalculates the next timing for review.

[0605] Prompt Sentence Examples

[0606] "Generate a student question based on the following text: 'The molecular weight of water is 18 g / mol. The chemical formula for water is H2O. Water is essential for most life on Earth.'"

[0607] In this way, by incorporating emotion recognition technology, it is possible to maintain learners' motivation and maximize the effectiveness of review.

[0608] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0609] Step 1:

[0610] A user takes a photo of a page in a reference book or textbook using the camera on their smartphone or tablet. The input is the captured image data, and the output is that image data. Specifically, the user launches the device's camera app and takes a photo of the learning material page. At this time, the device temporarily saves the image data.

[0611] Step 2:

[0612] The device sends the captured image data to the server. The input is the image data stored on the device, and the output is the image data sent to the server. Specifically, the device's application uses the server data sending function to upload the image data to the server.

[0613] Step 3:

[0614] The server extracts text from the image data it receives using optical character recognition (OCR). The input is the image data sent to the server, and the output is the extracted text. Specifically, OCR software running on the server analyzes the image and recognizes it as text data.

[0615] Step 4:

[0616] The server analyzes the extracted text using natural language processing (NLP) technology and generates question sentences and candidate answers based on the learning content. The input is the extracted text, and the output is the generated question sentences and candidate answers. Specifically, the NLP algorithm analyzes the text on the server and converts it into an appropriate question format.

[0617] Step 5:

[0618] The generated question sentences and answer candidates are saved in a database on the server. The input is the generated question sentences and answer candidates, and the output is saved in the database. Specifically, the questions and answers are recorded in the database using a database management system on the server.

[0619] Step 6:

[0620] The user sets learning goals and deadlines on the application. The input is the learning goals and deadlines entered by the user, and the output is the transmission of that information from the terminal to the server. Specifically, the user enters learning goals and deadlines using the application's schedule setting function.

[0621] Step 7:

[0622] The server applies Ebbinghaus's forgetting curve model to calculate the optimal review timing. The input is learning goals and deadline information, and the output is the calculated review timing. Specifically, an algorithm running on the server predicts forgetting of the learning content and determines the optimal review time.

[0623] Step 8:

[0624] The server generates a notification when review is required based on the set review timing and sends it to the device. The input is the calculated review timing and the output is a notification message. Specifically, the server uses the notification system to send reminders to the learner at the set timing.

[0625] Step 9:

[0626] The notification message received by the terminal is displayed to the user. The input is the notification message sent from the server, and the output is the notification that the user sees on the display. Specifically, a popup message is displayed to the user using the terminal's notification function.

[0627] Step 10:

[0628] The user confirms the notification, opens the application, and begins working on the review questions. The input is the notified review question, and the output is the answer entered by the user. Specifically, the user enters the answer to the question displayed in the application and presses the submit button.

[0629] Step 11:

[0630] The terminal sends the user's answer data to the server. The input is the answer data entered by the user, and the output is the answer data sent to the server. Specifically, the terminal uploads the answer data to the server via the network.

[0631] Step 12:

[0632] The server analyzes the answer data and generates a feedback message. The input is the user's answer data, and the output is the generated feedback message. Specifically, an evaluation algorithm running on the server scores the answers and generates appropriate feedback.

[0633] Step 13:

[0634] The server uses an emotion engine to generate a feedback message based on the user's emotion and sends it to the device. The input is the user's emotion data, and the output is a customized feedback message. Specifically, the server creates a message to increase the learner's motivation based on the emotion recognition results.

[0635] Step 14:

[0636] The terminal displays a feedback message to the user. The input is the feedback message sent from the server, and the output is the feedback that the user sees on the display. Specifically, the feedback message pops up on the terminal screen.

[0637] Step 15:

[0638] The server updates the learning history and recalculates the next review timing as necessary. The input is the user's learning history data, and the output is the updated learning history and the recalculated review timing. Specifically, the server updates the database and sets a new review timing.

[0639] Through these processing steps, the system optimizes the user's learning experience and provides customized feedback according to their emotional state, maximizing learning effectiveness.

[0640] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0641] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0642] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0643] [Second embodiment]

[0644] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0645] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0646] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0647] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0648] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0649] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0650] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0651] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0652] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0653] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0654] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0655] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0656] This invention is a system for helping learners efficiently solidify their memories. In particular, it provides a function that takes pictures of reference books and textbooks and automatically notifies learners when it is time to review them based on the Ebbinghaus forgetting curve.

[0657] System Overview

[0658] Image input of learning materials and question generation

[0659] User

[0660] Learners use the camera on their smartphone or PC to take pictures of pages from reference books or textbooks.

[0661] Terminal

[0662] The captured image data is sent to the server via an application on the terminal.

[0663] server

[0664] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image.

[0665] The extracted text is interpreted and questions and candidate answers are generated using natural language processing (NLP) techniques.

[0666] The generated questions are stored in a database and saved for later access by the user.

[0667] Setting a study schedule

[0668] User

[0669] Learners log in to the application and set their own learning goals and deadlines.

[0670] For example, if you set an exam to take place at the end of the month, that date information will be entered.

[0671] server

[0672] The server applies Ebbinghaus' forgetting curve model based on the input learning goals and deadlines to calculate the optimal timing for review.

[0673] The calculated schedule is stored in a database and saved as user profile information.

[0674] Notification of review timing

[0675] server

[0676] Regularly check learners' schedules based on set review timings.

[0677] When review is required, the server generates a notification and sends it to the learner's device.

[0678] Terminal

[0679] The terminal displays the received notification message to the user.

[0680] For example, you might be notified, "Here's today's review question: What is the speed of light in m / s?"

[0681] Review and feedback

[0682] User

[0683] The learner checks the notification, opens the terminal application, and works on the review questions.

[0684] A question is displayed and the learner enters and submits the answer.

[0685] Terminal

[0686] The user's answer data is sent to the server.

[0687] server

[0688] The server analyzes the received answers and determines whether they are correct or incorrect.

[0689] Generate a feedback message, such as "Correct! We'll review this in a week."

[0690] The user's learning history information is updated and the next review timing is recalculated as necessary.

[0691] Specific examples

[0692] 1. A user takes a photo of a page about the "speed of light" in a physics textbook.

[0693] 2. The device sends the image data to the server.

[0694] 3. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?" and saves it in the database.

[0695] 4. The user sets the exam date one month in the future.

[0696] 5. The server sets the review timing based on Ebbinghaus's forgetting curve.

[0697] 6. When the first review time comes (for example, one day later), the server generates a notification and sends it to the device.

[0698] 7. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?"

[0699] 8. The user enters the answer and submits it.

[0700] 9. The server analyzes the answer and generates feedback to the user saying "That's right!"

[0701] 10. The server recalculates the next review timing and updates the learning history.

[0702] In this way, learners can efficiently review and solidify their memories.

[0703] The processing flow will be explained below.

[0704] Step 1:

[0705] The user launches the application and takes a photo of a page from a reference book or textbook.

[0706] Step 2:

[0707] The terminal transmits the captured image data to the server.

[0708] Step 3:

[0709] The server receives the transmitted image data and uses optical character recognition (OCR) technology to extract text from the image.

[0710] Step 4:

[0711] The server analyzes the extracted text using natural language processing (NLP) technology and generates questions from the text based on the learning content.

[0712] Step 5:

[0713] The server stores the generated questions and answer candidates in a database.

[0714] Step 6:

[0715] Users log in to the application and set their learning goals and deadlines.

[0716] Step 7:

[0717] The terminal transmits the user's setting information to the server.

[0718] Step 8:

[0719] The server applies Ebbinghaus's forgetting curve model to calculate the optimal timing for review based on learning goals and deadlines.

[0720] Step 9:

[0721] The server stores the calculated review timing in a database.

[0722] Step 10:

[0723] The server periodically checks the learner's schedule based on the review timing.

[0724] Step 11:

[0725] The server generates a notification when review is necessary and sends it to the learner's device.

[0726] Step 12:

[0727] The device displays a notification to the learner.

[0728] Step 13:

[0729] The user sees the notification, opens the application and begins working on the review questions.

[0730] Step 14:

[0731] The user enters answers to the review questions and presses the submit button.

[0732] Step 15:

[0733] The terminal transmits the user's answer data to the server.

[0734] Step 16:

[0735] The server receives the answer data, scores it, and determines whether it is correct or incorrect.

[0736] Step 17:

[0737] The server generates and sends a feedback message to the user.

[0738] Step 18:

[0739] The server updates the user's learning history information and recalculates the next review timing as needed.

[0740] Step 19:

[0741] The terminal displays a feedback message to the user.

[0742] This series of steps allows learners to review the material efficiently and solidify their memory.

[0743] Example 1

[0744] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0745] Conventional learning systems lacked efficient methods for learners to solidify their knowledge. Furthermore, they were unable to properly schedule review sessions, preventing the learner from maximizing their learning effectiveness. Furthermore, manually generating and managing questions was time-consuming and burdensome for learners. The purpose of this invention is to provide a system that solves these problems and enables learners to acquire knowledge efficiently.

[0746] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0747] In this invention, the server includes optical character recognition means, natural language processing means, means for automatically generating practice questions, data storage means, means for calculating review timing, means for sending notifications at optimal review timings, means for accepting answers from learners, means for evaluating answers and generating feedback, means for periodically checking the review schedule based on the calculated review timings, means for generating questions and answer candidates using a generative AI model with natural language processing technology, and means for displaying notification messages on the learner's computer device, thereby enabling learners to efficiently review and effectively solidify their memories.

[0748] A "learner" is an individual or group whose purpose is to acquire the learning content.

[0749] "Reference books and textbooks" are printed or electronic media used by learners for the purpose of acquiring knowledge.

[0750] A "computer device" is an electronic device for processing, storing, and communicating data, and includes personal computers, smartphones, and the like.

[0751] An "information processing device" is a server or equivalent system for receiving and processing data over a network.

[0752] "Optical character recognition" is technology or software that extracts text from image data.

[0753] "Natural language processing" is technology or software for analyzing and understanding extracted text.

[0754] "Means for automatically generating practice questions" refers to technology or software that automatically generates questions for learners based on the results of natural language processing analysis.

[0755] The "data storage means" is a recording medium or database for storing generated questions and related data.

[0756] The "means for calculating the timing of review" is a technology or software that calculates the optimal time for review based on Ebbinghaus's forgetting curve.

[0757] The "means for sending notifications" refers to technology or software that sends notifications to learners based on the calculated review timing.

[0758] The "means for accepting learner's answers" is a system or software that receives and processes answer data from learners.

[0759] The "means for evaluating answers and generating feedback" refers to technology or software that analyzes the answers received from learners, evaluates their correctness, and generates appropriate feedback.

[0760] The "means for checking the review schedule" refers to technology or software that periodically checks the learner's schedule based on the calculated review timing.

[0761] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate questions and potential answers.

[0762] A "means for displaying a notification message" is a technique or software that generates and transmits a notification message to be displayed on a learner's computing device.

[0763] This invention relates to a system that helps learners efficiently solidify their memories. In particular, it has the function of taking pictures of reference books and textbooks and automatically notifying learners when it is time to review them based on the Ebbinghaus forgetting curve.

[0764] Image input of learning materials and question generation

[0765] User: The learner takes a photo of a page from a reference book or textbook using the camera on their smartphone or personal computer. This image data is stored on a computer device.

[0766] Terminal: The terminal transmits the captured image data to an information processing device (server) through an application in the computer device. For example, the image data is uploaded to the server using an application on a smartphone.

[0767] Server: The server uses optical character recognition (OCR) technology on the received image data to extract text. OCR technology such as Google Cloud Vision API or Tesseract is used. The extracted text is then analyzed using natural language processing (NLP) technology. A generative AI model (e.g., OpenAI's GPT-3) is used for this analysis, and the question and candidate answers are generated. The generated questions are saved in a data storage device.

[0768] Setting a study schedule

[0769] User: Learners log in to the application and set deadlines such as learning goals and exam dates. For example, learners can "set exam dates one month from now" within the application.

[0770] Server: The server applies Ebbinghaus's forgetting curve model based on the input learning goals and deadlines to calculate the optimal review timing. The calculated schedule is saved in a data storage device and managed as the user's profile information.

[0771] Notification of review timing

[0772] Server: The server periodically checks the learner's schedule based on the set review timing. When a review is required, it generates a notification and sends it to the user's computer device.

[0773] Terminal: The terminal displays the received notification message to the user. For example, using the smartphone's notification function, a message such as "Today's review question: What is the speed of light in m / s?" is displayed.

[0774] Review and feedback

[0775] User: The learner checks the notification, opens the application on their device, and begins working on the review questions. The questions are displayed, and they can enter and submit their answers immediately.

[0776] Terminal: The user's answer data is sent to the server.

[0777] Server: The server analyzes the received answers and determines whether they are correct or incorrect. Based on the analysis results, it generates a feedback message and sends it to the user. It also updates the user's learning history information and recalculates the next review timing if necessary.

[0778] Specific examples

[0779] 1. A user takes a photo of a page about the "speed of light" in a physics textbook.

[0780] 2. The device sends the image data to the server.

[0781] 3. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?" and saves it in the database.

[0782] 4. The user sets the exam date one month in the future.

[0783] 5. The server sets the review timing based on Ebbinghaus's forgetting curve.

[0784] 6. When the first review time comes (for example, one day later), the server generates a notification and sends it to the device.

[0785] 7. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?"

[0786] 8. The user enters the answer and submits it.

[0787] 9. The server analyzes the answer and generates feedback to the user saying "That's right!"

[0788] 10. The server recalculates the next review timing and updates the learning history.

[0789] Prompt Sentence Examples

[0790] 1. Extract text from images

[0791] Extract the text from this image.

[0792] 2. Problem statement generation

[0793] "Generate an appropriate question from this text. Create a question from the text 'The speed of light is 299,792,458 m / s.'"

[0794] 3. Calculating review timing

[0795] "Calculate the next time to review based on Ebbinghaus's forgetting curve. The first time to review is today."

[0796] This invention enables learners to efficiently review and effectively solidify their memories.

[0797] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0798] Step 1:

[0799] The user takes a photo of a page from a reference book or textbook using the camera on their smartphone or personal computer. The input is image data of the learning material, and the output is image data saved on the device.

[0800] Specific action: A student takes a photo of a page about the speed of light in a physics textbook using their smartphone.

[0801] Step 2:

[0802] The terminal sends the captured image data to the server. The input is the image data stored in the terminal, and the output is the image data sent to the server.

[0803] Specific operation: The captured image is uploaded to the server through the application, and the message "Sending to server" is displayed.

[0804] Step 3:

[0805] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image. The input is the image data sent to the server, and the output is the extracted text data. The OCR technology used is Google Cloud Vision API and Tesseract.

[0806] What it does: The text "The speed of light is 299,792,458 m / s" is extracted from the image.

[0807] Step 4:

[0808] The server uses natural language processing (NLP) techniques to analyze the extracted text. The input is the text data from the OCR, and the output is the analysis results. A generative AI model (e.g., OpenAI's GPT-3) is used for this analysis to generate the question and candidate answers.

[0809] Specific operation: From the text "The speed of light is 299,792,458 m / s," the question "What is the speed of light in m / s?" is generated.

[0810] Step 5:

[0811] The server stores the generated questions in a data storage means, and the input is the generated question data, and the output is the question data stored in the database.

[0812] Specific operation: The generated question "What is the speed of light in m / s?" is saved in the database.

[0813] Step 6:

[0814] A user logs in to the application and sets learning goals and deadlines such as exam dates. The input is the goal and deadline data set by the learner, and the output is the setting data sent to the server.

[0815] Specific behavior: The learner "sets the exam date one month in the future" within the application.

[0816] Step 7:

[0817] The server applies the Ebbinghaus forgetting curve model based on the input learning goals and deadlines to calculate the optimal review timing. The input is the learning goals and deadline data, and the output is the calculated review schedule.

[0818] Specific operation: The server calculates the review timing, such as "the next review is one day later, then three days later, then one week later," and records it in the profile.

[0819] Step 8:

[0820] The server periodically checks the learner's schedule based on the calculated review timing. The input is review schedule data, and the output is notification data.

[0821] What happens: The server checks the schedule and generates a notification such as "Today's review questions are here."

[0822] Step 9:

[0823] The terminal displays the received notification message to the user. The input is the notification data sent from the server, and the output is the notification message displayed to the user.

[0824] Specific behavior: A notification will appear on your device saying, "Here's today's review question: What is the speed of light in m / s?"

[0825] Step 10:

[0826] The user checks the notification, opens the application on the device, and works on the review questions. The input is the displayed question text, and the output is the answer data entered by the user.

[0827] Specific operation: The user enters "299,792,458" in response to the question "What is the speed of light in m / s?" and submits it.

[0828] Step 11:

[0829] The terminal transmits the user's answer data to the server. The input is the answer data entered by the user, and the output is the answer data transmitted to the server.

[0830] Specific behavior: The device will display "Sending answers."

[0831] Step 12:

[0832] The server analyzes the received answers and determines whether they are correct or incorrect. The input is the user's answer data, and the output is the evaluation result and a feedback message. The feedback message is generated based on the analysis result.

[0833] Specific operation: The server determines that the user's answer "299,792,458" is correct, generates feedback to the user saying "That's correct! We'll review it in a week," and sends it to the user.

[0834] Step 13:

[0835] The server updates the user's learning history information and recalculates the next review timing as necessary. The input is the updated learning history data, and the output is the recalculated review schedule.

[0836] Specific operation: The server recalculates the next review timing based on the learning history and records it in the profile.

[0837] (Application example 1)

[0838] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0839] Conventional learning support systems lack the ability to notify learners of effective review timing and automatic problem generation functions to help them efficiently solidify their memories. Furthermore, more efficient educational methods are required for supporting learning for workers in practical environments such as factories, but no systems currently exist to address this. Therefore, there is a need for a system that automatically generates questions from images in reference books and textbooks and notifies learners of review timing based on Ebbinghaus's forgetting curve, thereby promoting efficient learning and memory consolidation. It is also necessary to provide similar review support to workers in factories, thereby helping them master work procedures and improve safety.

[0840] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0841] In this invention, the server includes a terminal for a learner to take an image of a reference book or textbook, a means for receiving the image data taken by the terminal, an optical character recognition means for extracting text from the image data within the server, a natural language processing means for analyzing the extracted text, a means for automatically generating questions based on the analysis results, a database for storing the generated questions, a means for calculating the timing of review based on Ebbinghaus's forgetting curve, a means for sending a notification to the learner of the optimal timing for review, a means for accepting the learner's answers, a means for evaluating the answers and generating feedback, smart glasses for a worker to take an image of a work procedure manual, a means for receiving the image data taken by the smart glasses, and a means for calculating the timing of review based on Ebbinghaus's forgetting curve and notifying the worker. This allows learners and workers to efficiently review, solidify their memories, and master work procedures.

[0842] A "learner" is someone who photographs materials from reference books and textbooks and reviews them by solving automatically generated problems, with the aim of effective learning and memory consolidation.

[0843] A "terminal" is a device used by learners to take images of reference books and textbooks, specifically a smartphone or tablet.

[0844] A "server" is a central processing unit that receives image data sent from a terminal, processes and stores the data, calculates review timing, and so on.

[0845] "Optical character recognition means" is a technology that automatically extracts character information from image data, and examples of this include OCR (Optical Character Recognition) engines.

[0846] "Natural language processing means" is a technology that analyzes extracted text information, understands the context, and automatically generates questions. Specifically, NLP (Natural Language Processing) algorithms are used.

[0847] The "database" is a system for efficiently managing and storing generated questions and learners' answer data.

[0848] The "Ebbinghaus forgetting curve" is a theory that shows the decline in memory over time and is used to calculate the optimal timing for review.

[0849] "Notification means" refers to a system that notifies learners when they need to review, and specifically includes push notifications from applications and email notifications.

[0850] "Smart glasses" are a wearable device that allows workers to take images of work procedures and send the image data to a server.

[0851] The "means for evaluating answers and generating feedback" is a system that automatically evaluates answers entered by learners and provides feedback including whether the answers are correct or incorrect and when the next review should be done.

[0852] This invention is a system for enabling learners and workers to efficiently solidify their memories, and in particular provides a function for taking images of reference books, textbooks, and work procedure manuals, and automatically notifying them of the timing for review based on Ebbinghaus's forgetting curve.

[0853] System Overview

[0854] Image input of learning materials and question generation

[0855] 1. User (learner or worker)

[0856] Learners use smartphones or tablets, while workers use smart glasses to take photos of pages from reference books, textbooks, and work instructions.

[0857] 2. Terminal

[0858] Image data captured by learners is sent to a server via an application on the device (smartphone, tablet, smart glasses).

[0859] 3. Server

[0860] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image.

[0861] uses "pytesseract" to extract text information from image data using OCR.

[0862] The extracted text is then used to generate question statements and candidate answers using natural language processing (NLP) techniques, utilizing NLP algorithms such as "scikit-learn."

[0863] The generated questions are stored in a database and saved for later access by the user.

[0864] Setting a study schedule

[0865] 1. Users

[0866] Learners log in to the application and set their own learning goals and deadlines.

[0867] For example, if you set an exam to take place at the end of the month, that date information will be entered.

[0868] 2. Server

[0869] The server applies Ebbinghaus' forgetting curve model based on the input learning goals and deadlines to calculate the optimal timing for review.

[0870] The calculated schedule is stored in a database and saved as user profile information.

[0871] Notification of review timing

[0872] 1. Server

[0873] Regularly check the schedules of learners and workers based on the set review timings.

[0874] When review is required, the server generates a notification and sends it to the learner's or worker's terminal.

[0875] 2. Terminal

[0876] The terminal displays the received notification message to the user.

[0877] For example, you might be notified, "Here's today's review question: What is the speed of light in m / s?"

[0878] Review and feedback

[0879] 1. Users

[0880] The learner or worker checks the notification, opens the terminal application, and works on the review questions.

[0881] A question is displayed and the user enters and submits the answer.

[0882] 2. Terminal

[0883] The user's answer data is sent to the server.

[0884] 3. Server

[0885] The server analyzes the received answers and determines whether they are correct or incorrect.

[0886] Generate a feedback message, such as "Correct! We'll review this in a week."

[0887] The user's learning history information is updated and the next review timing is recalculated as necessary.

[0888] Specific Examples

[0889] For example, when a learner takes a photo of a page about the "speed of light" in a physics textbook, the device sends the image data to the server. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?", and stores it in a database. If the user sets the exam date one month in the future, the server sets a review timing based on Ebbinghaus's forgetting curve. When the first review timing arrives (for example, one day later), the server generates a notification and sends it to the device. The device notifies the user, "Here is today's review question: What is the speed of light in m / s?" When the user enters and submits the answer, the server analyzes the answer and generates feedback saying "That's correct!" and sends it to the user. The next review timing is recalculated, and the user's learning history is updated.

[0890] Prompt Sentence Examples

[0891] For example, use the following prompt for a generative AI model:

[0892] plaintext

[0893] We have scanned specific pages of the manual using OCR technology. Please generate questions based on the following text to help students review effectively:

[0894] The machine operation procedure is as follows: 1. Turn on the power. 2. Perform the initial setup. 3. Select the required operation on the operation panel.

[0895] In this way, the system helps learners and workers to efficiently review, solidify their memories, and master work procedures.

[0896] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0897] Step 1:

[0898] A learner or worker uses a device (smartphone, tablet, smart glasses) to take a photo of a page from a reference book, textbook, or work procedure manual.

[0899] Input: Images of reference books, textbooks, and work instructions

[0900] Output: Image data is saved on the device

[0901] Step 2:

[0902] The terminal transmits the captured image data to the server.

[0903] Input: Image data

[0904] Output: The server receives the image data.

[0905] Step 3:

[0906] The server uses optical character recognition (OCR) technology on the image data it receives to extract text from the image.

[0907] Input: Image data

[0908] Output: Extracted text data

[0909] Specific operation: The server uses "pytesseract" to analyze and extract text information from the image.

[0910] Step 4:

[0911] The server analyzes the extracted text data using natural language processing (NLP) methods to generate question statements and candidate answers.

[0912] Input: Text data

[0913] Output: Generated question and answer candidates

[0914] What it does: The server uses NLP algorithms such as scikit-learn to analyze the text and generate appropriate questions.

[0915] Step 5:

[0916] The server stores the generated questions and answer candidates in a database.

[0917] Input: Generated question and answer candidates

[0918] Output: Questions and answer candidates stored in the database

[0919] Step 6:

[0920] Learners and workers log in to the application and set learning goals and deadlines.

[0921] Input: Learning objectives and due dates

[0922] Output: Objectives and due dates data sent to the server

[0923] Step 7:

[0924] The server applies Ebbinghaus' forgetting curve based on the learning goals and deadlines entered and calculates the optimal timing for review.

[0925] Input: Learning objectives and due dates

[0926] Output: Calculated review timing

[0927] Specific operation: The server creates a review schedule using Ebbinghaus's forgetting curve model.

[0928] Step 8:

[0929] When it is time to review, the server generates a notification and sends it to the learner's or worker's terminal.

[0930] Input:Review timing

[0931] Output: Notification message sent to the terminal

[0932] Specific operation: The server sends a notification using a push notification system or email notification system.

[0933] Step 9:

[0934] The learner or worker checks the notification, opens the terminal application and works on the review questions.

[0935] Input: Notification message

[0936] Output: Answered questions

[0937] Specific behavior: The user receives a notification on their device, opens the application, and answers the question.

[0938] Step 10:

[0939] The terminal transmits the user's answer data to the server.

[0940] Input: Answer data

[0941] Output: Answer data sent to the server

[0942] Step 11:

[0943] The server analyzes the received answers and determines whether they are correct or incorrect.

[0944] Input: Answer data

[0945] Output: Analysis results and feedback messages

[0946] Specific behavior: The server determines whether the answer is correct or incorrect and generates feedback.

[0947] Step 12:

[0948] The server sends the generated feedback to the user's terminal, recalculates the next review timing, and updates the learning history.

[0949] Input: Feedback message

[0950] Output: Feedback sent to the user and an updated review schedule

[0951] Specific operation: The server creates a feedback message and sends it to the device. It also recalculates the next review timing based on the user's learning history.

[0952] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0953] This invention is a system for helping learners efficiently solidify their memories, and aims to improve the learning experience by incorporating an emotion engine. It takes images of reference books and textbooks, automatically generates questions based on their contents, and provides a function to notify users when to review based on Ebbinghaus's forgetting curve. It can also recognize the learner's emotions and adjust the learning content and feedback based on those emotions.

[0954] System Overview

[0955] Image input of learning materials and question generation

[0956] User

[0957] Learners use the camera on their smartphone or PC to take pictures of pages from reference books or textbooks.

[0958] Terminal

[0959] The captured image data is sent to the server via an application on the terminal.

[0960] server

[0961] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image.

[0962] The extracted text is analyzed using natural language processing (NLP) technology to generate question statements and candidate answers based on the learning content.

[0963] The generated questions are stored in a database for later access by the user.

[0964] emotion recognition

[0965] User

[0966] While learners are solving problems, emotional data is collected using facial recognition cameras and microphones.

[0967] Terminal

[0968] The emotion data is processed by the terminal and transmitted to the server.

[0969] server

[0970] The server uses an emotion engine to recognize the learner's emotion from the received data.

[0971] Based on the emotion recognition results, the difficulty level and timing of questions are adjusted, and feedback messages are also customized according to the emotion.

[0972] Setting a study schedule

[0973] User

[0974] The application allows users to set learning goals and deadlines, for example, entering the date of an upcoming exam.

[0975] Terminal

[0976] The entered learning objectives and deadline information are sent to the server.

[0977] server

[0978] The server applies Ebbinghaus' forgetting curve model to calculate the optimal timing for review.

[0979] The calculated schedule is stored in a database and managed as user profile information.

[0980] Notification of review timing

[0981] server

[0982] Regularly check learners' schedules based on set review timings.

[0983] When review is necessary, a notification is generated and sent to the learner's device.

[0984] Terminal

[0985] The device will then display the received notification message to the user, for example, "Here's today's review question: What is the speed of light in m / s?"

[0986] Review and feedback

[0987] User

[0988] The learner sees the notification, opens the application and begins working on the review questions.

[0989] A question is displayed and the learner enters and submits the answer.

[0990] Terminal

[0991] The user's answer data is sent to the server.

[0992] server

[0993] The server analyzes the answer data, scores it, and determines whether it is correct or incorrect.

[0994] Generate a feedback message saying "Correct! We'll review this in a week."

[0995] Updates the user's learning history information and recalculates the next review timing as needed.

[0996] It also uses an emotion engine to generate feedback based on the user's emotions, sending encouraging messages such as "You look good today, keep it up next time!"

[0997] Specific examples

[0998] 1. A user takes a photo of a page about the "speed of light" in a physics textbook.

[0999] 2. The device sends the image data to the server.

[1000] 3. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?" and saves it in the database.

[1001] 4. The user sets the exam date one month in the future.

[1002] 5. The server sets the review timing based on Ebbinghaus's forgetting curve.

[1003] 6. The server generates a notification each time the review time arrives and sends it to the device.

[1004] 7. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?"

[1005] 8. The user enters the answer and submits it.

[1006] 9. The server analyzes the answer and generates feedback to the user saying "That's right!"

[1007] 10. The server also uses an emotion engine to provide feedback based on the user's emotions, such as "Great! You look motivated today."

[1008] 11. The server updates the learning history and recalculates the next review timing.

[1009] In this way, by incorporating emotion recognition technology, it is possible to maintain learners' motivation and maximize the effectiveness of review.

[1010] The processing flow will be explained below.

[1011] Step 1:

[1012] The user launches the application and takes a photo of a page from a reference book or textbook using the camera on their smartphone or PC.

[1013] Step 2:

[1014] The terminal transmits the captured image data to the server.

[1015] Step 3:

[1016] The server receives the transmitted image data and uses optical character recognition (OCR) technology to extract text from the image.

[1017] Step 4:

[1018] The server analyzes the extracted text using natural language processing (NLP) technology and generates questions and candidate answers based on the learning content from the text.

[1019] Step 5:

[1020] The server stores the generated questions and answer candidates in a database.

[1021] Step 6:

[1022] A user logs into the application and sets learning goals and deadlines, for example, entering an exam date.

[1023] Step 7:

[1024] The device sends the learning goals and deadline information set to the server.

[1025] Step 8:

[1026] The server applies Ebbinghaus' forgetting curve model to calculate the optimal time to review based on the set deadline.

[1027] Step 9:

[1028] The server stores the calculated review timing in a database.

[1029] Step 10:

[1030] While the user is solving the problem, emotional data is collected using a camera and microphone.

[1031] Step 11:

[1032] The device processes the emotion data and sends it to the server.

[1033] Step 12:

[1034] The server uses an emotion engine to recognize the learner's emotion from the received emotion data.

[1035] Step 13:

[1036] The server adjusts the difficulty level and timing of questions based on the emotion recognition results.

[1037] Step 14:

[1038] The server generates a notification according to the review timing and sends it to the learner's terminal.

[1039] Step 15:

[1040] The device displays a notification to the learner, for example, "Here's today's review question: What is the speed of light in m / s?"

[1041] Step 16:

[1042] The user sees the notification, opens the app, and begins reviewing the questions.

[1043] Step 17:

[1044] The user enters answers to the review questions and presses the submit button.

[1045] Step 18:

[1046] The terminal transmits the user's answer data to the server.

[1047] Step 19:

[1048] The server receives the answer data, scores the answers, and determines whether they are correct or incorrect.

[1049] Step 20:

[1050] The server generates a feedback message and creates feedback based on the learner's emotions along with the correct or incorrect result.

[1051] Step 21:

[1052] The server sends a feedback message to the learner's terminal.

[1053] Step 22:

[1054] The device displays feedback messages to the learner, such as "Correct! We'll review this in a week" or "Great! You look good today."

[1055] Step 23:

[1056] The server updates the learning history information and recalculates the next review timing as needed.

[1057] This detailed processing flow allows learners to utilize emotion recognition functions to review at the optimal time, helping to solidify their memories.

[1058] Example 2

[1059] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1060] Conventional learning support systems have the problem of reduced learning efficiency because they do not optimize the timing of learners' memory consolidation or review. Furthermore, they do not take learners' emotions into consideration when improving the learning experience, making it difficult to maintain their motivation. This makes it difficult to achieve long-term learning outcomes.

[1061] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1062] In this invention, the server includes optical character recognition means, natural language processing means, means for automatically generating questions, a database for storing the generated questions, means for collecting and transmitting emotional data of learners, means for analyzing the emotional data and adjusting the learning content and feedback, means for accepting answers from learners, and means for evaluating the answers and generating feedback, thereby enabling learners to review at the optimal time and to receive appropriate feedback according to their emotions.

[1063] "Terminal" means a device used by a learner to take images of learning materials and process and transmit other data.

[1064] "Server" is a computer system that receives data sent from a terminal and processes and analyzes it.

[1065] "Optical character recognition means" refers to a technical means for extracting text information from image data.

[1066] "Natural language processing means" refers to technical means for analyzing extracted text and understanding its meaning and structure.

[1067] "Means for automatically generating questions" refers to technical means for automatically generating questions for learners based on the analysis results.

[1068] "Database" refers to the information system that stores and manages generated questions and other learning-related data.

[1069] The "Ebbinghaus forgetting curve" is a theory that describes how human memories are forgotten over time.

[1070] The "means for calculating the timing of review" is a technical means for calculating the optimal timing of review based on Ebbinghaus's forgetting curve.

[1071] The "means for sending notifications" refers to the technical means for sending notifications to learners when it is time to review.

[1072] The "means for collecting and transmitting learner's emotional data" refers to a technical means for detecting the learner's emotions during learning and transmitting the data to a server.

[1073] "Means for analyzing emotional data and adjusting learning content and feedback" refers to technical means for analyzing collected emotional data and adjusting learning content and feedback based on the results.

[1074] "Means for accepting learner's answers" refers to an interface that allows learners to input answers and the technical means for receiving that data.

[1075] The "means for evaluating answers and generating feedback" refers to a technical means for evaluating a learner's answers and generating appropriate feedback based on the results.

[1076] This invention is a system that helps learners efficiently consolidate their memories, and aims to improve the learning experience by combining it with an emotion engine. The system is composed mainly of users (learners), terminals, and servers.

[1077] Image input of learning materials and question generation

[1078] User

[1079] A user can use a camera on a smartphone or computer to take a picture of a page from a reference book or textbook. For example, they can take a picture of a page about the "speed of light" that is part of a physics textbook.

[1080] Terminal

[1081] The captured image data is sent to the server via the device's application, where it is compressed and encrypted.

[1082] server

[1083] The server then uses optical character recognition (OCR) technology on the received image data to extract text from the image, using OCR tools such as Tesseract as a concrete example of the software used for this process.

[1084] The extracted text is analyzed using natural language processing (NLP) techniques, such as spaCy and NLTK, to generate a question and answer candidate based on the learning content.

[1085] The generated problems are stored in a database, and users can solve them later. The database uses a common database system (such as MySQL or PostgreSQL).

[1086] emotion recognition

[1087] User

[1088] While the user is solving the problem, emotion data is collected using facial recognition cameras and microphones, for example by analyzing the user's facial expressions and tone of voice.

[1089] Terminal

[1090] The emotion data is processed by the device and sent to a server, where the collected data is processed in real time or in batches.

[1091] server

[1092] The server uses an emotion engine to recognize the user's emotions from the received data, using Microsoft Azure's Emotion API and Amazon Rekognition.

[1093] The difficulty level and timing of questions are adjusted based on the emotion recognition results. Feedback messages are also customized based on the emotion. For example, a message such as "You seem to be doing well today. Let's keep it up next time!" is generated.

[1094] Setting a study schedule

[1095] User

[1096] The application allows users to set learning goals and deadlines, for example, entering the date of an upcoming exam.

[1097] Terminal

[1098] The entered learning objectives and deadline information are sent to the server.

[1099] server

[1100] Once the server receives the user's input, it applies the Ebbinghaus forgetting curve model to calculate the optimal timing for review. This calculation uses Python libraries such as Scikit-learn and SciPy.

[1101] The calculated schedule is stored in a database and managed as user profile information.

[1102] Notification of review timing

[1103] server

[1104] Based on the set review timing, the server periodically checks the database and generates a notification to the user when the time for review has come.

[1105] The generated notification is sent to the device, for example, "Here's a review question for today: What is the speed of light in m / s?"

[1106] Terminal

[1107] The device will display the received notification message to the user. The push notification function is used for the notification, and when the user taps the notification, the application will open and the user will be taken to the review screen.

[1108] Review and feedback

[1109] User

[1110] The user acknowledges the notification, opens the application, and completes the review questions, such as "What is the speed of light in m / s?", answering "299,792,458 m / s."

[1111] Terminal

[1112] The user's answer data is sent to the server, where it is formatted and encrypted before being sent.

[1113] server

[1114] The server analyzes the received answer data and determines whether it is correct or incorrect, using rule-based algorithms or simple string matching.

[1115] A feedback message is generated and sent to the user, such as "Correct! We'll review it again in a week."

[1116] It also uses an emotion engine to generate feedback based on the user's emotions, such as "Great! You look motivated today."

[1117] Example prompts for generative AI models

[1118] "From an image of the text in a physics textbook, extract the information that "the speed of light is 299,792,458 m / s" and generate questions based on this information."

[1119] "Based on the user's target date for the next exam, create an optimal review schedule based on the Ebbinghaus forgetting curve model."

[1120] "Use facial recognition data from the user to recognize emotions and generate feedback based on that."

[1121] This allows users to study efficiently and maximize their learning effectiveness by receiving personalized feedback based on their emotions.

[1122] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1123] Step 1:

[1124] The user takes a photo of the page of the educational material using the device's camera.

[1125] Specific operation: The user takes a photo of a page in a textbook or reference book using the camera on their smartphone or PC, making sure to focus the camera properly to capture a clear image.

[1126] Input: Teaching material page

[1127] Output: Captured image data

[1128] Step 2:

[1129] The terminal transmits the captured image data to the server.

[1130] Specific operation: An application installed on the device receives image data, compresses and encrypts it, and then sends it to a server using a secure communication protocol (e.g., HTTPS).

[1131] Input: Photographed image data

[1132] Output: Image data sent to the server

[1133] Step 3:

[1134] The server uses OCR technology to extract text from the image data.

[1135] Specific operation: The server uses an OCR tool such as Tesseract to extract text information from the received image data.

[1136] Input: Image data

[1137] Output: Extracted text data

[1138] Step 4:

[1139] The server uses NLP technology to analyze the extracted text and generate question statements and candidate answers.

[1140] Specific operation: The server uses NLP libraries such as spaCy and NLTK to analyze text data. Based on the analysis results, it generates questions and multiple answer candidates appropriate for the learning content.

[1141] Input: Extracted text data

[1142] Output: Generated question and answer candidates

[1143] Step 5:

[1144] The server stores the generated questions in a database.

[1145] Specific operation: The generated question statements and answer candidates are stored in a database system (e.g., MySQL, PostgreSQL) so that the user can access them later.

[1146] Input: Generated question and answer candidates

[1147] Output: Problem data stored in a database

[1148] Step 6:

[1149] Users set learning goals and deadlines in the application.

[1150] Specific operations: The user inputs the next exam date and study goals through the application interface.

[1151] Input: learning goals and exam dates

[1152] Output: Input learning objectives and test date data

[1153] Step 7:

[1154] The terminal transmits the input data to the server.

[1155] Specific operation: Collected learning objectives and test date data are sent to the server. At this time, the data is verified and sanitized to ensure secure transmission.

[1156] Input: Learning objectives and test date data

[1157] Output: Learning objectives and test date data sent to the server

[1158] Step 8:

[1159] The server calculates the timing of review using Ebbinghaus' forgetting curve model.

[1160] Specific operation: The server uses Python's Scikit-learn and SciPy to calculate the optimal timing for review based on Ebbinghaus' forgetting curve.

[1161] Input: Learning objectives and test date data

[1162] Output: Calculated review schedule

[1163] Step 9:

[1164] The server stores the calculated schedule in a database.

[1165] Specific operation: The calculated review schedule is saved in a database and managed as user profile information.

[1166] Input: Calculated review schedule

[1167] Output: Review schedule saved in the database

[1168] Step 10:

[1169] When the time for review arrives, the server generates a notification and sends it to the terminal.

[1170] Specific operation: The server periodically checks the user's study schedule, and when it is time to review, it generates a notification message and sends it to the terminal.

[1171] Input:Review Schedule

[1172] Output: The generated notification message

[1173] Step 11:

[1174] The terminal displays the received notification message to the user.

[1175] Specific operation: The device displays the received notification message to the user and notifies the user as a push notification or in-app notification.

[1176] Input: Notification message

[1177] Output: Notification message displayed to the user

[1178] Step 12:

[1179] The user checks the notification and works through the review questions.

[1180] Specific behavior: The user confirms the notification, opens the application, and answers the review questions. For example, the user answers "299,792,458 m / s" to the question "What is the speed of light in m / s?"

[1181] Input: Notification message

[1182] Output: User's answer data

[1183] Step 13:

[1184] The terminal transmits the user's answer data to the server.

[1185] Specific operation: After preprocessing the user's answer data, it is securely sent to the server.

[1186] Input: User's answer data

[1187] Output: Answer data sent to the server

[1188] Step 14:

[1189] The server analyzes the answer data, determines whether it is correct or incorrect, and generates feedback.

[1190] Specific operation: The server analyzes the received answer data and determines whether it is correct or incorrect. Based on the result of the analysis, it generates a feedback message and sends it to the user. For example, it could give feedback such as "That's right! We'll review it in a week." It can also generate emotion-based feedback using an emotion engine. For example, it could give feedback such as "Great! You look motivated today."

[1191] Input: User's answer data

[1192] Output: The generated feedback message

[1193] (Application example 2)

[1194] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1195] Conventional learning management systems do not take into account the individual emotions and motivation of learners. As a result, if a learner is presented with a difficult question when they are tired or have low concentration, their learning effectiveness and motivation decrease. Furthermore, while there are systems that determine the optimal review timing to maximize learning efficiency and provide notifications at that timing, there are few systems that adjust feedback based on emotion recognition. The purpose of this invention is to solve these problems and provide a learning experience optimized for each individual learner.

[1196] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting emotional data of the learner, means for adjusting the difficulty of questions and the timing of questions based on the collected emotional data, and means for generating feedback messages based on the emotions. This makes it possible to provide an optimal learning experience according to the learner's emotional state.

[1197] A "terminal" is a device used by learners to take images of reference books and textbooks, and includes smartphones, tablets, etc.

[1198] The "server" is a central processing unit that receives and processes image data captured by the terminal.

[1199] "Optical character recognition means" refers to a technology for extracting text from image data, and includes OCR (Optical Character Recognition) technology.

[1200] "Natural language processing means" refers to technology for analyzing extracted text and understanding its meaning, and includes NLP (Natural Language Processing) technology.

[1201] "Means for automatically generating questions" refers to technology that automatically generates questions for learners based on the analysis results.

[1202] A "database" is a digital storage system for storing and managing generated questions.

[1203] The "means for calculating review timing" is a technology that calculates the optimal review timing based on Ebbinghaus's forgetting curve.

[1204] The "means for sending notifications" refers to a technique for sending notifications to learners based on the optimal timing for review.

[1205] The "means for accepting answers" is an interface for receiving and processing the learner's answers as input.

[1206] "Means for generating feedback" are techniques for evaluating learners' answers and providing results and advice.

[1207] "Means for collecting emotional data" refers to technology that recognizes learners' emotions and extracts them as data, and includes facial recognition cameras and voice input devices.

[1208] "Means for adjusting the difficulty of questions and the timing of questions" refers to technology that individually adjusts the difficulty of questions and the timing of questions based on collected emotional data.

[1209] The "means for generating a feedback message" is a technique for creating a customized feedback message based on the learner's emotions.

[1210] This invention is a system that helps learners efficiently solidify their memories. It aims to improve the learning experience by incorporating an emotion engine. Learners can take pictures of textbooks and reference books, automatically generate questions based on the content, and provide reminders for review based on Ebbinghaus's forgetting curve. It can also recognize learners' emotions and adjust learning content and feedback based on those emotions.

[1211] System Overview

[1212] Image input of learning materials and question generation

[1213] Users use the camera on their smartphones, tablets, or other devices to take a photo of a page from a reference book or textbook. The captured image data is sent to a server via an application on the device. The server then uses optical character recognition (OCR) technology to extract text from the image. The extracted text is then analyzed using natural language processing (NLP) technology to generate questions and candidate answers based on the learning content. The generated questions are stored in a database so that users can access them later.

[1214] emotion recognition

[1215] While the user is solving the problems, emotional data is collected using a facial recognition camera and microphone. The emotional data is processed by the device and sent to the server. The server uses an emotion engine to recognize the learner's emotions from the received data. Based on the emotion recognition results, the difficulty of the questions and the timing of questions are adjusted. Feedback messages are also customized according to the emotions.

[1216] Setting a study schedule

[1217] The user sets learning goals and deadlines on the application. For example, they input the date of the next exam. The entered learning goals and deadline information is sent to the server. The server applies Ebbinghaus' forgetting curve model to calculate the optimal review timing. The calculated schedule is saved in a database and managed as user profile information.

[1218] Notification of review timing

[1219] The server periodically checks the learner's schedule based on the set review timing. When a review is required, it generates a notification and sends it to the learner's device. The device then displays the received notification message to the user. For example, the notification may say, "Here is today's review question: What is the speed of light in m / s?"

[1220] Review and feedback

[1221] The user confirms the notification, opens the application, and begins working on the review questions. The questions are displayed, and the learner enters and submits their answers. The user's answer data is sent to the server. The server analyzes the answer data, scores it, and determines whether it is correct or incorrect. It generates a feedback message, displaying "That's right!" The server also uses an emotion engine to provide feedback based on the user's emotions. It may also send encouraging messages such as "You look like you're doing well today, keep it up next time!" The server updates the learning history and recalculates the next review timing as necessary.

[1222] Specific examples

[1223] A user takes a photo of a page in a physics textbook about the "speed of light." The image data of this page is sent to the server. The server uses OCR technology to extract the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?", and saves it in a database. The user sets the exam date one month in the future. The server calculates the timing for review based on Ebbinghaus's forgetting curve. Each time the timing for review arrives, the server generates a notification and sends it to the device. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?" When the user enters and submits the correct answer, the server analyzes it and provides feedback such as "That's correct!" It also provides emotion-based feedback such as "Great! You seem motivated today." The server updates the learning history and recalculates the next timing for review.

[1224] Prompt Sentence Examples

[1225] "Generate a student question based on the following text: 'The molecular weight of water is 18 g / mol. The chemical formula for water is H2O. Water is essential for most life on Earth.'"

[1226] In this way, by incorporating emotion recognition technology, it is possible to maintain learners' motivation and maximize the effectiveness of review.

[1227] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1228] Step 1:

[1229] A user takes a photo of a page in a reference book or textbook using the camera on their smartphone or tablet. The input is the captured image data, and the output is that image data. Specifically, the user launches the device's camera app and takes a photo of the learning material page. At this time, the device temporarily saves the image data.

[1230] Step 2:

[1231] The device sends the captured image data to the server. The input is the image data stored on the device, and the output is the image data sent to the server. Specifically, the device's application uses the server data sending function to upload the image data to the server.

[1232] Step 3:

[1233] The server extracts text from the image data it receives using optical character recognition (OCR). The input is the image data sent to the server, and the output is the extracted text. Specifically, OCR software running on the server analyzes the image and recognizes it as text data.

[1234] Step 4:

[1235] The server analyzes the extracted text using natural language processing (NLP) technology and generates question sentences and candidate answers based on the learning content. The input is the extracted text, and the output is the generated question sentences and candidate answers. Specifically, the NLP algorithm analyzes the text on the server and converts it into an appropriate question format.

[1236] Step 5:

[1237] The generated question sentences and answer candidates are saved in a database on the server. The input is the generated question sentences and answer candidates, and the output is saved in the database. Specifically, the questions and answers are recorded in the database using a database management system on the server.

[1238] Step 6:

[1239] The user sets learning goals and deadlines on the application. The input is the learning goals and deadlines entered by the user, and the output is the transmission of that information from the terminal to the server. Specifically, the user enters learning goals and deadlines using the application's schedule setting function.

[1240] Step 7:

[1241] The server applies Ebbinghaus's forgetting curve model to calculate the optimal review timing. The input is learning goals and deadline information, and the output is the calculated review timing. Specifically, an algorithm running on the server predicts forgetting of the learning content and determines the optimal review time.

[1242] Step 8:

[1243] The server generates a notification when review is required based on the set review timing and sends it to the device. The input is the calculated review timing and the output is a notification message. Specifically, the server uses the notification system to send reminders to the learner at the set timing.

[1244] Step 9:

[1245] The notification message received by the terminal is displayed to the user. The input is the notification message sent from the server, and the output is the notification that the user sees on the display. Specifically, a popup message is displayed to the user using the terminal's notification function.

[1246] Step 10:

[1247] The user confirms the notification, opens the application, and begins working on the review questions. The input is the notified review question, and the output is the answer entered by the user. Specifically, the user enters the answer to the question displayed in the application and presses the submit button.

[1248] Step 11:

[1249] The terminal sends the user's answer data to the server. The input is the answer data entered by the user, and the output is the answer data sent to the server. Specifically, the terminal uploads the answer data to the server via the network.

[1250] Step 12:

[1251] The server analyzes the answer data and generates a feedback message. The input is the user's answer data, and the output is the generated feedback message. Specifically, an evaluation algorithm running on the server scores the answers and generates appropriate feedback.

[1252] Step 13:

[1253] The server uses an emotion engine to generate a feedback message based on the user's emotion and sends it to the device. The input is the user's emotion data, and the output is a customized feedback message. Specifically, the server creates a message to increase the learner's motivation based on the emotion recognition results.

[1254] Step 14:

[1255] The terminal displays a feedback message to the user. The input is the feedback message sent from the server, and the output is the feedback that the user sees on the display. Specifically, the feedback message pops up on the terminal screen.

[1256] Step 15:

[1257] The server updates the learning history and recalculates the next review timing as necessary. The input is the user's learning history data, and the output is the updated learning history and the recalculated review timing. Specifically, the server updates the database and sets a new review timing.

[1258] Through these processing steps, the system optimizes the user's learning experience and provides customized feedback according to their emotional state, maximizing learning effectiveness.

[1259] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1260] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1261] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1262] [Third embodiment]

[1263] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1264] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1265] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1266] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1267] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1268] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1269] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1270] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1271] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1272] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1273] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1274] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1275] This invention is a system for helping learners efficiently solidify their memories. In particular, it provides a function that takes pictures of reference books and textbooks and automatically notifies learners when it is time to review them based on the Ebbinghaus forgetting curve.

[1276] System Overview

[1277] Image input of learning materials and question generation

[1278] User

[1279] Learners use the camera on their smartphone or PC to take pictures of pages from reference books or textbooks.

[1280] Terminal

[1281] The captured image data is sent to the server via an application on the terminal.

[1282] server

[1283] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image.

[1284] The extracted text is interpreted and questions and candidate answers are generated using natural language processing (NLP) techniques.

[1285] The generated questions are stored in a database and saved for later access by the user.

[1286] Setting a study schedule

[1287] User

[1288] Learners log in to the application and set their own learning goals and deadlines.

[1289] For example, if you set an exam to take place at the end of the month, that date information will be entered.

[1290] server

[1291] The server applies Ebbinghaus' forgetting curve model based on the input learning goals and deadlines to calculate the optimal timing for review.

[1292] The calculated schedule is stored in a database and saved as user profile information.

[1293] Notification of review timing

[1294] server

[1295] Regularly check learners' schedules based on set review timings.

[1296] When review is required, the server generates a notification and sends it to the learner's device.

[1297] Terminal

[1298] The terminal displays the received notification message to the user.

[1299] For example, you might be notified, "Here's today's review question: What is the speed of light in m / s?"

[1300] Review and feedback

[1301] User

[1302] The learner checks the notification, opens the terminal application, and works on the review questions.

[1303] A question is displayed and the learner enters and submits the answer.

[1304] Terminal

[1305] The user's answer data is sent to the server.

[1306] server

[1307] The server analyzes the received answers and determines whether they are correct or incorrect.

[1308] Generate a feedback message, such as "Correct! We'll review this in a week."

[1309] The user's learning history information is updated and the next review timing is recalculated as necessary.

[1310] Specific examples

[1311] 1. A user takes a photo of a page about the "speed of light" in a physics textbook.

[1312] 2. The device sends the image data to the server.

[1313] 3. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?" and saves it in the database.

[1314] 4. The user sets the exam date one month in the future.

[1315] 5. The server sets the review timing based on Ebbinghaus's forgetting curve.

[1316] 6. When the first review time comes (for example, one day later), the server generates a notification and sends it to the device.

[1317] 7. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?"

[1318] 8. The user enters the answer and submits it.

[1319] 9. The server analyzes the answer and generates feedback to the user saying "That's right!"

[1320] 10. The server recalculates the next review timing and updates the learning history.

[1321] In this way, learners can efficiently review and solidify their memories.

[1322] The processing flow will be explained below.

[1323] Step 1:

[1324] The user launches the application and takes a photo of a page from a reference book or textbook.

[1325] Step 2:

[1326] The terminal transmits the captured image data to the server.

[1327] Step 3:

[1328] The server receives the transmitted image data and uses optical character recognition (OCR) technology to extract text from the image.

[1329] Step 4:

[1330] The server analyzes the extracted text using natural language processing (NLP) technology and generates questions from the text based on the learning content.

[1331] Step 5:

[1332] The server stores the generated questions and answer candidates in a database.

[1333] Step 6:

[1334] Users log in to the application and set their learning goals and deadlines.

[1335] Step 7:

[1336] The terminal transmits the user's setting information to the server.

[1337] Step 8:

[1338] The server applies Ebbinghaus's forgetting curve model to calculate the optimal timing for review based on learning goals and deadlines.

[1339] Step 9:

[1340] The server stores the calculated review timing in a database.

[1341] Step 10:

[1342] The server periodically checks the learner's schedule based on the review timing.

[1343] Step 11:

[1344] The server generates a notification when review is necessary and sends it to the learner's device.

[1345] Step 12:

[1346] The device displays a notification to the learner.

[1347] Step 13:

[1348] The user sees the notification, opens the application and begins working on the review questions.

[1349] Step 14:

[1350] The user enters answers to the review questions and presses the submit button.

[1351] Step 15:

[1352] The terminal transmits the user's answer data to the server.

[1353] Step 16:

[1354] The server receives the answer data, scores it, and determines whether it is correct or incorrect.

[1355] Step 17:

[1356] The server generates and sends a feedback message to the user.

[1357] Step 18:

[1358] The server updates the user's learning history information and recalculates the next review timing as needed.

[1359] Step 19:

[1360] The terminal displays a feedback message to the user.

[1361] This series of steps allows learners to review the material efficiently and solidify their memory.

[1362] Example 1

[1363] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1364] Conventional learning systems lacked efficient methods for learners to solidify their knowledge. Furthermore, they were unable to properly schedule review sessions, preventing the learner from maximizing their learning effectiveness. Furthermore, manually generating and managing questions was time-consuming and burdensome for learners. The purpose of this invention is to provide a system that solves these problems and enables learners to acquire knowledge efficiently.

[1365] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1366] In this invention, the server includes optical character recognition means, natural language processing means, means for automatically generating practice questions, data storage means, means for calculating review timing, means for sending notifications at optimal review timings, means for accepting answers from learners, means for evaluating answers and generating feedback, means for periodically checking the review schedule based on the calculated review timings, means for generating questions and answer candidates using a generative AI model with natural language processing technology, and means for displaying notification messages on the learner's computer device, thereby enabling learners to efficiently review and effectively solidify their memories.

[1367] A "learner" is an individual or group whose purpose is to acquire the learning content.

[1368] "Reference books and textbooks" are printed or electronic media used by learners for the purpose of acquiring knowledge.

[1369] A "computer device" is an electronic device for processing, storing, and communicating data, and includes personal computers, smartphones, and the like.

[1370] An "information processing device" is a server or equivalent system for receiving and processing data over a network.

[1371] "Optical character recognition" is technology or software that extracts text from image data.

[1372] "Natural language processing" is technology or software for analyzing and understanding extracted text.

[1373] "Means for automatically generating practice questions" refers to technology or software that automatically generates questions for learners based on the results of natural language processing analysis.

[1374] The "data storage means" is a recording medium or database for storing generated questions and related data.

[1375] The "means for calculating the timing of review" is a technology or software that calculates the optimal time for review based on Ebbinghaus's forgetting curve.

[1376] The "means for sending notifications" refers to technology or software that sends notifications to learners based on the calculated review timing.

[1377] The "means for accepting learner's answers" is a system or software that receives and processes answer data from learners.

[1378] The "means for evaluating answers and generating feedback" refers to technology or software that analyzes the answers received from learners, evaluates their correctness, and generates appropriate feedback.

[1379] The "means for checking the review schedule" refers to technology or software that periodically checks the learner's schedule based on the calculated review timing.

[1380] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate questions and potential answers.

[1381] A "means for displaying a notification message" is a technique or software that generates and transmits a notification message to be displayed on a learner's computing device.

[1382] This invention relates to a system that helps learners efficiently solidify their memories. In particular, it has the function of taking pictures of reference books and textbooks and automatically notifying learners when it is time to review them based on the Ebbinghaus forgetting curve.

[1383] Image input of learning materials and question generation

[1384] User: The learner takes a photo of a page from a reference book or textbook using the camera on their smartphone or personal computer. This image data is stored on a computer device.

[1385] Terminal: The terminal transmits the captured image data to an information processing device (server) through an application in the computer device. For example, the image data is uploaded to the server using an application on a smartphone.

[1386] Server: The server uses optical character recognition (OCR) technology on the received image data to extract text. OCR technology such as Google Cloud Vision API or Tesseract is used. The extracted text is then analyzed using natural language processing (NLP) technology. A generative AI model (e.g., OpenAI's GPT-3) is used for this analysis, and the question and candidate answers are generated. The generated questions are saved in a data storage device.

[1387] Setting a study schedule

[1388] User: Learners log in to the application and set deadlines such as learning goals and exam dates. For example, learners can "set exam dates one month from now" within the application.

[1389] Server: The server applies Ebbinghaus's forgetting curve model based on the input learning goals and deadlines to calculate the optimal review timing. The calculated schedule is saved in a data storage device and managed as the user's profile information.

[1390] Notification of review timing

[1391] Server: The server periodically checks the learner's schedule based on the set review timing. When a review is required, it generates a notification and sends it to the user's computer device.

[1392] Terminal: The terminal displays the received notification message to the user. For example, using the smartphone's notification function, a message such as "Today's review question: What is the speed of light in m / s?" is displayed.

[1393] Review and feedback

[1394] User: The learner checks the notification, opens the application on their device, and begins working on the review questions. The questions are displayed, and they can enter and submit their answers immediately.

[1395] Terminal: The user's answer data is sent to the server.

[1396] Server: The server analyzes the received answers and determines whether they are correct or incorrect. Based on the analysis results, it generates a feedback message and sends it to the user. It also updates the user's learning history information and recalculates the next review timing if necessary.

[1397] Specific examples

[1398] 1. A user takes a photo of a page about the "speed of light" in a physics textbook.

[1399] 2. The device sends the image data to the server.

[1400] 3. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?" and saves it in the database.

[1401] 4. The user sets the exam date one month in the future.

[1402] 5. The server sets the review timing based on Ebbinghaus's forgetting curve.

[1403] 6. When the first review time comes (for example, one day later), the server generates a notification and sends it to the device.

[1404] 7. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?"

[1405] 8. The user enters the answer and submits it.

[1406] 9. The server analyzes the answer and generates feedback to the user saying "That's right!"

[1407] 10. The server recalculates the next review timing and updates the learning history.

[1408] Prompt Sentence Examples

[1409] 1. Extract text from images

[1410] Extract the text from this image.

[1411] 2. Problem statement generation

[1412] "Generate an appropriate question from this text. Create a question from the text 'The speed of light is 299,792,458 m / s.'"

[1413] 3. Calculating review timing

[1414] "Calculate the next time to review based on Ebbinghaus's forgetting curve. The first time to review is today."

[1415] This invention enables learners to efficiently review and effectively solidify their memories.

[1416] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1417] Step 1:

[1418] The user takes a photo of a page from a reference book or textbook using the camera on their smartphone or personal computer. The input is image data of the learning material, and the output is image data saved on the device.

[1419] Specific action: A student takes a photo of a page about the speed of light in a physics textbook using their smartphone.

[1420] Step 2:

[1421] The terminal sends the captured image data to the server. The input is the image data stored in the terminal, and the output is the image data sent to the server.

[1422] Specific operation: The captured image is uploaded to the server through the application, and the message "Sending to server" is displayed.

[1423] Step 3:

[1424] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image. The input is the image data sent to the server, and the output is the extracted text data. The OCR technology used is Google Cloud Vision API and Tesseract.

[1425] What it does: The text "The speed of light is 299,792,458 m / s" is extracted from the image.

[1426] Step 4:

[1427] The server uses natural language processing (NLP) techniques to analyze the extracted text. The input is the text data from the OCR, and the output is the analysis results. A generative AI model (e.g., OpenAI's GPT-3) is used for this analysis to generate the question and candidate answers.

[1428] Specific operation: From the text "The speed of light is 299,792,458 m / s," the question "What is the speed of light in m / s?" is generated.

[1429] Step 5:

[1430] The server stores the generated questions in a data storage means, and the input is the generated question data, and the output is the question data stored in the database.

[1431] Specific operation: The generated question "What is the speed of light in m / s?" is saved in the database.

[1432] Step 6:

[1433] A user logs in to the application and sets learning goals and deadlines such as exam dates. The input is the goal and deadline data set by the learner, and the output is the setting data sent to the server.

[1434] Specific behavior: The learner "sets the exam date one month in the future" within the application.

[1435] Step 7:

[1436] The server applies the Ebbinghaus forgetting curve model based on the input learning goals and deadlines to calculate the optimal review timing. The input is the learning goals and deadline data, and the output is the calculated review schedule.

[1437] Specific operation: The server calculates the review timing, such as "the next review is one day later, then three days later, then one week later," and records it in the profile.

[1438] Step 8:

[1439] The server periodically checks the learner's schedule based on the calculated review timing. The input is review schedule data, and the output is notification data.

[1440] What happens: The server checks the schedule and generates a notification such as "Today's review questions are here."

[1441] Step 9:

[1442] The terminal displays the received notification message to the user. The input is the notification data sent from the server, and the output is the notification message displayed to the user.

[1443] Specific behavior: A notification will appear on your device saying, "Here's today's review question: What is the speed of light in m / s?"

[1444] Step 10:

[1445] The user checks the notification, opens the application on the device, and works on the review questions. The input is the displayed question text, and the output is the answer data entered by the user.

[1446] Specific operation: The user enters "299,792,458" in response to the question "What is the speed of light in m / s?" and submits it.

[1447] Step 11:

[1448] The terminal transmits the user's answer data to the server. The input is the answer data entered by the user, and the output is the answer data transmitted to the server.

[1449] Specific behavior: The device will display "Sending answers."

[1450] Step 12:

[1451] The server analyzes the received answers and determines whether they are correct or incorrect. The input is the user's answer data, and the output is the evaluation result and a feedback message. The feedback message is generated based on the analysis result.

[1452] Specific operation: The server determines that the user's answer "299,792,458" is correct, generates feedback to the user saying "That's correct! We'll review it in a week," and sends it to the user.

[1453] Step 13:

[1454] The server updates the user's learning history information and recalculates the next review timing as necessary. The input is the updated learning history data, and the output is the recalculated review schedule.

[1455] Specific operation: The server recalculates the next review timing based on the learning history and records it in the profile.

[1456] (Application example 1)

[1457] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1458] Conventional learning support systems lack the ability to notify learners of effective review timing and automatic problem generation functions to help them efficiently solidify their memories. Furthermore, more efficient educational methods are required for supporting learning for workers in practical environments such as factories, but no systems currently exist to address this. Therefore, there is a need for a system that automatically generates questions from images in reference books and textbooks and notifies learners of review timing based on Ebbinghaus's forgetting curve, thereby promoting efficient learning and memory consolidation. It is also necessary to provide similar review support to workers in factories, thereby helping them master work procedures and improve safety.

[1459] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1460] In this invention, the server includes a terminal for a learner to take an image of a reference book or textbook, a means for receiving the image data taken by the terminal, an optical character recognition means for extracting text from the image data within the server, a natural language processing means for analyzing the extracted text, a means for automatically generating questions based on the analysis results, a database for storing the generated questions, a means for calculating the timing of review based on Ebbinghaus's forgetting curve, a means for sending a notification to the learner of the optimal timing for review, a means for accepting the learner's answers, a means for evaluating the answers and generating feedback, smart glasses for a worker to take an image of a work procedure manual, a means for receiving the image data taken by the smart glasses, and a means for calculating the timing of review based on Ebbinghaus's forgetting curve and notifying the worker. This allows learners and workers to efficiently review, solidify their memories, and master work procedures.

[1461] A "learner" is someone who photographs materials from reference books and textbooks and reviews them by solving automatically generated problems, with the aim of effective learning and memory consolidation.

[1462] A "terminal" is a device used by learners to take images of reference books and textbooks, specifically a smartphone or tablet.

[1463] A "server" is a central processing unit that receives image data sent from a terminal, processes and stores the data, calculates review timing, and so on.

[1464] "Optical character recognition means" is a technology that automatically extracts character information from image data, and examples of this include OCR (Optical Character Recognition) engines.

[1465] "Natural language processing means" is a technology that analyzes extracted text information, understands the context, and automatically generates questions. Specifically, NLP (Natural Language Processing) algorithms are used.

[1466] The "database" is a system for efficiently managing and storing generated questions and learners' answer data.

[1467] The "Ebbinghaus forgetting curve" is a theory that shows the decline in memory over time and is used to calculate the optimal timing for review.

[1468] "Notification means" refers to a system that notifies learners when they need to review, and specifically includes push notifications from applications and email notifications.

[1469] "Smart glasses" are a wearable device that allows workers to take images of work procedures and send the image data to a server.

[1470] The "means for evaluating answers and generating feedback" is a system that automatically evaluates answers entered by learners and provides feedback including whether the answers are correct or incorrect and when the next review should be done.

[1471] This invention is a system for enabling learners and workers to efficiently solidify their memories, and in particular provides a function for taking images of reference books, textbooks, and work procedure manuals, and automatically notifying them of the timing for review based on Ebbinghaus's forgetting curve.

[1472] System Overview

[1473] Image input of learning materials and question generation

[1474] 1. User (learner or worker)

[1475] Learners use smartphones or tablets, while workers use smart glasses to take photos of pages from reference books, textbooks, and work instructions.

[1476] 2. Terminal

[1477] Image data captured by learners is sent to a server via an application on the device (smartphone, tablet, smart glasses).

[1478] 3. Server

[1479] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image.

[1480] uses "pytesseract" to extract text information from image data using OCR.

[1481] The extracted text is then used to generate question statements and candidate answers using natural language processing (NLP) techniques, utilizing NLP algorithms such as "scikit-learn."

[1482] The generated questions are stored in a database and saved for later access by the user.

[1483] Setting a study schedule

[1484] 1. Users

[1485] Learners log in to the application and set their own learning goals and deadlines.

[1486] For example, if you set an exam to take place at the end of the month, that date information will be entered.

[1487] 2. Server

[1488] The server applies Ebbinghaus' forgetting curve model based on the input learning goals and deadlines to calculate the optimal timing for review.

[1489] The calculated schedule is stored in a database and saved as user profile information.

[1490] Notification of review timing

[1491] 1. Server

[1492] Regularly check the schedules of learners and workers based on the set review timings.

[1493] When review is required, the server generates a notification and sends it to the learner's or worker's terminal.

[1494] 2. Terminal

[1495] The terminal displays the received notification message to the user.

[1496] For example, you might be notified, "Here's today's review question: What is the speed of light in m / s?"

[1497] Review and feedback

[1498] 1. Users

[1499] The learner or worker checks the notification, opens the terminal application, and works on the review questions.

[1500] A question is displayed and the user enters and submits the answer.

[1501] 2. Terminal

[1502] The user's answer data is sent to the server.

[1503] 3. Server

[1504] The server analyzes the received answers and determines whether they are correct or incorrect.

[1505] Generate a feedback message, such as "Correct! We'll review this in a week."

[1506] The user's learning history information is updated and the next review timing is recalculated as necessary.

[1507] Specific Examples

[1508] For example, when a learner takes a photo of a page about the "speed of light" in a physics textbook, the device sends the image data to the server. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?", and stores it in a database. If the user sets the exam date one month in the future, the server sets a review timing based on Ebbinghaus's forgetting curve. When the first review timing arrives (for example, one day later), the server generates a notification and sends it to the device. The device notifies the user, "Here is today's review question: What is the speed of light in m / s?" When the user enters and submits the answer, the server analyzes the answer and generates feedback saying "That's correct!" and sends it to the user. The next review timing is recalculated, and the user's learning history is updated.

[1509] Prompt Sentence Examples

[1510] For example, use the following prompt for a generative AI model:

[1511] plaintext

[1512] We have scanned specific pages of the manual using OCR technology. Please generate questions based on the following text to help students review effectively:

[1513] The machine operation procedure is as follows: 1. Turn on the power. 2. Perform the initial setup. 3. Select the required operation on the operation panel.

[1514] In this way, the system helps learners and workers to efficiently review, solidify their memories, and master work procedures.

[1515] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1516] Step 1:

[1517] A learner or worker uses a device (smartphone, tablet, smart glasses) to take a photo of a page from a reference book, textbook, or work procedure manual.

[1518] Input: Images of reference books, textbooks, and work instructions

[1519] Output: Image data is saved on the device

[1520] Step 2:

[1521] The terminal transmits the captured image data to the server.

[1522] Input: Image data

[1523] Output: The server receives the image data.

[1524] Step 3:

[1525] The server uses optical character recognition (OCR) technology on the image data it receives to extract text from the image.

[1526] Input: Image data

[1527] Output: Extracted text data

[1528] Specific operation: The server uses "pytesseract" to analyze and extract text information from the image.

[1529] Step 4:

[1530] The server analyzes the extracted text data using natural language processing (NLP) methods to generate question statements and candidate answers.

[1531] Input: Text data

[1532] Output: Generated question and answer candidates

[1533] What it does: The server uses NLP algorithms such as scikit-learn to analyze the text and generate appropriate questions.

[1534] Step 5:

[1535] The server stores the generated questions and answer candidates in a database.

[1536] Input: Generated question and answer candidates

[1537] Output: Questions and answer candidates stored in the database

[1538] Step 6:

[1539] Learners and workers log in to the application and set learning goals and deadlines.

[1540] Input: Learning objectives and due dates

[1541] Output: Objectives and due dates data sent to the server

[1542] Step 7:

[1543] The server applies Ebbinghaus' forgetting curve based on the learning goals and deadlines entered and calculates the optimal timing for review.

[1544] Input: Learning objectives and due dates

[1545] Output: Calculated review timing

[1546] Specific operation: The server creates a review schedule using Ebbinghaus's forgetting curve model.

[1547] Step 8:

[1548] When it is time to review, the server generates a notification and sends it to the learner's or worker's terminal.

[1549] Input:Review timing

[1550] Output: Notification message sent to the terminal

[1551] Specific operation: The server sends a notification using a push notification system or email notification system.

[1552] Step 9:

[1553] The learner or worker checks the notification, opens the terminal application and works on the review questions.

[1554] Input: Notification message

[1555] Output: Answered questions

[1556] Specific behavior: The user receives a notification on their device, opens the application, and answers the question.

[1557] Step 10:

[1558] The terminal transmits the user's answer data to the server.

[1559] Input: Answer data

[1560] Output: Answer data sent to the server

[1561] Step 11:

[1562] The server analyzes the received answers and determines whether they are correct or incorrect.

[1563] Input: Answer data

[1564] Output: Analysis results and feedback messages

[1565] Specific behavior: The server determines whether the answer is correct or incorrect and generates feedback.

[1566] Step 12:

[1567] The server sends the generated feedback to the user's terminal, recalculates the next review timing, and updates the learning history.

[1568] Input: Feedback message

[1569] Output: Feedback sent to the user and an updated review schedule

[1570] Specific operation: The server creates a feedback message and sends it to the device. It also recalculates the next review timing based on the user's learning history.

[1571] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1572] This invention is a system for helping learners efficiently solidify their memories, and aims to improve the learning experience by incorporating an emotion engine. It takes images of reference books and textbooks, automatically generates questions based on their contents, and provides a function to notify users when to review based on Ebbinghaus's forgetting curve. It can also recognize the learner's emotions and adjust the learning content and feedback based on those emotions.

[1573] System Overview

[1574] Image input of learning materials and question generation

[1575] User

[1576] Learners use the camera on their smartphone or PC to take pictures of pages from reference books or textbooks.

[1577] Terminal

[1578] The captured image data is sent to the server via an application on the terminal.

[1579] server

[1580] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image.

[1581] The extracted text is analyzed using natural language processing (NLP) technology to generate question statements and candidate answers based on the learning content.

[1582] The generated questions are stored in a database for later access by the user.

[1583] emotion recognition

[1584] User

[1585] While learners are solving problems, emotional data is collected using facial recognition cameras and microphones.

[1586] Terminal

[1587] The emotion data is processed by the terminal and transmitted to the server.

[1588] server

[1589] The server uses an emotion engine to recognize the learner's emotion from the received data.

[1590] Based on the emotion recognition results, the difficulty level and timing of questions are adjusted, and feedback messages are also customized according to the emotion.

[1591] Setting a study schedule

[1592] User

[1593] The application allows users to set learning goals and deadlines, for example, entering the date of an upcoming exam.

[1594] Terminal

[1595] The entered learning objectives and deadline information are sent to the server.

[1596] server

[1597] The server applies Ebbinghaus' forgetting curve model to calculate the optimal timing for review.

[1598] The calculated schedule is stored in a database and managed as user profile information.

[1599] Notification of review timing

[1600] server

[1601] Regularly check learners' schedules based on set review timings.

[1602] When review is necessary, a notification is generated and sent to the learner's device.

[1603] Terminal

[1604] The device will then display the received notification message to the user, for example, "Here's today's review question: What is the speed of light in m / s?"

[1605] Review and feedback

[1606] User

[1607] The learner sees the notification, opens the application and begins working on the review questions.

[1608] A question is displayed and the learner enters and submits the answer.

[1609] Terminal

[1610] The user's answer data is sent to the server.

[1611] server

[1612] The server analyzes the answer data, scores it, and determines whether it is correct or incorrect.

[1613] Generate a feedback message saying "Correct! We'll review this in a week."

[1614] Updates the user's learning history information and recalculates the next review timing as needed.

[1615] It also uses an emotion engine to generate feedback based on the user's emotions, sending encouraging messages such as "You look good today, keep it up next time!"

[1616] Specific examples

[1617] 1. A user takes a photo of a page about the "speed of light" in a physics textbook.

[1618] 2. The device sends the image data to the server.

[1619] 3. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?" and saves it in the database.

[1620] 4. The user sets the exam date one month in the future.

[1621] 5. The server sets the review timing based on Ebbinghaus's forgetting curve.

[1622] 6. The server generates a notification each time the review time arrives and sends it to the device.

[1623] 7. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?"

[1624] 8. The user enters the answer and submits it.

[1625] 9. The server analyzes the answer and generates feedback to the user saying "That's right!"

[1626] 10. The server also uses an emotion engine to provide feedback based on the user's emotions, such as "Great! You look motivated today."

[1627] 11. The server updates the learning history and recalculates the next review timing.

[1628] In this way, by incorporating emotion recognition technology, it is possible to maintain learners' motivation and maximize the effectiveness of review.

[1629] The processing flow will be explained below.

[1630] Step 1:

[1631] The user launches the application and takes a photo of a page from a reference book or textbook using the camera on their smartphone or PC.

[1632] Step 2:

[1633] The terminal transmits the captured image data to the server.

[1634] Step 3:

[1635] The server receives the transmitted image data and uses optical character recognition (OCR) technology to extract text from the image.

[1636] Step 4:

[1637] The server analyzes the extracted text using natural language processing (NLP) technology and generates questions and candidate answers based on the learning content from the text.

[1638] Step 5:

[1639] The server stores the generated questions and answer candidates in a database.

[1640] Step 6:

[1641] A user logs into the application and sets learning goals and deadlines, for example, entering an exam date.

[1642] Step 7:

[1643] The device sends the learning goals and deadline information set to the server.

[1644] Step 8:

[1645] The server applies Ebbinghaus' forgetting curve model to calculate the optimal time to review based on the set deadline.

[1646] Step 9:

[1647] The server stores the calculated review timing in a database.

[1648] Step 10:

[1649] While the user is solving the problem, emotional data is collected using a camera and microphone.

[1650] Step 11:

[1651] The device processes the emotion data and sends it to the server.

[1652] Step 12:

[1653] The server uses an emotion engine to recognize the learner's emotion from the received emotion data.

[1654] Step 13:

[1655] The server adjusts the difficulty level and timing of questions based on the emotion recognition results.

[1656] Step 14:

[1657] The server generates a notification according to the review timing and sends it to the learner's terminal.

[1658] Step 15:

[1659] The device displays a notification to the learner, for example, "Here's today's review question: What is the speed of light in m / s?"

[1660] Step 16:

[1661] The user sees the notification, opens the app, and begins reviewing the questions.

[1662] Step 17:

[1663] The user enters answers to the review questions and presses the submit button.

[1664] Step 18:

[1665] The terminal transmits the user's answer data to the server.

[1666] Step 19:

[1667] The server receives the answer data, scores the answers, and determines whether they are correct or incorrect.

[1668] Step 20:

[1669] The server generates a feedback message and creates feedback based on the learner's emotions along with the correct or incorrect result.

[1670] Step 21:

[1671] The server sends a feedback message to the learner's terminal.

[1672] Step 22:

[1673] The device displays feedback messages to the learner, such as "Correct! We'll review this in a week" or "Great! You look good today."

[1674] Step 23:

[1675] The server updates the learning history information and recalculates the next review timing as needed.

[1676] This detailed processing flow allows learners to utilize emotion recognition functions to review at the optimal time, helping to solidify their memories.

[1677] Example 2

[1678] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1679] Conventional learning support systems have the problem of reduced learning efficiency because they do not optimize the timing of learners' memory consolidation or review. Furthermore, they do not take learners' emotions into consideration when improving the learning experience, making it difficult to maintain their motivation. This makes it difficult to achieve long-term learning outcomes.

[1680] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1681] In this invention, the server includes optical character recognition means, natural language processing means, means for automatically generating questions, a database for storing the generated questions, means for collecting and transmitting emotional data of learners, means for analyzing the emotional data and adjusting the learning content and feedback, means for accepting answers from learners, and means for evaluating the answers and generating feedback, thereby enabling learners to review at the optimal time and to receive appropriate feedback according to their emotions.

[1682] "Terminal" means a device used by a learner to take images of learning materials and process and transmit other data.

[1683] "Server" is a computer system that receives data sent from a terminal and processes and analyzes it.

[1684] "Optical character recognition means" refers to a technical means for extracting text information from image data.

[1685] "Natural language processing means" refers to technical means for analyzing extracted text and understanding its meaning and structure.

[1686] "Means for automatically generating questions" refers to technical means for automatically generating questions for learners based on the analysis results.

[1687] "Database" refers to the information system that stores and manages generated questions and other learning-related data.

[1688] The "Ebbinghaus forgetting curve" is a theory that describes how human memories are forgotten over time.

[1689] The "means for calculating the timing of review" is a technical means for calculating the optimal timing of review based on Ebbinghaus's forgetting curve.

[1690] The "means for sending notifications" refers to the technical means for sending notifications to learners when it is time to review.

[1691] The "means for collecting and transmitting learner's emotional data" refers to a technical means for detecting the learner's emotions during learning and transmitting the data to a server.

[1692] "Means for analyzing emotional data and adjusting learning content and feedback" refers to technical means for analyzing collected emotional data and adjusting learning content and feedback based on the results.

[1693] "Means for accepting learner's answers" refers to an interface that allows learners to input answers and the technical means for receiving that data.

[1694] The "means for evaluating answers and generating feedback" refers to a technical means for evaluating a learner's answers and generating appropriate feedback based on the results.

[1695] This invention is a system that helps learners efficiently consolidate their memories, and aims to improve the learning experience by combining it with an emotion engine. The system is composed mainly of users (learners), terminals, and servers.

[1696] Image input of learning materials and question generation

[1697] User

[1698] A user can use a camera on a smartphone or computer to take a picture of a page from a reference book or textbook. For example, they can take a picture of a page about the "speed of light" that is part of a physics textbook.

[1699] Terminal

[1700] The captured image data is sent to the server via the device's application, where it is compressed and encrypted.

[1701] server

[1702] The server then uses optical character recognition (OCR) technology on the received image data to extract text from the image, using OCR tools such as Tesseract as a concrete example of the software used for this process.

[1703] The extracted text is analyzed using natural language processing (NLP) techniques, such as spaCy and NLTK, to generate a question and answer candidate based on the learning content.

[1704] The generated problems are stored in a database, and users can solve them later. The database uses a common database system (such as MySQL or PostgreSQL).

[1705] emotion recognition

[1706] User

[1707] While the user is solving the problem, emotion data is collected using facial recognition cameras and microphones, for example by analyzing the user's facial expressions and tone of voice.

[1708] Terminal

[1709] The emotion data is processed by the device and sent to a server, where the collected data is processed in real time or in batches.

[1710] server

[1711] The server uses an emotion engine to recognize the user's emotions from the received data, using Microsoft Azure's Emotion API and Amazon Rekognition.

[1712] The difficulty level and timing of questions are adjusted based on the emotion recognition results. Feedback messages are also customized based on the emotion. For example, a message such as "You seem to be doing well today. Let's keep it up next time!" is generated.

[1713] Setting a study schedule

[1714] User

[1715] The application allows users to set learning goals and deadlines, for example, entering the date of an upcoming exam.

[1716] Terminal

[1717] The entered learning objectives and deadline information are sent to the server.

[1718] server

[1719] Once the server receives the user's input, it applies the Ebbinghaus forgetting curve model to calculate the optimal timing for review. This calculation uses Python libraries such as Scikit-learn and SciPy.

[1720] The calculated schedule is stored in a database and managed as user profile information.

[1721] Notification of review timing

[1722] server

[1723] Based on the set review timing, the server periodically checks the database and generates a notification to the user when the time for review has come.

[1724] The generated notification is sent to the device, for example, "Here's a review question for today: What is the speed of light in m / s?"

[1725] Terminal

[1726] The device will display the received notification message to the user. The push notification function is used for the notification, and when the user taps the notification, the application will open and the user will be taken to the review screen.

[1727] Review and feedback

[1728] User

[1729] The user acknowledges the notification, opens the application, and completes the review questions, such as "What is the speed of light in m / s?", answering "299,792,458 m / s."

[1730] Terminal

[1731] The user's answer data is sent to the server, where it is formatted and encrypted before being sent.

[1732] server

[1733] The server analyzes the received answer data and determines whether it is correct or incorrect, using rule-based algorithms or simple string matching.

[1734] A feedback message is generated and sent to the user, such as "Correct! We'll review it again in a week."

[1735] It also uses an emotion engine to generate feedback based on the user's emotions, such as "Great! You look motivated today."

[1736] Example prompts for generative AI models

[1737] "From an image of the text in a physics textbook, extract the information that "the speed of light is 299,792,458 m / s" and generate questions based on this information."

[1738] "Based on the user's target date for the next exam, create an optimal review schedule based on the Ebbinghaus forgetting curve model."

[1739] "Use facial recognition data from the user to recognize emotions and generate feedback based on that."

[1740] This allows users to study efficiently and maximize their learning effectiveness by receiving personalized feedback based on their emotions.

[1741] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1742] Step 1:

[1743] The user takes a photo of the page of the educational material using the device's camera.

[1744] Specific operation: The user takes a photo of a page in a textbook or reference book using the camera on their smartphone or PC, making sure to focus the camera properly to capture a clear image.

[1745] Input: Teaching material page

[1746] Output: Captured image data

[1747] Step 2:

[1748] The terminal transmits the captured image data to the server.

[1749] Specific operation: An application installed on the device receives image data, compresses and encrypts it, and then sends it to a server using a secure communication protocol (e.g., HTTPS).

[1750] Input: Photographed image data

[1751] Output: Image data sent to the server

[1752] Step 3:

[1753] The server uses OCR technology to extract text from the image data.

[1754] Specific operation: The server uses an OCR tool such as Tesseract to extract text information from the received image data.

[1755] Input: Image data

[1756] Output: Extracted text data

[1757] Step 4:

[1758] The server uses NLP technology to analyze the extracted text and generate question statements and candidate answers.

[1759] Specific operation: The server uses NLP libraries such as spaCy and NLTK to analyze text data. Based on the analysis results, it generates questions and multiple answer candidates appropriate for the learning content.

[1760] Input: Extracted text data

[1761] Output: Generated question and answer candidates

[1762] Step 5:

[1763] The server stores the generated questions in a database.

[1764] Specific operation: The generated question statements and answer candidates are stored in a database system (e.g., MySQL, PostgreSQL) so that the user can access them later.

[1765] Input: Generated question and answer candidates

[1766] Output: Problem data stored in a database

[1767] Step 6:

[1768] Users set learning goals and deadlines in the application.

[1769] Specific operations: The user inputs the next exam date and study goals through the application interface.

[1770] Input: learning goals and exam dates

[1771] Output: Input learning objectives and test date data

[1772] Step 7:

[1773] The terminal transmits the input data to the server.

[1774] Specific operation: Collected learning objectives and test date data are sent to the server. At this time, the data is verified and sanitized to ensure secure transmission.

[1775] Input: Learning objectives and test date data

[1776] Output: Learning objectives and test date data sent to the server

[1777] Step 8:

[1778] The server calculates the timing of review using Ebbinghaus' forgetting curve model.

[1779] Specific operation: The server uses Python's Scikit-learn and SciPy to calculate the optimal timing for review based on Ebbinghaus' forgetting curve.

[1780] Input: Learning objectives and test date data

[1781] Output: Calculated review schedule

[1782] Step 9:

[1783] The server stores the calculated schedule in a database.

[1784] Specific operation: The calculated review schedule is saved in a database and managed as user profile information.

[1785] Input: Calculated review schedule

[1786] Output: Review schedule saved in the database

[1787] Step 10:

[1788] When the time for review arrives, the server generates a notification and sends it to the terminal.

[1789] Specific operation: The server periodically checks the user's study schedule, and when it is time to review, it generates a notification message and sends it to the terminal.

[1790] Input:Review Schedule

[1791] Output: The generated notification message

[1792] Step 11:

[1793] The terminal displays the received notification message to the user.

[1794] Specific operation: The device displays the received notification message to the user and notifies the user as a push notification or in-app notification.

[1795] Input: Notification message

[1796] Output: Notification message displayed to the user

[1797] Step 12:

[1798] The user checks the notification and works through the review questions.

[1799] Specific behavior: The user confirms the notification, opens the application, and answers the review questions. For example, the user answers "299,792,458 m / s" to the question "What is the speed of light in m / s?"

[1800] Input: Notification message

[1801] Output: User's answer data

[1802] Step 13:

[1803] The terminal transmits the user's answer data to the server.

[1804] Specific operation: After preprocessing the user's answer data, it is securely sent to the server.

[1805] Input: User's answer data

[1806] Output: Answer data sent to the server

[1807] Step 14:

[1808] The server analyzes the answer data, determines whether it is correct or incorrect, and generates feedback.

[1809] Specific operation: The server analyzes the received answer data and determines whether it is correct or incorrect. Based on the result of the analysis, it generates a feedback message and sends it to the user. For example, it could give feedback such as "That's right! We'll review it in a week." It can also generate emotion-based feedback using an emotion engine. For example, it could give feedback such as "Great! You look motivated today."

[1810] Input: User's answer data

[1811] Output: The generated feedback message

[1812] (Application example 2)

[1813] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1814] Conventional learning management systems do not take into account the individual emotions and motivation of learners. As a result, if a learner is presented with a difficult question when they are tired or have low concentration, their learning effectiveness and motivation decrease. Furthermore, while there are systems that determine the optimal review timing to maximize learning efficiency and provide notifications at that timing, there are few systems that adjust feedback based on emotion recognition. The purpose of this invention is to solve these problems and provide a learning experience optimized for each individual learner.

[1815] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting emotional data of the learner, means for adjusting the difficulty of questions and the timing of questions based on the collected emotional data, and means for generating feedback messages based on the emotions. This makes it possible to provide an optimal learning experience according to the learner's emotional state.

[1816] A "terminal" is a device used by learners to take images of reference books and textbooks, and includes smartphones, tablets, etc.

[1817] The "server" is a central processing unit that receives and processes image data captured by the terminal.

[1818] "Optical character recognition means" refers to a technology for extracting text from image data, and includes OCR (Optical Character Recognition) technology.

[1819] "Natural language processing means" refers to technology for analyzing extracted text and understanding its meaning, and includes NLP (Natural Language Processing) technology.

[1820] "Means for automatically generating questions" refers to technology that automatically generates questions for learners based on the analysis results.

[1821] A "database" is a digital storage system for storing and managing generated questions.

[1822] The "means for calculating review timing" is a technology that calculates the optimal review timing based on Ebbinghaus's forgetting curve.

[1823] The "means for sending notifications" refers to a technique for sending notifications to learners based on the optimal timing for review.

[1824] The "means for accepting answers" is an interface for receiving and processing the learner's answers as input.

[1825] "Means for generating feedback" are techniques for evaluating learners' answers and providing results and advice.

[1826] "Means for collecting emotional data" refers to technology that recognizes learners' emotions and extracts them as data, and includes facial recognition cameras and voice input devices.

[1827] "Means for adjusting the difficulty of questions and the timing of questions" refers to technology that individually adjusts the difficulty of questions and the timing of questions based on collected emotional data.

[1828] The "means for generating a feedback message" is a technique for creating a customized feedback message based on the learner's emotions.

[1829] This invention is a system that helps learners efficiently solidify their memories. It aims to improve the learning experience by incorporating an emotion engine. Learners can take pictures of textbooks and reference books, automatically generate questions based on the content, and provide reminders for review based on Ebbinghaus's forgetting curve. It can also recognize learners' emotions and adjust learning content and feedback based on those emotions.

[1830] System Overview

[1831] Image input of learning materials and question generation

[1832] Users use the camera on their smartphones, tablets, or other devices to take a photo of a page from a reference book or textbook. The captured image data is sent to a server via an application on the device. The server then uses optical character recognition (OCR) technology to extract text from the image. The extracted text is then analyzed using natural language processing (NLP) technology to generate questions and candidate answers based on the learning content. The generated questions are stored in a database so that users can access them later.

[1833] emotion recognition

[1834] While the user is solving the problems, emotional data is collected using a facial recognition camera and microphone. The emotional data is processed by the device and sent to the server. The server uses an emotion engine to recognize the learner's emotions from the received data. Based on the emotion recognition results, the difficulty of the questions and the timing of questions are adjusted. Feedback messages are also customized according to the emotions.

[1835] Setting a study schedule

[1836] The user sets learning goals and deadlines on the application. For example, they input the date of the next exam. The entered learning goals and deadline information is sent to the server. The server applies Ebbinghaus' forgetting curve model to calculate the optimal review timing. The calculated schedule is saved in a database and managed as user profile information.

[1837] Notification of review timing

[1838] The server periodically checks the learner's schedule based on the set review timing. When a review is required, it generates a notification and sends it to the learner's device. The device then displays the received notification message to the user. For example, the notification may say, "Here is today's review question: What is the speed of light in m / s?"

[1839] Review and feedback

[1840] The user confirms the notification, opens the application, and begins working on the review questions. The questions are displayed, and the learner enters and submits their answers. The user's answer data is sent to the server. The server analyzes the answer data, scores it, and determines whether it is correct or incorrect. It generates a feedback message, displaying "That's right!" The server also uses an emotion engine to provide feedback based on the user's emotions. It may also send encouraging messages such as "You look like you're doing well today, keep it up next time!" The server updates the learning history and recalculates the next review timing as necessary.

[1841] Specific examples

[1842] A user takes a photo of a page in a physics textbook about the "speed of light." The image data of this page is sent to the server. The server uses OCR technology to extract the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?", and saves it in a database. The user sets the exam date one month in the future. The server calculates the timing for review based on Ebbinghaus's forgetting curve. Each time the timing for review arrives, the server generates a notification and sends it to the device. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?" When the user enters and submits the correct answer, the server analyzes it and provides feedback such as "That's correct!" It also provides emotion-based feedback such as "Great! You seem motivated today." The server updates the learning history and recalculates the next timing for review.

[1843] Prompt Sentence Examples

[1844] "Generate a student question based on the following text: 'The molecular weight of water is 18 g / mol. The chemical formula for water is H2O. Water is essential for most life on Earth.'"

[1845] In this way, by incorporating emotion recognition technology, it is possible to maintain learners' motivation and maximize the effectiveness of review.

[1846] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1847] Step 1:

[1848] A user takes a photo of a page in a reference book or textbook using the camera on their smartphone or tablet. The input is the captured image data, and the output is that image data. Specifically, the user launches the device's camera app and takes a photo of the learning material page. At this time, the device temporarily saves the image data.

[1849] Step 2:

[1850] The device sends the captured image data to the server. The input is the image data stored on the device, and the output is the image data sent to the server. Specifically, the device's application uses the server data sending function to upload the image data to the server.

[1851] Step 3:

[1852] The server extracts text from the image data it receives using optical character recognition (OCR). The input is the image data sent to the server, and the output is the extracted text. Specifically, OCR software running on the server analyzes the image and recognizes it as text data.

[1853] Step 4:

[1854] The server analyzes the extracted text using natural language processing (NLP) technology and generates question sentences and candidate answers based on the learning content. The input is the extracted text, and the output is the generated question sentences and candidate answers. Specifically, the NLP algorithm analyzes the text on the server and converts it into an appropriate question format.

[1855] Step 5:

[1856] The generated question sentences and answer candidates are saved in a database on the server. The input is the generated question sentences and answer candidates, and the output is saved in the database. Specifically, the questions and answers are recorded in the database using a database management system on the server.

[1857] Step 6:

[1858] The user sets learning goals and deadlines on the application. The input is the learning goals and deadlines entered by the user, and the output is the transmission of that information from the terminal to the server. Specifically, the user enters learning goals and deadlines using the application's schedule setting function.

[1859] Step 7:

[1860] The server applies Ebbinghaus's forgetting curve model to calculate the optimal review timing. The input is learning goals and deadline information, and the output is the calculated review timing. Specifically, an algorithm running on the server predicts forgetting of the learning content and determines the optimal review time.

[1861] Step 8:

[1862] The server generates a notification when review is required based on the set review timing and sends it to the device. The input is the calculated review timing and the output is a notification message. Specifically, the server uses the notification system to send reminders to the learner at the set timing.

[1863] Step 9:

[1864] The notification message received by the terminal is displayed to the user. The input is the notification message sent from the server, and the output is the notification that the user sees on the display. Specifically, a popup message is displayed to the user using the terminal's notification function.

[1865] Step 10:

[1866] The user confirms the notification, opens the application, and begins working on the review questions. The input is the notified review question, and the output is the answer entered by the user. Specifically, the user enters the answer to the question displayed in the application and presses the submit button.

[1867] Step 11:

[1868] The terminal sends the user's answer data to the server. The input is the answer data entered by the user, and the output is the answer data sent to the server. Specifically, the terminal uploads the answer data to the server via the network.

[1869] Step 12:

[1870] The server analyzes the answer data and generates a feedback message. The input is the user's answer data, and the output is the generated feedback message. Specifically, an evaluation algorithm running on the server scores the answers and generates appropriate feedback.

[1871] Step 13:

[1872] The server uses an emotion engine to generate a feedback message based on the user's emotion and sends it to the device. The input is the user's emotion data, and the output is a customized feedback message. Specifically, the server creates a message to increase the learner's motivation based on the emotion recognition results.

[1873] Step 14:

[1874] The terminal displays a feedback message to the user. The input is the feedback message sent from the server, and the output is the feedback that the user sees on the display. Specifically, the feedback message pops up on the terminal screen.

[1875] Step 15:

[1876] The server updates the learning history and recalculates the next review timing as necessary. The input is the user's learning history data, and the output is the updated learning history and the recalculated review timing. Specifically, the server updates the database and sets a new review timing.

[1877] Through these processing steps, the system optimizes the user's learning experience and provides customized feedback according to their emotional state, maximizing learning effectiveness.

[1878] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1879] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1880] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1881] [Fourth embodiment]

[1882] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1883] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1884] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1885] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1886] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1887] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1888] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1889] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1890] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1891] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1892] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1893] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1894] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1895] This invention is a system for helping learners efficiently solidify their memories. In particular, it provides a function that takes pictures of reference books and textbooks and automatically notifies learners when it is time to review them based on the Ebbinghaus forgetting curve.

[1896] System Overview

[1897] Image input of learning materials and question generation

[1898] User

[1899] Learners use the camera on their smartphone or PC to take pictures of pages from reference books or textbooks.

[1900] Terminal

[1901] The captured image data is sent to the server via an application on the terminal.

[1902] server

[1903] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image.

[1904] The extracted text is interpreted and questions and candidate answers are generated using natural language processing (NLP) techniques.

[1905] The generated questions are stored in a database and saved for later access by the user.

[1906] Setting a study schedule

[1907] User

[1908] Learners log in to the application and set their own learning goals and deadlines.

[1909] For example, if you set an exam to take place at the end of the month, that date information will be entered.

[1910] server

[1911] The server applies Ebbinghaus' forgetting curve model based on the input learning goals and deadlines to calculate the optimal timing for review.

[1912] The calculated schedule is stored in a database and saved as user profile information.

[1913] Notification of review timing

[1914] server

[1915] Regularly check learners' schedules based on set review timings.

[1916] When review is required, the server generates a notification and sends it to the learner's device.

[1917] Terminal

[1918] The terminal displays the received notification message to the user.

[1919] For example, you might be notified, "Here's today's review question: What is the speed of light in m / s?"

[1920] Review and feedback

[1921] User

[1922] The learner checks the notification, opens the terminal application, and works on the review questions.

[1923] A question is displayed and the learner enters and submits the answer.

[1924] Terminal

[1925] The user's answer data is sent to the server.

[1926] server

[1927] The server analyzes the received answers and determines whether they are correct or incorrect.

[1928] Generate a feedback message, such as "Correct! We'll review this in a week."

[1929] The user's learning history information is updated and the next review timing is recalculated as necessary.

[1930] Specific examples

[1931] 1. A user takes a photo of a page about the "speed of light" in a physics textbook.

[1932] 2. The device sends the image data to the server.

[1933] 3. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?" and saves it in the database.

[1934] 4. The user sets the exam date one month in the future.

[1935] 5. The server sets the review timing based on Ebbinghaus's forgetting curve.

[1936] 6. When the first review time comes (for example, one day later), the server generates a notification and sends it to the device.

[1937] 7. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?"

[1938] 8. The user enters the answer and submits it.

[1939] 9. The server analyzes the answer and generates feedback to the user saying "That's right!"

[1940] 10. The server recalculates the next review timing and updates the learning history.

[1941] In this way, learners can efficiently review and solidify their memories.

[1942] The processing flow will be explained below.

[1943] Step 1:

[1944] The user launches the application and takes a photo of a page from a reference book or textbook.

[1945] Step 2:

[1946] The terminal transmits the captured image data to the server.

[1947] Step 3:

[1948] The server receives the transmitted image data and uses optical character recognition (OCR) technology to extract text from the image.

[1949] Step 4:

[1950] The server analyzes the extracted text using natural language processing (NLP) technology and generates questions from the text based on the learning content.

[1951] Step 5:

[1952] The server stores the generated questions and answer candidates in a database.

[1953] Step 6:

[1954] Users log in to the application and set their learning goals and deadlines.

[1955] Step 7:

[1956] The terminal transmits the user's setting information to the server.

[1957] Step 8:

[1958] The server applies Ebbinghaus's forgetting curve model to calculate the optimal timing for review based on learning goals and deadlines.

[1959] Step 9:

[1960] The server stores the calculated review timing in a database.

[1961] Step 10:

[1962] The server periodically checks the learner's schedule based on the review timing.

[1963] Step 11:

[1964] The server generates a notification when review is necessary and sends it to the learner's device.

[1965] Step 12:

[1966] The device displays a notification to the learner.

[1967] Step 13:

[1968] The user sees the notification, opens the application and begins working on the review questions.

[1969] Step 14:

[1970] The user enters answers to the review questions and presses the submit button.

[1971] Step 15:

[1972] The terminal transmits the user's answer data to the server.

[1973] Step 16:

[1974] The server receives the answer data, scores it, and determines whether it is correct or incorrect.

[1975] Step 17:

[1976] The server generates and sends a feedback message to the user.

[1977] Step 18:

[1978] The server updates the user's learning history information and recalculates the next review timing as needed.

[1979] Step 19:

[1980] The terminal displays a feedback message to the user.

[1981] This series of steps allows learners to review the material efficiently and solidify their memory.

[1982] Example 1

[1983] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1984] Conventional learning systems lacked efficient methods for learners to solidify their knowledge. Furthermore, they were unable to properly schedule review sessions, preventing the learner from maximizing their learning effectiveness. Furthermore, manually generating and managing questions was time-consuming and burdensome for learners. The purpose of this invention is to provide a system that solves these problems and enables learners to acquire knowledge efficiently.

[1985] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1986] In this invention, the server includes optical character recognition means, natural language processing means, means for automatically generating practice questions, data storage means, means for calculating review timing, means for sending notifications at optimal review timings, means for accepting answers from learners, means for evaluating answers and generating feedback, means for periodically checking the review schedule based on the calculated review timings, means for generating questions and answer candidates using a generative AI model with natural language processing technology, and means for displaying notification messages on the learner's computer device, thereby enabling learners to efficiently review and effectively solidify their memories.

[1987] A "learner" is an individual or group whose purpose is to acquire the learning content.

[1988] "Reference books and textbooks" are printed or electronic media used by learners for the purpose of acquiring knowledge.

[1989] A "computer device" is an electronic device for processing, storing, and communicating data, and includes personal computers, smartphones, and the like.

[1990] An "information processing device" is a server or equivalent system for receiving and processing data over a network.

[1991] "Optical character recognition" is technology or software that extracts text from image data.

[1992] "Natural language processing" is technology or software for analyzing and understanding extracted text.

[1993] "Means for automatically generating practice questions" refers to technology or software that automatically generates questions for learners based on the results of natural language processing analysis.

[1994] The "data storage means" is a recording medium or database for storing generated questions and related data.

[1995] The "means for calculating the timing of review" is a technology or software that calculates the optimal time for review based on Ebbinghaus's forgetting curve.

[1996] The "means for sending notifications" refers to technology or software that sends notifications to learners based on the calculated review timing.

[1997] The "means for accepting learner's answers" is a system or software that receives and processes answer data from learners.

[1998] The "means for evaluating answers and generating feedback" refers to technology or software that analyzes the answers received from learners, evaluates their correctness, and generates appropriate feedback.

[1999] The "means for checking the review schedule" refers to technology or software that periodically checks the learner's schedule based on the calculated review timing.

[2000] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate questions and potential answers.

[2001] A "means for displaying a notification message" is a technique or software that generates and transmits a notification message to be displayed on a learner's computing device.

[2002] This invention relates to a system that helps learners efficiently solidify their memories. In particular, it has the function of taking pictures of reference books and textbooks and automatically notifying learners when it is time to review them based on the Ebbinghaus forgetting curve.

[2003] Image input of learning materials and question generation

[2004] User: The learner takes a photo of a page from a reference book or textbook using the camera on their smartphone or personal computer. This image data is stored on a computer device.

[2005] Terminal: The terminal transmits the captured image data to an information processing device (server) through an application in the computer device. For example, the image data is uploaded to the server using an application on a smartphone.

[2006] Server: The server uses optical character recognition (OCR) technology on the received image data to extract text. OCR technology such as Google Cloud Vision API or Tesseract is used. The extracted text is then analyzed using natural language processing (NLP) technology. A generative AI model (e.g., OpenAI's GPT-3) is used for this analysis, and the question and candidate answers are generated. The generated questions are saved in a data storage device.

[2007] Setting a study schedule

[2008] User: Learners log in to the application and set deadlines such as learning goals and exam dates. For example, learners can "set exam dates one month from now" within the application.

[2009] Server: The server applies Ebbinghaus's forgetting curve model based on the input learning goals and deadlines to calculate the optimal review timing. The calculated schedule is saved in a data storage device and managed as the user's profile information.

[2010] Notification of review timing

[2011] Server: The server periodically checks the learner's schedule based on the set review timing. When a review is required, it generates a notification and sends it to the user's computer device.

[2012] Terminal: The terminal displays the received notification message to the user. For example, using the smartphone's notification function, a message such as "Today's review question: What is the speed of light in m / s?" is displayed.

[2013] Review and feedback

[2014] User: The learner checks the notification, opens the application on their device, and begins working on the review questions. The questions are displayed, and they can enter and submit their answers immediately.

[2015] Terminal: The user's answer data is sent to the server.

[2016] Server: The server analyzes the received answers and determines whether they are correct or incorrect. Based on the analysis results, it generates a feedback message and sends it to the user. It also updates the user's learning history information and recalculates the next review timing if necessary.

[2017] Specific examples

[2018] 1. A user takes a photo of a page about the "speed of light" in a physics textbook.

[2019] 2. The device sends the image data to the server.

[2020] 3. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?" and saves it in the database.

[2021] 4. The user sets the exam date one month in the future.

[2022] 5. The server sets the review timing based on Ebbinghaus's forgetting curve.

[2023] 6. When the first review time comes (for example, one day later), the server generates a notification and sends it to the device.

[2024] 7. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?"

[2025] 8. The user enters the answer and submits it.

[2026] 9. The server analyzes the answer and generates feedback to the user saying "That's right!"

[2027] 10. The server recalculates the next review timing and updates the learning history.

[2028] Prompt Sentence Examples

[2029] 1. Extract text from images

[2030] Extract the text from this image.

[2031] 2. Problem statement generation

[2032] "Generate an appropriate question from this text. Create a question from the text 'The speed of light is 299,792,458 m / s.'"

[2033] 3. Calculating review timing

[2034] "Calculate the next time to review based on Ebbinghaus's forgetting curve. The first time to review is today."

[2035] This invention enables learners to efficiently review and effectively solidify their memories.

[2036] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2037] Step 1:

[2038] The user takes a photo of a page from a reference book or textbook using the camera on their smartphone or personal computer. The input is image data of the learning material, and the output is image data saved on the device.

[2039] Specific action: A student takes a photo of a page about the speed of light in a physics textbook using their smartphone.

[2040] Step 2:

[2041] The terminal sends the captured image data to the server. The input is the image data stored in the terminal, and the output is the image data sent to the server.

[2042] Specific operation: The captured image is uploaded to the server through the application, and the message "Sending to server" is displayed.

[2043] Step 3:

[2044] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image. The input is the image data sent to the server, and the output is the extracted text data. The OCR technology used is Google Cloud Vision API and Tesseract.

[2045] What it does: The text "The speed of light is 299,792,458 m / s" is extracted from the image.

[2046] Step 4:

[2047] The server uses natural language processing (NLP) techniques to analyze the extracted text. The input is the text data from the OCR, and the output is the analysis results. A generative AI model (e.g., OpenAI's GPT-3) is used for this analysis to generate the question and candidate answers.

[2048] Specific operation: From the text "The speed of light is 299,792,458 m / s," the question "What is the speed of light in m / s?" is generated.

[2049] Step 5:

[2050] The server stores the generated questions in a data storage means, and the input is the generated question data, and the output is the question data stored in the database.

[2051] Specific operation: The generated question "What is the speed of light in m / s?" is saved in the database.

[2052] Step 6:

[2053] A user logs in to the application and sets learning goals and deadlines such as exam dates. The input is the goal and deadline data set by the learner, and the output is the setting data sent to the server.

[2054] Specific behavior: The learner "sets the exam date one month in the future" within the application.

[2055] Step 7:

[2056] The server applies the Ebbinghaus forgetting curve model based on the input learning goals and deadlines to calculate the optimal review timing. The input is the learning goals and deadline data, and the output is the calculated review schedule.

[2057] Specific operation: The server calculates the review timing, such as "the next review is one day later, then three days later, then one week later," and records it in the profile.

[2058] Step 8:

[2059] The server periodically checks the learner's schedule based on the calculated review timing. The input is review schedule data, and the output is notification data.

[2060] What happens: The server checks the schedule and generates a notification such as "Today's review questions are here."

[2061] Step 9:

[2062] The terminal displays the received notification message to the user. The input is the notification data sent from the server, and the output is the notification message displayed to the user.

[2063] Specific behavior: A notification will appear on your device saying, "Here's today's review question: What is the speed of light in m / s?"

[2064] Step 10:

[2065] The user checks the notification, opens the application on the device, and works on the review questions. The input is the displayed question text, and the output is the answer data entered by the user.

[2066] Specific operation: The user enters "299,792,458" in response to the question "What is the speed of light in m / s?" and submits it.

[2067] Step 11:

[2068] The terminal transmits the user's answer data to the server. The input is the answer data entered by the user, and the output is the answer data transmitted to the server.

[2069] Specific behavior: The device will display "Sending answers."

[2070] Step 12:

[2071] The server analyzes the received answers and determines whether they are correct or incorrect. The input is the user's answer data, and the output is the evaluation result and a feedback message. The feedback message is generated based on the analysis result.

[2072] Specific operation: The server determines that the user's answer "299,792,458" is correct, generates feedback to the user saying "That's correct! We'll review it in a week," and sends it to the user.

[2073] Step 13:

[2074] The server updates the user's learning history information and recalculates the next review timing as necessary. The input is the updated learning history data, and the output is the recalculated review schedule.

[2075] Specific operation: The server recalculates the next review timing based on the learning history and records it in the profile.

[2076] (Application example 1)

[2077] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2078] Conventional learning support systems lack the ability to notify learners of effective review timing and automatic problem generation functions to help them efficiently solidify their memories. Furthermore, more efficient educational methods are required for supporting learning for workers in practical environments such as factories, but no systems currently exist to address this. Therefore, there is a need for a system that automatically generates questions from images in reference books and textbooks and notifies learners of review timing based on Ebbinghaus's forgetting curve, thereby promoting efficient learning and memory consolidation. It is also necessary to provide similar review support to workers in factories, thereby helping them master work procedures and improve safety.

[2079] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2080] In this invention, the server includes a terminal for a learner to take an image of a reference book or textbook, a means for receiving the image data taken by the terminal, an optical character recognition means for extracting text from the image data within the server, a natural language processing means for analyzing the extracted text, a means for automatically generating questions based on the analysis results, a database for storing the generated questions, a means for calculating the timing of review based on Ebbinghaus's forgetting curve, a means for sending a notification to the learner of the optimal timing for review, a means for accepting the learner's answers, a means for evaluating the answers and generating feedback, smart glasses for a worker to take an image of a work procedure manual, a means for receiving the image data taken by the smart glasses, and a means for calculating the timing of review based on Ebbinghaus's forgetting curve and notifying the worker. This allows learners and workers to efficiently review, solidify their memories, and master work procedures.

[2081] A "learner" is someone who photographs materials from reference books and textbooks and reviews them by solving automatically generated problems, with the aim of effective learning and memory consolidation.

[2082] A "terminal" is a device used by learners to take images of reference books and textbooks, specifically a smartphone or tablet.

[2083] A "server" is a central processing unit that receives image data sent from a terminal, processes and stores the data, calculates review timing, and so on.

[2084] "Optical character recognition means" is a technology that automatically extracts character information from image data, and examples of this include OCR (Optical Character Recognition) engines.

[2085] "Natural language processing means" is a technology that analyzes extracted text information, understands the context, and automatically generates questions. Specifically, NLP (Natural Language Processing) algorithms are used.

[2086] The "database" is a system for efficiently managing and storing generated questions and learners' answer data.

[2087] The "Ebbinghaus forgetting curve" is a theory that shows the decline in memory over time and is used to calculate the optimal timing for review.

[2088] "Notification means" refers to a system that notifies learners when they need to review, and specifically includes push notifications from applications and email notifications.

[2089] "Smart glasses" are a wearable device that allows workers to take images of work procedures and send the image data to a server.

[2090] The "means for evaluating answers and generating feedback" is a system that automatically evaluates answers entered by learners and provides feedback including whether the answers are correct or incorrect and when the next review should be done.

[2091] This invention is a system for enabling learners and workers to efficiently solidify their memories, and in particular provides a function for taking images of reference books, textbooks, and work procedure manuals, and automatically notifying them of the timing for review based on Ebbinghaus's forgetting curve.

[2092] System Overview

[2093] Image input of learning materials and question generation

[2094] 1. User (learner or worker)

[2095] Learners use smartphones or tablets, while workers use smart glasses to take photos of pages from reference books, textbooks, and work instructions.

[2096] 2. Terminal

[2097] Image data captured by learners is sent to a server via an application on the device (smartphone, tablet, smart glasses).

[2098] 3. Server

[2099] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image.

[2100] uses "pytesseract" to extract text information from image data using OCR.

[2101] The extracted text is then used to generate question statements and candidate answers using natural language processing (NLP) techniques, utilizing NLP algorithms such as "scikit-learn."

[2102] The generated questions are stored in a database and saved for later access by the user.

[2103] Setting a study schedule

[2104] 1. Users

[2105] Learners log in to the application and set their own learning goals and deadlines.

[2106] For example, if you set an exam to take place at the end of the month, that date information will be entered.

[2107] 2. Server

[2108] The server applies Ebbinghaus' forgetting curve model based on the input learning goals and deadlines to calculate the optimal timing for review.

[2109] The calculated schedule is stored in a database and saved as user profile information.

[2110] Notification of review timing

[2111] 1. Server

[2112] Regularly check the schedules of learners and workers based on the set review timings.

[2113] When review is required, the server generates a notification and sends it to the learner's or worker's terminal.

[2114] 2. Terminal

[2115] The terminal displays the received notification message to the user.

[2116] For example, you might be notified, "Here's today's review question: What is the speed of light in m / s?"

[2117] Review and feedback

[2118] 1. Users

[2119] The learner or worker checks the notification, opens the terminal application, and works on the review questions.

[2120] A question is displayed and the user enters and submits the answer.

[2121] 2. Terminal

[2122] The user's answer data is sent to the server.

[2123] 3. Server

[2124] The server analyzes the received answers and determines whether they are correct or incorrect.

[2125] Generate a feedback message, such as "Correct! We'll review this in a week."

[2126] The user's learning history information is updated and the next review timing is recalculated as necessary.

[2127] Specific Examples

[2128] For example, when a learner takes a photo of a page about the "speed of light" in a physics textbook, the device sends the image data to the server. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?", and stores it in a database. If the user sets the exam date one month in the future, the server sets a review timing based on Ebbinghaus's forgetting curve. When the first review timing arrives (for example, one day later), the server generates a notification and sends it to the device. The device notifies the user, "Here is today's review question: What is the speed of light in m / s?" When the user enters and submits the answer, the server analyzes the answer and generates feedback saying "That's correct!" and sends it to the user. The next review timing is recalculated, and the user's learning history is updated.

[2129] Prompt Sentence Examples

[2130] For example, use the following prompt for a generative AI model:

[2131] plaintext

[2132] We have scanned specific pages of the manual using OCR technology. Please generate questions based on the following text to help students review effectively:

[2133] The machine operation procedure is as follows: 1. Turn on the power. 2. Perform the initial setup. 3. Select the required operation on the operation panel.

[2134] In this way, the system helps learners and workers to efficiently review, solidify their memories, and master work procedures.

[2135] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2136] Step 1:

[2137] A learner or worker uses a device (smartphone, tablet, smart glasses) to take a photo of a page from a reference book, textbook, or work procedure manual.

[2138] Input: Images of reference books, textbooks, and work instructions

[2139] Output: Image data is saved on the device

[2140] Step 2:

[2141] The terminal transmits the captured image data to the server.

[2142] Input: Image data

[2143] Output: The server receives the image data.

[2144] Step 3:

[2145] The server uses optical character recognition (OCR) technology on the image data it receives to extract text from the image.

[2146] Input: Image data

[2147] Output: Extracted text data

[2148] Specific operation: The server uses "pytesseract" to analyze and extract text information from the image.

[2149] Step 4:

[2150] The server analyzes the extracted text data using natural language processing (NLP) methods to generate question statements and candidate answers.

[2151] Input: Text data

[2152] Output: Generated question and answer candidates

[2153] What it does: The server uses NLP algorithms such as scikit-learn to analyze the text and generate appropriate questions.

[2154] Step 5:

[2155] The server stores the generated questions and answer candidates in a database.

[2156] Input: Generated question and answer candidates

[2157] Output: Questions and answer candidates stored in the database

[2158] Step 6:

[2159] Learners and workers log in to the application and set learning goals and deadlines.

[2160] Input: Learning objectives and due dates

[2161] Output: Objectives and due dates data sent to the server

[2162] Step 7:

[2163] The server applies Ebbinghaus' forgetting curve based on the learning goals and deadlines entered and calculates the optimal timing for review.

[2164] Input: Learning objectives and due dates

[2165] Output: Calculated review timing

[2166] Specific operation: The server creates a review schedule using Ebbinghaus's forgetting curve model.

[2167] Step 8:

[2168] When it is time to review, the server generates a notification and sends it to the learner's or worker's terminal.

[2169] Input:Review timing

[2170] Output: Notification message sent to the terminal

[2171] Specific operation: The server sends a notification using a push notification system or email notification system.

[2172] Step 9:

[2173] The learner or worker checks the notification, opens the terminal application and works on the review questions.

[2174] Input: Notification message

[2175] Output: Answered questions

[2176] Specific behavior: The user receives a notification on their device, opens the application, and answers the question.

[2177] Step 10:

[2178] The terminal transmits the user's answer data to the server.

[2179] Input: Answer data

[2180] Output: Answer data sent to the server

[2181] Step 11:

[2182] The server analyzes the received answers and determines whether they are correct or incorrect.

[2183] Input: Answer data

[2184] Output: Analysis results and feedback messages

[2185] Specific behavior: The server determines whether the answer is correct or incorrect and generates feedback.

[2186] Step 12:

[2187] The server sends the generated feedback to the user's terminal, recalculates the next review timing, and updates the learning history.

[2188] Input: Feedback message

[2189] Output: Feedback sent to the user and an updated review schedule

[2190] Specific operation: The server creates a feedback message and sends it to the device. It also recalculates the next review timing based on the user's learning history.

[2191] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2192] This invention is a system for helping learners efficiently solidify their memories, and aims to improve the learning experience by incorporating an emotion engine. It takes images of reference books and textbooks, automatically generates questions based on their contents, and provides a function to notify users when to review based on Ebbinghaus's forgetting curve. It can also recognize the learner's emotions and adjust the learning content and feedback based on those emotions.

[2193] System Overview

[2194] Image input of learning materials and question generation

[2195] User

[2196] Learners use the camera on their smartphone or PC to take pictures of pages from reference books or textbooks.

[2197] Terminal

[2198] The captured image data is sent to the server via an application on the terminal.

[2199] server

[2200] The server uses optical character recognition (OCR) technology on the received image data to extract text from the image.

[2201] The extracted text is analyzed using natural language processing (NLP) technology to generate question statements and candidate answers based on the learning content.

[2202] The generated questions are stored in a database for later access by the user.

[2203] emotion recognition

[2204] User

[2205] While learners are solving problems, emotional data is collected using facial recognition cameras and microphones.

[2206] Terminal

[2207] The emotion data is processed by the terminal and transmitted to the server.

[2208] server

[2209] The server uses an emotion engine to recognize the learner's emotion from the received data.

[2210] Based on the emotion recognition results, the difficulty level and timing of questions are adjusted, and feedback messages are also customized according to the emotion.

[2211] Setting a study schedule

[2212] User

[2213] The application allows users to set learning goals and deadlines, for example, entering the date of an upcoming exam.

[2214] Terminal

[2215] The entered learning objectives and deadline information are sent to the server.

[2216] server

[2217] The server applies Ebbinghaus' forgetting curve model to calculate the optimal timing for review.

[2218] The calculated schedule is stored in a database and managed as user profile information.

[2219] Notification of review timing

[2220] server

[2221] Regularly check learners' schedules based on set review timings.

[2222] When review is necessary, a notification is generated and sent to the learner's device.

[2223] Terminal

[2224] The device will then display the received notification message to the user, for example, "Here's today's review question: What is the speed of light in m / s?"

[2225] Review and feedback

[2226] User

[2227] The learner sees the notification, opens the application and begins working on the review questions.

[2228] A question is displayed and the learner enters and submits the answer.

[2229] Terminal

[2230] The user's answer data is sent to the server.

[2231] server

[2232] The server analyzes the answer data, scores it, and determines whether it is correct or incorrect.

[2233] Generate a feedback message saying "Correct! We'll review this in a week."

[2234] Updates the user's learning history information and recalculates the next review timing as needed.

[2235] It also uses an emotion engine to generate feedback based on the user's emotions, sending encouraging messages such as "You look good today, keep it up next time!"

[2236] Specific examples

[2237] 1. A user takes a photo of a page about the "speed of light" in a physics textbook.

[2238] 2. The device sends the image data to the server.

[2239] 3. The server extracts the text "The speed of light is 299,792,458 m / s" from the image data, generates the question "What is the speed of light in m / s?" and saves it in the database.

[2240] 4. The user sets the exam date one month in the future.

[2241] 5. The server sets the review timing based on Ebbinghaus's forgetting curve.

[2242] 6. The server generates a notification each time the review time arrives and sends it to the device.

[2243] 7. The device notifies the user, "Here's today's review question: What is the speed of light in m / s?"

[2244] 8. The user enters the answer and submits it.

[2245] 9. The server analyzes the answer and generates feedback to the user saying "That's right!"

[2246] 10. The server also uses an emotion engine to provide feedback based on the user's emotions, such as "Great! You look motivated today."

[2247] 11. The server updates the learning history and recalculates the next review timing.

[2248] In this way, by incorporating emotion recognition technology, it is possible to maintain learners' motivation and maximize the effectiveness of review.

[2249] The processing flow will be explained below.

[2250] Step 1:

[2251] The user launches the application and takes a photo of a page from a reference book or textbook using the camera on their smartphone or PC.

[2252] Step 2:

[2253] The terminal transmits the captured image data to the server.

[2254] Step 3:

[2255] The server receives the transmitted image data and uses optical character recognition (OCR) technology to extract text from the image.

[2256] Step 4:

[2257] The server analyzes the extracted text using natural language processing (NLP) technology and generates questions and candidate answers based on the learning content from the text.

[2258] Step 5:

[2259] The server stores the generated questions and answer candidates in a database.

[2260] Step 6:

[2261] A user logs into the application and sets learning goals and deadlines, for example, entering an exam date.

[2262] Step 7:

[2263] The device sends the learning goals and deadline information set to the server.

[2264] Step 8:

[2265] The server applies Ebbinghaus' forgetting curve model to calculate the optimal time to review based on the set deadline.

[2266] Step 9:

[2267] The server stores the calculated review timing in a database.

[2268] Step 10:

[2269] While the user is solving the problem, emotional data is collected using a camera and microphone.

[2270] Step 11:

[2271] The device processes the emotion data and sends it to the server.

[2272] Step 12:

[2273] The server uses an emotion engine to recognize the learner's emotion from the received emotion data.

[2274] Step 13:

[2275] The server adjusts the difficulty level and timing of questions based on the emotion recognition results.

[2276] Step 14:

[2277] The server generates a notification according to the review timing and sends it to the learner's terminal.

[2278] Step 15:

[2279] The device displays a notification to the learner, for example, "Here's today's review question: What is the speed of light in m / s?"

[2280] Step 16:

[2281] The user sees the notification, opens the app, and begins reviewing the questions.

[2282] Step 17:

[2283] The user enters answers to the review questions and presses the submit button.

[2284] Step 18:

[2285] The terminal transmits the user's answer data to the server.

[2286] Step 19:

[2287] The server receives the answer data, scores the answers, and determines whether they are correct or incorrect.

[2288] Step 20:

[2289] The server generates a feedback message and creates feedback based on the learner's emotions along with the correct or incorrect result.

[2290] Step 21:

[2291] The server sends a feedback message to the learner's terminal.

[2292] Step 22:

[2293] The device displays feedback messages to the learner, such as "Correct! We'll review this in a week" or "Great! You look good today."

[2294] Step 23:

[2295] The server updates the learning history information and recalculates the next review timing as needed.

[2296] This detailed processing flow allows learners to utilize emotion recognition functions to review at the optimal time, helping to solidify their memories.

[2297] Example 2

[2298] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2299] Conventional learning support systems have the problem of reduced learning efficiency because they do not optimize the timing of learners' memory consolidation or review. Furthermore, they do not take learners' emotions into consideration when improving the learning experience, making it difficult to maintain their motivation. This makes it difficult to achieve long-term learning outcomes.

[2300] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2301] In this invention, the server includes optical character recognition means, natural language processing means, means for automatically generating questions, a database for storing the generated questions, means for collecting and transmitting emotional data of learners, means for analyzing the emotional data and adjusting the learning content and feedback, means for accepting answers from learners, and means for evaluating the answers and generating feedback, thereby enabling learners to review at the optimal time and to receive appropriate feedback according to their emotions.

[2302] "Terminal" means a device used by a learner to take images of learning materials and process and transmit other data.

[2303] "Server" is a computer system that receives data sent from a terminal and processes and analyzes it.

[2304] "Optical character recognition means" refers to a technical means for extracting text information from image data.

[2305] "Natural language processing means" refers to technical means for analyzing extracted text and understanding its meaning and structure.

[2306] "Means for automatically generating questions" refers to technical means for automatically generating questions for learners based on the analysis results.

[2307] "Database" refers to the information system that stores and manages generated questions and other learning-related data.

[2308] The "Ebbinghaus forgetting curve" is a theory that describes how human memories are forgotten over time.

[2309] The "means for calculating the timing of review" is a technical means for calculating the optimal timing of review based on Ebbinghaus's forgetting curve.

[2310] The "means for sending notifications" refers to the technical means for sending notifications to learners when it is time to review.

[2311] The "means for collecting and transmitting learner's emotional data" refers to a technical means for detecting the learner's emotions during learning and transmitting the data to a server.

[2312] "Means for analyzing emotional data and adjusting learning content and feedback" refers to technical means for analyzing collected emotional data and adjusting learning content and feedback based on the results.

[2313] "Means for accepting learner's answers" refers to an interface that allows learners to input answers and the technical means for receiving that data.

[2314] The "means for evaluating answers and generating feedback" refers to a technical means for evaluating a learner's answers and generating appropriate feedback based on the results.

[2315] This invention is a system that helps learners efficiently consolidate their memories, and aims to improve the learning experience by combining it with an emotion engine. The system is composed mainly of users (learners), terminals, and servers.

[2316] Image input of learning materials and question generation

[2317] User

[2318] A user can use a camera on a smartphone or computer to take a picture of a page from a reference book or textbook. For example, they can take a picture of a page about the "speed of light" that is part of a physics textbook.

[2319] Terminal

[2320] The captured image data is sent to the server via the device's application, where it is compressed and encrypted.

[2321] server

[2322] The server then uses optical character recognition (OCR) technology on the received image data to extract text from the image, using OCR tools such as Tesseract as a concrete example of the software used for this process.

[2323] The extracted text is analyzed using natural language processing (NLP) techniques, such as spaCy and NLTK, to generate a question and answer candidate based on the learning content.

[2324] The generated problems are stored in a database, and users can solve them later. The database uses a common database system (such as MySQL or PostgreSQL).

[2325] emotion recognition

[2326] User

[2327] While the user is solving the problem, emotion data is collected using facial recognition cameras and microphones, for example by analyzing the user's facial expressions and tone of voice.

[2328] Terminal

[2329] The emotion data is processed by the device and sent to a server, where the collected data is processed in real time or in batches.

[2330] server

[2331] The server uses an emotion engine to recognize the user's emotions from the received data, using Microsoft Azure's Emotion API and Amazon Rekognition.

[2332] The difficulty level and timing of questions are adjusted based on the emotion recognition results. Feedback messages are also customized based on the emotion. For example, a message such as "You seem to be doing well today. Let's keep it up next time!" is generated.

[2333] Setting a study schedule

[2334] User

[2335] The application allows users to set learning goals and deadlines, for example, entering the date of an upcoming exam.

[2336] Terminal

[2337] The entered learning objectives and deadline information are sent to the server.

[2338] server

[2339] Once the server receives the user's input, it applies the Ebbinghaus forgetting curve model to calculate the optimal timing for review. This calculation uses Python libraries such as Scikit-learn and SciPy.

[2340] The calculated schedule is stored in a database and managed as user profile information.

[2341] Notification of review timing

[2342] server

[2343] Based on the set review timing, the server periodically checks the database and generates a notification to the user when the time for review has come.

[2344] The generated notification is sent to the device, for example, "Here's a review question for today: What is the speed of light in m / s?"

[2345] Terminal

[2346] The device will display the received notification message to the user. The push notification function is used for the notification, and when the user taps the notification, the application will open and the user will be taken to the review screen.

[2347] Review and feedback

[2348] User

[2349] The user acknowledges the notification, opens the application, and completes the review questions, such as "What is the speed of light in m / s?", answering "299,792,458 m / s."

[2350] Terminal

[2351] The user's answer data is sent to the server, where it is formatted and encrypted before being sent.

[2352] server

[2353] The server analyzes the received answer data and determines whether it is correct or incorrect, using rule-based algorithms or simple string matching.

[2354] A feedback message is generated and sent to the user, such as "Correct! We'll review it again in a week."

[2355] It also uses an emotion engine to generate feedback based on the user's emotions, such as "Great! You look motivated today."

[2356] Example prompts for generative AI models

[2357] "From an image of the text in a physics textbook, extract the information that "the speed of light is 299,792,458 m / s" and generate questions based on this information."

[2358] "Based on the user's target date for the next exam, create an optimal review schedule based on the Ebbinghaus forgetting curve model."

[2359] "Use facial recognition data from the user to recognize emotions and generate feedback based on that."

[2360] This allows users to study efficiently and maximize their learning effectiveness by receiving personalized feedback based on their emotions.

[2361] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2362] Step 1:

[2363] The user takes a photo of the page of the educational material using the device's camera.

[2364] Specific operation: The user takes a photo of a page in a textbook or reference book using the camera on their smartphone or PC, making sure to focus the camera properly to capture a clear image.

[2365] Input: Teaching material page

[2366] Output: Captured image data

[2367] Step 2:

[2368] The terminal transmits the captured image data to the server.

[2369] Specific operation: An application installed on the device receives image data, compresses and encrypts it, and then sends it to a server using a secure communication protocol (e.g., HTTPS).

[2370] Input: Photographed image data

[2371] Output: Image data sent to the server

[2372] Step 3:

[2373] The server uses OCR technology to extract text from the image data.

[2374] Specific operation: The server uses an OCR tool such as Tesseract to extract text information from the received image data.

[2375] Input: Image data

[2376] Output: Extracted text data

[2377] Step 4:

[2378] The server uses NLP technology to analyze the extracted text and generate question statements and candidate answers.

[2379] Specific operation: The server uses NLP libraries such as spaCy and NLTK to analyze text data. Based on the analysis results, it generates questions and multiple answer candidates appropriate for the learning content.

[2380] Input: Extracted text data

[2381] Output: Generated question and answer candidates

[2382] Step 5:

[2383] The server stores the generated questions in a database.

[2384] Specific operation: The generated question statements and answer candidates are stored in a database system (e.g., MySQL, PostgreSQL) so that the user can access them later.

[2385] Input: Generated question and answer candidates

[2386] Output: Problem data stored in a database

[2387] Step 6:

[2388] Users set learning goals and deadlines in the application.

[2389] Specific operations: The user inputs the next exam date and study goals through the application interface.

[2390] Input: learning goals and exam dates

[2391] Output: Input learning objectives and test date data

[2392] Step 7:

[2393] The terminal transmits the input data to the server.

[2394] Specific operation: Collected learning objectives and test date data are sent to the server. At this time, the data is verified and sanitized to ensure secure transmission.

[2395] Input: Learning objectives and test date data

[2396] Output: Learning objectives and test date data sent to the server

[2397] Step 8:

[2398] The server calculates the timing of review using Ebbinghaus' forgetting curve model.

[2399] Specific operation: The server uses Python's Scikit-learn and SciPy to calculate the optimal timing for review based on Ebbinghaus' forgetting curve.

[2400] Input: Learning objectives and test date data

[2401] Output: Calculated review schedule

[2402] Step 9:

[2403] The server stores the calculated schedule in a database.

[2404] Specific operation: The calculated review schedule is saved in a database and managed as user profile information.

[2405] Input: Calculated review schedule

[2406] Output: Review schedule saved in the database

[2407] Step 10:

[2408] When the time for review arrives, the server generates a notification and sends it to the terminal.

[2409] Specific operation: The server periodically checks the user's study schedule, and when it is time to review, it generates a notification message and sends it to the terminal.

[2410] Input:Review Schedule

[2411] Output: The generated notification message

[2412] Step 11:

[2413] The terminal displays the received notification message to the user.

[2414] Specific operation: The device displays the received notification message to the user and notifies the user as a push notification or in-app notification.

[2415] Input: Notification message

[2416] Output: Notification message displayed to the user

[2417] Step 12:

[2418] The user checks the notification and works through the review questions.

[2419] Specific behavior: The user confirms the notification, opens the application, and answers the review questions. For example, the user answers "299,792,458 m / s" to the question "What is the speed of light in m / s?"

[2420] Input: Notification message

[2421] Output: User's answer data

[2422] Step 13:

[2423] The terminal transmits the user's answer data to the server.

[2424] Specific operation: After preprocessing the user's answer data, it is securely sent to the server.

[2425] Input: User's answer data

[2426] Output: Answer data sent to the server

[2427] Step 14:

[2428] The server analyzes the answer data, determines whether it is correct or incorrect, and generates feedback.

[2429] Specific operation: The server analyzes the received answer data and determines whether it is correct or incorrect. Based on the result of the analysis, it generates a feedback message and sends it to the user. For example, it could give feedback such as "That's right! We'll review it in a week." It can also generate emotion-based feedback using an emotion engine. For example, it could give feedback such as "Great! You look motivated today."

[2430] Input: User's answer data

[2431] Output: The generated feedback message

[2432] (Application example 2)

[2433] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2434] Conventional learning management systems do not take into account the individual emotions and motivation of learners. As a result, if a learner is presented with a difficult question when they are tired or have low concentration, their learning effectiveness and motivation decrease. Furthermore, while there are systems that determine the optimal review timing to maximize learning efficiency and provide notifications at that timing, there are few systems that adjust feedback based on emotion recognition. The purpose of this invention is to solve these problems and provide a learning experience optimized for each individual learner.

[2435] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting emotional data of the learner, means for adjusting the difficulty of questions and the timing of questions based on the collected emotional data, and means for generating feedback messages based on the emotions. This makes it possible to provide an optimal learning experience according to the learner's emotional state.

[2436] A "terminal" is a device used by learners to take images of reference books and textbooks, and includes smart...

Claims

1. A device for students to take pictures of reference books and textbooks, a server that receives image data captured by the terminal; optical character recognition means within said server for extracting text from image data; natural language processing means for analyzing the extracted text; means for automatically generating questions based on the analysis results; a database for storing the generated questions; A means for calculating the timing of review based on Ebbinghaus's forgetting curve; A means to notify learners at the optimal time for review; a means for accepting learner responses; The system includes means for evaluating answers and generating feedback.

2. 2. The system according to claim 1, further comprising means for inputting a target study period for the learner in order to set the review timing.

3. 2. The system according to claim 1, wherein said notification means includes means for notifying the terminal of the timing of review.

Images