System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving image data of an answer sheet; means for analyzing the received image data and converting the image data into text data; means for evaluating accuracy of an answer based on the converted text data and specifying an error portion; means for generating an appropriate explanation and guidance for the specified error portion; and means for providing the generated explanation and guidance to a recipient.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional learning methods have the problem of reducing learning efficiency because it takes time to provide feedback after grading tests and workbooks. Effective relearning is also difficult because test takers cannot immediately identify their mistakes and receive appropriate explanations and guidance. Furthermore, it is difficult to provide individual test takers with a dedicated learning plan, resulting in a lack of support to effectively overcome test takers' weaknesses. [Means for solving the problem]

[0005] The present invention provides a means for receiving image data of an answer sheet, analyzing the image data, and converting it into text data. It also includes a means for evaluating the accuracy of the answer based on the converted text data and identifying errors, and a means for generating appropriate explanations and guidance for the identified errors. The generated explanations and guidance are provided to the answerer in real time. The present invention also includes a means for responding to questions from the answerer in real time using voice recognition technology, and a means for preprocessing the image data and correcting tilt. The present invention also provides a means for automatically generating and presenting repractice questions corresponding to the errors, a means for analyzing the user's learning progress based on the received answer data and proposing an optimal learning plan, and a means for providing individually customized learning content by saving and analyzing past answer data, thereby significantly improving the answerer's learning efficiency and providing effective learning support.

[0006] An "answer sheet" is a paper medium on which a respondent writes their answers to questions in a test or workbook.

[0007] "Image data" refers to digital image data taken with a camera or scanner, such as an answer sheet.

[0008] "Text data" refers to data in the form of digitized character strings, such as characters or mathematical expressions, recognized from image data.

[0009] "Evaluating accuracy" means judging whether the answer is correct based on a standard.

[0010] "Mistakes" refer to incorrect parts of the answer or incorrect answers.

[0011] An "explanation" is an explanatory text that provides reasons and evidence for whether the answer is correct or incorrect, and helps with understanding.

[0012] "Guidance" means providing specific advice and guidance on how to improve answers and study methods.

[0013] "Real-time" refers to the fact that the processing is immediate, with almost no delay between the time the user submits the answer data and the time feedback is returned.

[0014] "Voice recognition technology" is a technology that analyzes human voice as a digital signal and converts it into text data.

[0015] "Preprocessing" refers to processing image data in advance before analysis or recognition, specifically, correcting tilt or trimming.

[0016] "Refresher questions" are additional practice questions provided to help test takers improve their mistakes or areas of weakness.

[0017] "Learning progress" refers to the degree of learning progress and achievement within a certain period of time.

[0018] A "learning plan" is a systematic learning policy that includes the selection of learning materials and a learning schedule optimized for each individual learner.

[0019] "Customized learning content" refers to learning materials and exercises that are individually tailored to each learner's weaknesses and learning needs.

[0020] "An answerer" refers to a person who answers a test or question set, and is primarily a student or learner. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] This invention is an efficient learning support system that receives image data of answer sheets, analyzes and understands the data, generates feedback, and provides it to test takers in real time. This system significantly improves test takers' learning efficiency and can provide appropriate guidance without being restricted by time or place.

[0043] Server-side behavior

[0044] The server plays a central role in the system and performs the following functions:

[0045] 1. Receiving answer data

[0046] The server receives the image data of the answer sheet sent by the contestant (user) through the terminal.

[0047] Example: Taking a photo of a math test answer sheet with your smartphone and sending it via an app.

[0048] 2. Image data preprocessing

[0049] The server performs image pre-processing such as tilt correction and cropping.

[0050] For example, if the edges of the answer sheet are cut, make appropriate corrections so that the whole sheet can be analyzed.

[0051] 3. OCR Processing and Analysis

[0052] The server uses OCR technology to recognize characters and formulas from image data and convert them into text data.

[0053] Example: Recognize "2x + 3 = 7" using OCR and convert it to text format.

[0054] 4. Marking answers

[0055] The server analyzes the converted text data and determines whether the answer is correct or incorrect.

[0056] Example: If the respondent answers "x = 3", it is determined that this is incorrect.

[0057] 5. Identifying and analyzing errors

[0058] The server analyzes the incorrect answers to determine at which step the error occurred.

[0059] Example: Identifying where parentheses were incorrectly used during a calculation.

[0060] 6. Feedback Generation

[0061] The server generates appropriate text-based explanations and instructions, including the correct answers.

[0062] Example: Generate an explanation such as "You have processed the parentheses incorrectly, so please check the parentheses rules again."

[0063] 7. Presenting Retraining Questions

[0064] The server automatically generates relearning questions related to the incorrect answers and presents them to the test taker.

[0065] Example: Provide the respondent with a "similar equation problem" again.

[0066] Operation on the terminal side

[0067] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[0068] 1. Sending answer data

[0069] The terminal transmits the image data of the answer sheet photographed by the user to the server.

[0070] Example: Uploading an image of the answer sheet through the application.

[0071] 2. Receiving and Viewing Feedback

[0072] The terminal receives the feedback sent from the server and displays it on the user interface.

[0073] Example: The app displays the message "An error occurred during the calculation process."

[0074] User behavior

[0075] The user enters and submits answer data, checks feedback, and re-studies.

[0076] 1. Answers to the questions

[0077] Users write their answers to tests and questions on paper.

[0078] Example: Handwrite the answer to the math problem "2x + 3 = 7."

[0079] 2. Sending the answer data

[0080] The user takes a photo of the answer sheet with their device and sends it to the server.

[0081] Example: Take a photo of your answer sheet with your smartphone camera and send it via the app.

[0082] 3. Review feedback and relearn

[0083] The user checks the feedback displayed on the device and understands the mistakes and explanations.

[0084] For example, read the feedback on your mistakes in the app and then try the same type of relearning exercise.

[0085] In this way, the present invention realizes a system that analyzes answers, provides feedback, and supports re-learning in real time through the cooperation of the server, terminal, and user. This system allows test takers to receive immediate feedback and quickly correct errors, greatly improving learning efficiency.

[0086] The processing flow will be explained below.

[0087] Step 1:

[0088] Users write their answers on a paper test or workbook, and then use their smartphone camera to take a photo of the answer sheet.

[0089] Step 2:

[0090] After taking a picture of the answer sheet, the user launches the dedicated application and selects the captured image. When ready, the image data is sent from the device to the server.

[0091] Step 3:

[0092] The device performs preprocessing on the received image data, including correcting the tilt of the answer sheet and cropping it so that the answer area is clearly visible.

[0093] Step 4:

[0094] The device sends the pre-processed image data to the server via the API, and the server receives the image data sent from the device.

[0095] Step 5:

[0096] The server analyzes the received image data using OCR technology, which also includes handwriting recognition technology, to convert characters and mathematical expressions in the image into text data.

[0097] Step 6:

[0098] The server runs the converted text data through a scoring algorithm to evaluate the accuracy of the answer, thereby determining whether the answer is correct or incorrect.

[0099] Step 7:

[0100] The server analyzes the incorrect answers to identify where the error occurred, for example, by identifying calculation errors or errors between steps in the solution.

[0101] Step 8:

[0102] Based on the identified mistakes, the server generates a correct solution and explanation, including procedural and theoretical explanations.

[0103] Step 9:

[0104] The server generates feedback and provides it to the user, including explanatory notes, reference materials, and even refresher questions.

[0105] Step 10:

[0106] To facilitate re-learning, the server analyzes the user's past answer data and generates practice questions that are optimally tailored to each individual user. These practice questions are designed to reinforce the user's weak points.

[0107] Step 11:

[0108] The device receives the feedback data sent from the server and displays it on the user interface. The feedback includes whether the answer was correct or incorrect, the cause of the error, and advice for relearning.

[0109] Step 12:

[0110] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then works on the re-learning questions to deepen their understanding.

[0111] Step 13:

[0112] The user can ask additional questions as needed through the application, and the device uses voice recognition to convert the user's questions into text data and send it to the server.

[0113] Step 14:

[0114] The server analyzes the user's question and generates an appropriate answer, which is provided in real time in voice or text format.

[0115] Step 15:

[0116] The device displays or plays aloud the received answers to the user to help them understand, and the user can use this information to further their learning.

[0117] Through this series of processes, the system provides users with real-time feedback and re-learning support, improving learning efficiency and comprehension.

[0118] Example 1

[0119] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0120] In conventional learning support systems, the processing of image data of answer sheets after receiving them is too manual, making it difficult to provide efficient guidance in real time. Furthermore, there are issues with the system, such as poor identification of the causes of errors and insufficient provision of re-study questions, which reduces the learner's learning efficiency.

[0121] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0122] In this invention, the server includes means for receiving image data of the answer sheet, means for preprocessing the received image data and correcting tilt and cropping, means for converting the image-processed data into text data using OCR technology, means for evaluating the accuracy of the answers based on the converted text data and identifying errors, means for generating appropriate explanations and guidance for the identified errors using a generative AI model, means for providing the generated explanations and guidance to the recipient, and means for presenting related relearning questions, which enable real-time and efficient analysis of the answer sheet, feedback on errors, and the presentation of appropriate relearning questions.

[0123] "Answer Sheet" means the paper document on which a respondent completes a question.

[0124] "Image data" refers to a file that digitally represents the answer sheet, usually as a photograph or scanned image.

[0125] "Preprocessing" refers to processes such as tilt correction and cropping that are performed to improve the quality of image data.

[0126] "OCR technology" is an abbreviation for "Optical Character Recognition," and is a technology that recognizes characters and mathematical formulas from image data and converts them into text data.

[0127] "Text data" means digital data containing textual information extracted from an image using OCR technology.

[0128] "Means for assessing accuracy" refers to algorithms or programs used to determine whether a respondent's answer is correct.

[0129] "Incorrect part" refers to the part of the answer given by the respondent that is not correct.

[0130] A "generative AI model" is an artificial intelligence model trained using large amounts of data, and refers to a technology that generates sentences and explanations based on specific conditions.

[0131] "Explanation and Guidance" means explanations or advice provided for specific mistakes.

[0132] A "re-study question" is a new question presented to the respondent related to the part where the respondent made a mistake, with the aim of relearning.

[0133] MODE FOR CARRYING OUT THE INVENTION

[0134] This invention is an efficient learning support system that receives image data of answer sheets, analyzes and understands the data, generates feedback, and provides it to answerers in real time. The specific operations of each component, the server, the terminal, and the user, are described in detail below.

[0135] Server-side behavior

[0136] The server plays a central role in the system, performing a series of processes from receiving image data of answer sheets to generating feedback and presenting re-study questions.

[0137] 1. Receiving answer data

[0138] The server receives the image data of the answer sheet from the contestant (user). At this time, it uses an HTTP POST request and saves the uploaded image data. As a framework, Flask or Django can be used.

[0139] 2. Image data preprocessing

[0140] The server uses image processing libraries such as OpenCV to perform preprocessing such as tilt correction and cropping, improving the image quality and making it easier to analyze. For example, the server uses cv2.getRotationMatrix2D and cv2.warpAffine to correct the tilt of the image.

[0141] 3. OCR Processing and Analysis

[0142] The server uses OCR technology such as Tesseract to recognize characters and formulas from the image data and convert them into text data. For example, it uses pytesseract.image_to_string to convert image data into text data.

[0143] 4. Marking answers

[0144] The server uses the converted text data to evaluate the accuracy of the answer, analyzing the answer using a mathematical library such as SymPy. For example, it uses sympy.solve to determine the user's answer.

[0145] 5. Identifying and analyzing errors

[0146] The server identifies where the error occurred and analyzes its cause. It uses the debug library to see at which step of the solution the error occurred, for example by using sympy.simplify to evaluate intermediate results.

[0147] 6. Feedback Generation

[0148] The server uses a generative AI model (e.g., GPT-4) to generate appropriate feedback, including details of the mistake and an explanation of the correct answer. For example, it generates an explanation such as, "You made a mistake in handling the parentheses, so please check the parentheses rules again."

[0149] 7. Presenting Retraining Questions

[0150] The server generates relevant relearning problems and presents them to the solver. For example, it uses a generative AI model to generate similar equation problems such as "x + 5 = 12."

[0151] Operation on the terminal side

[0152] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[0153] 1. Sending answer data

[0154] The device sends the image data of the answer sheet taken by the user to the server. At this time, by using React Native or Flutter, it is possible to easily send an HTTP POST request.

[0155] 2. Receiving and Viewing Feedback

[0156] The device receives the feedback data sent from the server and displays it in a user interface. For example, a React Native app uses fetch to retrieve the feedback and display it on the screen.

[0157] User behavior

[0158] The user enters and submits answer data, checks feedback, and re-studies.

[0159] 1. Answers to the questions

[0160] Users write their answers on a paper-based question sheet. For example, they write their answers to the math problem "2x + 3 = 7" on paper.

[0161] 2. Sending the answer data

[0162] Users take a photo of their answer sheet with their device and send it to the server, or they use their smartphone camera to upload the photo of their answer sheet through a dedicated app.

[0163] 3. Review feedback and relearn

[0164] The user checks the feedback displayed on the device, understands the error, reads the explanation displayed through the application, and then tackles the presented re-learning problem.

[0165] Examples of prompt statements

[0166] Prompt example 1 (preprocessing of image)

[0167] "Generate Python code to skew and crop the answer sheet."

[0168] Prompt sentence example 2 (OCR processing)

[0169] "Write a code to convert mathematical formulas from PDF to text using Tesseract"

[0170] Prompt example 3 (scoring system)

[0171] "Create a Python script that uses SymPy to determine the solution to an equation."

[0172] In this way, the present invention allows the server, terminal, and user to work together to efficiently analyze image data on answer sheets and provide immediate feedback, significantly improving the learner's learning efficiency and enabling them to receive appropriate guidance regardless of time or place.

[0173] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0174] Step 1:

[0175] Sending answer data (terminal side)

[0176] Input: Image data of the answer sheet taken by the user with a smartphone camera

[0177] How it works: The device sends the image data of the answer sheet to the server using an HTTP POST request. Applications using React Native or Flutter can be used. For example, the image of the answer sheet can be selected on the device and sent by pressing the upload button.

[0178] Output: The image data is uploaded to the server.

[0179] Step 2:

[0180] Receiving answer data (server side)

[0181] Input: Image data of the answer sheet sent from the device

[0182] How it works: The server receives an HTTP POST request and saves the uploaded image data. You can use a framework like Flask or Django, and retrieve the file using the request.files['file'] method.

[0183] Output: Image data stored on the server

[0184] Step 3:

[0185] Image data preprocessing (server side)

[0186] Input: Saved image data of answer sheet

[0187] Operation: Uses an image processing library such as OpenCV to perform preprocessing such as tilt correction and cropping. Specifically, calculates the rotation matrix using cv2.getRotationMatrix2D and corrects the image using cv2.warpAffine. This improves image quality and makes it easier to analyze.

[0188] Output: Preprocessed image data

[0189] Step 4:

[0190] OCR processing and analysis (server side)

[0191] Input: Preprocessed image data

[0192] How it works: Using OCR technology such as Tesseract, it recognizes characters and formulas from image data and converts them into text data. Specifically, it extracts the text data using the pytesseract.image_to_string method.

[0193] Output: Extracted text data

[0194] Step 5:

[0195] Answer scoring (server side)

[0196] Input: Extracted text data

[0197] How it works: It uses mathematical libraries such as SymPy to evaluate the accuracy of the answer. Specifically, it uses sympy.solve to solve the equation and compares it with the user's answer to determine whether it is correct or incorrect.

[0198] Output: Correct / incorrect result

[0199] Step 6:

[0200] Identifying and analyzing errors (server side)

[0201] Input: Correct / incorrect result

[0202] How it works: Using the debug library and SymPy functions, we analyze at which step of the solution process the error occurred. Specifically, we identify the cause of the error by evaluating intermediate results using sympy.simplify.

[0203] Output: Detailed information about the error

[0204] Step 7:

[0205] Feedback generation (server side)

[0206] Input:Detailed information about the error

[0207] How it works: Using a generative AI model (e.g., GPT-4), it generates a detailed explanation of the error and a correct answer. For example, it generates feedback such as, "You made a mistake with the parentheses. Please check the parentheses rules again."

[0208] Output: Text data of generated feedback

[0209] Step 8:

[0210] Presenting retraining questions (server side)

[0211] Input:Detailed information about the error

[0212] How it works: It uses a generative AI model to automatically generate related retraining problems, such as similar equation problems like "x + 5 = 12."

[0213] Output: Generated retraining problem

[0214] Step 9:

[0215] Receiving and displaying feedback (device side)

[0216] Input: Feedback and retraining questions sent by the server

[0217] How it works: The device receives the feedback data using an HTTP GET request and displays it in the user interface. If it's a React Native app, it uses the fetch method to get the feedback and display it on the screen.

[0218] Output: Feedback and relearning questions displayed in the user interface

[0219] Step 10:

[0220] Reviewing feedback and relearning (user side)

[0221] Input: Feedback and relearning questions displayed on the device

[0222] How it works: The user reviews the displayed feedback, understands the explanation, and then works on the presented re-learning problem. For example, they can learn by following the "Review parentheses processing" within the application.

[0223] Output: Understanding the error and completing the re-learning

[0224] (Application example 1)

[0225] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0226] Conventional learning support systems have the problem that analyzing answers and generating feedback takes time, making it difficult to provide real-time support. Furthermore, bookstores and stores selling learning materials have limited ways for purchasers to receive on-site learning support. This means that they are unable to receive immediate explanations or guidance, and it can take time to resolve concerns or questions. Furthermore, there is a need for a system that can support efficient learning without being tied to a specific location or time.

[0227] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0228] In this invention, the server includes means for receiving image data of the answer sheet, means for analyzing the received image data and converting it into text data, means for evaluating the accuracy of the answers based on the converted text data and identifying errors, means for generating appropriate explanations and guidance for the identified errors, means for providing the generated explanations and guidance to the recipient, means for transmitting image data of the photographed or uploaded answer sheet to the server, means for providing a user interface for displaying feedback in real time, and devices for providing learning support in commercial facilities and means for using the same, which enable immediate analysis of answers and feedback, making learning support possible, particularly in bookstores and stores that sell learning materials.

[0229] "Answer Sheet" means a medium containing answers to a set of questions in paper or electronic format.

[0230] "Image data" is data containing visual information stored in digital form.

[0231] The "means for receiving" is a method or device for taking in data from the outside.

[0232] An "analyzing means" is a method or device for processing data and understanding its content.

[0233] "Text data" is data stored as character information.

[0234] A "means for converting" is a method or device for changing data from one format to another.

[0235] A "means for assessing accuracy" is a method or device that checks whether the information entered is accurate.

[0236] The "means for identifying the error portion" is a method or device for detecting the error portion.

[0237] A "means for generating appropriate explanations and guidance" is a method or device for creating correct solutions and study advice.

[0238] The "means for providing" refers to a method or device for delivering the generated information or data to the user.

[0239] "Means for taking pictures" refers to a method or device for taking pictures or videos.

[0240] A "server" is a computer system on a network that processes and stores data.

[0241] An "uploading means" is a method or apparatus for transmitting data from a local device to a remote server.

[0242] A "user interface" is an interface through which a user interacts with a system.

[0243] "Real-time display means" refers to a method or device for immediately displaying generated data or information to a user.

[0244] A "commercial establishment" is a public or private place that offers goods or services.

[0245] "Equipment for providing learning support" is hardware or software used to aid in learning.

[0246] "Method of use" refers to a procedure or method for achieving a specific purpose.

[0247] The present invention is an efficient learning support system that receives image data of answer sheets, analyzes and understands the data, and generates and provides feedback. Specific embodiments of the system are described below.

[0248] Server-side behavior

[0249] The server plays a central role in the system and performs the following functions:

[0250] 1. Receiving answer data

[0251] The server receives image data of the answer sheet sent by the test taker via the device. For example, a user takes a photo of the answer sheet for a math test with their smartphone and sends the data to the server via an application.

[0252] 2. Image data preprocessing

[0253] The server performs image preprocessing such as deskewing and cropping. For example, if the edges of an answer sheet are cut off, it will be properly corrected so that the entire sheet can be analyzed.

[0254] 3. OCR Processing and Analysis

[0255] The server uses OCR (Optical Character Recognition) technology to recognize characters and formulas from image data and convert them into text data. Here, we use "pytesseract" for OCR. As an example, OCR recognizes "2x + 3 = 7" and converts it into text format.

[0256] 4. Marking answers

[0257] The server analyzes the converted text data and determines whether the answer is correct or incorrect. For example, if the answerer answers "x = 3," the server determines that this is incorrect.

[0258] 5. Identifying and analyzing errors

[0259] The server analyzes the incorrect answer to determine at which step the error occurred, for example by identifying the incorrect parentheses in the calculation process.

[0260] 6. Feedback Generation

[0261] The server generates appropriate explanations and guidance in text format, including the correct answer. For example, it generates an explanation such as "You made a mistake in handling the parentheses. Please check the parentheses rules again."

[0262] 7. Presenting Retraining Questions

[0263] The server automatically generates relearning questions related to the mistakes and presents them to the test taker. For example, it provides the test taker with "similar equation problems."

[0264] Operation on the terminal side

[0265] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[0266] 1. Sending answer data

[0267] The device sends the image data of the answer sheet taken by the user to the server. For example, the image of the answer sheet is uploaded through an application.

[0268] 2. Receiving and Viewing Feedback

[0269] The device receives the feedback sent from the server and displays it in the user interface. For example, the app might display "An error occurred during the calculation."

[0270] User behavior

[0271] The user enters and submits answer data, checks feedback, and re-studies.

[0272] 1. Answers to the questions

[0273] Users write answers to tests and workbooks on paper, for example, by handwriting the answer to the math problem "2x + 3 = 7."

[0274] 2. Sending the answer data

[0275] The user takes a photo of the answer sheet with their device and sends it to the server. For example, they can take a photo of the answer sheet with their smartphone camera and send it using the app.

[0276] 3. Review feedback and relearn

[0277] The user checks the feedback displayed on the device and understands the mistakes and explanations. For example, the user reads the feedback on the mistakes in the app and then works on the same type of relearning problem.

[0278] In this way, a system has been realized that links the server, terminals, and users to analyze answers in real time, provide feedback, and support re-learning. This system makes it possible to provide learning support in bookstores and stores that sell learning materials, and users can receive immediate feedback, significantly improving their learning efficiency.

[0279] Prompt Sentence Examples

[0280] Here is an example of a prompt that the user can use to submit their answer data:

[0281] Please upload the image data of your answer sheet:

[0282] The correct answer to this math problem is "x = 2." What was your answer to this problem? Take a photo of your answer sheet with your smartphone and upload it to this app.

[0283] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0284] Step 1:

[0285] The user takes a picture of the answer sheet with their smartphone.

[0286] Input: Handwritten answer sheet

[0287] Output: Image data of the answer sheet

[0288] Step 2:

[0289] The device displays the captured image data on the application and asks the user to confirm the upload.

[0290] Input: Image data of answer sheet

[0291] Output: Upload confirmation message

[0292] Step 3:

[0293] The user taps the "Upload" button on the application to send the image data of the answer sheet to the server.

[0294] Input: image data of answer sheet, user operation

[0295] Output: Image data sent to the server

[0296] Step 4:

[0297] The server preprocesses the received image data, correcting for skew and cropping, and uses OpenCV to reshape the image.

[0298] Input: Image data of answer sheet

[0299] Output: Preprocessed image data

[0300] Step 5:

[0301] The server performs OCR processing on the preprocessed image data using pytesseract and converts it into text data.

[0302] Input: Preprocessed image data

[0303] Output: Converted text data

[0304] Step 6:

[0305] The server analyzes the converted text data and compares it with the correct answer to evaluate the accuracy of the answer.

[0306] Input: converted text data, correct answer data

[0307] Output: Determine whether the answer is correct or incorrect

[0308] Step 7:

[0309] The server identifies the error and analyzes the step that caused the error.

[0310] Input: Answer accuracy evaluation result

[0311] Output: Specific information about the error

[0312] Step 8:

[0313] The server generates appropriate explanations and guidance based on the mistakes, using a generative AI model to generate the feedback content.

[0314] Input: Specific information about the mistake, correct answer data

[0315] Output: Explanatory and instructional text

[0316] Step 9:

[0317] The server sends feedback to the user's device, including a description of the error and an appropriate explanation.

[0318] Input: Explanation and instructional text

[0319] Output: Feedback sent to the user's device

[0320] Step 10:

[0321] The terminal displays the received feedback on a user interface.

[0322] Input: Feedback data from the server

[0323] Output: Feedback displayed in the user interface

[0324] Step 11:

[0325] The user reviews and understands the displayed feedback and, if necessary, works through the provided refresher questions.

[0326] Input: Displayed feedback

[0327] Output: Improve the user's learning behavior and perform re-learning.

[0328] This series of processes provides efficient learning support in real time, from sending answer data to receiving feedback.

[0329] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0330] This invention is an efficient learning support system that receives image data of answer sheets, analyzes the data, generates feedback, and provides it to test takers in real time. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more appropriate instruction can be provided. This system significantly improves test takers' learning efficiency and can provide appropriate instruction regardless of time or place.

[0331] Server-side behavior

[0332] The server plays a central role in the system and performs the following functions:

[0333] 1. Receiving answer data

[0334] The server receives the image data of the answer sheet sent by the contestant (user) through the terminal.

[0335] Example: Taking a photo of a math test answer sheet with your smartphone and sending it via an app.

[0336] 2. Image data preprocessing

[0337] The server preprocesses the image data of the answer sheet, correcting tilt and cropping it.

[0338] Example: If the edges of the answer sheet are cut, make the appropriate corrections so that the answer can be analyzed.

[0339] 3. OCR Processing and Analysis

[0340] The server uses OCR technology to recognize characters and formulas from image data and convert them into text data.

[0341] Example: Recognizing handwritten "2x + 3 = 7" and converting it to text form.

[0342] 4. Marking answers

[0343] The server runs the converted text data through a scoring algorithm to assess the accuracy of the answer.

[0344] Example: If the answer is "x = 3", determine that this is incorrect.

[0345] 5. Identifying and analyzing errors

[0346] The server analyzes the incorrect answer to determine where the mistake occurred and identifies the mistake.

[0347] Example: Identifying calculation errors or incorrect steps in a solution.

[0348] 6. Feedback Generation

[0349] The server generates appropriate explanations and instructions, including the correct answers.

[0350] Example: Generate an explanation such as "There was an error in the calculation process, so please check the calculation rules again."

[0351] 7. Presenting Retraining Questions

[0352] The server automatically generates relearning questions related to the error and presents them to the solver.

[0353] Example: Provide the respondent with a "similar equation problem" again.

[0354] 8. How the Emotion Engine Works

[0355] The server uses an emotion engine to analyze the user's facial expressions and tone of voice to recognize their emotional state.

[0356] For example, using a camera and microphone to monitor a user's facial expressions and tone of voice in real time.

[0357] 9. Use of Emotional Data

[0358] Based on the recognized emotional data, the server evaluates the user's stress level and motivation, and adjusts the learning guidance and feedback content.

[0359] Example: If the user is feeling stressed, offer encouraging comments or suggest a break.

[0360] Operation on the terminal side

[0361] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[0362] 1. Sending answer data

[0363] The terminal transmits the image data of the answer sheet photographed by the user to the server.

[0364] Example: Uploading an image of the answer sheet through the application.

[0365] 2. Receiving and Viewing Feedback

[0366] The terminal receives the feedback sent from the server and displays it on the user interface.

[0367] Example: The app displays advice stating that "there was an error in the calculation process."

[0368] 3. Collecting Emotional Data

[0369] The device uses a camera and microphone to collect the user's facial expressions and voice in real time and transmits them to a server.

[0370] Example: Capture the user's face with a camera and use that data for analysis.

[0371] User behavior

[0372] Users input and submit answer data, check and re-study feedback, and provide emotion data.

[0373] 1. Answers to the questions

[0374] Users write their answers to tests and questions on paper.

[0375] Example: Handwrite the answer to the math problem "2x + 3 = 7."

[0376] 2. Sending the answer data

[0377] The user takes a photo of the answer sheet with their device and sends it to the server.

[0378] Example: Take a photo of your answer sheet with your smartphone camera and send it via the app.

[0379] 3. Review feedback and relearn

[0380] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then works on the re-learning questions.

[0381] For example, read the feedback on your mistakes in the app and try the same type of relearning exercise.

[0382] 4. Providing Emotion Data

[0383] Users use the device's camera and microphone to provide emotional data to the system through facial expressions and voice.

[0384] For example, you turn to the device's camera so that the emotion engine can analyze your facial expressions and tone of voice while solving a problem.

[0385] In this way, the present invention realizes a system that analyzes answers in real time, provides feedback, and supports relearning through collaboration between the server, terminal, and user. Furthermore, the introduction of an emotion engine enables individualized instruction based on the user's emotional state, further improving learning efficiency and providing psychological support.

[0386] The processing flow will be explained below.

[0387] Step 1:

[0388] Users write their answers on a paper test or workbook, and then use their smartphone camera to take a photo of the answer sheet.

[0389] Step 2:

[0390] After taking a picture of the answer sheet, the user launches the dedicated application and selects the captured image. When ready, the image data is sent from the device to the server.

[0391] Step 3:

[0392] The device performs preprocessing on the received image data, including correcting the tilt of the answer sheet and cropping it so that the answer area is clearly visible.

[0393] Step 4:

[0394] The device sends the pre-processed image data to the server via the API, and the server receives the image data sent from the device.

[0395] Step 5:

[0396] The server analyzes the received image data using OCR technology, which also includes handwriting recognition technology, to convert characters and mathematical expressions in the image into text data.

[0397] Step 6:

[0398] The server runs the converted text data through a scoring algorithm to evaluate the accuracy of the answer, thereby determining whether the answer is correct or incorrect.

[0399] Step 7:

[0400] The server analyzes the incorrect answers to identify where the error occurred, for example, by identifying calculation errors or errors between steps in the solution.

[0401] Step 8:

[0402] Based on the identified mistakes, the server generates a correct solution and explanation, including procedural and theoretical explanations.

[0403] Step 9:

[0404] The device receives the feedback data sent from the server and displays it on the user interface. The feedback includes whether the answer was correct or incorrect, the cause of the error, and advice for relearning.

[0405] Step 10:

[0406] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then attempts to re-learn the problems presented by the server.

[0407] Step 11:

[0408] The device uses a camera and microphone to capture the user's facial expressions and voice tone, and transmits them to a server in real time, thereby monitoring their emotional state.

[0409] Step 12:

[0410] The server uses an emotion engine to analyze the user's emotional state from their facial expressions and tone of voice, thereby determining their stress level, motivation, etc.

[0411] Step 13:

[0412] The server adjusts the learning guidance and feedback based on the user's emotional state. For example, if the user is in a high-stress state, it will suggest encouraging messages or a break.

[0413] Step 14:

[0414] The server then sends tailored feedback and instruction to the device, including emotionally tailored content.

[0415] Step 15:

[0416] The device displays the feedback and guidance sent from the server on the user interface, allowing the user to receive support tailored to their own emotional state.

[0417] Through this series of processes, the system provides users with real-time feedback and support for relearning, and furthermore, the emotion engine enables individualized instruction tailored to the user's psychological state.

[0418] Example 2

[0419] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0420] Conventional learning support systems lacked the accuracy of receiving and analyzing image data from answer sheets, resulting in problems such as not being able to provide appropriate feedback to learners. Furthermore, there was also the issue that instruction did not take into account the learner's emotional state, preventing the effectiveness of learning from being maximized. This resulted in a decrease in learners' learning efficiency and made it difficult to provide appropriate individual instruction.

[0421] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0422] In this invention, the server includes means for receiving image data of the answer sheet, means for preprocessing the received image data to correct skew and trim it, means for analyzing the preprocessed image data, recognizing characters and formulas and converting them into text data, means for identifying incorrect answers and generating appropriate explanations and guidance, means for providing the generated explanations and guidance to the recipient, means for analyzing the user's facial expressions and voice to recognize their emotional state, means for adjusting the feedback content based on the recognized emotional data, and means for generating and presenting relearning questions related to the mistakes. This not only improves the accuracy of answers but also enables individual guidance that corresponds to the learner's emotional state, thereby improving learning efficiency and providing enhanced psychological support.

[0423] "Image data of the answer sheet" refers to data that has been photographed or captured with a camera or scanner of the answers written by the user on paper and converted into digital format.

[0424] "Means for receiving" refers to a combination of hardware and software for capturing image data of the answer sheet to the server via a network.

[0425] The "pre-processing means for tilt correction and trimming" is a combination of hardware and software that uses image processing technology on the image data of the answer sheet to correct the tilt and cut out unnecessary parts.

[0426] The "means for analyzing, recognizing characters and mathematical formulas, and converting them into text data" refers to a combination of hardware and software that uses OCR technology to extract characters and mathematical formulas from the image data of the answer sheet and convert them into text format.

[0427] The "means for identifying incorrect answers and generating appropriate explanations and guidance" is a combination of hardware and software that applies evaluation algorithms to the converted text data, identifies incorrect answers, and generates appropriate explanations and guidance using natural language processing techniques.

[0428] The "means for providing the generated commentary and instruction to the recipient" is a combination of hardware and software for transmitting the generated commentary and instruction to the user's terminal via a network and displaying it on a user interface.

[0429] The "means for recognizing the emotional state by analyzing the user's facial expression and voice" is a combination of hardware and software that uses a camera and microphone to collect data on the user's facial expression and voice, and analyzes it using an emotion recognition algorithm.

[0430] The "means for adjusting the feedback content based on the recognized emotional data" is a combination of hardware and software for evaluating the user's emotional state based on the analysis results and appropriately changing the feedback and instruction content as necessary.

[0431] The "means for generating and presenting relearning questions related to the error portion" is a combination of hardware and software for generating new questions related to the identified error portion and presenting them to the user.

[0432] MODE FOR CARRYING OUT THE INVENTION

[0433] The present invention is an efficient learning support system that receives image data of answer sheets, analyzes the data, generates feedback, and provides it to test takers in real time. Furthermore, by using an emotion engine, it is possible to provide appropriate guidance based on the user's emotional state. This system improves the test taker's learning efficiency and can provide appropriate guidance regardless of time or location. A specific embodiment of this system is described below.

[0434] Server-side behavior

[0435] 1. Receiving answer data

[0436] The server has a means to receive image data of answer sheets sent by users from their devices. The hardware used is the server for receiving data, and the software is the data transfer protocol. For example, a user takes a photo of a math test answer sheet using a smartphone app and sends it to the server. The server saves the sent image file in a specific directory and returns a response confirming receipt.

[0437] 2. Image data preprocessing

[0438] The server has the means to preprocess the image data it receives. The software used is the OpenCV library, which corrects the image's skew and crops out unnecessary parts. For example, if the edges of an answer sheet are cut off, this is corrected so that the image can be analyzed.

[0439] 3. OCR Processing and Analysis

[0440] The server uses OCR technology to recognize characters and mathematical formulas from image data and convert them into text data. The software used is the Tesseract OCR library. For example, it recognizes handwritten text such as "2x + 3 = 7" and converts it into text format. The converted text data is then stored in a database.

[0441] 4. Marking answers

[0442] The server has a means to evaluate the converted text data by running it through a scoring algorithm. The scoring algorithm is executed by a dedicated program. For example, if the answer is "x = 3," it will determine that this is incorrect, generate an evaluation result, and save it in JSON format.

[0443] 5. Identifying and analyzing errors

[0444] The server has a means to identify the incorrect answers and analyze their causes. It uses an error analysis algorithm to identify the causes of the incorrect answers by matching them with specific error patterns. For example, it identifies calculation errors or incorrect steps in the solution.

[0445] 6. Feedback Generation

[0446] The server has the means to generate appropriate explanations and instructional feedback. The technology used is natural language processing (NLP) using a generative AI model (e.g., GPT-3). For example, it generates an explanation such as, "There was an error in the calculation process. Please check the calculation rules again."

[0447] 7. Presenting Retraining Questions

[0448] The server has a means to automatically generate relearning questions related to the mistake and present them to the solver. It uses a problem generation algorithm. For example, it generates "similar equation problems" and provides them to the solver.

[0449] 8. How the Emotion Engine Works

[0450] The server has a means to analyze the user's facial expressions and voice to recognize their emotional state. It uses emotion recognition software (e.g., Emotion API) and uses a camera and microphone to analyze the user's facial expressions and tone of voice in real time.

[0451] 9. Use of Emotional Data

[0452] The server has a means to adjust the feedback content based on the emotional data. The feedback and instruction content is appropriately changed based on the analysis results. For example, if the user is feeling stressed, an encouraging comment is added.

[0453] Operation on the terminal side

[0454] 1. Sending answer data

[0455] The device has a means to send image data of the answer sheet taken by the user to the server. The hardware used is a smartphone, and the software is a dedicated application. For example, the user can take a photo of the answer sheet with the smartphone camera and send it to the server via the application.

[0456] 2. Receiving and Viewing Feedback

[0457] The device has a means to receive and display feedback from the server. This is done using a UI component within the application. For example, the app might display feedback indicating that an error occurred in the calculation process.

[0458] 3. Collecting Emotional Data

[0459] The device has a means for collecting the user's facial expressions and voice in real time using a camera and microphone and sending the data to a server. For example, the device captures the user's face with a camera and sends the data to a server.

[0460] User behavior

[0461] 1. Answers to the questions

[0462] Users write answers to tests and workbooks on paper, for example, by handwriting the answer to the math problem "2x + 3 = 7."

[0463] 2. Sending the answer data

[0464] The user takes a photo of the answer sheet with their device and sends it to the server. The device used is a smartphone, and a dedicated app is used. For example, the answer sheet is photographed with the smartphone camera and sent via the app.

[0465] 3. Review feedback and relearn

[0466] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then attempts to re-test the questions. For example, the user reads the feedback on the mistakes made on the app and attempts to re-test the questions.

[0467] 4. Providing Emotion Data

[0468] Users can provide emotional data through facial expressions and voice using the device's camera and microphone. For example, when solving a problem, they answer in front of the camera, and their voice is also recorded.

[0469] Examples of prompt statements

[0470] example:

[0471] "Take a picture of your 2x + 3 = 7 answer sheet and submit it. Then, check your answer and receive feedback to try again if you made a mistake. Our emotion engine will also analyze your emotional state to see if you're feeling stressed."

[0472] This system works by linking the server, terminals, and users to analyze answers in real time, provide feedback, and support re-learning.The introduction of an emotion engine enables individualized instruction based on the user's emotional state, improving learning efficiency and providing psychological support.

[0473] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0474] Step 1:

[0475] Sending answer data

[0476] The terminal transmits image data of the answer sheet photographed by the user to the server.

[0477] Input: An image file of the answer sheet taken by the user with their smartphone

[0478] Output: Notification of completion of image data upload to the server

[0479] Specific operation: Using the dedicated application on the device, the user takes a photo of the answer sheet and taps the send button to upload the image to the server. After the upload is complete, the device displays a notification that the upload was successful.

[0480] Step 2:

[0481] Receiving answer data

[0482] The server receives the image data of the answer sheet sent from the terminal.

[0483] Input: Image data from the device

[0484] Output: Save image data and confirm receipt response

[0485] Specific operation: The server receives the image data of the answer sheet via an HTTP request, saves the image file in a specific directory, and returns a success message to the terminal after completion of the reception.

[0486] Step 3:

[0487] Image data preprocessing

[0488] The server performs pre-processing on the received image data.

[0489] Input: Saved image data

[0490] Output: Deskewed and cropped image data

[0491] How it works: The server uses the OpenCV library to straighten the image and crop unnecessary parts. The processed image data is then re-saved in a format suitable for analysis.

[0492] Step 4:

[0493] OCR Processing and Analysis

[0494] The server analyzes the image data using OCR technology and converts it into text data.

[0495] Input: Preprocessed image data

[0496] Output: Parsed text data

[0497] How it works: The server uses the Tesseract OCR library to extract characters and formulas from the image data and convert them into text format, which is then stored in a database.

[0498] Step 5:

[0499] Grading answers

[0500] The server evaluates the text data using a scoring algorithm.

[0501] Input: Text data generated by OCR processing

[0502] Output: Scoring result data (correct / incorrect)

[0503] How it works: The server runs a dedicated scoring algorithm, compares the text data with the correct answers, and generates a score, which is saved in JSON format.

[0504] Step 6:

[0505] Identifying and analyzing errors

[0506] The server identifies and analyzes the incorrect answers.

[0507] Input: Grade result data

[0508] Output: Analysis data of the error location and its cause

[0509] Specific operation: The server runs an error analysis algorithm to identify the incorrect answers and analyze the causes (calculation errors, incorrect solutions, etc.). The analysis results are stored in a database.

[0510] Step 7:

[0511] Feedback Generation

[0512] The server generates appropriate commentary and instructional feedback.

[0513] Input: Analysis data of the error location and cause

[0514] Output: The generated feedback text

[0515] Specific operation: The server uses a generative AI model (e.g., GPT-3) to generate feedback text based on the analyzed data. The generated feedback is stored in a database.

[0516] Step 8:

[0517] Presenting re-learning questions

[0518] The server generates and presents relearning questions related to the error location.

[0519] Input: Analysis data of the error

[0520] Output: Retraining problem data

[0521] Specific operation: The server uses a problem generation algorithm to generate new retraining problems based on the identified mistakes. The generated problems are stored in a database.

[0522] Step 9:

[0523] Emotion Engine Operation

[0524] The server analyzes the user's facial expression and voice data to recognize their emotional state.

[0525] Input: Facial expression and voice data sent from the device

[0526] Output: Recognized emotion data

[0527] How it works: The server uses emotion recognition software to analyze the transmitted data and recognize the user's emotional state (e.g., stress, excitement, concentration). The analysis results are stored in a database.

[0528] Step 10:

[0529] Use of Emotional Data

[0530] The server adjusts the feedback content based on the recognized emotion data.

[0531] Input: Recognized emotion data

[0532] Output: Adjusted feedback text

[0533] Specific operation: The server refers to the emotion data and adjusts the feedback content appropriately according to the user's emotional state. For example, if the user is feeling stressed, it adds encouraging comments. The adjusted feedback is stored in a database and sent to the device.

[0534] (Application example 2)

[0535] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0536] While conventional learning support systems focus on evaluating answers and providing feedback, they do not adequately provide individualized instruction tailored to the learner's emotional state. This makes it difficult to provide appropriate instruction tailored to the learner's motivation and level of understanding, resulting in limited improvements in learning efficiency. Furthermore, while real-time feedback and appropriate instruction tailored to the learner's emotional state are required in training scenarios such as factories, current systems have difficulty achieving this.

[0537] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving image data of an answer sheet, means for analyzing the received image data and converting it into text data, means for evaluating the accuracy of the answer based on the converted text data and identifying errors, means for generating appropriate explanations and guidance for the identified errors, means for providing the generated explanations and guidance to the recipient, means for identifying the user's emotional state via a camera and a voice input device, and means for adjusting the feedback content based on the identified emotional state. This allows individualized instruction and appropriate feedback to be provided in real time according to the learner's emotional state, improving learning efficiency. Furthermore, similar effects can be expected in training scenarios such as factories, contributing to the efficiency of employee skill acquisition.

[0538] An "answer sheet" is a paper sheet on which answers are written by hand and which contains questions and answers.

[0539] "Image data" refers to digital data of the answer sheet obtained using a device such as a camera or scanner.

[0540] "Means for receiving" refers to an interface or function for importing image data into a system such as a server.

[0541] The "means for analyzing and converting into text data" is a function that uses optical character recognition (OCR) technology to analyze image data and convert characters and mathematical formulas into text format.

[0542] "Means for assessing the accuracy of answers and identifying errors" refers to algorithms or methods for determining whether answers are correct or incorrect based on text data and for identifying incorrect parts of the answers.

[0543] The "means for generating appropriate explanations and guidance" is a function that automatically generates the correct answer, its explanation, and further guidance for incorrect answers.

[0544] The "means for providing" refers to an interface or function for notifying the user of the generated feedback and instruction content.

[0545] "Camera and audio input device" refers to a hardware device for capturing the user's facial expressions and voice.

[0546] The "means for identifying emotional state" refers to algorithms or software that analyze captured facial and voice data to interpret the user's emotions.

[0547] The "means for adjusting the feedback content" is a function that dynamically changes and adjusts the content of the feedback provided, taking into account the identified emotional state.

[0548] This invention is a learning support system that receives image data of answer sheets, analyzes them, generates feedback, and provides it to users in real time. It also has the ability to identify the user's emotional state using a camera and a voice input device and adjust the feedback content accordingly. The system aims to improve learning and training efficiency.

[0549] Server side behavior:

[0550] The server plays a central role in the system and performs multiple processes to achieve the following functions:

[0551] 1. Receiving answer data: The server receives image data of the answer sheet sent from a terminal equipped with image data transmission functionality. This data can also be used for things like training checklists in factories.

[0552] 2. Image data preprocessing: The server preprocesses the image data of the answer sheet, correcting tilt and removing noise. For example, it uses an image processing library such as OpenCV.

[0553] 3. OCR processing and analysis: The server uses optical character recognition (OCR) technology to convert handwritten characters and mathematical formulas from the image data into text data. The pytesseract library assists in this process.

[0554] 4. Answer scoring: Based on the text data, the scoring algorithm evaluates the accuracy of the answer and identifies incorrect parts. The scoring algorithm compares the answer with predefined correct answer data.

[0555] 5. Feedback generation: The server generates appropriate explanations and instruction methods based on the incorrect answers. For example, if there is a calculation error, the server generates feedback that explains the details.

[0556] 6. Operation of the Emotion Engine: The server analyzes facial images and voice data sent by the user to identify the user's emotional state. By using an emotion analysis library, the stress level and motivation of employees and learners are evaluated.

[0557] 7. Feedback Adjustment: Dynamically adjust the feedback content based on the emotional state identified by the emotion engine, thereby providing guidance that is optimal for the user's psychological state.

[0558] 8. Presenting Rehearsal Questions: The server automatically generates rehearsal questions related to incorrect answers and presents them to the user, helping to correct misunderstandings and deepen understanding.

[0559] On the device side:

[0560] The terminal is responsible for sending input from the user to the server and for receiving and displaying feedback.

[0561] 1. Sending answer data: The terminal sends the image data of the answer sheet taken by the user using the camera to the server.

[0562] 2. Receiving and displaying feedback: The terminal receives the feedback sent by the server and displays it on the user interface, allowing the user to identify their mistakes and find ways to improve.

[0563] 3. Emotion data collection: The device uses a camera and microphone to collect the user's facial expressions and voice in real time and transmits them to the server.

[0564] User behavior:

[0565] The user inputs and submits answer data, checks and re-learns feedback, and provides emotion data.

[0566] 1. Answering questions: The user writes down the answers to the training checklist or questions on paper.

[0567] 2. Sending answer data: The user takes a photo of their handwritten answer on their device and sends it to the server.

[0568] 3. Checking feedback and relearning: The user checks the feedback displayed on the device, understands the mistakes and explanations, and works on relearning questions to solidify their understanding.

[0569] 4. Providing emotional data: The user provides emotional data through facial expressions and voice using the device's camera and microphone.

[0570] This system allows users to study and train efficiently and receive appropriate feedback based on their emotional state.It can also be used in factories and other workplaces to provide appropriate training that takes into account the emotional state of employees, thereby improving their skills and managing stress.

[0571] Examples:

[0572] Below are some example prompts for a generative AI model:

[0573] Build an application that analyzes a user's checklist image and provides feedback. This is a cloud-based learning support system that recognizes the user's emotions from video and audio feeds and tailors the feedback. Specifically, it uses OCR technology to recognize handwritten text, generates feedback based on the response data, and tailors the feedback content through sentiment analysis.

[0574] This prompt conveys specific system requirements and functions to the generative AI model, providing appropriate learning assistance to the user.

[0575] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0576] Step 1:

[0577] The server receives the image data of the answer sheet sent from the terminal. The image data of the answer sheet is received as input and imported into the server. The received data is stored in the internal memory and passed to the next pre-processing step.

[0578] Step 2:

[0579] The server preprocesses the received image data. It uses the received image data as input and performs deskew and noise removal. Specifically, it uses the OpenCV library to convert the image to grayscale and threshold it. The result is a binary image that can be analyzed.

[0580] Step 3:

[0581] The server performs OCR processing on the preprocessed image data to generate text data. Using the preprocessed binary image as input, it reads characters and formulas using the pytesseract library and outputs text data. This allows the text format answer data to be extracted from the handwritten answer sheet.

[0582] Step 4:

[0583] The server evaluates the accuracy of the answers based on the extracted text data. It uses the generated text data as input and compares it with a pre-defined database of correct answers. Specifically, an algorithm determines whether each answer is correct and identifies any errors. The output is a result of the accuracy assessment and data indicating the errors.

[0584] Step 5:

[0585] The server generates appropriate explanations and guidance based on the mistakes. Using the evaluation results and data on the mistakes as input, it automatically generates the explanations using a generative AI model. The output is text data of the explanations and guidance.

[0586] Step 6:

[0587] The server provides the generated explanations and instructions to the terminal, using the text data of the explanations and instructions as input and transferring the data to the terminal, which receives it and displays it on the user interface.

[0588] Step 7:

[0589] The device collects the user's emotional data and sends it to the server. It captures the user's facial expressions and voice in real time using a camera and microphone. It receives facial image data and voice data as input and sends them to the server.

[0590] Step 8:

[0591] The server uses an emotion engine to identify the user's emotional state. It analyzes facial image and voice data received as input to evaluate the user's emotional state (e.g., stress level and motivation). The identified emotion data is obtained as output.

[0592] Step 9:

[0593] The server adjusts the feedback content based on the identified emotional data. Using the emotional data and the generated commentary and instruction data as input, the server changes and adjusts the feedback content according to the emotional state. The adjusted feedback text data is obtained as output.

[0594] Step 10:

[0595] The server sends the adjusted feedback back to the terminal and provides it to the user. The server uses the adjusted feedback text data as input and sends the data to the terminal. The terminal receives the data and displays it on the user interface. This allows the user to receive feedback optimized for their emotional state.

[0596] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0597] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0598] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0599] [Second embodiment]

[0600] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0601] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0602] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0603] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0604] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0605] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0606] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0607] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0608] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0609] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0610] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0611] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0612] This invention is an efficient learning support system that receives image data of answer sheets, analyzes and understands the data, generates feedback, and provides it to test takers in real time. This system significantly improves test takers' learning efficiency and can provide appropriate guidance without being restricted by time or place.

[0613] Server-side behavior

[0614] The server plays a central role in the system and performs the following functions:

[0615] 1. Receiving answer data

[0616] The server receives the image data of the answer sheet sent by the contestant (user) through the terminal.

[0617] Example: Taking a photo of a math test answer sheet with your smartphone and sending it via an app.

[0618] 2. Image data preprocessing

[0619] The server performs image pre-processing such as tilt correction and cropping.

[0620] For example, if the edges of the answer sheet are cut, make appropriate corrections so that the whole sheet can be analyzed.

[0621] 3. OCR Processing and Analysis

[0622] The server uses OCR technology to recognize characters and formulas from image data and convert them into text data.

[0623] Example: Recognize "2x + 3 = 7" using OCR and convert it to text format.

[0624] 4. Marking answers

[0625] The server analyzes the converted text data and determines whether the answer is correct or incorrect.

[0626] Example: If the respondent answers "x = 3", it is determined that this is incorrect.

[0627] 5. Identifying and analyzing errors

[0628] The server analyzes the incorrect answers to determine at which step the error occurred.

[0629] Example: Identifying where parentheses were incorrectly used during a calculation.

[0630] 6. Feedback Generation

[0631] The server generates appropriate text-based explanations and instructions, including the correct answers.

[0632] Example: Generate an explanation such as "You have processed the parentheses incorrectly, so please check the parentheses rules again."

[0633] 7. Presenting Retraining Questions

[0634] The server automatically generates relearning questions related to the incorrect answers and presents them to the test taker.

[0635] Example: Provide the respondent with a "similar equation problem" again.

[0636] Operation on the terminal side

[0637] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[0638] 1. Sending answer data

[0639] The terminal transmits the image data of the answer sheet photographed by the user to the server.

[0640] Example: Uploading an image of the answer sheet through the application.

[0641] 2. Receiving and Viewing Feedback

[0642] The terminal receives the feedback sent from the server and displays it on the user interface.

[0643] Example: The app displays the message "An error occurred during the calculation process."

[0644] User behavior

[0645] The user enters and submits answer data, checks feedback, and re-studies.

[0646] 1. Answers to the questions

[0647] Users write their answers to tests and questions on paper.

[0648] Example: Handwrite the answer to the math problem "2x + 3 = 7."

[0649] 2. Sending the answer data

[0650] The user takes a photo of the answer sheet with their device and sends it to the server.

[0651] Example: Take a photo of your answer sheet with your smartphone camera and send it via the app.

[0652] 3. Review feedback and relearn

[0653] The user checks the feedback displayed on the device and understands the mistakes and explanations.

[0654] For example, read the feedback on your mistakes in the app and then try the same type of relearning exercise.

[0655] In this way, the present invention realizes a system that analyzes answers, provides feedback, and supports re-learning in real time through the cooperation of the server, terminal, and user. This system allows test takers to receive immediate feedback and quickly correct errors, greatly improving learning efficiency.

[0656] The processing flow will be explained below.

[0657] Step 1:

[0658] Users write their answers on a paper test or workbook, and then use their smartphone camera to take a photo of the answer sheet.

[0659] Step 2:

[0660] After taking a picture of the answer sheet, the user launches the dedicated application and selects the captured image. When ready, the image data is sent from the device to the server.

[0661] Step 3:

[0662] The device performs preprocessing on the received image data, including correcting the tilt of the answer sheet and cropping it so that the answer area is clearly visible.

[0663] Step 4:

[0664] The device sends the pre-processed image data to the server via the API, and the server receives the image data sent from the device.

[0665] Step 5:

[0666] The server analyzes the received image data using OCR technology, which also includes handwriting recognition technology, to convert characters and mathematical expressions in the image into text data.

[0667] Step 6:

[0668] The server runs the converted text data through a scoring algorithm to evaluate the accuracy of the answer, thereby determining whether the answer is correct or incorrect.

[0669] Step 7:

[0670] The server analyzes the incorrect answers to identify where the error occurred, for example, by identifying calculation errors or errors between steps in the solution.

[0671] Step 8:

[0672] Based on the identified mistakes, the server generates a correct solution and explanation, including procedural and theoretical explanations.

[0673] Step 9:

[0674] The server generates feedback and provides it to the user, including explanatory notes, reference materials, and even refresher questions.

[0675] Step 10:

[0676] To facilitate re-learning, the server analyzes the user's past answer data and generates practice questions that are optimally tailored to each individual user. These practice questions are designed to reinforce the user's weak points.

[0677] Step 11:

[0678] The device receives the feedback data sent from the server and displays it on the user interface. The feedback includes whether the answer was correct or incorrect, the cause of the error, and advice for relearning.

[0679] Step 12:

[0680] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then works on the re-learning questions to deepen their understanding.

[0681] Step 13:

[0682] The user can ask additional questions as needed through the application, and the device uses voice recognition to convert the user's questions into text data and send it to the server.

[0683] Step 14:

[0684] The server analyzes the user's question and generates an appropriate answer, which is provided in real time in voice or text format.

[0685] Step 15:

[0686] The device displays or plays aloud the received answers to the user to help them understand, and the user can use this information to further their learning.

[0687] Through this series of processes, the system provides users with real-time feedback and re-learning support, improving learning efficiency and comprehension.

[0688] Example 1

[0689] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0690] In conventional learning support systems, the processing of image data of answer sheets after receiving them is too manual, making it difficult to provide efficient guidance in real time. Furthermore, there are issues with the system, such as poor identification of the causes of errors and insufficient provision of re-study questions, which reduces the learner's learning efficiency.

[0691] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0692] In this invention, the server includes means for receiving image data of the answer sheet, means for preprocessing the received image data and correcting tilt and cropping, means for converting the image-processed data into text data using OCR technology, means for evaluating the accuracy of the answers based on the converted text data and identifying errors, means for generating appropriate explanations and guidance for the identified errors using a generative AI model, means for providing the generated explanations and guidance to the recipient, and means for presenting related relearning questions, which enable real-time and efficient analysis of the answer sheet, feedback on errors, and the presentation of appropriate relearning questions.

[0693] "Answer Sheet" means the paper document on which a respondent completes a question.

[0694] "Image data" refers to a file that digitally represents the answer sheet, usually as a photograph or scanned image.

[0695] "Preprocessing" refers to processes such as tilt correction and cropping that are performed to improve the quality of image data.

[0696] "OCR technology" is an abbreviation for "Optical Character Recognition," and is a technology that recognizes characters and mathematical formulas from image data and converts them into text data.

[0697] "Text data" means digital data containing textual information extracted from an image using OCR technology.

[0698] "Means for assessing accuracy" refers to algorithms or programs used to determine whether a respondent's answer is correct.

[0699] "Incorrect part" refers to the part of the answer given by the respondent that is not correct.

[0700] A "generative AI model" is an artificial intelligence model trained using large amounts of data, and refers to a technology that generates sentences and explanations based on specific conditions.

[0701] "Explanation and Guidance" means explanations or advice provided for specific mistakes.

[0702] A "re-study question" is a new question presented to the respondent related to the part where the respondent made a mistake, with the aim of relearning.

[0703] MODE FOR CARRYING OUT THE INVENTION

[0704] This invention is an efficient learning support system that receives image data of answer sheets, analyzes and understands the data, generates feedback, and provides it to answerers in real time. The specific operations of each component, the server, the terminal, and the user, are described in detail below.

[0705] Server-side behavior

[0706] The server plays a central role in the system, performing a series of processes from receiving image data of answer sheets to generating feedback and presenting re-study questions.

[0707] 1. Receiving answer data

[0708] The server receives the image data of the answer sheet from the contestant (user). At this time, it uses an HTTP POST request and saves the uploaded image data. As a framework, Flask or Django can be used.

[0709] 2. Image data preprocessing

[0710] The server uses image processing libraries such as OpenCV to perform preprocessing such as tilt correction and cropping, improving the image quality and making it easier to analyze. For example, the server uses cv2.getRotationMatrix2D and cv2.warpAffine to correct the tilt of the image.

[0711] 3. OCR Processing and Analysis

[0712] The server uses OCR technology such as Tesseract to recognize characters and formulas from the image data and convert them into text data. For example, it uses pytesseract.image_to_string to convert image data into text data.

[0713] 4. Marking answers

[0714] The server uses the converted text data to evaluate the accuracy of the answer, analyzing the answer using a mathematical library such as SymPy. For example, it uses sympy.solve to determine the user's answer.

[0715] 5. Identifying and analyzing errors

[0716] The server identifies where the error occurred and analyzes its cause. It uses the debug library to see at which step of the solution the error occurred, for example by using sympy.simplify to evaluate intermediate results.

[0717] 6. Feedback Generation

[0718] The server uses a generative AI model (e.g., GPT-4) to generate appropriate feedback, including details of the mistake and an explanation of the correct answer. For example, it generates an explanation such as, "You made a mistake in handling the parentheses, so please check the parentheses rules again."

[0719] 7. Presenting Retraining Questions

[0720] The server generates relevant relearning problems and presents them to the solver. For example, it uses a generative AI model to generate similar equation problems such as "x + 5 = 12."

[0721] Operation on the terminal side

[0722] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[0723] 1. Sending answer data

[0724] The device sends the image data of the answer sheet taken by the user to the server. At this time, by using React Native or Flutter, it is possible to easily send an HTTP POST request.

[0725] 2. Receiving and Viewing Feedback

[0726] The device receives the feedback data sent from the server and displays it in a user interface. For example, a React Native app uses fetch to retrieve the feedback and display it on the screen.

[0727] User behavior

[0728] The user enters and submits answer data, checks feedback, and re-studies.

[0729] 1. Answers to the questions

[0730] Users write their answers on a paper-based question sheet. For example, they write their answers to the math problem "2x + 3 = 7" on paper.

[0731] 2. Sending the answer data

[0732] Users take a photo of their answer sheet with their device and send it to the server, or they use their smartphone camera to upload the photo of their answer sheet through a dedicated app.

[0733] 3. Review feedback and relearn

[0734] The user checks the feedback displayed on the device, understands the error, reads the explanation displayed through the application, and then tackles the presented re-learning problem.

[0735] Examples of prompt statements

[0736] Prompt example 1 (preprocessing of image)

[0737] "Generate Python code to skew and crop the answer sheet."

[0738] Prompt sentence example 2 (OCR processing)

[0739] "Write a code to convert mathematical formulas from PDF to text using Tesseract"

[0740] Prompt example 3 (scoring system)

[0741] "Create a Python script that uses SymPy to determine the solution to an equation."

[0742] In this way, the present invention allows the server, terminal, and user to work together to efficiently analyze image data on answer sheets and provide immediate feedback, significantly improving the learner's learning efficiency and enabling them to receive appropriate guidance regardless of time or place.

[0743] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0744] Step 1:

[0745] Sending answer data (terminal side)

[0746] Input: Image data of the answer sheet taken by the user with a smartphone camera

[0747] How it works: The device sends the image data of the answer sheet to the server using an HTTP POST request. Applications using React Native or Flutter can be used. For example, the image of the answer sheet can be selected on the device and sent by pressing the upload button.

[0748] Output: The image data is uploaded to the server.

[0749] Step 2:

[0750] Receiving answer data (server side)

[0751] Input: Image data of the answer sheet sent from the device

[0752] How it works: The server receives an HTTP POST request and saves the uploaded image data. You can use a framework like Flask or Django, and retrieve the file using the request.files['file'] method.

[0753] Output: Image data stored on the server

[0754] Step 3:

[0755] Image data preprocessing (server side)

[0756] Input: Saved image data of answer sheet

[0757] Operation: Uses an image processing library such as OpenCV to perform preprocessing such as tilt correction and cropping. Specifically, calculates the rotation matrix using cv2.getRotationMatrix2D and corrects the image using cv2.warpAffine. This improves image quality and makes it easier to analyze.

[0758] Output: Preprocessed image data

[0759] Step 4:

[0760] OCR processing and analysis (server side)

[0761] Input: Preprocessed image data

[0762] How it works: Using OCR technology such as Tesseract, it recognizes characters and formulas from image data and converts them into text data. Specifically, it extracts the text data using the pytesseract.image_to_string method.

[0763] Output: Extracted text data

[0764] Step 5:

[0765] Answer scoring (server side)

[0766] Input: Extracted text data

[0767] How it works: It uses mathematical libraries such as SymPy to evaluate the accuracy of the answer. Specifically, it uses sympy.solve to solve the equation and compares it with the user's answer to determine whether it is correct or incorrect.

[0768] Output: Correct / incorrect result

[0769] Step 6:

[0770] Identifying and analyzing errors (server side)

[0771] Input: Correct / incorrect result

[0772] How it works: Using the debug library and SymPy functions, we analyze at which step of the solution process the error occurred. Specifically, we identify the cause of the error by evaluating intermediate results using sympy.simplify.

[0773] Output: Detailed information about the error

[0774] Step 7:

[0775] Feedback generation (server side)

[0776] Input:Detailed information about the error

[0777] How it works: Using a generative AI model (e.g., GPT-4), it generates a detailed explanation of the error and a correct answer. For example, it generates feedback such as, "You made a mistake with the parentheses. Please check the parentheses rules again."

[0778] Output: Text data of generated feedback

[0779] Step 8:

[0780] Presenting retraining questions (server side)

[0781] Input:Detailed information about the error

[0782] How it works: It uses a generative AI model to automatically generate related retraining problems, such as similar equation problems like "x + 5 = 12."

[0783] Output: Generated retraining problem

[0784] Step 9:

[0785] Receiving and displaying feedback (device side)

[0786] Input: Feedback and retraining questions sent by the server

[0787] How it works: The device receives the feedback data using an HTTP GET request and displays it in the user interface. If it's a React Native app, it uses the fetch method to get the feedback and display it on the screen.

[0788] Output: Feedback and relearning questions displayed in the user interface

[0789] Step 10:

[0790] Reviewing feedback and relearning (user side)

[0791] Input: Feedback and relearning questions displayed on the device

[0792] How it works: The user reviews the displayed feedback, understands the explanation, and then works on the presented re-learning problem. For example, they can learn by following the "Review parentheses processing" within the application.

[0793] Output: Understanding the error and completing the re-learning

[0794] (Application example 1)

[0795] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0796] Conventional learning support systems have the problem that analyzing answers and generating feedback takes time, making it difficult to provide real-time support. Furthermore, bookstores and stores selling learning materials have limited ways for purchasers to receive on-site learning support. This means that they are unable to receive immediate explanations or guidance, and it can take time to resolve concerns or questions. Furthermore, there is a need for a system that can support efficient learning without being tied to a specific location or time.

[0797] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0798] In this invention, the server includes means for receiving image data of the answer sheet, means for analyzing the received image data and converting it into text data, means for evaluating the accuracy of the answers based on the converted text data and identifying errors, means for generating appropriate explanations and guidance for the identified errors, means for providing the generated explanations and guidance to the recipient, means for transmitting image data of the photographed or uploaded answer sheet to the server, means for providing a user interface for displaying feedback in real time, and devices for providing learning support in commercial facilities and means for using the same, which enable immediate analysis of answers and feedback, making learning support possible, particularly in bookstores and stores that sell learning materials.

[0799] "Answer Sheet" means a medium containing answers to a set of questions in paper or electronic format.

[0800] "Image data" is data containing visual information stored in digital form.

[0801] The "means for receiving" is a method or device for taking in data from the outside.

[0802] An "analyzing means" is a method or device for processing data and understanding its content.

[0803] "Text data" is data stored as character information.

[0804] A "means for converting" is a method or device for changing data from one format to another.

[0805] A "means for assessing accuracy" is a method or device that checks whether the information entered is accurate.

[0806] The "means for identifying the error portion" is a method or device for detecting the error portion.

[0807] A "means for generating appropriate explanations and guidance" is a method or device for creating correct solutions and study advice.

[0808] The "means for providing" refers to a method or device for delivering the generated information or data to the user.

[0809] "Means for taking pictures" refers to a method or device for taking pictures or videos.

[0810] A "server" is a computer system on a network that processes and stores data.

[0811] An "uploading means" is a method or apparatus for transmitting data from a local device to a remote server.

[0812] A "user interface" is an interface through which a user interacts with a system.

[0813] "Real-time display means" refers to a method or device for immediately displaying generated data or information to a user.

[0814] A "commercial establishment" is a public or private place that offers goods or services.

[0815] "Equipment for providing learning support" is hardware or software used to aid in learning.

[0816] "Method of use" refers to a procedure or method for achieving a specific purpose.

[0817] The present invention is an efficient learning support system that receives image data of answer sheets, analyzes and understands the data, and generates and provides feedback. Specific embodiments of the system are described below.

[0818] Server-side behavior

[0819] The server plays a central role in the system and performs the following functions:

[0820] 1. Receiving answer data

[0821] The server receives image data of the answer sheet sent by the test taker via the device. For example, a user takes a photo of the answer sheet for a math test with their smartphone and sends the data to the server via an application.

[0822] 2. Image data preprocessing

[0823] The server performs image preprocessing such as deskewing and cropping. For example, if the edges of an answer sheet are cut off, it will be properly corrected so that the entire sheet can be analyzed.

[0824] 3. OCR Processing and Analysis

[0825] The server uses OCR (Optical Character Recognition) technology to recognize characters and formulas from image data and convert them into text data. Here, we use "pytesseract" for OCR. As an example, OCR recognizes "2x + 3 = 7" and converts it into text format.

[0826] 4. Marking answers

[0827] The server analyzes the converted text data and determines whether the answer is correct or incorrect. For example, if the answerer answers "x = 3," the server determines that this is incorrect.

[0828] 5. Identifying and analyzing errors

[0829] The server analyzes the incorrect answer to determine at which step the error occurred, for example by identifying the incorrect parentheses in the calculation process.

[0830] 6. Feedback Generation

[0831] The server generates appropriate explanations and guidance in text format, including the correct answer. For example, it generates an explanation such as "You made a mistake in handling the parentheses. Please check the parentheses rules again."

[0832] 7. Presenting Retraining Questions

[0833] The server automatically generates relearning questions related to the mistakes and presents them to the test taker. For example, it provides the test taker with "similar equation problems."

[0834] Operation on the terminal side

[0835] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[0836] 1. Sending answer data

[0837] The device sends the image data of the answer sheet taken by the user to the server. For example, the image of the answer sheet is uploaded through an application.

[0838] 2. Receiving and Viewing Feedback

[0839] The device receives the feedback sent from the server and displays it in the user interface. For example, the app might display "An error occurred during the calculation."

[0840] User behavior

[0841] The user enters and submits answer data, checks feedback, and re-studies.

[0842] 1. Answers to the questions

[0843] Users write answers to tests and workbooks on paper, for example, by handwriting the answer to the math problem "2x + 3 = 7."

[0844] 2. Sending the answer data

[0845] The user takes a photo of the answer sheet with their device and sends it to the server. For example, they can take a photo of the answer sheet with their smartphone camera and send it using the app.

[0846] 3. Review feedback and relearn

[0847] The user checks the feedback displayed on the device and understands the mistakes and explanations. For example, the user reads the feedback on the mistakes in the app and then works on the same type of relearning problem.

[0848] In this way, a system has been realized that links the server, terminals, and users to analyze answers in real time, provide feedback, and support re-learning. This system makes it possible to provide learning support in bookstores and stores that sell learning materials, and users can receive immediate feedback, significantly improving their learning efficiency.

[0849] Prompt Sentence Examples

[0850] Here is an example of a prompt that the user can use to submit their answer data:

[0851] Please upload the image data of your answer sheet:

[0852] The correct answer to this math problem is "x = 2." What was your answer to this problem? Take a photo of your answer sheet with your smartphone and upload it to this app.

[0853] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0854] Step 1:

[0855] The user takes a picture of the answer sheet with their smartphone.

[0856] Input: Handwritten answer sheet

[0857] Output: Image data of the answer sheet

[0858] Step 2:

[0859] The device displays the captured image data on the application and asks the user to confirm the upload.

[0860] Input: Image data of answer sheet

[0861] Output: Upload confirmation message

[0862] Step 3:

[0863] The user taps the "Upload" button on the application to send the image data of the answer sheet to the server.

[0864] Input: image data of answer sheet, user operation

[0865] Output: Image data sent to the server

[0866] Step 4:

[0867] The server preprocesses the received image data, correcting for skew and cropping, and uses OpenCV to reshape the image.

[0868] Input: Image data of answer sheet

[0869] Output: Preprocessed image data

[0870] Step 5:

[0871] The server performs OCR processing on the preprocessed image data using pytesseract and converts it into text data.

[0872] Input: Preprocessed image data

[0873] Output: Converted text data

[0874] Step 6:

[0875] The server analyzes the converted text data and compares it with the correct answer to evaluate the accuracy of the answer.

[0876] Input: converted text data, correct answer data

[0877] Output: Determine whether the answer is correct or incorrect

[0878] Step 7:

[0879] The server identifies the error and analyzes the step that caused the error.

[0880] Input: Answer accuracy evaluation result

[0881] Output: Specific information about the error

[0882] Step 8:

[0883] The server generates appropriate explanations and guidance based on the mistakes, using a generative AI model to generate the feedback content.

[0884] Input: Specific information about the mistake, correct answer data

[0885] Output: Explanatory and instructional text

[0886] Step 9:

[0887] The server sends feedback to the user's device, including a description of the error and an appropriate explanation.

[0888] Input: Explanation and instructional text

[0889] Output: Feedback sent to the user's device

[0890] Step 10:

[0891] The terminal displays the received feedback on a user interface.

[0892] Input: Feedback data from the server

[0893] Output: Feedback displayed in the user interface

[0894] Step 11:

[0895] The user reviews and understands the displayed feedback and, if necessary, works through the provided refresher questions.

[0896] Input: Displayed feedback

[0897] Output: Improve the user's learning behavior and perform re-learning.

[0898] This series of processes provides efficient learning support in real time, from sending answer data to receiving feedback.

[0899] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0900] This invention is an efficient learning support system that receives image data of answer sheets, analyzes the data, generates feedback, and provides it to test takers in real time. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more appropriate instruction can be provided. This system significantly improves test takers' learning efficiency and can provide appropriate instruction regardless of time or place.

[0901] Server-side behavior

[0902] The server plays a central role in the system and performs the following functions:

[0903] 1. Receiving answer data

[0904] The server receives the image data of the answer sheet sent by the contestant (user) through the terminal.

[0905] Example: Taking a photo of a math test answer sheet with your smartphone and sending it via an app.

[0906] 2. Image data preprocessing

[0907] The server preprocesses the image data of the answer sheet, correcting tilt and cropping it.

[0908] Example: If the edges of the answer sheet are cut, make the appropriate corrections so that the answer can be analyzed.

[0909] 3. OCR Processing and Analysis

[0910] The server uses OCR technology to recognize characters and formulas from image data and convert them into text data.

[0911] Example: Recognizing handwritten "2x + 3 = 7" and converting it to text form.

[0912] 4. Marking answers

[0913] The server runs the converted text data through a scoring algorithm to assess the accuracy of the answer.

[0914] Example: If the answer is "x = 3", determine that this is incorrect.

[0915] 5. Identifying and analyzing errors

[0916] The server analyzes the incorrect answer to determine where the mistake occurred and identifies the mistake.

[0917] Example: Identifying calculation errors or incorrect steps in a solution.

[0918] 6. Feedback Generation

[0919] The server generates appropriate explanations and instructions, including the correct answers.

[0920] Example: Generate an explanation such as "There was an error in the calculation process, so please check the calculation rules again."

[0921] 7. Presenting Retraining Questions

[0922] The server automatically generates relearning questions related to the error and presents them to the solver.

[0923] Example: Provide the respondent with a "similar equation problem" again.

[0924] 8. How the Emotion Engine Works

[0925] The server uses an emotion engine to analyze the user's facial expressions and tone of voice to recognize their emotional state.

[0926] For example, using a camera and microphone to monitor a user's facial expressions and tone of voice in real time.

[0927] 9. Use of Emotional Data

[0928] Based on the recognized emotional data, the server evaluates the user's stress level and motivation, and adjusts the learning guidance and feedback content.

[0929] Example: If the user is feeling stressed, offer encouraging comments or suggest a break.

[0930] Operation on the terminal side

[0931] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[0932] 1. Sending answer data

[0933] The terminal transmits the image data of the answer sheet photographed by the user to the server.

[0934] Example: Uploading an image of the answer sheet through the application.

[0935] 2. Receiving and Viewing Feedback

[0936] The terminal receives the feedback sent from the server and displays it on the user interface.

[0937] Example: The app displays advice stating that "there was an error in the calculation process."

[0938] 3. Collecting Emotional Data

[0939] The device uses a camera and microphone to collect the user's facial expressions and voice in real time and transmits them to a server.

[0940] Example: Capture the user's face with a camera and use that data for analysis.

[0941] User behavior

[0942] Users input and submit answer data, check and re-study feedback, and provide emotion data.

[0943] 1. Answers to the questions

[0944] Users write their answers to tests and questions on paper.

[0945] Example: Handwrite the answer to the math problem "2x + 3 = 7."

[0946] 2. Sending the answer data

[0947] The user takes a photo of the answer sheet with their device and sends it to the server.

[0948] Example: Take a photo of your answer sheet with your smartphone camera and send it via the app.

[0949] 3. Review feedback and relearn

[0950] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then works on the re-learning questions.

[0951] For example, read the feedback on your mistakes in the app and try the same type of relearning exercise.

[0952] 4. Providing Emotion Data

[0953] Users use the device's camera and microphone to provide emotional data to the system through facial expressions and voice.

[0954] For example, you turn to the device's camera so that the emotion engine can analyze your facial expressions and tone of voice while solving a problem.

[0955] In this way, the present invention realizes a system that analyzes answers in real time, provides feedback, and supports relearning through collaboration between the server, terminal, and user. Furthermore, the introduction of an emotion engine enables individualized instruction based on the user's emotional state, further improving learning efficiency and providing psychological support.

[0956] The processing flow will be explained below.

[0957] Step 1:

[0958] Users write their answers on a paper test or workbook, and then use their smartphone camera to take a photo of the answer sheet.

[0959] Step 2:

[0960] After taking a picture of the answer sheet, the user launches the dedicated application and selects the captured image. When ready, the image data is sent from the device to the server.

[0961] Step 3:

[0962] The device performs preprocessing on the received image data, including correcting the tilt of the answer sheet and cropping it so that the answer area is clearly visible.

[0963] Step 4:

[0964] The device sends the pre-processed image data to the server via the API, and the server receives the image data sent from the device.

[0965] Step 5:

[0966] The server analyzes the received image data using OCR technology, which also includes handwriting recognition technology, to convert characters and mathematical expressions in the image into text data.

[0967] Step 6:

[0968] The server runs the converted text data through a scoring algorithm to evaluate the accuracy of the answer, thereby determining whether the answer is correct or incorrect.

[0969] Step 7:

[0970] The server analyzes the incorrect answers to identify where the error occurred, for example, by identifying calculation errors or errors between steps in the solution.

[0971] Step 8:

[0972] Based on the identified mistakes, the server generates a correct solution and explanation, including procedural and theoretical explanations.

[0973] Step 9:

[0974] The device receives the feedback data sent from the server and displays it on the user interface. The feedback includes whether the answer was correct or incorrect, the cause of the error, and advice for relearning.

[0975] Step 10:

[0976] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then attempts to re-learn the problems presented by the server.

[0977] Step 11:

[0978] The device uses a camera and microphone to capture the user's facial expressions and voice tone, and transmits them to a server in real time, thereby monitoring their emotional state.

[0979] Step 12:

[0980] The server uses an emotion engine to analyze the user's emotional state from their facial expressions and tone of voice, thereby determining their stress level, motivation, etc.

[0981] Step 13:

[0982] The server adjusts the learning guidance and feedback based on the user's emotional state. For example, if the user is in a high-stress state, it will suggest encouraging messages or a break.

[0983] Step 14:

[0984] The server then sends tailored feedback and instruction to the device, including emotionally tailored content.

[0985] Step 15:

[0986] The device displays the feedback and guidance sent from the server on the user interface, allowing the user to receive support tailored to their own emotional state.

[0987] Through this series of processes, the system provides users with real-time feedback and support for relearning, and furthermore, the emotion engine enables individualized instruction tailored to the user's psychological state.

[0988] Example 2

[0989] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0990] Conventional learning support systems lacked the accuracy of receiving and analyzing image data from answer sheets, resulting in problems such as not being able to provide appropriate feedback to learners. Furthermore, there was also the issue that instruction did not take into account the learner's emotional state, preventing the effectiveness of learning from being maximized. This resulted in a decrease in learners' learning efficiency and made it difficult to provide appropriate individual instruction.

[0991] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0992] In this invention, the server includes means for receiving image data of the answer sheet, means for preprocessing the received image data to correct skew and trim it, means for analyzing the preprocessed image data, recognizing characters and formulas and converting them into text data, means for identifying incorrect answers and generating appropriate explanations and guidance, means for providing the generated explanations and guidance to the recipient, means for analyzing the user's facial expressions and voice to recognize their emotional state, means for adjusting the feedback content based on the recognized emotional data, and means for generating and presenting relearning questions related to the mistakes. This not only improves the accuracy of answers but also enables individual guidance that corresponds to the learner's emotional state, thereby improving learning efficiency and providing enhanced psychological support.

[0993] "Image data of the answer sheet" refers to data that has been photographed or captured with a camera or scanner of the answers written by the user on paper and converted into digital format.

[0994] "Means for receiving" refers to a combination of hardware and software for capturing image data of the answer sheet to the server via a network.

[0995] The "pre-processing means for tilt correction and trimming" is a combination of hardware and software that uses image processing technology on the image data of the answer sheet to correct the tilt and cut out unnecessary parts.

[0996] The "means for analyzing, recognizing characters and mathematical formulas, and converting them into text data" refers to a combination of hardware and software that uses OCR technology to extract characters and mathematical formulas from the image data of the answer sheet and convert them into text format.

[0997] The "means for identifying incorrect answers and generating appropriate explanations and guidance" is a combination of hardware and software that applies evaluation algorithms to the converted text data, identifies incorrect answers, and generates appropriate explanations and guidance using natural language processing techniques.

[0998] The "means for providing the generated commentary and instruction to the recipient" is a combination of hardware and software for transmitting the generated commentary and instruction to the user's terminal via a network and displaying it on a user interface.

[0999] The "means for recognizing the emotional state by analyzing the user's facial expression and voice" is a combination of hardware and software that uses a camera and microphone to collect data on the user's facial expression and voice, and analyzes it using an emotion recognition algorithm.

[1000] The "means for adjusting the feedback content based on the recognized emotional data" is a combination of hardware and software for evaluating the user's emotional state based on the analysis results and appropriately changing the feedback and instruction content as necessary.

[1001] The "means for generating and presenting relearning questions related to the error portion" is a combination of hardware and software for generating new questions related to the identified error portion and presenting them to the user.

[1002] MODE FOR CARRYING OUT THE INVENTION

[1003] The present invention is an efficient learning support system that receives image data of answer sheets, analyzes the data, generates feedback, and provides it to test takers in real time. Furthermore, by using an emotion engine, it is possible to provide appropriate guidance based on the user's emotional state. This system improves the test taker's learning efficiency and can provide appropriate guidance regardless of time or location. A specific embodiment of this system is described below.

[1004] Server-side behavior

[1005] 1. Receiving answer data

[1006] The server has a means to receive image data of answer sheets sent by users from their devices. The hardware used is the server for receiving data, and the software is the data transfer protocol. For example, a user takes a photo of a math test answer sheet using a smartphone app and sends it to the server. The server saves the sent image file in a specific directory and returns a response confirming receipt.

[1007] 2. Image data preprocessing

[1008] The server has the means to preprocess the image data it receives. The software used is the OpenCV library, which corrects the image's skew and crops out unnecessary parts. For example, if the edges of an answer sheet are cut off, this is corrected so that the image can be analyzed.

[1009] 3. OCR Processing and Analysis

[1010] The server uses OCR technology to recognize characters and mathematical formulas from image data and convert them into text data. The software used is the Tesseract OCR library. For example, it recognizes handwritten text such as "2x + 3 = 7" and converts it into text format. The converted text data is then stored in a database.

[1011] 4. Marking answers

[1012] The server has a means to evaluate the converted text data by running it through a scoring algorithm. The scoring algorithm is executed by a dedicated program. For example, if the answer is "x = 3," it will determine that this is incorrect, generate an evaluation result, and save it in JSON format.

[1013] 5. Identifying and analyzing errors

[1014] The server has a means to identify the incorrect answers and analyze their causes. It uses an error analysis algorithm to identify the causes of the incorrect answers by matching them with specific error patterns. For example, it identifies calculation errors or incorrect steps in the solution.

[1015] 6. Feedback Generation

[1016] The server has the means to generate appropriate explanations and instructional feedback. The technology used is natural language processing (NLP) using a generative AI model (e.g., GPT-3). For example, it generates an explanation such as, "There was an error in the calculation process. Please check the calculation rules again."

[1017] 7. Presenting Retraining Questions

[1018] The server has a means to automatically generate relearning questions related to the mistake and present them to the solver. It uses a problem generation algorithm. For example, it generates "similar equation problems" and provides them to the solver.

[1019] 8. How the Emotion Engine Works

[1020] The server has a means to analyze the user's facial expressions and voice to recognize their emotional state. It uses emotion recognition software (e.g., Emotion API) and uses a camera and microphone to analyze the user's facial expressions and tone of voice in real time.

[1021] 9. Use of Emotional Data

[1022] The server has a means to adjust the feedback content based on the emotional data. The feedback and instruction content is appropriately changed based on the analysis results. For example, if the user is feeling stressed, an encouraging comment is added.

[1023] Operation on the terminal side

[1024] 1. Sending answer data

[1025] The device has a means to send image data of the answer sheet taken by the user to the server. The hardware used is a smartphone, and the software is a dedicated application. For example, the user can take a photo of the answer sheet with the smartphone camera and send it to the server via the application.

[1026] 2. Receiving and Viewing Feedback

[1027] The device has a means to receive and display feedback from the server. This is done using a UI component within the application. For example, the app might display feedback indicating that an error occurred in the calculation process.

[1028] 3. Collecting Emotional Data

[1029] The device has a means for collecting the user's facial expressions and voice in real time using a camera and microphone and sending the data to a server. For example, the device captures the user's face with a camera and sends the data to a server.

[1030] User behavior

[1031] 1. Answers to the questions

[1032] Users write answers to tests and workbooks on paper, for example, by handwriting the answer to the math problem "2x + 3 = 7."

[1033] 2. Sending the answer data

[1034] The user takes a photo of the answer sheet with their device and sends it to the server. The device used is a smartphone, and a dedicated app is used. For example, the answer sheet is photographed with the smartphone camera and sent via the app.

[1035] 3. Review feedback and relearn

[1036] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then attempts to re-test the questions. For example, the user reads the feedback on the mistakes made on the app and attempts to re-test the questions.

[1037] 4. Providing Emotion Data

[1038] Users can provide emotional data through facial expressions and voice using the device's camera and microphone. For example, when solving a problem, they answer in front of the camera, and their voice is also recorded.

[1039] Examples of prompt statements

[1040] example:

[1041] "Take a picture of your 2x + 3 = 7 answer sheet and submit it. Then, check your answer and receive feedback to try again if you made a mistake. Our emotion engine will also analyze your emotional state to see if you're feeling stressed."

[1042] This system works by linking the server, terminals, and users to analyze answers in real time, provide feedback, and support re-learning.The introduction of an emotion engine enables individualized instruction based on the user's emotional state, improving learning efficiency and providing psychological support.

[1043] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1044] Step 1:

[1045] Sending answer data

[1046] The terminal transmits image data of the answer sheet photographed by the user to the server.

[1047] Input: An image file of the answer sheet taken by the user with their smartphone

[1048] Output: Notification of completion of image data upload to the server

[1049] Specific operation: Using the dedicated application on the device, the user takes a photo of the answer sheet and taps the send button to upload the image to the server. After the upload is complete, the device displays a notification that the upload was successful.

[1050] Step 2:

[1051] Receiving answer data

[1052] The server receives the image data of the answer sheet sent from the terminal.

[1053] Input: Image data from the device

[1054] Output: Save image data and confirm receipt response

[1055] Specific operation: The server receives the image data of the answer sheet via an HTTP request, saves the image file in a specific directory, and returns a success message to the terminal after completion of the reception.

[1056] Step 3:

[1057] Image data preprocessing

[1058] The server performs pre-processing on the received image data.

[1059] Input: Saved image data

[1060] Output: Deskewed and cropped image data

[1061] How it works: The server uses the OpenCV library to straighten the image and crop unnecessary parts. The processed image data is then re-saved in a format suitable for analysis.

[1062] Step 4:

[1063] OCR Processing and Analysis

[1064] The server analyzes the image data using OCR technology and converts it into text data.

[1065] Input: Preprocessed image data

[1066] Output: Parsed text data

[1067] How it works: The server uses the Tesseract OCR library to extract characters and formulas from the image data and convert them into text format, which is then stored in a database.

[1068] Step 5:

[1069] Grading answers

[1070] The server evaluates the text data using a scoring algorithm.

[1071] Input: Text data generated by OCR processing

[1072] Output: Scoring result data (correct / incorrect)

[1073] How it works: The server runs a dedicated scoring algorithm, compares the text data with the correct answers, and generates a score, which is saved in JSON format.

[1074] Step 6:

[1075] Identifying and analyzing errors

[1076] The server identifies and analyzes the incorrect answers.

[1077] Input: Grade result data

[1078] Output: Analysis data of the error location and its cause

[1079] Specific operation: The server runs an error analysis algorithm to identify the incorrect answers and analyze the causes (calculation errors, incorrect solutions, etc.). The analysis results are stored in a database.

[1080] Step 7:

[1081] Feedback Generation

[1082] The server generates appropriate commentary and instructional feedback.

[1083] Input: Analysis data of the error location and cause

[1084] Output: The generated feedback text

[1085] Specific operation: The server uses a generative AI model (e.g., GPT-3) to generate feedback text based on the analyzed data. The generated feedback is stored in a database.

[1086] Step 8:

[1087] Presenting re-learning questions

[1088] The server generates and presents relearning questions related to the error location.

[1089] Input: Analysis data of the error

[1090] Output: Retraining problem data

[1091] Specific operation: The server uses a problem generation algorithm to generate new retraining problems based on the identified mistakes. The generated problems are stored in a database.

[1092] Step 9:

[1093] Emotion Engine Operation

[1094] The server analyzes the user's facial expression and voice data to recognize their emotional state.

[1095] Input: Facial expression and voice data sent from the device

[1096] Output: Recognized emotion data

[1097] How it works: The server uses emotion recognition software to analyze the transmitted data and recognize the user's emotional state (e.g., stress, excitement, concentration). The analysis results are stored in a database.

[1098] Step 10:

[1099] Use of Emotional Data

[1100] The server adjusts the feedback content based on the recognized emotion data.

[1101] Input: Recognized emotion data

[1102] Output: Adjusted feedback text

[1103] Specific operation: The server refers to the emotion data and adjusts the feedback content appropriately according to the user's emotional state. For example, if the user is feeling stressed, it adds encouraging comments. The adjusted feedback is stored in a database and sent to the device.

[1104] (Application example 2)

[1105] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1106] While conventional learning support systems focus on evaluating answers and providing feedback, they do not adequately provide individualized instruction tailored to the learner's emotional state. This makes it difficult to provide appropriate instruction tailored to the learner's motivation and level of understanding, resulting in limited improvements in learning efficiency. Furthermore, while real-time feedback and appropriate instruction tailored to the learner's emotional state are required in training scenarios such as factories, current systems have difficulty achieving this.

[1107] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving image data of an answer sheet, means for analyzing the received image data and converting it into text data, means for evaluating the accuracy of the answer based on the converted text data and identifying errors, means for generating appropriate explanations and guidance for the identified errors, means for providing the generated explanations and guidance to the recipient, means for identifying the user's emotional state via a camera and a voice input device, and means for adjusting the feedback content based on the identified emotional state. This allows individualized instruction and appropriate feedback to be provided in real time according to the learner's emotional state, improving learning efficiency. Furthermore, similar effects can be expected in training scenarios such as factories, contributing to the efficiency of employee skill acquisition.

[1108] An "answer sheet" is a paper sheet on which answers are written by hand and which contains questions and answers.

[1109] "Image data" refers to digital data of the answer sheet obtained using a device such as a camera or scanner.

[1110] "Means for receiving" refers to an interface or function for importing image data into a system such as a server.

[1111] The "means for analyzing and converting into text data" is a function that uses optical character recognition (OCR) technology to analyze image data and convert characters and mathematical formulas into text format.

[1112] "Means for assessing the accuracy of answers and identifying errors" refers to algorithms or methods for determining whether answers are correct or incorrect based on text data and for identifying incorrect parts of the answers.

[1113] The "means for generating appropriate explanations and guidance" is a function that automatically generates the correct answer, its explanation, and further guidance for incorrect answers.

[1114] The "means for providing" refers to an interface or function for notifying the user of the generated feedback and instruction content.

[1115] "Camera and audio input device" refers to a hardware device for capturing the user's facial expressions and voice.

[1116] The "means for identifying emotional state" refers to algorithms or software that analyze captured facial and voice data to interpret the user's emotions.

[1117] The "means for adjusting the feedback content" is a function that dynamically changes and adjusts the content of the feedback provided, taking into account the identified emotional state.

[1118] This invention is a learning support system that receives image data of answer sheets, analyzes them, generates feedback, and provides it to users in real time. It also has the ability to identify the user's emotional state using a camera and a voice input device and adjust the feedback content accordingly. The system aims to improve learning and training efficiency.

[1119] Server side behavior:

[1120] The server plays a central role in the system and performs multiple processes to achieve the following functions:

[1121] 1. Receiving answer data: The server receives image data of the answer sheet sent from a terminal equipped with image data transmission functionality. This data can also be used for things like training checklists in factories.

[1122] 2. Image data preprocessing: The server preprocesses the image data of the answer sheet, correcting tilt and removing noise. For example, it uses an image processing library such as OpenCV.

[1123] 3. OCR processing and analysis: The server uses optical character recognition (OCR) technology to convert handwritten characters and mathematical formulas from the image data into text data. The pytesseract library assists in this process.

[1124] 4. Answer scoring: Based on the text data, the scoring algorithm evaluates the accuracy of the answer and identifies incorrect parts. The scoring algorithm compares the answer with predefined correct answer data.

[1125] 5. Feedback generation: The server generates appropriate explanations and instruction methods based on the incorrect answers. For example, if there is a calculation error, the server generates feedback that explains the details.

[1126] 6. Operation of the Emotion Engine: The server analyzes facial images and voice data sent by the user to identify the user's emotional state. By using an emotion analysis library, the stress level and motivation of employees and learners are evaluated.

[1127] 7. Feedback Adjustment: Dynamically adjust the feedback content based on the emotional state identified by the emotion engine, thereby providing guidance that is optimal for the user's psychological state.

[1128] 8. Presenting Rehearsal Questions: The server automatically generates rehearsal questions related to incorrect answers and presents them to the user, helping to correct misunderstandings and deepen understanding.

[1129] On the device side:

[1130] The terminal is responsible for sending input from the user to the server and for receiving and displaying feedback.

[1131] 1. Sending answer data: The terminal sends the image data of the answer sheet taken by the user using the camera to the server.

[1132] 2. Receiving and displaying feedback: The terminal receives the feedback sent by the server and displays it on the user interface, allowing the user to identify their mistakes and find ways to improve.

[1133] 3. Emotion data collection: The device uses a camera and microphone to collect the user's facial expressions and voice in real time and transmits them to the server.

[1134] User behavior:

[1135] The user inputs and submits answer data, checks and re-learns feedback, and provides emotion data.

[1136] 1. Answering questions: The user writes down the answers to the training checklist or questions on paper.

[1137] 2. Sending answer data: The user takes a photo of their handwritten answer on their device and sends it to the server.

[1138] 3. Checking feedback and relearning: The user checks the feedback displayed on the device, understands the mistakes and explanations, and works on relearning questions to solidify their understanding.

[1139] 4. Providing emotional data: The user provides emotional data through facial expressions and voice using the device's camera and microphone.

[1140] This system allows users to study and train efficiently and receive appropriate feedback based on their emotional state.It can also be used in factories and other workplaces to provide appropriate training that takes into account the emotional state of employees, thereby improving their skills and managing stress.

[1141] Examples:

[1142] Below are some example prompts for a generative AI model:

[1143] Build an application that analyzes a user's checklist image and provides feedback. This is a cloud-based learning support system that recognizes the user's emotions from video and audio feeds and tailors the feedback. Specifically, it uses OCR technology to recognize handwritten text, generates feedback based on the response data, and tailors the feedback content through sentiment analysis.

[1144] This prompt conveys specific system requirements and functions to the generative AI model, providing appropriate learning assistance to the user.

[1145] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1146] Step 1:

[1147] The server receives the image data of the answer sheet sent from the terminal. The image data of the answer sheet is received as input and imported into the server. The received data is stored in the internal memory and passed to the next pre-processing step.

[1148] Step 2:

[1149] The server preprocesses the received image data. It uses the received image data as input and performs deskew and noise removal. Specifically, it uses the OpenCV library to convert the image to grayscale and threshold it. The result is a binary image that can be analyzed.

[1150] Step 3:

[1151] The server performs OCR processing on the preprocessed image data to generate text data. Using the preprocessed binary image as input, it reads characters and formulas using the pytesseract library and outputs text data. This allows the text format answer data to be extracted from the handwritten answer sheet.

[1152] Step 4:

[1153] The server evaluates the accuracy of the answers based on the extracted text data. It uses the generated text data as input and compares it with a pre-defined database of correct answers. Specifically, an algorithm determines whether each answer is correct and identifies any errors. The output is a result of the accuracy assessment and data indicating the errors.

[1154] Step 5:

[1155] The server generates appropriate explanations and guidance based on the mistakes. Using the evaluation results and data on the mistakes as input, it automatically generates the explanations using a generative AI model. The output is text data of the explanations and guidance.

[1156] Step 6:

[1157] The server provides the generated explanations and instructions to the terminal, using the text data of the explanations and instructions as input and transferring the data to the terminal, which receives it and displays it on the user interface.

[1158] Step 7:

[1159] The device collects the user's emotional data and sends it to the server. It captures the user's facial expressions and voice in real time using a camera and microphone. It receives facial image data and voice data as input and sends them to the server.

[1160] Step 8:

[1161] The server uses an emotion engine to identify the user's emotional state. It analyzes facial image and voice data received as input to evaluate the user's emotional state (e.g., stress level and motivation). The identified emotion data is obtained as output.

[1162] Step 9:

[1163] The server adjusts the feedback content based on the identified emotional data. Using the emotional data and the generated commentary and instruction data as input, the server changes and adjusts the feedback content according to the emotional state. The adjusted feedback text data is obtained as output.

[1164] Step 10:

[1165] The server sends the adjusted feedback back to the terminal and provides it to the user. The server uses the adjusted feedback text data as input and sends the data to the terminal. The terminal receives the data and displays it on the user interface. This allows the user to receive feedback optimized for their emotional state.

[1166] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1167] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1168] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1169] [Third embodiment]

[1170] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1171] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1172] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1173] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1174] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1175] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1176] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1177] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1178] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1179] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1180] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1181] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1182] This invention is an efficient learning support system that receives image data of answer sheets, analyzes and understands the data, generates feedback, and provides it to test takers in real time. This system significantly improves test takers' learning efficiency and can provide appropriate guidance without being restricted by time or place.

[1183] Server-side behavior

[1184] The server plays a central role in the system and performs the following functions:

[1185] 1. Receiving answer data

[1186] The server receives the image data of the answer sheet sent by the contestant (user) through the terminal.

[1187] Example: Taking a photo of a math test answer sheet with your smartphone and sending it via an app.

[1188] 2. Image data preprocessing

[1189] The server performs image pre-processing such as tilt correction and cropping.

[1190] For example, if the edges of the answer sheet are cut, make appropriate corrections so that the whole sheet can be analyzed.

[1191] 3. OCR Processing and Analysis

[1192] The server uses OCR technology to recognize characters and formulas from image data and convert them into text data.

[1193] Example: Recognize "2x + 3 = 7" using OCR and convert it to text format.

[1194] 4. Marking answers

[1195] The server analyzes the converted text data and determines whether the answer is correct or incorrect.

[1196] Example: If the respondent answers "x = 3", it is determined that this is incorrect.

[1197] 5. Identifying and analyzing errors

[1198] The server analyzes the incorrect answers to determine at which step the error occurred.

[1199] Example: Identifying where parentheses were incorrectly used during a calculation.

[1200] 6. Feedback Generation

[1201] The server generates appropriate text-based explanations and instructions, including the correct answers.

[1202] Example: Generate an explanation such as "You have processed the parentheses incorrectly, so please check the parentheses rules again."

[1203] 7. Presenting Retraining Questions

[1204] The server automatically generates relearning questions related to the incorrect answers and presents them to the test taker.

[1205] Example: Provide the respondent with a "similar equation problem" again.

[1206] Operation on the terminal side

[1207] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[1208] 1. Sending answer data

[1209] The terminal transmits the image data of the answer sheet photographed by the user to the server.

[1210] Example: Uploading an image of the answer sheet through the application.

[1211] 2. Receiving and Viewing Feedback

[1212] The terminal receives the feedback sent from the server and displays it on the user interface.

[1213] Example: The app displays the message "An error occurred during the calculation process."

[1214] User behavior

[1215] The user enters and submits answer data, checks feedback, and re-studies.

[1216] 1. Answers to the questions

[1217] Users write their answers to tests and questions on paper.

[1218] Example: Handwrite the answer to the math problem "2x + 3 = 7."

[1219] 2. Sending the answer data

[1220] The user takes a photo of the answer sheet with their device and sends it to the server.

[1221] Example: Take a photo of your answer sheet with your smartphone camera and send it via the app.

[1222] 3. Review feedback and relearn

[1223] The user checks the feedback displayed on the device and understands the mistakes and explanations.

[1224] For example, read the feedback on your mistakes in the app and then try the same type of relearning exercise.

[1225] In this way, the present invention realizes a system that analyzes answers, provides feedback, and supports re-learning in real time through the cooperation of the server, terminal, and user. This system allows test takers to receive immediate feedback and quickly correct errors, greatly improving learning efficiency.

[1226] The processing flow will be explained below.

[1227] Step 1:

[1228] Users write their answers on a paper test or workbook, and then use their smartphone camera to take a photo of the answer sheet.

[1229] Step 2:

[1230] After taking a picture of the answer sheet, the user launches the dedicated application and selects the captured image. When ready, the image data is sent from the device to the server.

[1231] Step 3:

[1232] The device performs preprocessing on the received image data, including correcting the tilt of the answer sheet and cropping it so that the answer area is clearly visible.

[1233] Step 4:

[1234] The device sends the pre-processed image data to the server via the API, and the server receives the image data sent from the device.

[1235] Step 5:

[1236] The server analyzes the received image data using OCR technology, which also includes handwriting recognition technology, to convert characters and mathematical expressions in the image into text data.

[1237] Step 6:

[1238] The server runs the converted text data through a scoring algorithm to evaluate the accuracy of the answer, thereby determining whether the answer is correct or incorrect.

[1239] Step 7:

[1240] The server analyzes the incorrect answers to identify where the error occurred, for example, by identifying calculation errors or errors between steps in the solution.

[1241] Step 8:

[1242] Based on the identified mistakes, the server generates a correct solution and explanation, including procedural and theoretical explanations.

[1243] Step 9:

[1244] The server generates feedback and provides it to the user, including explanatory notes, reference materials, and even refresher questions.

[1245] Step 10:

[1246] To facilitate re-learning, the server analyzes the user's past answer data and generates practice questions that are optimally tailored to each individual user. These practice questions are designed to reinforce the user's weak points.

[1247] Step 11:

[1248] The device receives the feedback data sent from the server and displays it on the user interface. The feedback includes whether the answer was correct or incorrect, the cause of the error, and advice for relearning.

[1249] Step 12:

[1250] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then works on the re-learning questions to deepen their understanding.

[1251] Step 13:

[1252] The user can ask additional questions as needed through the application, and the device uses voice recognition to convert the user's questions into text data and send it to the server.

[1253] Step 14:

[1254] The server analyzes the user's question and generates an appropriate answer, which is provided in real time in voice or text format.

[1255] Step 15:

[1256] The device displays or plays aloud the received answers to the user to help them understand, and the user can use this information to further their learning.

[1257] Through this series of processes, the system provides users with real-time feedback and re-learning support, improving learning efficiency and comprehension.

[1258] Example 1

[1259] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1260] In conventional learning support systems, the processing of image data of answer sheets after receiving them is too manual, making it difficult to provide efficient guidance in real time. Furthermore, there are issues with the system, such as poor identification of the causes of errors and insufficient provision of re-study questions, which reduces the learner's learning efficiency.

[1261] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1262] In this invention, the server includes means for receiving image data of the answer sheet, means for preprocessing the received image data and correcting tilt and cropping, means for converting the image-processed data into text data using OCR technology, means for evaluating the accuracy of the answers based on the converted text data and identifying errors, means for generating appropriate explanations and guidance for the identified errors using a generative AI model, means for providing the generated explanations and guidance to the recipient, and means for presenting related relearning questions, which enable real-time and efficient analysis of the answer sheet, feedback on errors, and the presentation of appropriate relearning questions.

[1263] "Answer Sheet" means the paper document on which a respondent completes a question.

[1264] "Image data" refers to a file that digitally represents the answer sheet, usually as a photograph or scanned image.

[1265] "Preprocessing" refers to processes such as tilt correction and cropping that are performed to improve the quality of image data.

[1266] "OCR technology" is an abbreviation for "Optical Character Recognition," and is a technology that recognizes characters and mathematical formulas from image data and converts them into text data.

[1267] "Text data" means digital data containing textual information extracted from an image using OCR technology.

[1268] "Means for assessing accuracy" refers to algorithms or programs used to determine whether a respondent's answer is correct.

[1269] "Incorrect part" refers to the part of the answer given by the respondent that is not correct.

[1270] A "generative AI model" is an artificial intelligence model trained using large amounts of data, and refers to a technology that generates sentences and explanations based on specific conditions.

[1271] "Explanation and Guidance" means explanations or advice provided for specific mistakes.

[1272] A "re-study question" is a new question presented to the respondent related to the part where the respondent made a mistake, with the aim of relearning.

[1273] MODE FOR CARRYING OUT THE INVENTION

[1274] This invention is an efficient learning support system that receives image data of answer sheets, analyzes and understands the data, generates feedback, and provides it to answerers in real time. The specific operations of each component, the server, the terminal, and the user, are described in detail below.

[1275] Server-side behavior

[1276] The server plays a central role in the system, performing a series of processes from receiving image data of answer sheets to generating feedback and presenting re-study questions.

[1277] 1. Receiving answer data

[1278] The server receives the image data of the answer sheet from the contestant (user). At this time, it uses an HTTP POST request and saves the uploaded image data. As a framework, Flask or Django can be used.

[1279] 2. Image data preprocessing

[1280] The server uses image processing libraries such as OpenCV to perform preprocessing such as tilt correction and cropping, improving the image quality and making it easier to analyze. For example, the server uses cv2.getRotationMatrix2D and cv2.warpAffine to correct the tilt of the image.

[1281] 3. OCR Processing and Analysis

[1282] The server uses OCR technology such as Tesseract to recognize characters and formulas from the image data and convert them into text data. For example, it uses pytesseract.image_to_string to convert image data into text data.

[1283] 4. Marking answers

[1284] The server uses the converted text data to evaluate the accuracy of the answer, analyzing the answer using a mathematical library such as SymPy. For example, it uses sympy.solve to determine the user's answer.

[1285] 5. Identifying and analyzing errors

[1286] The server identifies where the error occurred and analyzes its cause. It uses the debug library to see at which step of the solution the error occurred, for example by using sympy.simplify to evaluate intermediate results.

[1287] 6. Feedback Generation

[1288] The server uses a generative AI model (e.g., GPT-4) to generate appropriate feedback, including details of the mistake and an explanation of the correct answer. For example, it generates an explanation such as, "You made a mistake in handling the parentheses, so please check the parentheses rules again."

[1289] 7. Presenting Retraining Questions

[1290] The server generates relevant relearning problems and presents them to the solver. For example, it uses a generative AI model to generate similar equation problems such as "x + 5 = 12."

[1291] Operation on the terminal side

[1292] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[1293] 1. Sending answer data

[1294] The device sends the image data of the answer sheet taken by the user to the server. At this time, by using React Native or Flutter, it is possible to easily send an HTTP POST request.

[1295] 2. Receiving and Viewing Feedback

[1296] The device receives the feedback data sent from the server and displays it in a user interface. For example, a React Native app uses fetch to retrieve the feedback and display it on the screen.

[1297] User behavior

[1298] The user enters and submits answer data, checks feedback, and re-studies.

[1299] 1. Answers to the questions

[1300] Users write their answers on a paper-based question sheet. For example, they write their answers to the math problem "2x + 3 = 7" on paper.

[1301] 2. Sending the answer data

[1302] Users take a photo of their answer sheet with their device and send it to the server, or they use their smartphone camera to upload the photo of their answer sheet through a dedicated app.

[1303] 3. Review feedback and relearn

[1304] The user checks the feedback displayed on the device, understands the error, reads the explanation displayed through the application, and then tackles the presented re-learning problem.

[1305] Examples of prompt statements

[1306] Prompt example 1 (preprocessing of image)

[1307] "Generate Python code to skew and crop the answer sheet."

[1308] Prompt sentence example 2 (OCR processing)

[1309] "Write a code to convert mathematical formulas from PDF to text using Tesseract"

[1310] Prompt example 3 (scoring system)

[1311] "Create a Python script that uses SymPy to determine the solution to an equation."

[1312] In this way, the present invention allows the server, terminal, and user to work together to efficiently analyze image data on answer sheets and provide immediate feedback, significantly improving the learner's learning efficiency and enabling them to receive appropriate guidance regardless of time or place.

[1313] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1314] Step 1:

[1315] Sending answer data (terminal side)

[1316] Input: Image data of the answer sheet taken by the user with a smartphone camera

[1317] How it works: The device sends the image data of the answer sheet to the server using an HTTP POST request. Applications using React Native or Flutter can be used. For example, the image of the answer sheet can be selected on the device and sent by pressing the upload button.

[1318] Output: The image data is uploaded to the server.

[1319] Step 2:

[1320] Receiving answer data (server side)

[1321] Input: Image data of the answer sheet sent from the device

[1322] How it works: The server receives an HTTP POST request and saves the uploaded image data. You can use a framework like Flask or Django, and retrieve the file using the request.files['file'] method.

[1323] Output: Image data stored on the server

[1324] Step 3:

[1325] Image data preprocessing (server side)

[1326] Input: Saved image data of answer sheet

[1327] Operation: Uses an image processing library such as OpenCV to perform preprocessing such as tilt correction and cropping. Specifically, calculates the rotation matrix using cv2.getRotationMatrix2D and corrects the image using cv2.warpAffine. This improves image quality and makes it easier to analyze.

[1328] Output: Preprocessed image data

[1329] Step 4:

[1330] OCR processing and analysis (server side)

[1331] Input: Preprocessed image data

[1332] How it works: Using OCR technology such as Tesseract, it recognizes characters and formulas from image data and converts them into text data. Specifically, it extracts the text data using the pytesseract.image_to_string method.

[1333] Output: Extracted text data

[1334] Step 5:

[1335] Answer scoring (server side)

[1336] Input: Extracted text data

[1337] How it works: It uses mathematical libraries such as SymPy to evaluate the accuracy of the answer. Specifically, it uses sympy.solve to solve the equation and compares it with the user's answer to determine whether it is correct or incorrect.

[1338] Output: Correct / incorrect result

[1339] Step 6:

[1340] Identifying and analyzing errors (server side)

[1341] Input: Correct / incorrect result

[1342] How it works: Using the debug library and SymPy functions, we analyze at which step of the solution process the error occurred. Specifically, we identify the cause of the error by evaluating intermediate results using sympy.simplify.

[1343] Output: Detailed information about the error

[1344] Step 7:

[1345] Feedback generation (server side)

[1346] Input:Detailed information about the error

[1347] How it works: Using a generative AI model (e.g., GPT-4), it generates a detailed explanation of the error and a correct answer. For example, it generates feedback such as, "You made a mistake with the parentheses. Please check the parentheses rules again."

[1348] Output: Text data of generated feedback

[1349] Step 8:

[1350] Presenting retraining questions (server side)

[1351] Input:Detailed information about the error

[1352] How it works: It uses a generative AI model to automatically generate related retraining problems, such as similar equation problems like "x + 5 = 12."

[1353] Output: Generated retraining problem

[1354] Step 9:

[1355] Receiving and displaying feedback (device side)

[1356] Input: Feedback and retraining questions sent by the server

[1357] How it works: The device receives the feedback data using an HTTP GET request and displays it in the user interface. If it's a React Native app, it uses the fetch method to get the feedback and display it on the screen.

[1358] Output: Feedback and relearning questions displayed in the user interface

[1359] Step 10:

[1360] Reviewing feedback and relearning (user side)

[1361] Input: Feedback and relearning questions displayed on the device

[1362] How it works: The user reviews the displayed feedback, understands the explanation, and then works on the presented re-learning problem. For example, they can learn by following the "Review parentheses processing" within the application.

[1363] Output: Understanding the error and completing the re-learning

[1364] (Application example 1)

[1365] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1366] Conventional learning support systems have the problem that analyzing answers and generating feedback takes time, making it difficult to provide real-time support. Furthermore, bookstores and stores selling learning materials have limited ways for purchasers to receive on-site learning support. This means that they are unable to receive immediate explanations or guidance, and it can take time to resolve concerns or questions. Furthermore, there is a need for a system that can support efficient learning without being tied to a specific location or time.

[1367] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1368] In this invention, the server includes means for receiving image data of the answer sheet, means for analyzing the received image data and converting it into text data, means for evaluating the accuracy of the answers based on the converted text data and identifying errors, means for generating appropriate explanations and guidance for the identified errors, means for providing the generated explanations and guidance to the recipient, means for transmitting image data of the photographed or uploaded answer sheet to the server, means for providing a user interface for displaying feedback in real time, and devices for providing learning support in commercial facilities and means for using the same, which enable immediate analysis of answers and feedback, making learning support possible, particularly in bookstores and stores that sell learning materials.

[1369] "Answer Sheet" means a medium containing answers to a set of questions in paper or electronic format.

[1370] "Image data" is data containing visual information stored in digital form.

[1371] The "means for receiving" is a method or device for taking in data from the outside.

[1372] An "analyzing means" is a method or device for processing data and understanding its content.

[1373] "Text data" is data stored as character information.

[1374] A "means for converting" is a method or device for changing data from one format to another.

[1375] A "means for assessing accuracy" is a method or device that checks whether the information entered is accurate.

[1376] The "means for identifying the error portion" is a method or device for detecting the error portion.

[1377] A "means for generating appropriate explanations and guidance" is a method or device for creating correct solutions and study advice.

[1378] The "means for providing" refers to a method or device for delivering the generated information or data to the user.

[1379] "Means for taking pictures" refers to a method or device for taking pictures or videos.

[1380] A "server" is a computer system on a network that processes and stores data.

[1381] An "uploading means" is a method or apparatus for transmitting data from a local device to a remote server.

[1382] A "user interface" is an interface through which a user interacts with a system.

[1383] "Real-time display means" refers to a method or device for immediately displaying generated data or information to a user.

[1384] A "commercial establishment" is a public or private place that offers goods or services.

[1385] "Equipment for providing learning support" is hardware or software used to aid in learning.

[1386] "Method of use" refers to a procedure or method for achieving a specific purpose.

[1387] The present invention is an efficient learning support system that receives image data of answer sheets, analyzes and understands the data, and generates and provides feedback. Specific embodiments of the system are described below.

[1388] Server-side behavior

[1389] The server plays a central role in the system and performs the following functions:

[1390] 1. Receiving answer data

[1391] The server receives image data of the answer sheet sent by the test taker via the device. For example, a user takes a photo of the answer sheet for a math test with their smartphone and sends the data to the server via an application.

[1392] 2. Image data preprocessing

[1393] The server performs image preprocessing such as deskewing and cropping. For example, if the edges of an answer sheet are cut off, it will be properly corrected so that the entire sheet can be analyzed.

[1394] 3. OCR Processing and Analysis

[1395] The server uses OCR (Optical Character Recognition) technology to recognize characters and formulas from image data and convert them into text data. Here, we use "pytesseract" for OCR. As an example, OCR recognizes "2x + 3 = 7" and converts it into text format.

[1396] 4. Marking answers

[1397] The server analyzes the converted text data and determines whether the answer is correct or incorrect. For example, if the answerer answers "x = 3," the server determines that this is incorrect.

[1398] 5. Identifying and analyzing errors

[1399] The server analyzes the incorrect answer to determine at which step the error occurred, for example by identifying the incorrect parentheses in the calculation process.

[1400] 6. Feedback Generation

[1401] The server generates appropriate explanations and guidance in text format, including the correct answer. For example, it generates an explanation such as "You made a mistake in handling the parentheses. Please check the parentheses rules again."

[1402] 7. Presenting Retraining Questions

[1403] The server automatically generates relearning questions related to the mistakes and presents them to the test taker. For example, it provides the test taker with "similar equation problems."

[1404] Operation on the terminal side

[1405] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[1406] 1. Sending answer data

[1407] The device sends the image data of the answer sheet taken by the user to the server. For example, the image of the answer sheet is uploaded through an application.

[1408] 2. Receiving and Viewing Feedback

[1409] The device receives the feedback sent from the server and displays it in the user interface. For example, the app might display "An error occurred during the calculation."

[1410] User behavior

[1411] The user enters and submits answer data, checks feedback, and re-studies.

[1412] 1. Answers to the questions

[1413] Users write answers to tests and workbooks on paper, for example, by handwriting the answer to the math problem "2x + 3 = 7."

[1414] 2. Sending the answer data

[1415] The user takes a photo of the answer sheet with their device and sends it to the server. For example, they can take a photo of the answer sheet with their smartphone camera and send it using the app.

[1416] 3. Review feedback and relearn

[1417] The user checks the feedback displayed on the device and understands the mistakes and explanations. For example, the user reads the feedback on the mistakes in the app and then works on the same type of relearning problem.

[1418] In this way, a system has been realized that links the server, terminals, and users to analyze answers in real time, provide feedback, and support re-learning. This system makes it possible to provide learning support in bookstores and stores that sell learning materials, and users can receive immediate feedback, significantly improving their learning efficiency.

[1419] Prompt Sentence Examples

[1420] Here is an example of a prompt that the user can use to submit their answer data:

[1421] Please upload the image data of your answer sheet:

[1422] The correct answer to this math problem is "x = 2." What was your answer to this problem? Take a photo of your answer sheet with your smartphone and upload it to this app.

[1423] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1424] Step 1:

[1425] The user takes a picture of the answer sheet with their smartphone.

[1426] Input: Handwritten answer sheet

[1427] Output: Image data of the answer sheet

[1428] Step 2:

[1429] The device displays the captured image data on the application and asks the user to confirm the upload.

[1430] Input: Image data of answer sheet

[1431] Output: Upload confirmation message

[1432] Step 3:

[1433] The user taps the "Upload" button on the application to send the image data of the answer sheet to the server.

[1434] Input: image data of answer sheet, user operation

[1435] Output: Image data sent to the server

[1436] Step 4:

[1437] The server preprocesses the received image data, correcting for skew and cropping, and uses OpenCV to reshape the image.

[1438] Input: Image data of answer sheet

[1439] Output: Preprocessed image data

[1440] Step 5:

[1441] The server performs OCR processing on the preprocessed image data using pytesseract and converts it into text data.

[1442] Input: Preprocessed image data

[1443] Output: Converted text data

[1444] Step 6:

[1445] The server analyzes the converted text data and compares it with the correct answer to evaluate the accuracy of the answer.

[1446] Input: converted text data, correct answer data

[1447] Output: Determine whether the answer is correct or incorrect

[1448] Step 7:

[1449] The server identifies the error and analyzes the step that caused the error.

[1450] Input: Answer accuracy evaluation result

[1451] Output: Specific information about the error

[1452] Step 8:

[1453] The server generates appropriate explanations and guidance based on the mistakes, using a generative AI model to generate the feedback content.

[1454] Input: Specific information about the mistake, correct answer data

[1455] Output: Explanatory and instructional text

[1456] Step 9:

[1457] The server sends feedback to the user's device, including a description of the error and an appropriate explanation.

[1458] Input: Explanation and instructional text

[1459] Output: Feedback sent to the user's device

[1460] Step 10:

[1461] The terminal displays the received feedback on a user interface.

[1462] Input: Feedback data from the server

[1463] Output: Feedback displayed in the user interface

[1464] Step 11:

[1465] The user reviews and understands the displayed feedback and, if necessary, works through the provided refresher questions.

[1466] Input: Displayed feedback

[1467] Output: Improve the user's learning behavior and perform re-learning.

[1468] This series of processes provides efficient learning support in real time, from sending answer data to receiving feedback.

[1469] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1470] This invention is an efficient learning support system that receives image data of answer sheets, analyzes the data, generates feedback, and provides it to test takers in real time. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more appropriate instruction can be provided. This system significantly improves test takers' learning efficiency and can provide appropriate instruction regardless of time or place.

[1471] Server-side behavior

[1472] The server plays a central role in the system and performs the following functions:

[1473] 1. Receiving answer data

[1474] The server receives the image data of the answer sheet sent by the contestant (user) through the terminal.

[1475] Example: Taking a photo of a math test answer sheet with your smartphone and sending it via an app.

[1476] 2. Image data preprocessing

[1477] The server preprocesses the image data of the answer sheet, correcting tilt and cropping it.

[1478] Example: If the edges of the answer sheet are cut, make the appropriate corrections so that the answer can be analyzed.

[1479] 3. OCR Processing and Analysis

[1480] The server uses OCR technology to recognize characters and formulas from image data and convert them into text data.

[1481] Example: Recognizing handwritten "2x + 3 = 7" and converting it to text form.

[1482] 4. Marking answers

[1483] The server runs the converted text data through a scoring algorithm to assess the accuracy of the answer.

[1484] Example: If the answer is "x = 3", determine that this is incorrect.

[1485] 5. Identifying and analyzing errors

[1486] The server analyzes the incorrect answer to determine where the mistake occurred and identifies the mistake.

[1487] Example: Identifying calculation errors or incorrect steps in a solution.

[1488] 6. Feedback Generation

[1489] The server generates appropriate explanations and instructions, including the correct answers.

[1490] Example: Generate an explanation such as "There was an error in the calculation process, so please check the calculation rules again."

[1491] 7. Presenting Retraining Questions

[1492] The server automatically generates relearning questions related to the error and presents them to the solver.

[1493] Example: Provide the respondent with a "similar equation problem" again.

[1494] 8. How the Emotion Engine Works

[1495] The server uses an emotion engine to analyze the user's facial expressions and tone of voice to recognize their emotional state.

[1496] For example, using a camera and microphone to monitor a user's facial expressions and tone of voice in real time.

[1497] 9. Use of Emotional Data

[1498] Based on the recognized emotional data, the server evaluates the user's stress level and motivation, and adjusts the learning guidance and feedback content.

[1499] Example: If the user is feeling stressed, offer encouraging comments or suggest a break.

[1500] Operation on the terminal side

[1501] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[1502] 1. Sending answer data

[1503] The terminal transmits the image data of the answer sheet photographed by the user to the server.

[1504] Example: Uploading an image of the answer sheet through the application.

[1505] 2. Receiving and Viewing Feedback

[1506] The terminal receives the feedback sent from the server and displays it on the user interface.

[1507] Example: The app displays advice stating that "there was an error in the calculation process."

[1508] 3. Collecting Emotional Data

[1509] The device uses a camera and microphone to collect the user's facial expressions and voice in real time and transmits them to a server.

[1510] Example: Capture the user's face with a camera and use that data for analysis.

[1511] User behavior

[1512] Users input and submit answer data, check and re-study feedback, and provide emotion data.

[1513] 1. Answers to the questions

[1514] Users write their answers to tests and questions on paper.

[1515] Example: Handwrite the answer to the math problem "2x + 3 = 7."

[1516] 2. Sending the answer data

[1517] The user takes a photo of the answer sheet with their device and sends it to the server.

[1518] Example: Take a photo of your answer sheet with your smartphone camera and send it via the app.

[1519] 3. Review feedback and relearn

[1520] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then works on the re-learning questions.

[1521] For example, read the feedback on your mistakes in the app and try the same type of relearning exercise.

[1522] 4. Providing Emotion Data

[1523] Users use the device's camera and microphone to provide emotional data to the system through facial expressions and voice.

[1524] For example, you turn to the device's camera so that the emotion engine can analyze your facial expressions and tone of voice while solving a problem.

[1525] In this way, the present invention realizes a system that analyzes answers in real time, provides feedback, and supports relearning through collaboration between the server, terminal, and user. Furthermore, the introduction of an emotion engine enables individualized instruction based on the user's emotional state, further improving learning efficiency and providing psychological support.

[1526] The processing flow will be explained below.

[1527] Step 1:

[1528] Users write their answers on a paper test or workbook, and then use their smartphone camera to take a photo of the answer sheet.

[1529] Step 2:

[1530] After taking a picture of the answer sheet, the user launches the dedicated application and selects the captured image. When ready, the image data is sent from the device to the server.

[1531] Step 3:

[1532] The device performs preprocessing on the received image data, including correcting the tilt of the answer sheet and cropping it so that the answer area is clearly visible.

[1533] Step 4:

[1534] The device sends the pre-processed image data to the server via the API, and the server receives the image data sent from the device.

[1535] Step 5:

[1536] The server analyzes the received image data using OCR technology, which also includes handwriting recognition technology, to convert characters and mathematical expressions in the image into text data.

[1537] Step 6:

[1538] The server runs the converted text data through a scoring algorithm to evaluate the accuracy of the answer, thereby determining whether the answer is correct or incorrect.

[1539] Step 7:

[1540] The server analyzes the incorrect answers to identify where the error occurred, for example, by identifying calculation errors or errors between steps in the solution.

[1541] Step 8:

[1542] Based on the identified mistakes, the server generates a correct solution and explanation, including procedural and theoretical explanations.

[1543] Step 9:

[1544] The device receives the feedback data sent from the server and displays it on the user interface. The feedback includes whether the answer was correct or incorrect, the cause of the error, and advice for relearning.

[1545] Step 10:

[1546] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then attempts to re-learn the problems presented by the server.

[1547] Step 11:

[1548] The device uses a camera and microphone to capture the user's facial expressions and voice tone, and transmits them to a server in real time, thereby monitoring their emotional state.

[1549] Step 12:

[1550] The server uses an emotion engine to analyze the user's emotional state from their facial expressions and tone of voice, thereby determining their stress level, motivation, etc.

[1551] Step 13:

[1552] The server adjusts the learning guidance and feedback based on the user's emotional state. For example, if the user is in a high-stress state, it will suggest encouraging messages or a break.

[1553] Step 14:

[1554] The server then sends tailored feedback and instruction to the device, including emotionally tailored content.

[1555] Step 15:

[1556] The device displays the feedback and guidance sent from the server on the user interface, allowing the user to receive support tailored to their own emotional state.

[1557] Through this series of processes, the system provides users with real-time feedback and support for relearning, and furthermore, the emotion engine enables individualized instruction tailored to the user's psychological state.

[1558] Example 2

[1559] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1560] Conventional learning support systems lacked the accuracy of receiving and analyzing image data from answer sheets, resulting in problems such as not being able to provide appropriate feedback to learners. Furthermore, there was also the issue that instruction did not take into account the learner's emotional state, preventing the effectiveness of learning from being maximized. This resulted in a decrease in learners' learning efficiency and made it difficult to provide appropriate individual instruction.

[1561] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1562] In this invention, the server includes means for receiving image data of the answer sheet, means for preprocessing the received image data to correct skew and trim it, means for analyzing the preprocessed image data, recognizing characters and formulas and converting them into text data, means for identifying incorrect answers and generating appropriate explanations and guidance, means for providing the generated explanations and guidance to the recipient, means for analyzing the user's facial expressions and voice to recognize their emotional state, means for adjusting the feedback content based on the recognized emotional data, and means for generating and presenting relearning questions related to the mistakes. This not only improves the accuracy of answers but also enables individual guidance that corresponds to the learner's emotional state, thereby improving learning efficiency and providing enhanced psychological support.

[1563] "Image data of the answer sheet" refers to data that has been photographed or captured with a camera or scanner of the answers written by the user on paper and converted into digital format.

[1564] "Means for receiving" refers to a combination of hardware and software for capturing image data of the answer sheet to the server via a network.

[1565] The "pre-processing means for tilt correction and trimming" is a combination of hardware and software that uses image processing technology on the image data of the answer sheet to correct the tilt and cut out unnecessary parts.

[1566] The "means for analyzing, recognizing characters and mathematical formulas, and converting them into text data" refers to a combination of hardware and software that uses OCR technology to extract characters and mathematical formulas from the image data of the answer sheet and convert them into text format.

[1567] The "means for identifying incorrect answers and generating appropriate explanations and guidance" is a combination of hardware and software that applies evaluation algorithms to the converted text data, identifies incorrect answers, and generates appropriate explanations and guidance using natural language processing techniques.

[1568] The "means for providing the generated commentary and instruction to the recipient" is a combination of hardware and software for transmitting the generated commentary and instruction to the user's terminal via a network and displaying it on a user interface.

[1569] The "means for recognizing the emotional state by analyzing the user's facial expression and voice" is a combination of hardware and software that uses a camera and microphone to collect data on the user's facial expression and voice, and analyzes it using an emotion recognition algorithm.

[1570] The "means for adjusting the feedback content based on the recognized emotional data" is a combination of hardware and software for evaluating the user's emotional state based on the analysis results and appropriately changing the feedback and instruction content as necessary.

[1571] The "means for generating and presenting relearning questions related to the error portion" is a combination of hardware and software for generating new questions related to the identified error portion and presenting them to the user.

[1572] MODE FOR CARRYING OUT THE INVENTION

[1573] The present invention is an efficient learning support system that receives image data of answer sheets, analyzes the data, generates feedback, and provides it to test takers in real time. Furthermore, by using an emotion engine, it is possible to provide appropriate guidance based on the user's emotional state. This system improves the test taker's learning efficiency and can provide appropriate guidance regardless of time or location. A specific embodiment of this system is described below.

[1574] Server-side behavior

[1575] 1. Receiving answer data

[1576] The server has a means to receive image data of answer sheets sent by users from their devices. The hardware used is the server for receiving data, and the software is the data transfer protocol. For example, a user takes a photo of a math test answer sheet using a smartphone app and sends it to the server. The server saves the sent image file in a specific directory and returns a response confirming receipt.

[1577] 2. Image data preprocessing

[1578] The server has the means to preprocess the image data it receives. The software used is the OpenCV library, which corrects the image's skew and crops out unnecessary parts. For example, if the edges of an answer sheet are cut off, this is corrected so that the image can be analyzed.

[1579] 3. OCR Processing and Analysis

[1580] The server uses OCR technology to recognize characters and mathematical formulas from image data and convert them into text data. The software used is the Tesseract OCR library. For example, it recognizes handwritten text such as "2x + 3 = 7" and converts it into text format. The converted text data is then stored in a database.

[1581] 4. Marking answers

[1582] The server has a means to evaluate the converted text data by running it through a scoring algorithm. The scoring algorithm is executed by a dedicated program. For example, if the answer is "x = 3," it will determine that this is incorrect, generate an evaluation result, and save it in JSON format.

[1583] 5. Identifying and analyzing errors

[1584] The server has a means to identify the incorrect answers and analyze their causes. It uses an error analysis algorithm to identify the causes of the incorrect answers by matching them with specific error patterns. For example, it identifies calculation errors or incorrect steps in the solution.

[1585] 6. Feedback Generation

[1586] The server has the means to generate appropriate explanations and instructional feedback. The technology used is natural language processing (NLP) using a generative AI model (e.g., GPT-3). For example, it generates an explanation such as, "There was an error in the calculation process. Please check the calculation rules again."

[1587] 7. Presenting Retraining Questions

[1588] The server has a means to automatically generate relearning questions related to the mistake and present them to the solver. It uses a problem generation algorithm. For example, it generates "similar equation problems" and provides them to the solver.

[1589] 8. How the Emotion Engine Works

[1590] The server has a means to analyze the user's facial expressions and voice to recognize their emotional state. It uses emotion recognition software (e.g., Emotion API) and uses a camera and microphone to analyze the user's facial expressions and tone of voice in real time.

[1591] 9. Use of Emotional Data

[1592] The server has a means to adjust the feedback content based on the emotional data. The feedback and instruction content is appropriately changed based on the analysis results. For example, if the user is feeling stressed, an encouraging comment is added.

[1593] Operation on the terminal side

[1594] 1. Sending answer data

[1595] The device has a means to send image data of the answer sheet taken by the user to the server. The hardware used is a smartphone, and the software is a dedicated application. For example, the user can take a photo of the answer sheet with the smartphone camera and send it to the server via the application.

[1596] 2. Receiving and Viewing Feedback

[1597] The device has a means to receive and display feedback from the server. This is done using a UI component within the application. For example, the app might display feedback indicating that an error occurred in the calculation process.

[1598] 3. Collecting Emotional Data

[1599] The device has a means for collecting the user's facial expressions and voice in real time using a camera and microphone and sending the data to a server. For example, the device captures the user's face with a camera and sends the data to a server.

[1600] User behavior

[1601] 1. Answers to the questions

[1602] Users write answers to tests and workbooks on paper, for example, by handwriting the answer to the math problem "2x + 3 = 7."

[1603] 2. Sending the answer data

[1604] The user takes a photo of the answer sheet with their device and sends it to the server. The device used is a smartphone, and a dedicated app is used. For example, the answer sheet is photographed with the smartphone camera and sent via the app.

[1605] 3. Review feedback and relearn

[1606] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then attempts to re-test the questions. For example, the user reads the feedback on the mistakes made on the app and attempts to re-test the questions.

[1607] 4. Providing Emotion Data

[1608] Users can provide emotional data through facial expressions and voice using the device's camera and microphone. For example, when solving a problem, they answer in front of the camera, and their voice is also recorded.

[1609] Examples of prompt statements

[1610] example:

[1611] "Take a picture of your 2x + 3 = 7 answer sheet and submit it. Then, check your answer and receive feedback to try again if you made a mistake. Our emotion engine will also analyze your emotional state to see if you're feeling stressed."

[1612] This system works by linking the server, terminals, and users to analyze answers in real time, provide feedback, and support re-learning.The introduction of an emotion engine enables individualized instruction based on the user's emotional state, improving learning efficiency and providing psychological support.

[1613] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1614] Step 1:

[1615] Sending answer data

[1616] The terminal transmits image data of the answer sheet photographed by the user to the server.

[1617] Input: An image file of the answer sheet taken by the user with their smartphone

[1618] Output: Notification of completion of image data upload to the server

[1619] Specific operation: Using the dedicated application on the device, the user takes a photo of the answer sheet and taps the send button to upload the image to the server. After the upload is complete, the device displays a notification that the upload was successful.

[1620] Step 2:

[1621] Receiving answer data

[1622] The server receives the image data of the answer sheet sent from the terminal.

[1623] Input: Image data from the device

[1624] Output: Save image data and confirm receipt response

[1625] Specific operation: The server receives the image data of the answer sheet via an HTTP request, saves the image file in a specific directory, and returns a success message to the terminal after completion of the reception.

[1626] Step 3:

[1627] Image data preprocessing

[1628] The server performs pre-processing on the received image data.

[1629] Input: Saved image data

[1630] Output: Deskewed and cropped image data

[1631] How it works: The server uses the OpenCV library to straighten the image and crop unnecessary parts. The processed image data is then re-saved in a format suitable for analysis.

[1632] Step 4:

[1633] OCR Processing and Analysis

[1634] The server analyzes the image data using OCR technology and converts it into text data.

[1635] Input: Preprocessed image data

[1636] Output: Parsed text data

[1637] How it works: The server uses the Tesseract OCR library to extract characters and formulas from the image data and convert them into text format, which is then stored in a database.

[1638] Step 5:

[1639] Grading answers

[1640] The server evaluates the text data using a scoring algorithm.

[1641] Input: Text data generated by OCR processing

[1642] Output: Scoring result data (correct / incorrect)

[1643] How it works: The server runs a dedicated scoring algorithm, compares the text data with the correct answers, and generates a score, which is saved in JSON format.

[1644] Step 6:

[1645] Identifying and analyzing errors

[1646] The server identifies and analyzes the incorrect answers.

[1647] Input: Grade result data

[1648] Output: Analysis data of the error location and its cause

[1649] Specific operation: The server runs an error analysis algorithm to identify the incorrect answers and analyze the causes (calculation errors, incorrect solutions, etc.). The analysis results are stored in a database.

[1650] Step 7:

[1651] Feedback Generation

[1652] The server generates appropriate commentary and instructional feedback.

[1653] Input: Analysis data of the error location and cause

[1654] Output: The generated feedback text

[1655] Specific operation: The server uses a generative AI model (e.g., GPT-3) to generate feedback text based on the analyzed data. The generated feedback is stored in a database.

[1656] Step 8:

[1657] Presenting re-learning questions

[1658] The server generates and presents relearning questions related to the error location.

[1659] Input: Analysis data of the error

[1660] Output: Retraining problem data

[1661] Specific operation: The server uses a problem generation algorithm to generate new retraining problems based on the identified mistakes. The generated problems are stored in a database.

[1662] Step 9:

[1663] Emotion Engine Operation

[1664] The server analyzes the user's facial expression and voice data to recognize their emotional state.

[1665] Input: Facial expression and voice data sent from the device

[1666] Output: Recognized emotion data

[1667] How it works: The server uses emotion recognition software to analyze the transmitted data and recognize the user's emotional state (e.g., stress, excitement, concentration). The analysis results are stored in a database.

[1668] Step 10:

[1669] Use of Emotional Data

[1670] The server adjusts the feedback content based on the recognized emotion data.

[1671] Input: Recognized emotion data

[1672] Output: Adjusted feedback text

[1673] Specific operation: The server refers to the emotion data and adjusts the feedback content appropriately according to the user's emotional state. For example, if the user is feeling stressed, it adds encouraging comments. The adjusted feedback is stored in a database and sent to the device.

[1674] (Application example 2)

[1675] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1676] While conventional learning support systems focus on evaluating answers and providing feedback, they do not adequately provide individualized instruction tailored to the learner's emotional state. This makes it difficult to provide appropriate instruction tailored to the learner's motivation and level of understanding, resulting in limited improvements in learning efficiency. Furthermore, while real-time feedback and appropriate instruction tailored to the learner's emotional state are required in training scenarios such as factories, current systems have difficulty achieving this.

[1677] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving image data of an answer sheet, means for analyzing the received image data and converting it into text data, means for evaluating the accuracy of the answer based on the converted text data and identifying errors, means for generating appropriate explanations and guidance for the identified errors, means for providing the generated explanations and guidance to the recipient, means for identifying the user's emotional state via a camera and a voice input device, and means for adjusting the feedback content based on the identified emotional state. This allows individualized instruction and appropriate feedback to be provided in real time according to the learner's emotional state, improving learning efficiency. Furthermore, similar effects can be expected in training scenarios such as factories, contributing to the efficiency of employee skill acquisition.

[1678] An "answer sheet" is a paper sheet on which answers are written by hand and which contains questions and answers.

[1679] "Image data" refers to digital data of the answer sheet obtained using a device such as a camera or scanner.

[1680] "Means for receiving" refers to an interface or function for importing image data into a system such as a server.

[1681] The "means for analyzing and converting into text data" is a function that uses optical character recognition (OCR) technology to analyze image data and convert characters and mathematical formulas into text format.

[1682] "Means for assessing the accuracy of answers and identifying errors" refers to algorithms or methods for determining whether answers are correct or incorrect based on text data and for identifying incorrect parts of the answers.

[1683] The "means for generating appropriate explanations and guidance" is a function that automatically generates the correct answer, its explanation, and further guidance for incorrect answers.

[1684] The "means for providing" refers to an interface or function for notifying the user of the generated feedback and instruction content.

[1685] "Camera and audio input device" refers to a hardware device for capturing the user's facial expressions and voice.

[1686] The "means for identifying emotional state" refers to algorithms or software that analyze captured facial and voice data to interpret the user's emotions.

[1687] The "means for adjusting the feedback content" is a function that dynamically changes and adjusts the content of the feedback provided, taking into account the identified emotional state.

[1688] This invention is a learning support system that receives image data of answer sheets, analyzes them, generates feedback, and provides it to users in real time. It also has the ability to identify the user's emotional state using a camera and a voice input device and adjust the feedback content accordingly. The system aims to improve learning and training efficiency.

[1689] Server side behavior:

[1690] The server plays a central role in the system and performs multiple processes to achieve the following functions:

[1691] 1. Receiving answer data: The server receives image data of the answer sheet sent from a terminal equipped with image data transmission functionality. This data can also be used for things like training checklists in factories.

[1692] 2. Image data preprocessing: The server preprocesses the image data of the answer sheet, correcting tilt and removing noise. For example, it uses an image processing library such as OpenCV.

[1693] 3. OCR processing and analysis: The server uses optical character recognition (OCR) technology to convert handwritten characters and mathematical formulas from the image data into text data. The pytesseract library assists in this process.

[1694] 4. Answer scoring: Based on the text data, the scoring algorithm evaluates the accuracy of the answer and identifies incorrect parts. The scoring algorithm compares the answer with predefined correct answer data.

[1695] 5. Feedback generation: The server generates appropriate explanations and instruction methods based on the incorrect answers. For example, if there is a calculation error, the server generates feedback that explains the details.

[1696] 6. Operation of the Emotion Engine: The server analyzes facial images and voice data sent by the user to identify the user's emotional state. By using an emotion analysis library, the stress level and motivation of employees and learners are evaluated.

[1697] 7. Feedback Adjustment: Dynamically adjust the feedback content based on the emotional state identified by the emotion engine, thereby providing guidance that is optimal for the user's psychological state.

[1698] 8. Presenting Rehearsal Questions: The server automatically generates rehearsal questions related to incorrect answers and presents them to the user, helping to correct misunderstandings and deepen understanding.

[1699] On the device side:

[1700] The terminal is responsible for sending input from the user to the server and for receiving and displaying feedback.

[1701] 1. Sending answer data: The terminal sends the image data of the answer sheet taken by the user using the camera to the server.

[1702] 2. Receiving and displaying feedback: The terminal receives the feedback sent by the server and displays it on the user interface, allowing the user to identify their mistakes and find ways to improve.

[1703] 3. Emotion data collection: The device uses a camera and microphone to collect the user's facial expressions and voice in real time and transmits them to the server.

[1704] User behavior:

[1705] The user inputs and submits answer data, checks and re-learns feedback, and provides emotion data.

[1706] 1. Answering questions: The user writes down the answers to the training checklist or questions on paper.

[1707] 2. Sending answer data: The user takes a photo of their handwritten answer on their device and sends it to the server.

[1708] 3. Checking feedback and relearning: The user checks the feedback displayed on the device, understands the mistakes and explanations, and works on relearning questions to solidify their understanding.

[1709] 4. Providing emotional data: The user provides emotional data through facial expressions and voice using the device's camera and microphone.

[1710] This system allows users to study and train efficiently and receive appropriate feedback based on their emotional state.It can also be used in factories and other workplaces to provide appropriate training that takes into account the emotional state of employees, thereby improving their skills and managing stress.

[1711] Examples:

[1712] Below are some example prompts for a generative AI model:

[1713] Build an application that analyzes a user's checklist image and provides feedback. This is a cloud-based learning support system that recognizes the user's emotions from video and audio feeds and tailors the feedback. Specifically, it uses OCR technology to recognize handwritten text, generates feedback based on the response data, and tailors the feedback content through sentiment analysis.

[1714] This prompt conveys specific system requirements and functions to the generative AI model, providing appropriate learning assistance to the user.

[1715] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1716] Step 1:

[1717] The server receives the image data of the answer sheet sent from the terminal. The image data of the answer sheet is received as input and imported into the server. The received data is stored in the internal memory and passed to the next pre-processing step.

[1718] Step 2:

[1719] The server preprocesses the received image data. It uses the received image data as input and performs deskew and noise removal. Specifically, it uses the OpenCV library to convert the image to grayscale and threshold it. The result is a binary image that can be analyzed.

[1720] Step 3:

[1721] The server performs OCR processing on the preprocessed image data to generate text data. Using the preprocessed binary image as input, it reads characters and formulas using the pytesseract library and outputs text data. This allows the text format answer data to be extracted from the handwritten answer sheet.

[1722] Step 4:

[1723] The server evaluates the accuracy of the answers based on the extracted text data. It uses the generated text data as input and compares it with a pre-defined database of correct answers. Specifically, an algorithm determines whether each answer is correct and identifies any errors. The output is a result of the accuracy assessment and data indicating the errors.

[1724] Step 5:

[1725] The server generates appropriate explanations and guidance based on the mistakes. Using the evaluation results and data on the mistakes as input, it automatically generates the explanations using a generative AI model. The output is text data of the explanations and guidance.

[1726] Step 6:

[1727] The server provides the generated explanations and instructions to the terminal, using the text data of the explanations and instructions as input and transferring the data to the terminal, which receives it and displays it on the user interface.

[1728] Step 7:

[1729] The device collects the user's emotional data and sends it to the server. It captures the user's facial expressions and voice in real time using a camera and microphone. It receives facial image data and voice data as input and sends them to the server.

[1730] Step 8:

[1731] The server uses an emotion engine to identify the user's emotional state. It analyzes facial image and voice data received as input to evaluate the user's emotional state (e.g., stress level and motivation). The identified emotion data is obtained as output.

[1732] Step 9:

[1733] The server adjusts the feedback content based on the identified emotional data. Using the emotional data and the generated commentary and instruction data as input, the server changes and adjusts the feedback content according to the emotional state. The adjusted feedback text data is obtained as output.

[1734] Step 10:

[1735] The server sends the adjusted feedback back to the terminal and provides it to the user. The server uses the adjusted feedback text data as input and sends the data to the terminal. The terminal receives the data and displays it on the user interface. This allows the user to receive feedback optimized for their emotional state.

[1736] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1737] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1738] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1739] [Fourth embodiment]

[1740] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1741] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1742] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1743] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1744] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1745] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1746] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1747] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1748] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1749] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1750] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1751] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1752] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1753] This invention is an efficient learning support system that receives image data of answer sheets, analyzes and understands the data, generates feedback, and provides it to test takers in real time. This system significantly improves test takers' learning efficiency and can provide appropriate guidance without being restricted by time or place.

[1754] Server-side behavior

[1755] The server plays a central role in the system and performs the following functions:

[1756] 1. Receiving answer data

[1757] The server receives the image data of the answer sheet sent by the contestant (user) through the terminal.

[1758] Example: Taking a photo of a math test answer sheet with your smartphone and sending it via an app.

[1759] 2. Image data preprocessing

[1760] The server performs image pre-processing such as tilt correction and cropping.

[1761] For example, if the edges of the answer sheet are cut, make appropriate corrections so that the whole sheet can be analyzed.

[1762] 3. OCR Processing and Analysis

[1763] The server uses OCR technology to recognize characters and formulas from image data and convert them into text data.

[1764] Example: Recognize "2x + 3 = 7" using OCR and convert it to text format.

[1765] 4. Marking answers

[1766] The server analyzes the converted text data and determines whether the answer is correct or incorrect.

[1767] Example: If the respondent answers "x = 3", it is determined that this is incorrect.

[1768] 5. Identifying and analyzing errors

[1769] The server analyzes the incorrect answers to determine at which step the error occurred.

[1770] Example: Identifying where parentheses were incorrectly used during a calculation.

[1771] 6. Feedback Generation

[1772] The server generates appropriate text-based explanations and instructions, including the correct answers.

[1773] Example: Generate an explanation such as "You have processed the parentheses incorrectly, so please check the parentheses rules again."

[1774] 7. Presenting Retraining Questions

[1775] The server automatically generates relearning questions related to the incorrect answers and presents them to the test taker.

[1776] Example: Provide the respondent with a "similar equation problem" again.

[1777] Operation on the terminal side

[1778] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[1779] 1. Sending answer data

[1780] The terminal transmits the image data of the answer sheet photographed by the user to the server.

[1781] Example: Uploading an image of the answer sheet through the application.

[1782] 2. Receiving and Viewing Feedback

[1783] The terminal receives the feedback sent from the server and displays it on the user interface.

[1784] Example: The app displays the message "An error occurred during the calculation process."

[1785] User behavior

[1786] The user enters and submits answer data, checks feedback, and re-studies.

[1787] 1. Answers to the questions

[1788] Users write their answers to tests and questions on paper.

[1789] Example: Handwrite the answer to the math problem "2x + 3 = 7."

[1790] 2. Sending the answer data

[1791] The user takes a photo of the answer sheet with their device and sends it to the server.

[1792] Example: Take a photo of your answer sheet with your smartphone camera and send it via the app.

[1793] 3. Review feedback and relearn

[1794] The user checks the feedback displayed on the device and understands the mistakes and explanations.

[1795] For example, read the feedback on your mistakes in the app and then try the same type of relearning exercise.

[1796] In this way, the present invention realizes a system that analyzes answers, provides feedback, and supports re-learning in real time through the cooperation of the server, terminal, and user. This system allows test takers to receive immediate feedback and quickly correct errors, greatly improving learning efficiency.

[1797] The processing flow will be explained below.

[1798] Step 1:

[1799] Users write their answers on a paper test or workbook, and then use their smartphone camera to take a photo of the answer sheet.

[1800] Step 2:

[1801] After taking a picture of the answer sheet, the user launches the dedicated application and selects the captured image. When ready, the image data is sent from the device to the server.

[1802] Step 3:

[1803] The device performs preprocessing on the received image data, including correcting the tilt of the answer sheet and cropping it so that the answer area is clearly visible.

[1804] Step 4:

[1805] The device sends the pre-processed image data to the server via the API, and the server receives the image data sent from the device.

[1806] Step 5:

[1807] The server analyzes the received image data using OCR technology, which also includes handwriting recognition technology, to convert characters and mathematical expressions in the image into text data.

[1808] Step 6:

[1809] The server runs the converted text data through a scoring algorithm to evaluate the accuracy of the answer, thereby determining whether the answer is correct or incorrect.

[1810] Step 7:

[1811] The server analyzes the incorrect answers to identify where the error occurred, for example, by identifying calculation errors or errors between steps in the solution.

[1812] Step 8:

[1813] Based on the identified mistakes, the server generates a correct solution and explanation, including procedural and theoretical explanations.

[1814] Step 9:

[1815] The server generates feedback and provides it to the user, including explanatory notes, reference materials, and even refresher questions.

[1816] Step 10:

[1817] To facilitate re-learning, the server analyzes the user's past answer data and generates practice questions that are optimally tailored to each individual user. These practice questions are designed to reinforce the user's weak points.

[1818] Step 11:

[1819] The device receives the feedback data sent from the server and displays it on the user interface. The feedback includes whether the answer was correct or incorrect, the cause of the error, and advice for relearning.

[1820] Step 12:

[1821] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then works on the re-learning questions to deepen their understanding.

[1822] Step 13:

[1823] The user can ask additional questions as needed through the application, and the device uses voice recognition to convert the user's questions into text data and send it to the server.

[1824] Step 14:

[1825] The server analyzes the user's question and generates an appropriate answer, which is provided in real time in voice or text format.

[1826] Step 15:

[1827] The device displays or plays aloud the received answers to the user to help them understand, and the user can use this information to further their learning.

[1828] Through this series of processes, the system provides users with real-time feedback and re-learning support, improving learning efficiency and comprehension.

[1829] Example 1

[1830] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1831] In conventional learning support systems, the processing of image data of answer sheets after receiving them is too manual, making it difficult to provide efficient guidance in real time. Furthermore, there are issues with the system, such as poor identification of the causes of errors and insufficient provision of re-study questions, which reduces the learner's learning efficiency.

[1832] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1833] In this invention, the server includes means for receiving image data of the answer sheet, means for preprocessing the received image data and correcting tilt and cropping, means for converting the image-processed data into text data using OCR technology, means for evaluating the accuracy of the answers based on the converted text data and identifying errors, means for generating appropriate explanations and guidance for the identified errors using a generative AI model, means for providing the generated explanations and guidance to the recipient, and means for presenting related relearning questions, which enable real-time and efficient analysis of the answer sheet, feedback on errors, and the presentation of appropriate relearning questions.

[1834] "Answer Sheet" means the paper document on which a respondent completes a question.

[1835] "Image data" refers to a file that digitally represents the answer sheet, usually as a photograph or scanned image.

[1836] "Preprocessing" refers to processes such as tilt correction and cropping that are performed to improve the quality of image data.

[1837] "OCR technology" is an abbreviation for "Optical Character Recognition," and is a technology that recognizes characters and mathematical formulas from image data and converts them into text data.

[1838] "Text data" means digital data containing textual information extracted from an image using OCR technology.

[1839] "Means for assessing accuracy" refers to algorithms or programs used to determine whether a respondent's answer is correct.

[1840] "Incorrect part" refers to the part of the answer given by the respondent that is not correct.

[1841] A "generative AI model" is an artificial intelligence model trained using large amounts of data, and refers to a technology that generates sentences and explanations based on specific conditions.

[1842] "Explanation and Guidance" means explanations or advice provided for specific mistakes.

[1843] A "re-study question" is a new question presented to the respondent related to the part where the respondent made a mistake, with the aim of relearning.

[1844] MODE FOR CARRYING OUT THE INVENTION

[1845] This invention is an efficient learning support system that receives image data of answer sheets, analyzes and understands the data, generates feedback, and provides it to answerers in real time. The specific operations of each component, the server, the terminal, and the user, are described in detail below.

[1846] Server-side behavior

[1847] The server plays a central role in the system, performing a series of processes from receiving image data of answer sheets to generating feedback and presenting re-study questions.

[1848] 1. Receiving answer data

[1849] The server receives the image data of the answer sheet from the contestant (user). At this time, it uses an HTTP POST request and saves the uploaded image data. As a framework, Flask or Django can be used.

[1850] 2. Image data preprocessing

[1851] The server uses image processing libraries such as OpenCV to perform preprocessing such as tilt correction and cropping, improving the image quality and making it easier to analyze. For example, the server uses cv2.getRotationMatrix2D and cv2.warpAffine to correct the tilt of the image.

[1852] 3. OCR Processing and Analysis

[1853] The server uses OCR technology such as Tesseract to recognize characters and formulas from the image data and convert them into text data. For example, it uses pytesseract.image_to_string to convert image data into text data.

[1854] 4. Marking answers

[1855] The server uses the converted text data to evaluate the accuracy of the answer, analyzing the answer using a mathematical library such as SymPy. For example, it uses sympy.solve to determine the user's answer.

[1856] 5. Identifying and analyzing errors

[1857] The server identifies where the error occurred and analyzes its cause. It uses the debug library to see at which step of the solution the error occurred, for example by using sympy.simplify to evaluate intermediate results.

[1858] 6. Feedback Generation

[1859] The server uses a generative AI model (e.g., GPT-4) to generate appropriate feedback, including details of the mistake and an explanation of the correct answer. For example, it generates an explanation such as, "You made a mistake in handling the parentheses, so please check the parentheses rules again."

[1860] 7. Presenting Retraining Questions

[1861] The server generates relevant relearning problems and presents them to the solver. For example, it uses a generative AI model to generate similar equation problems such as "x + 5 = 12."

[1862] Operation on the terminal side

[1863] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[1864] 1. Sending answer data

[1865] The device sends the image data of the answer sheet taken by the user to the server. At this time, by using React Native or Flutter, it is possible to easily send an HTTP POST request.

[1866] 2. Receiving and Viewing Feedback

[1867] The device receives the feedback data sent from the server and displays it in a user interface. For example, a React Native app uses fetch to retrieve the feedback and display it on the screen.

[1868] User behavior

[1869] The user enters and submits answer data, checks feedback, and re-studies.

[1870] 1. Answers to the questions

[1871] Users write their answers on a paper-based question sheet. For example, they write their answers to the math problem "2x + 3 = 7" on paper.

[1872] 2. Sending the answer data

[1873] Users take a photo of their answer sheet with their device and send it to the server, or they use their smartphone camera to upload the photo of their answer sheet through a dedicated app.

[1874] 3. Review feedback and relearn

[1875] The user checks the feedback displayed on the device, understands the error, reads the explanation displayed through the application, and then tackles the presented re-learning problem.

[1876] Examples of prompt statements

[1877] Prompt example 1 (preprocessing of image)

[1878] "Generate Python code to skew and crop the answer sheet."

[1879] Prompt sentence example 2 (OCR processing)

[1880] "Write a code to convert mathematical formulas from PDF to text using Tesseract"

[1881] Prompt example 3 (scoring system)

[1882] "Create a Python script that uses SymPy to determine the solution to an equation."

[1883] In this way, the present invention allows the server, terminal, and user to work together to efficiently analyze image data on answer sheets and provide immediate feedback, significantly improving the learner's learning efficiency and enabling them to receive appropriate guidance regardless of time or place.

[1884] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1885] Step 1:

[1886] Sending answer data (terminal side)

[1887] Input: Image data of the answer sheet taken by the user with a smartphone camera

[1888] How it works: The device sends the image data of the answer sheet to the server using an HTTP POST request. Applications using React Native or Flutter can be used. For example, the image of the answer sheet can be selected on the device and sent by pressing the upload button.

[1889] Output: The image data is uploaded to the server.

[1890] Step 2:

[1891] Receiving answer data (server side)

[1892] Input: Image data of the answer sheet sent from the device

[1893] How it works: The server receives an HTTP POST request and saves the uploaded image data. You can use a framework like Flask or Django, and retrieve the file using the request.files['file'] method.

[1894] Output: Image data stored on the server

[1895] Step 3:

[1896] Image data preprocessing (server side)

[1897] Input: Saved image data of answer sheet

[1898] Operation: Uses an image processing library such as OpenCV to perform preprocessing such as tilt correction and cropping. Specifically, calculates the rotation matrix using cv2.getRotationMatrix2D and corrects the image using cv2.warpAffine. This improves image quality and makes it easier to analyze.

[1899] Output: Preprocessed image data

[1900] Step 4:

[1901] OCR processing and analysis (server side)

[1902] Input: Preprocessed image data

[1903] How it works: Using OCR technology such as Tesseract, it recognizes characters and formulas from image data and converts them into text data. Specifically, it extracts the text data using the pytesseract.image_to_string method.

[1904] Output: Extracted text data

[1905] Step 5:

[1906] Answer scoring (server side)

[1907] Input: Extracted text data

[1908] How it works: It uses mathematical libraries such as SymPy to evaluate the accuracy of the answer. Specifically, it uses sympy.solve to solve the equation and compares it with the user's answer to determine whether it is correct or incorrect.

[1909] Output: Correct / incorrect result

[1910] Step 6:

[1911] Identifying and analyzing errors (server side)

[1912] Input: Correct / incorrect result

[1913] How it works: Using the debug library and SymPy functions, we analyze at which step of the solution process the error occurred. Specifically, we identify the cause of the error by evaluating intermediate results using sympy.simplify.

[1914] Output: Detailed information about the error

[1915] Step 7:

[1916] Feedback generation (server side)

[1917] Input:Detailed information about the error

[1918] How it works: Using a generative AI model (e.g., GPT-4), it generates a detailed explanation of the error and a correct answer. For example, it generates feedback such as, "You made a mistake with the parentheses. Please check the parentheses rules again."

[1919] Output: Text data of generated feedback

[1920] Step 8:

[1921] Presenting retraining questions (server side)

[1922] Input:Detailed information about the error

[1923] How it works: It uses a generative AI model to automatically generate related retraining problems, such as similar equation problems like "x + 5 = 12."

[1924] Output: Generated retraining problem

[1925] Step 9:

[1926] Receiving and displaying feedback (device side)

[1927] Input: Feedback and retraining questions sent by the server

[1928] How it works: The device receives the feedback data using an HTTP GET request and displays it in the user interface. If it's a React Native app, it uses the fetch method to get the feedback and display it on the screen.

[1929] Output: Feedback and relearning questions displayed in the user interface

[1930] Step 10:

[1931] Reviewing feedback and relearning (user side)

[1932] Input: Feedback and relearning questions displayed on the device

[1933] How it works: The user reviews the displayed feedback, understands the explanation, and then works on the presented re-learning problem. For example, they can learn by following the "Review parentheses processing" within the application.

[1934] Output: Understanding the error and completing the re-learning

[1935] (Application example 1)

[1936] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1937] Conventional learning support systems have the problem that analyzing answers and generating feedback takes time, making it difficult to provide real-time support. Furthermore, bookstores and stores selling learning materials have limited ways for purchasers to receive on-site learning support. This means that they are unable to receive immediate explanations or guidance, and it can take time to resolve concerns or questions. Furthermore, there is a need for a system that can support efficient learning without being tied to a specific location or time.

[1938] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1939] In this invention, the server includes means for receiving image data of the answer sheet, means for analyzing the received image data and converting it into text data, means for evaluating the accuracy of the answers based on the converted text data and identifying errors, means for generating appropriate explanations and guidance for the identified errors, means for providing the generated explanations and guidance to the recipient, means for transmitting image data of the photographed or uploaded answer sheet to the server, means for providing a user interface for displaying feedback in real time, and devices for providing learning support in commercial facilities and means for using the same, which enable immediate analysis of answers and feedback, making learning support possible, particularly in bookstores and stores that sell learning materials.

[1940] "Answer Sheet" means a medium containing answers to a set of questions in paper or electronic format.

[1941] "Image data" is data containing visual information stored in digital form.

[1942] The "means for receiving" is a method or device for taking in data from the outside.

[1943] An "analyzing means" is a method or device for processing data and understanding its content.

[1944] "Text data" is data stored as character information.

[1945] A "means for converting" is a method or device for changing data from one format to another.

[1946] A "means for assessing accuracy" is a method or device that checks whether the information entered is accurate.

[1947] The "means for identifying the error portion" is a method or device for detecting the error portion.

[1948] A "means for generating appropriate explanations and guidance" is a method or device for creating correct solutions and study advice.

[1949] The "means for providing" refers to a method or device for delivering the generated information or data to the user.

[1950] "Means for taking pictures" refers to a method or device for taking pictures or videos.

[1951] A "server" is a computer system on a network that processes and stores data.

[1952] An "uploading means" is a method or apparatus for transmitting data from a local device to a remote server.

[1953] A "user interface" is an interface through which a user interacts with a system.

[1954] "Real-time display means" refers to a method or device for immediately displaying generated data or information to a user.

[1955] A "commercial establishment" is a public or private place that offers goods or services.

[1956] "Equipment for providing learning support" is hardware or software used to aid in learning.

[1957] "Method of use" refers to a procedure or method for achieving a specific purpose.

[1958] The present invention is an efficient learning support system that receives image data of answer sheets, analyzes and understands the data, and generates and provides feedback. Specific embodiments of the system are described below.

[1959] Server-side behavior

[1960] The server plays a central role in the system and performs the following functions:

[1961] 1. Receiving answer data

[1962] The server receives image data of the answer sheet sent by the test taker via the device. For example, a user takes a photo of the answer sheet for a math test with their smartphone and sends the data to the server via an application.

[1963] 2. Image data preprocessing

[1964] The server performs image preprocessing such as deskewing and cropping. For example, if the edges of an answer sheet are cut off, it will be properly corrected so that the entire sheet can be analyzed.

[1965] 3. OCR Processing and Analysis

[1966] The server uses OCR (Optical Character Recognition) technology to recognize characters and formulas from image data and convert them into text data. Here, we use "pytesseract" for OCR. As an example, OCR recognizes "2x + 3 = 7" and converts it into text format.

[1967] 4. Marking answers

[1968] The server analyzes the converted text data and determines whether the answer is correct or incorrect. For example, if the answerer answers "x = 3," the server determines that this is incorrect.

[1969] 5. Identifying and analyzing errors

[1970] The server analyzes the incorrect answer to determine at which step the error occurred, for example by identifying the incorrect parentheses in the calculation process.

[1971] 6. Feedback Generation

[1972] The server generates appropriate explanations and guidance in text format, including the correct answer. For example, it generates an explanation such as "You made a mistake in handling the parentheses. Please check the parentheses rules again."

[1973] 7. Presenting Retraining Questions

[1974] The server automatically generates relearning questions related to the mistakes and presents them to the test taker. For example, it provides the test taker with "similar equation problems."

[1975] Operation on the terminal side

[1976] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[1977] 1. Sending answer data

[1978] The device sends the image data of the answer sheet taken by the user to the server. For example, the image of the answer sheet is uploaded through an application.

[1979] 2. Receiving and Viewing Feedback

[1980] The device receives the feedback sent from the server and displays it in the user interface. For example, the app might display "An error occurred during the calculation."

[1981] User behavior

[1982] The user enters and submits answer data, checks feedback, and re-studies.

[1983] 1. Answers to the questions

[1984] Users write answers to tests and workbooks on paper, for example, by handwriting the answer to the math problem "2x + 3 = 7."

[1985] 2. Sending the answer data

[1986] The user takes a photo of the answer sheet with their device and sends it to the server. For example, they can take a photo of the answer sheet with their smartphone camera and send it using the app.

[1987] 3. Review feedback and relearn

[1988] The user checks the feedback displayed on the device and understands the mistakes and explanations. For example, the user reads the feedback on the mistakes in the app and then works on the same type of relearning problem.

[1989] In this way, a system has been realized that links the server, terminals, and users to analyze answers in real time, provide feedback, and support re-learning. This system makes it possible to provide learning support in bookstores and stores that sell learning materials, and users can receive immediate feedback, significantly improving their learning efficiency.

[1990] Prompt Sentence Examples

[1991] Here is an example of a prompt that the user can use to submit their answer data:

[1992] Please upload the image data of your answer sheet:

[1993] The correct answer to this math problem is "x = 2." What was your answer to this problem? Take a photo of your answer sheet with your smartphone and upload it to this app.

[1994] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1995] Step 1:

[1996] The user takes a picture of the answer sheet with their smartphone.

[1997] Input: Handwritten answer sheet

[1998] Output: Image data of the answer sheet

[1999] Step 2:

[2000] The device displays the captured image data on the application and asks the user to confirm the upload.

[2001] Input: Image data of answer sheet

[2002] Output: Upload confirmation message

[2003] Step 3:

[2004] The user taps the "Upload" button on the application to send the image data of the answer sheet to the server.

[2005] Input: image data of answer sheet, user operation

[2006] Output: Image data sent to the server

[2007] Step 4:

[2008] The server preprocesses the received image data, correcting for skew and cropping, and uses OpenCV to reshape the image.

[2009] Input: Image data of answer sheet

[2010] Output: Preprocessed image data

[2011] Step 5:

[2012] The server performs OCR processing on the preprocessed image data using pytesseract and converts it into text data.

[2013] Input: Preprocessed image data

[2014] Output: Converted text data

[2015] Step 6:

[2016] The server analyzes the converted text data and compares it with the correct answer to evaluate the accuracy of the answer.

[2017] Input: converted text data, correct answer data

[2018] Output: Determine whether the answer is correct or incorrect

[2019] Step 7:

[2020] The server identifies the error and analyzes the step that caused the error.

[2021] Input: Answer accuracy evaluation result

[2022] Output: Specific information about the error

[2023] Step 8:

[2024] The server generates appropriate explanations and guidance based on the mistakes, using a generative AI model to generate the feedback content.

[2025] Input: Specific information about the mistake, correct answer data

[2026] Output: Explanatory and instructional text

[2027] Step 9:

[2028] The server sends feedback to the user's device, including a description of the error and an appropriate explanation.

[2029] Input: Explanation and instructional text

[2030] Output: Feedback sent to the user's device

[2031] Step 10:

[2032] The terminal displays the received feedback on a user interface.

[2033] Input: Feedback data from the server

[2034] Output: Feedback displayed in the user interface

[2035] Step 11:

[2036] The user reviews and understands the displayed feedback and, if necessary, works through the provided refresher questions.

[2037] Input: Displayed feedback

[2038] Output: Improve the user's learning behavior and perform re-learning.

[2039] This series of processes provides efficient learning support in real time, from sending answer data to receiving feedback.

[2040] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2041] This invention is an efficient learning support system that receives image data of answer sheets, analyzes the data, generates feedback, and provides it to test takers in real time. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more appropriate instruction can be provided. This system significantly improves test takers' learning efficiency and can provide appropriate instruction regardless of time or place.

[2042] Server-side behavior

[2043] The server plays a central role in the system and performs the following functions:

[2044] 1. Receiving answer data

[2045] The server receives the image data of the answer sheet sent by the contestant (user) through the terminal.

[2046] Example: Taking a photo of a math test answer sheet with your smartphone and sending it via an app.

[2047] 2. Image data preprocessing

[2048] The server preprocesses the image data of the answer sheet, correcting tilt and cropping it.

[2049] Example: If the edges of the answer sheet are cut, make the appropriate corrections so that the answer can be analyzed.

[2050] 3. OCR Processing and Analysis

[2051] The server uses OCR technology to recognize characters and formulas from image data and convert them into text data.

[2052] Example: Recognizing handwritten "2x + 3 = 7" and converting it to text form.

[2053] 4. Marking answers

[2054] The server runs the converted text data through a scoring algorithm to assess the accuracy of the answer.

[2055] Example: If the answer is "x = 3", determine that this is incorrect.

[2056] 5. Identifying and analyzing errors

[2057] The server analyzes the incorrect answer to determine where the mistake occurred and identifies the mistake.

[2058] Example: Identifying calculation errors or incorrect steps in a solution.

[2059] 6. Feedback Generation

[2060] The server generates appropriate explanations and instructions, including the correct answers.

[2061] Example: Generate an explanation such as "There was an error in the calculation process, so please check the calculation rules again."

[2062] 7. Presenting Retraining Questions

[2063] The server automatically generates relearning questions related to the error and presents them to the solver.

[2064] Example: Provide the respondent with a "similar equation problem" again.

[2065] 8. How the Emotion Engine Works

[2066] The server uses an emotion engine to analyze the user's facial expressions and tone of voice to recognize their emotional state.

[2067] For example, using a camera and microphone to monitor a user's facial expressions and tone of voice in real time.

[2068] 9. Use of Emotional Data

[2069] Based on the recognized emotional data, the server evaluates the user's stress level and motivation, and adjusts the learning guidance and feedback content.

[2070] Example: If the user is feeling stressed, offer encouraging comments or suggest a break.

[2071] Operation on the terminal side

[2072] The terminal is responsible for transmitting answer data, receiving feedback, and displaying the feedback.

[2073] 1. Sending answer data

[2074] The terminal transmits the image data of the answer sheet photographed by the user to the server.

[2075] Example: Uploading an image of the answer sheet through the application.

[2076] 2. Receiving and Viewing Feedback

[2077] The terminal receives the feedback sent from the server and displays it on the user interface.

[2078] Example: The app displays advice stating that "there was an error in the calculation process."

[2079] 3. Collecting Emotional Data

[2080] The device uses a camera and microphone to collect the user's facial expressions and voice in real time and transmits them to a server.

[2081] Example: Capture the user's face with a camera and use that data for analysis.

[2082] User behavior

[2083] Users input and submit answer data, check and re-study feedback, and provide emotion data.

[2084] 1. Answers to the questions

[2085] Users write their answers to tests and questions on paper.

[2086] Example: Handwrite the answer to the math problem "2x + 3 = 7."

[2087] 2. Sending the answer data

[2088] The user takes a photo of the answer sheet with their device and sends it to the server.

[2089] Example: Take a photo of your answer sheet with your smartphone camera and send it via the app.

[2090] 3. Review feedback and relearn

[2091] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then works on the re-learning questions.

[2092] For example, read the feedback on your mistakes in the app and try the same type of relearning exercise.

[2093] 4. Providing Emotion Data

[2094] Users use the device's camera and microphone to provide emotional data to the system through facial expressions and voice.

[2095] For example, you turn to the device's camera so that the emotion engine can analyze your facial expressions and tone of voice while solving a problem.

[2096] In this way, the present invention realizes a system that analyzes answers in real time, provides feedback, and supports relearning through collaboration between the server, terminal, and user. Furthermore, the introduction of an emotion engine enables individualized instruction based on the user's emotional state, further improving learning efficiency and providing psychological support.

[2097] The processing flow will be explained below.

[2098] Step 1:

[2099] Users write their answers on a paper test or workbook, and then use their smartphone camera to take a photo of the answer sheet.

[2100] Step 2:

[2101] After taking a picture of the answer sheet, the user launches the dedicated application and selects the captured image. When ready, the image data is sent from the device to the server.

[2102] Step 3:

[2103] The device performs preprocessing on the received image data, including correcting the tilt of the answer sheet and cropping it so that the answer area is clearly visible.

[2104] Step 4:

[2105] The device sends the pre-processed image data to the server via the API, and the server receives the image data sent from the device.

[2106] Step 5:

[2107] The server analyzes the received image data using OCR technology, which also includes handwriting recognition technology, to convert characters and mathematical expressions in the image into text data.

[2108] Step 6:

[2109] The server runs the converted text data through a scoring algorithm to evaluate the accuracy of the answer, thereby determining whether the answer is correct or incorrect.

[2110] Step 7:

[2111] The server analyzes the incorrect answers to identify where the error occurred, for example, by identifying calculation errors or errors between steps in the solution.

[2112] Step 8:

[2113] Based on the identified mistakes, the server generates a correct solution and explanation, including procedural and theoretical explanations.

[2114] Step 9:

[2115] The device receives the feedback data sent from the server and displays it on the user interface. The feedback includes whether the answer was correct or incorrect, the cause of the error, and advice for relearning.

[2116] Step 10:

[2117] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then attempts to re-learn the problems presented by the server.

[2118] Step 11:

[2119] The device uses a camera and microphone to capture the user's facial expressions and voice tone, and transmits them to a server in real time, thereby monitoring their emotional state.

[2120] Step 12:

[2121] The server uses an emotion engine to analyze the user's emotional state from their facial expressions and tone of voice, thereby determining their stress level, motivation, etc.

[2122] Step 13:

[2123] The server adjusts the learning guidance and feedback based on the user's emotional state. For example, if the user is in a high-stress state, it will suggest encouraging messages or a break.

[2124] Step 14:

[2125] The server then sends tailored feedback and instruction to the device, including emotionally tailored content.

[2126] Step 15:

[2127] The device displays the feedback and guidance sent from the server on the user interface, allowing the user to receive support tailored to their own emotional state.

[2128] Through this series of processes, the system provides users with real-time feedback and support for relearning, and furthermore, the emotion engine enables individualized instruction tailored to the user's psychological state.

[2129] Example 2

[2130] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2131] Conventional learning support systems lacked the accuracy of receiving and analyzing image data from answer sheets, resulting in problems such as not being able to provide appropriate feedback to learners. Furthermore, there was also the issue that instruction did not take into account the learner's emotional state, preventing the effectiveness of learning from being maximized. This resulted in a decrease in learners' learning efficiency and made it difficult to provide appropriate individual instruction.

[2132] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2133] In this invention, the server includes means for receiving image data of the answer sheet, means for preprocessing the received image data to correct skew and trim it, means for analyzing the preprocessed image data, recognizing characters and formulas and converting them into text data, means for identifying incorrect answers and generating appropriate explanations and guidance, means for providing the generated explanations and guidance to the recipient, means for analyzing the user's facial expressions and voice to recognize their emotional state, means for adjusting the feedback content based on the recognized emotional data, and means for generating and presenting relearning questions related to the mistakes. This not only improves the accuracy of answers but also enables individual guidance that corresponds to the learner's emotional state, thereby improving learning efficiency and providing enhanced psychological support.

[2134] "Image data of the answer sheet" refers to data that has been photographed or captured with a camera or scanner of the answers written by the user on paper and converted into digital format.

[2135] "Means for receiving" refers to a combination of hardware and software for capturing image data of the answer sheet to the server via a network.

[2136] The "pre-processing means for tilt correction and trimming" is a combination of hardware and software that uses image processing technology on the image data of the answer sheet to correct the tilt and cut out unnecessary parts.

[2137] The "means for analyzing, recognizing characters and mathematical formulas, and converting them into text data" refers to a combination of hardware and software that uses OCR technology to extract characters and mathematical formulas from the image data of the answer sheet and convert them into text format.

[2138] The "means for identifying incorrect answers and generating appropriate explanations and guidance" is a combination of hardware and software that applies evaluation algorithms to the converted text data, identifies incorrect answers, and generates appropriate explanations and guidance using natural language processing techniques.

[2139] The "means for providing the generated commentary and instruction to the recipient" is a combination of hardware and software for transmitting the generated commentary and instruction to the user's terminal via a network and displaying it on a user interface.

[2140] The "means for recognizing the emotional state by analyzing the user's facial expression and voice" is a combination of hardware and software that uses a camera and microphone to collect data on the user's facial expression and voice, and analyzes it using an emotion recognition algorithm.

[2141] The "means for adjusting the feedback content based on the recognized emotional data" is a combination of hardware and software for evaluating the user's emotional state based on the analysis results and appropriately changing the feedback and instruction content as necessary.

[2142] The "means for generating and presenting relearning questions related to the error portion" is a combination of hardware and software for generating new questions related to the identified error portion and presenting them to the user.

[2143] MODE FOR CARRYING OUT THE INVENTION

[2144] The present invention is an efficient learning support system that receives image data of answer sheets, analyzes the data, generates feedback, and provides it to test takers in real time. Furthermore, by using an emotion engine, it is possible to provide appropriate guidance based on the user's emotional state. This system improves the test taker's learning efficiency and can provide appropriate guidance regardless of time or location. A specific embodiment of this system is described below.

[2145] Server-side behavior

[2146] 1. Receiving answer data

[2147] The server has a means to receive image data of answer sheets sent by users from their devices. The hardware used is the server for receiving data, and the software is the data transfer protocol. For example, a user takes a photo of a math test answer sheet using a smartphone app and sends it to the server. The server saves the sent image file in a specific directory and returns a response confirming receipt.

[2148] 2. Image data preprocessing

[2149] The server has the means to preprocess the image data it receives. The software used is the OpenCV library, which corrects the image's skew and crops out unnecessary parts. For example, if the edges of an answer sheet are cut off, this is corrected so that the image can be analyzed.

[2150] 3. OCR Processing and Analysis

[2151] The server uses OCR technology to recognize characters and mathematical formulas from image data and convert them into text data. The software used is the Tesseract OCR library. For example, it recognizes handwritten text such as "2x + 3 = 7" and converts it into text format. The converted text data is then stored in a database.

[2152] 4. Marking answers

[2153] The server has a means to evaluate the converted text data by running it through a scoring algorithm. The scoring algorithm is executed by a dedicated program. For example, if the answer is "x = 3," it will determine that this is incorrect, generate an evaluation result, and save it in JSON format.

[2154] 5. Identifying and analyzing errors

[2155] The server has a means to identify the incorrect answers and analyze their causes. It uses an error analysis algorithm to identify the causes of the incorrect answers by matching them with specific error patterns. For example, it identifies calculation errors or incorrect steps in the solution.

[2156] 6. Feedback Generation

[2157] The server has the means to generate appropriate explanations and instructional feedback. The technology used is natural language processing (NLP) using a generative AI model (e.g., GPT-3). For example, it generates an explanation such as, "There was an error in the calculation process. Please check the calculation rules again."

[2158] 7. Presenting Retraining Questions

[2159] The server has a means to automatically generate relearning questions related to the mistake and present them to the solver. It uses a problem generation algorithm. For example, it generates "similar equation problems" and provides them to the solver.

[2160] 8. How the Emotion Engine Works

[2161] The server has a means to analyze the user's facial expressions and voice to recognize their emotional state. It uses emotion recognition software (e.g., Emotion API) and uses a camera and microphone to analyze the user's facial expressions and tone of voice in real time.

[2162] 9. Use of Emotional Data

[2163] The server has a means to adjust the feedback content based on the emotional data. The feedback and instruction content is appropriately changed based on the analysis results. For example, if the user is feeling stressed, an encouraging comment is added.

[2164] Operation on the terminal side

[2165] 1. Sending answer data

[2166] The device has a means to send image data of the answer sheet taken by the user to the server. The hardware used is a smartphone, and the software is a dedicated application. For example, the user can take a photo of the answer sheet with the smartphone camera and send it to the server via the application.

[2167] 2. Receiving and Viewing Feedback

[2168] The device has a means to receive and display feedback from the server. This is done using a UI component within the application. For example, the app might display feedback indicating that an error occurred in the calculation process.

[2169] 3. Collecting Emotional Data

[2170] The device has a means for collecting the user's facial expressions and voice in real time using a camera and microphone and sending the data to a server. For example, the device captures the user's face with a camera and sends the data to a server.

[2171] User behavior

[2172] 1. Answers to the questions

[2173] Users write answers to tests and workbooks on paper, for example, by handwriting the answer to the math problem "2x + 3 = 7."

[2174] 2. Sending the answer data

[2175] The user takes a photo of the answer sheet with their device and sends it to the server. The device used is a smartphone, and a dedicated app is used. For example, the answer sheet is photographed with the smartphone camera and sent via the app.

[2176] 3. Review feedback and relearn

[2177] The user checks the feedback displayed on the device, understands the mistakes and explanations, and then attempts to re-test the questions. For example, the user reads the feedback on the mistakes made on the app and attempts to re-test the questions.

[2178] 4. Providing Emotion Data

[2179] Users can provide emotional data through facial expressions and voice using the device's camera and microphone. For example, when solving a problem, they answer in front of the camera, and their voice is also recorded.

[2180] Examples of prompt statements

[2181] example:

[2182] "Take a picture of your 2x + 3 = 7 answer sheet and submit it. Then, check your answer and receive feedback to try again if you made a mistake. Our emotion engine will also analyze your emotional state to see if you're feeling stressed."

[2183] This system works by linking the server, terminals, and users to analyze answers in real time, provide feedback, and support re-learning.The introduction of an emotion engine enables individualized instruction based on the user's emotional state, improving learning efficiency and providing psychological support.

[2184] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2185] Step 1:

[2186] Sending answer data

[2187] The terminal transmits image data of the answer sheet photographed by the user to the server.

[2188] Input: An image file of the answer sheet taken by the user with their smartphone

[2189] Output: Notification of completion of image data upload to the server

[2190] Specific operation: Using the dedicated application on the device, the user takes a photo of the answer sheet and taps the send button to upload the image to the server. After the upload is complete, the device displays a notification that the upload was successful.

[2191] Step 2:

[2192] Receiving answer data

[2193] The server receives the image data of the answer sheet sent from the terminal.

[2194] Input: Image data from the device

[2195] Output: Save image data and confirm receipt response

[2196] Specific operation: The server receives the image data of the answer sheet via an HTTP request, saves the image file in a specific directory, and returns a success message to the terminal after completion of the reception.

[2197] Step 3:

[2198] Image data preprocessing

[2199] The server performs pre-processing on the received image data.

[2200] Input: Saved image data

[2201] Output: Deskewed and cropped image data

[2202] How it works: The server uses the OpenCV library to straighten the image and crop unnecessary parts. The processed image data is then re-saved in a format suitable for analysis.

[2203] Step 4:

[2204] OCR Processing and Analysis

[2205] The server analyzes the image data using OCR technology and converts it into text data.

[2206] Input: Preprocessed image data

[2207] Output: Parsed text data

[2208] How it works: The server uses the Tesseract OCR library to extract characters and formulas from the image data and convert them into text format, which is then stored in a database.

[2209] Step 5:

[2210] Grading answers

[2211] The server evaluates the text data using a scoring algorithm.

[2212] Input: Text data generated by OCR processing

[2213] Output: Scoring result data (correct / incorrect)

[2214] How it works: The server runs a dedicated scoring algorithm, compares the text data with the correct answers, and generates a score, which is saved in JSON format.

[2215] Step 6:

[2216] Identifying and analyzing errors

[2217] The server identifies and analyzes the incorrect answers.

[2218] Input: Grade result data

[2219] Output: Analysis data of the error location and its cause

[2220] Specific operation: The server runs an error analysis algorithm to identify the incorrect answers and analyze the causes (calculation errors, incorrect solutions, etc.). The analysis results are stored in a database.

[2221] Step 7:

[2222] Feedback Generation

[2223] The server generates appropriate commentary and instructional feedback.

[2224] Input: Analysis data of the error location and cause

[2225] Output: The generated feedback text

[2226] Specific operation: The server uses a generative AI model (e.g., GPT-3) to generate feedback text based on the analyzed data. The generated feedback is stored in a database.

[2227] Step 8:

[2228] Presenting re-learning questions

[2229] The server generates and presents relearning questions related to the error location.

[2230] Input: Analysis data of the error

[2231] Output: Retraining problem data

[2232] Specific operation: The server uses a problem generation algorithm to generate new retraining problems based on the identified mistakes. The generated problems are stored in a database.

[2233] Step 9:

[2234] Emotion Engine Operation

[2235] The server analyzes the user's facial expression and voice data to recognize their emotional state.

[2236] Input: Facial expression and voice data sent from the device

[2237] Output: Recognized emotion data

[2238] How it works: The server uses emotion recognition software to analyze the transmitted data and recognize the user's emotional state (e.g., stress, excitement, concentration). The analysis results are stored in a database.

[2239] Step 10:

[2240] Use of Emotional Data

[2241] The server adjusts the feedback content based on the recognized emotion data.

[2242] Input: Recognized emotion data

[2243] Output: Adjusted feedback text

[2244] Specific operation: The server refers to the emotion data and adjusts the feedback content appropriately according to the user's emotional state. For example, if the user is feeling stressed, it adds encouraging comments. The adjusted feedback is stored in a database and sent to the device.

[2245] (Application example 2)

[2246] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2247] While conventional learning support systems focus on evaluating answers and providing feedback, they do not adequately provide individualized instruction tailored to the learner's emotional state. This makes it difficult to provide appropriate instruction tailored to the learner's motivation and level of understanding, resulting in limited improvements in learning efficiency. Furthermore, while real-time feedback and appropriate instruction tailored to the learner's emotional state are required in training scenarios such as factories, current systems have difficulty achieving this.

[2248] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving image data of an answer sheet, means for analyzing the received image data and converting it into text data, means for evaluating the accuracy of the answer based on the converted text data and identifying errors, means for generating appropriate explanations and guidance for the identified errors, means for providing the generated explanations and guidance to the recipient, means for identifying the user's emotional state via a camera and a voice input device, and means for adjusting the feedback content based on the identified emotional state. This allows individualized instruction and appropriate feedback to be provided in real time according to the learner's emotional state, improving learning efficiency. Furthermore, similar effects can be expected in training scenarios such as factories, contributing to the efficiency of employee skill acquisition.

[2249] An "answer sheet" is a paper sheet on which answers are written by hand and which contains questions and answers.

[2250] "Image data" refers to digital data of the answer sheet obtained using a device such as a camera or scanner.

[2251] "Means for receiving" refers to an interface or function for importing image data into a system such as a server.

[2252] The "means for analyzing and converting into text data" is a function that uses optical character recognition (OCR) technology to analyze image data and convert characters and mathematical formulas into text format.

[2253] "Means for assessing the accuracy of answers and identifying errors" refers to algorithms or methods for determining whether answers are correct or incorrect based on text data and for identifying incorrect parts of the answers.

[2254] The "means for generating appropriate explanations and guidance" is a function that automatically generates the correct answer, its explanation, and further guidance for incorrect answers.

[2255] The "means for providing" refers to an interface or function for notifying the user of the generated feedback and instruction content.

[2256] "Camera and audio input device" refers to a hardware device for capturing the user's facial expressions and voice.

[2257] The "means for identifying emotional state" refers to algorithms or software that analyze captured facial and voice data to interpret the user's emotions.

[2258] The "means for adjusting the feedback content" is a function that dynamically changes and adjusts the content of the feedback provided, taking into account the identified emotional state.

[2259] This invention is a learning support system that receives image data of answer sheets, analyzes them, generates feedback, and provides it to users in real time. It also has the ability to identify the user's emotional state using a camera and a voice input device and adjust the feedback content accordingly. The system aims to improve learning and training efficiency.

[2260] Server side behavior:

[2261] The server plays a central role in the system and performs multiple processes to achieve the following functions:

[2262] 1. Receiving answer data: The server receives image data of the answer sheet sent from a terminal equipped with image data transmission functionality. This data can also be used for things like training checklists in factories.

[2263] 2. Image data preprocessing: The server preprocesses the image data of the answer sheet, correcting tilt and removing noise. For example, it uses an image processing library such as OpenCV.

[2264] 3. OCR processing and analysis: The server uses optical character recognition (OCR) technology to convert handwritten characters and mathematical formulas from the image data into text data. The pytesseract library assists in this process.

[2265] 4. Answer scoring: Based on the text data, the scoring algorithm evaluates the accuracy of the answer and identifies incorrect parts. The scoring algorithm compares the answer with predefined correct answer data.

[2266] 5. Feedback generation: The server generates appropriate explanations and instr...

Claims

1. means for receiving image data of the answer sheet; means for analyzing the received image data and converting it into text data; A method for evaluating the accuracy of answers based on the converted text data and identifying mistakes; means for generating appropriate explanations and guidance for the identified errors; The system includes a means for providing the generated commentary and instruction to a recipient.

2. 10. The system of claim 1, further comprising means for responding in real time to questions from the respondent using voice recognition technology.

3. 2. The system according to claim 1, further comprising means for pre-processing the image data and correcting for skew of the answer sheet.

4. 2. The system according to claim 1, further comprising means for automatically generating and presenting relearning questions corresponding to the error locations.

5. 2. The system according to claim 1, further comprising means for analyzing the user's learning progress based on the received answer data and proposing an optimal learning plan for each user.

6. 10. The system of claim 1, further comprising means for identifying areas of weakness of a user by storing and analyzing data of the user's past answers and providing individually customized learning content.

Images