System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving homework image data; means for converting the received image data into text data using optical character recognition technology; means for analyzing the converted text data to determine a question and an answer; means for generating feedback based on a result of the determination; and means for transmitting the generated feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Existing learning support systems for children lack sufficient support for checking homework answers and understanding learning, making it difficult to provide appropriate feedback and individualized instruction, especially for parents without specialized knowledge. Furthermore, there are insufficient means for seamless online tutoring. This can lead to children being unable to study efficiently and losing motivation to learn. Furthermore, many existing systems are not free to use, creating a financial burden. [Means for solving the problem]

[0005] The present invention provides a system that receives image data of homework, converts it into text data using optical character recognition technology, and then analyzes the questions and answers to generate feedback and send it to the user. It also includes a means for accepting questions about homework, analyzing them, generating appropriate answers, and providing them to the user. It also accepts online tutoring reservation information, manages reservations, and connects users with online tutors at specified times. This improves children's learning efficiency, provides appropriate support and feedback, and reduces the burden on parents, while also reducing financial burdens.

[0006] "Homework" refers to learning assignments given by a school or educational institution that students complete outside of school, such as at home.

[0007] "Image data" refers to data for storing and transmitting visual information in digital format, including photographs and scanned images.

[0008] Optical character recognition (OCR) is a technology that identifies character information within an image and extracts it as text data.

[0009] "Text data" is data in which characters and symbols are recorded electronically, in a format that can be read and processed by a computer.

[0010] "Analysis" is the process of examining given data or information in detail and deriving meaning and patterns.

[0011] A "problem" is a question or task that students are required to answer during a learning activity.

[0012] An "answer" is a response or solution to a problem.

[0013] "Feedback" refers to evaluations, opinions, and advice provided to users regarding their activities and answers.

[0014] A "question" is a logical inquiry that seeks specific information.

[0015] "Online tutoring" refers to instruction and support provided in real time over the Internet.

[0016] "Reservation Information" means data used to reserve a specific date, time, location, or service in advance.

[0017] "Reservation Management" is the process of organizing and maintaining reservation information and ensuring appropriate service delivery based on that information.

[0018] "Users" refer to individuals or groups who use this system, primarily children and their parents. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention is a system for effectively supporting children with their homework, and is implemented in a form in which a server, terminals, and users cooperate with each other. This system has functions for checking homework answers, providing feedback, answering questions, and online tutoring, and detailed embodiments of the system are described below.

[0041] Homework answers and feedback

[0042] A means of receiving image data for homework

[0043] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system.

[0044] Optical character recognition and text conversion methods

[0045] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. OCR technology can convert handwritten mathematical formulas and sentences into text with high accuracy.

[0046] Means of analyzing and judging answers

[0047] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, an AI model is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[0048] Feedback generation and transmission methods

[0049] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Well done! Try your best on the next question" is generated.

[0050] Question and Answer Function

[0051] How questions are received and analyzed

[0052] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[0053] Means of generating and providing appropriate answers

[0054] The server analyzes the received question using an AI model and generates an appropriate answer. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[0055] Online Tutoring

[0056] A means of accepting and managing tutoring bookings

[0057] The terminal accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database.

[0058] A way to connect online at a specified time

[0059] When the reserved time arrives, the server connects the online tutor with the user's device via video chat, etc. For example, if a user reserves English tutoring at 2:00 p.m., the server generates a video chat link at that time and notifies the device.

[0060] System usage examples

[0061] Checking math homework answers

[0062] A child takes a photo of a math homework page with their smartphone and uploads it to the system. The server uses OCR to convert the content into text, and AI determines the answer. Appropriate feedback is generated and displayed on the device.

[0063] English Questions and Answers

[0064] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses AI to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[0065] Online tutoring

[0066] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[0067] The above describes a specific embodiment of the present invention, which enables efficient confirmation of homework and provision of feedback, support for learning comprehension, and smooth operation of online tutoring.

[0068] The processing flow will be explained below.

[0069] Step 1:

[0070] The device activates the camera, and the user takes a photo of their homework. After taking the photo, the device displays a preview of the image for the user to review.

[0071] Step 2:

[0072] The user taps the "Send" button, and the device sends the homework image data to the server, where encryption protocols are used to ensure the data is secure.

[0073] Step 3:

[0074] The server receives the image data sent from the user terminal, and then temporarily stores the data for further processing.

[0075] Step 4:

[0076] The server starts OCR and extracts text information from the received image data. After the OCR process is complete, the extracted text data is stored in an internal data store.

[0077] Step 5:

[0078] The server analyzes the text data to identify the homework questions and answers. Specifically, it uses natural language processing (NLP) algorithms to analyze the structure of the text and distinguish between the questions and answers.

[0079] Step 6:

[0080] The server uses an AI model to determine whether the extracted answer is correct or incorrect. The AI ​​model makes highly accurate judgments based on pre-trained data.

[0081] Step 7:

[0082] The server generates feedback for each question. If the answer is correct, it generates a message such as "That's right, well done!", and if the answer is incorrect, it generates feedback including specific points for improvement such as "That's wrong. Please review this."

[0083] Step 8:

[0084] The server generates feedback and sends it to the user's device, collecting the feedback into a single response message before sending it.

[0085] Step 9:

[0086] The terminal receives the feedback from the server and displays it on the user interface, and the user confirms the displayed feedback.

[0087] Step 10:

[0088] If a user has a question about their homework, they can enter the question in text format into the terminal and send it to the server. For example, they can enter a question such as "Why is this answer correct?"

[0089] Step 11:

[0090] The server analyzes the received question and uses AI models to generate an appropriate answer, based on appropriate references and explanations to help the user understand the answer.

[0091] Step 12:

[0092] The server generates a response and sends it to the user's device. The user receives the response and reviews it to further understand it.

[0093] Step 13:

[0094] When a user desires online tutoring, they input the desired date and time of tutoring into their terminal and send the reservation information to the server. For example, they reserve the following Monday at 2:00 PM.

[0095] Step 14:

[0096] The server receives the reservation information, stores it in a reservation database, and generates and sends a reminder notification to the device when the reservation date and time approaches.

[0097] Step 15:

[0098] When the reserved time arrives, the server generates a link to connect the online tutor and the user's device via video chat and notifies the device. The device launches the video chat application, and the user joins the online tutoring session.

[0099] Step 16:

[0100] After the session ends, the server collects the usage history and stores it for billing purposes. At the end of the month or when the session ends, the server calculates the charges based on the user's usage history and sends the billing information.

[0101] Example 1

[0102] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0103] Currently, there is no integrated system for supporting children's homework by checking answers, understanding answers, answering questions, and providing online tutoring. This means that children do not receive appropriate feedback or learning support, resulting in a decrease in learning effectiveness. Furthermore, manually checking homework is a heavy burden for parents and teachers, and it also hinders children's independent learning.

[0104] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0105] In this invention, the server includes means for receiving image data of homework, means for converting the received image data into text data using optical character recognition technology, means for analyzing the converted text data and determining the problem and answer, means for generating feedback based on the determination results, means for transmitting the generated feedback, means for analyzing the question content and generating an appropriate answer, means for providing the appropriate answer to the user, means for accepting online tutoring reservation information, means for managing reservations, means for connecting the online tutor with the user at a specified time, and means for analyzing the answer and generating feedback using a generative AI model. This enables efficient and integrated support for children's homework. Specifically, this system realizes automatic homework answer checking, provision of appropriate feedback, immediate response to questions, and effective online tutoring, thereby improving learning effectiveness.

[0106] "Homework image data" is an image file of a homework page, which is taken by the user using a terminal and sent to the system.

[0107] "Optical character recognition technology" is a technology that extracts character information from an image and converts it into text data, and is also known as OCR (Optical Character Recognition).

[0108] "Text data" refers to character string data converted using optical character recognition technology, and includes homework content such as mathematical formulas and sentences.

[0109] "Analysis" is the process of identifying the type of question based on text data and evaluating the accuracy of the answer.

[0110] An "answer" is a response to a question included in a homework assignment, and is determined by analysis.

[0111] "Feedback" refers to messages and explanations generated based on the analysis results, and is learning support information provided to the user.

[0112] "Question content" refers to the question or request for explanation that the user inputs to the system in text format.

[0113] An "answer" is appropriate information or explanation provided in response to a user's question, and is generated by the system.

[0114] "Online tutoring" is a method of learning support in which a user connects with an online tutor via video chat or other means at a designated time and receives instruction in real time.

[0115] "Reservation information" refers to the date and time of the online tutoring session desired by the user and other detailed information, which is accepted and managed by the system.

[0116] A "generative AI model" is a machine learning model trained on large datasets and used to analyze answers, provide feedback, and generate answers to questions.

[0117] A "prompt sentence" is an input sentence given to a generative AI model, and contains information that will form the basis for the generated answer and analysis.

[0118] This invention is a system for effectively supporting children with their homework, and is implemented in a form where a server, terminals, and users work together. This system has the functions of checking homework answers, providing feedback, answering questions, and online tutoring.

[0119] Homework answers and feedback

[0120] Receiving image data for homework

[0121] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system. In this case, the device sends the captured image file to an API endpoint (e.g., / upload / homework) that sends the image file to the server.

[0122] Optical Character Recognition and Text Conversion

[0123] The server uses the received image data to perform optical character recognition (OCR) technology, converting the text information in the image into text data. Software used includes Google® Cloud Vision API and Tesseract OCR. This technology converts handwritten mathematical formulas and sentences into text with high accuracy.

[0124] Analysis and evaluation of answers

[0125] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, a generative AI model (e.g., OpenAI's GPT-4®) is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[0126] Feedback Generation and Submission

[0127] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Well done! Try your best on the next question" is generated.

[0128] Question and Answer Function

[0129] Receiving and analyzing questions

[0130] The device accepts a question from the user in text format and sends it to the server. For example, the user might ask, "Why does x equal 2?" The device then sends the question to an API endpoint (e.g., / submit / question) that sends the question to the server.

[0131] Generate and provide appropriate answers

[0132] The server analyzes the received question using a generative AI model and generates an appropriate answer. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[0133] Online Tutoring

[0134] Accepting and managing tutoring appointments

[0135] The device accepts the user's desired date and time for online tutoring and sends the reservation information to the server. The server manages this information and stores it in a reservation database. Specifically, the device sends the reservation information to the API endpoint (e.g., / schedule / tutoring) that sends the information to the server.

[0136] Connect online at a specified time

[0137] When the reserved time arrives, the server connects the online tutor and the device using video chat (e.g., Zoom or GOOGLE MEET (registered trademark)). For example, if a user reserves English tutoring at 2:00 p.m., the server generates a video chat link at that time and notifies the device.

[0138] Component examples and prompt sentence examples

[0139] Checking math homework answers

[0140] A user takes a photo of a math homework page with their smartphone and uploads it to the system. The server converts the content into text using OCR, and a generative AI model determines the answer. Appropriate feedback is generated and displayed on the device.

[0141] English Questions and Answers

[0142] The user enters a question such as "What does this sentence mean?" and sends it to the system. The server uses a generative AI model to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays "This sentence means..."

[0143] Examples of applying prompt statements

[0144] An example prompt might be: "The user asks, 'Why does 6 ÷ 2 = 3?' Please provide a clear explanation for answering this question."

[0145] These detailed explanations of embodiments of the present invention enable efficient checking of children's homework and providing feedback, support for learning comprehension, and smooth operation of online tutoring.

[0146] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0147] Step 1:

[0148] Receiving image data for homework

[0149] The device takes a photo of the homework. The user uses the device's camera to take a photo of the homework page and taps the "upload button" to send the image data to the system.

[0150] Input: Image data of homework taken by the user

[0151] Output: Image data sent to the server

[0152] Specific operation: The image file taken by the device is sent to the API endpoint (e.g., / upload / homework) that sends the image file to the server.

[0153] Step 2:

[0154] Optical Character Recognition and Text Conversion

[0155] The server retrieves the received image data and extracts text information from the image using optical character recognition (OCR) technology.

[0156] Input: Homework image data sent from the device

[0157] Output: Converted text data

[0158] Specific operation: The server uses the Google Cloud Vision API and Tesseract OCR to convert handwritten and printed character information in the image into text data.

[0159] Step 3:

[0160] Analysis and evaluation of answers

[0161] The server analyzes the converted text data to determine the type of question and sends prompts to the AI ​​model to generate and determine the answer.

[0162] Input: Converted text data (e.g. "3x + 5 = 11")

[0163] Output: Answer result (e.g. "x = 2")

[0164] Specific operation: The server sends a prompt to the generative AI model, such as "Solve this equation '3x + 5 = 11'," and analyzes the returned answer to determine its accuracy.

[0165] Step 4:

[0166] Feedback Generation and Submission

[0167] The server generates feedback based on the correctness of the answer and sends it to the user terminal.

[0168] Input: Answer result

[0169] Output: Generated feedback

[0170] Specific operation: The server uses the generative AI model to create a message such as "Good job! Try your best on the next question," and notifies the device.

[0171] Step 5:

[0172] Receiving and analyzing questions

[0173] The terminal accepts questions from the user in text format and sends them to the system.

[0174] Input: A question typed by the user (e.g., "Why is x equal to 2?")

[0175] Output: The question sent to the server

[0176] Specific operation: The device sends the question to the API endpoint (e.g., / submit / question) that sends the question to the server.

[0177] Step 6:

[0178] Generate and provide appropriate answers

[0179] The server analyzes the received question using a generative AI model, generates an appropriate answer, and sends it to the user's device.

[0180] Input: The question sent to the server

[0181] Output: The generated answer (e.g., "3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2")

[0182] Specific operation: The server sends a prompt to the generative AI model saying, "Please answer the user's question 'Why is 6 ÷ 2 = 3?'" and sends the answer to the user's device.

[0183] Step 7:

[0184] Accepting and managing tutoring appointments

[0185] The terminal accepts the user's desired date and time for online tutoring and transmits the reservation information to the server.

[0186] Input: Reservation information entered by the user (date, time, etc.)

[0187] Output: Reservation information sent to the server

[0188] Specific operation: The device sends the reservation information to the API endpoint (e.g., / schedule / tutoring) that sends the reservation information to the server.

[0189] Step 8:

[0190] Connect online at a specified time

[0191] When the reserved time arrives, the server arranges for the online tutor and the user terminal to be connected using video chat or the like.

[0192] Input: Reservation information and current time

[0193] Output: Video chat connection link

[0194] Specific operation: The server generates a video chat link using an API such as Zoom or Google Meet, and then connects by notifying the device at the specified time.

[0195] (Application example 1)

[0196] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0197] Supporting students with their daily homework is extremely important in today's educational environment. However, quickly checking homework answers, providing accurate feedback, and immediately answering students' questions places a heavy burden on educators. Furthermore, scheduling and managing online tutoring sessions is time-consuming, requiring efficient operation. Furthermore, the systems and applications needed to effectively implement this support are insufficient, making it difficult to provide consistent learning support.

[0198] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0199] In this invention, the server includes means for receiving image data of homework, means for converting the received image data into text data using optical character recognition technology, means for analyzing the converted text data and determining the problem and answer, means for generating feedback based on the determination result, means for transmitting the generated feedback, means for accepting questions about homework, means for analyzing the content of the questions and generating appropriate answers, means for providing the generated answers to the user, means for accepting online tutoring reservation information, means for managing reservations based on the reservation information, means for connecting the user with an online tutor at a specified time, means for installing a learning support application on a tablet or smartphone, means for taking and uploading image data of the homework using a camera function of the tablet or smartphone, means for providing the generated feedback to the student in an easy-to-understand manner using natural language processing technology, means for training an AI using structured data and evaluating the accuracy of the answers, and means for supporting online and offline tutoring. This enables fast and accurate homework answer checking, feedback provision, question and answering, and efficient operation of online tutoring in learning environments.

[0200] The "means for receiving image data of homework" is a function that allows a user to take a photo of their homework and send the image data to a server.

[0201] "Means for converting into text data using optical character recognition technology" refers to a function that converts character information in received image data into text data using optical character recognition technology (OCR).

[0202] The "means for analyzing the converted text data and determining the questions and answers" is a function that can automatically determine the homework questions and answers by analyzing the text data converted by OCR.

[0203] The "means for generating feedback based on the assessment result" is a function that automatically generates appropriate feedback based on whether the answer is correct or incorrect.

[0204] The "means for transmitting the generated feedback" is a function for transmitting the generated feedback to the user's terminal and displaying it.

[0205] The "means for accepting questions about homework" is a function that allows a user to input questions about homework in text format and transmit them to the server.

[0206] "Means for analyzing the question content and generating an appropriate answer" is a function that analyzes the received question and generates an appropriate answer using AI.

[0207] The "means for providing the generated answer to the user" is a function for transmitting the generated answer to the user's terminal and displaying it.

[0208] The "means for accepting online tutoring reservation information" is a function that allows a user to input online tutoring reservation information and transmit that information to the server.

[0209] The "means for managing reservations based on reservation information" is a function for managing reservation data based on received reservation information.

[0210] The "means for connecting an online tutor with a user at a designated time" is a function that connects an online tutor with a user at the reserved time and allows them to make a video call or chat.

[0211] The "means for installing the learning support application on a tablet or smartphone" is a function that allows the learning support application to be installed on the user's tablet or smartphone.

[0212] "Means for taking and uploading image data of homework using the camera function of a tablet or smartphone" refers to a function that allows a user to take a photo of their homework using the camera of a tablet or smartphone and upload the image to a server.

[0213] "Means for providing the generated feedback to the student in an easy-to-understand format using natural language processing technology" is a function for providing the generated feedback to the student in an easily understandable format using natural language processing technology.

[0214] "Means for training AI using structured data and evaluating the accuracy of answers" refers to a function that uses structured data to train an AI model and then uses that model to evaluate the accuracy of answers.

[0215] "Means for supporting online and offline tutoring" refers to functions for supporting both online and offline tutoring.

[0216] The present invention provides a system for effectively supporting children in their homework, and is configured so that a server, a terminal, and a user work together. A specific embodiment of the system will be described below.

[0217] Homework answers and feedback

[0218] A means of receiving image data for homework

[0219] The user takes a photo of their homework using the camera function of their smartphone or tablet and sends the image data to the server. The device used is a general smartphone or tablet. The photo is uploaded using, for example, a mobile application.

[0220] Optical character recognition and text conversion methods

[0221] The server then performs optical character recognition (OCR) on the received image data, converting the character information in the image into text data. This process uses OCR technology such as Google Cloud Vision API.

[0222] Means of analyzing and judging answers

[0223] The server analyzes the converted text data and determines the homework problem and answer. The analysis uses Azure® Machine Learning service to train an AI model. For example, it determines that the answer "x = 2" is correct when "3x + 5 = 11" is correct.

[0224] Feedback generation and transmission methods

[0225] The server generates feedback based on the results and sends it to the user's device. The feedback includes not only whether the answer was correct or incorrect, but also detailed explanations and suggestions for improvement. The generated feedback is provided in an easy-to-understand manner using natural language processing technology (e.g., Google Cloud Natural Language API).

[0226] Question and Answer Function

[0227] How questions are received and analyzed

[0228] Users type in a homework question and send it to the server via their device, for example, "Why is x equal to 2?"

[0229] Means of generating and providing appropriate answers

[0230] The server analyzes the received question and generates an appropriate answer using AI (for example, GPT-3 (registered trademark)) and provides it to the user. For example, the generated answer might be, "First, subtract 5 from 3x + 5 = 11 to get 3x = 6, then divide by 3 to get x = 2."

[0231] Online Tutoring

[0232] A means of accepting and managing tutoring bookings

[0233] The user inputs the desired date and time for online tutoring through the application, and the server receives and manages this information. Reservation management systems that can be used include Microsoft® Bookings.

[0234] A way to connect online at a specified time

[0235] When the scheduled time arrives, the server connects the user with the online tutor using video chat (e.g., Zoom API).

[0236] Specific examples

[0237] For example, after a student at a cram school completes their homework, they can take a photo of the page with their smartphone and upload it through an application. This data is then analyzed using optical character recognition on the server, and feedback is provided instantly. If a student asks a question about a problem they can't solve, the AI ​​can analyze it and provide an immediate answer. Furthermore, when parents schedule online tutoring, a video chat will automatically connect at the specified time.

[0238] Prompt Sentence Examples

[0239] Question: "Why does 3x + 5 = 11 result in x = 2?"

[0240] Prompt: "This student wants to know why 3x + 5 = 11 means x = 2. Please provide a detailed explanation."

[0241] Answer: "Let's first rearrange the equation 3x + 5 = 11. First, subtract 5 from both sides to get 3x = 6. Then, divide both sides by 3 to get x = 2. This will give you the answer."

[0242] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0243] Step 1:

[0244] Users take photos of their homework using the camera function of their smartphone or tablet and upload the images to the server through the application. The input is the image data of the homework, and the output is the image data sent to the server.

[0245] Step 2:

[0246] The server performs optical character recognition (OCR) on the received image data and converts the text information in the image into text data. Using the Google Cloud Vision API, the input for this step is image data and the output is text data.

[0247] Step 3:

[0248] The server uses the Azure Machine Learning service to analyze the text data obtained by OCR and automatically determine the homework questions and answers. The input for this step is the text data, and the output is the analysis result, which indicates whether the answer is correct or not.

[0249] Step 4:

[0250] The server generates feedback based on the results of the assessment and provides the generated feedback to the user in an easy-to-understand manner using natural language processing technology (Google Cloud Natural Language API). The input of this step is the correct or incorrect answer, and the output is a feedback message.

[0251] Step 5:

[0252] The user types in a text question about their homework and sends it to the server through the application. The input is the text question data, and the output is the transmission of the question data to the server.

[0253] Step 6:

[0254] The server analyzes the received question and generates an appropriate answer using an AI model such as GPT-3. The input of this step is the question data, and the output is the generated answer data.

[0255] Step 7:

[0256] The server sends the generated answer to the user's terminal and displays it. The input of this step is the generated answer data, and the output is the answer message displayed on the user's terminal.

[0257] Step 8:

[0258] A user inputs reservation information for online tutoring through the application and sends it to the server. The input is reservation information data, and the output is reservation information sent to the server.

[0259] Step 9:

[0260] The server accepts the reservation information and manages it using the reservation management system (Microsoft Bookings). The input of this step is reservation information data, and the output is reservation management data.

[0261] Step 10:

[0262] When the reservation time arrives, the server generates a video call link (e.g., Zoom API) to connect the user with the online tutor. The input of this step is the reservation information and the current time, and the output is the generated video call link.

[0263] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0264] The present invention is a system for effectively supporting children with their homework, implemented in a form in which a server, terminals, and users cooperate with each other. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide more advanced support. This system has the functions of checking homework answers, providing feedback, question answering, online tutoring, and emotion recognition, and detailed embodiments of the system are described below.

[0265] Homework answers and feedback

[0266] A means of receiving image data for homework

[0267] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system.

[0268] Optical character recognition and text conversion methods

[0269] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. OCR technology can convert handwritten mathematical formulas and sentences into text with high accuracy.

[0270] Means of analyzing and judging answers

[0271] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, an AI model is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[0272] Feedback generation and transmission methods

[0273] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Good job! Try your best on the next question" is generated.

[0274] Question and Answer Function

[0275] How questions are received and analyzed

[0276] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[0277] Means of generating and providing appropriate answers

[0278] The server analyzes the received question and generates an appropriate answer using an AI model. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[0279] Online Tutoring

[0280] A means of accepting and managing tutoring bookings

[0281] The terminal accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database.

[0282] A way to connect online at a specified time

[0283] When the reserved time arrives, the server connects the online tutor with the user's device via video chat, etc. For example, if a user reserves English tutoring at 2:00 p.m., the server generates a video chat link at that time and notifies the device.

[0284] Emotion recognition and feedback regulation

[0285] Emotion recognition engine configuration

[0286] The device acquires the user's facial expression data and sends it to the server. The emotion engine analyzes this data and identifies the user's emotions. For example, a camera can capture a user's facial expression while they are doing their homework and recognize their emotional state, such as "smiling," "confused," or "concentrated."

[0287] A means of regulating emotion-based feedback

[0288] The server automatically adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it can send an encouraging message like, "It might be a little difficult, but please try again."

[0289] Emotion-based online tutoring adjustment measures

[0290] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor may recommend teaching that will help the user relax.

[0291] System usage examples

[0292] Checking math homework answers

[0293] A child takes a photo of a math homework page with their smartphone and uploads it to the system. The server uses OCR to convert the content into text, and AI determines the answer. Appropriate feedback is generated and displayed on the device.

[0294] English Questions and Answers

[0295] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses AI to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[0296] Online tutoring

[0297] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[0298] Feedback adjustment using emotion recognition

[0299] While a child is doing their homework, the device's camera captures facial expression data and sends it to a server. The server then uses an emotion engine to analyze the user's emotional state and adjusts the feedback accordingly. For example, if a child is confused, the system will send an encouraging message.

[0300] The above describes specific embodiments of the present invention, which enable efficient homework confirmation and feedback, support for learning comprehension, smooth operation of online tutoring, and advanced emotion-based feedback and guidance.

[0301] The processing flow will be explained below.

[0302] Step 1:

[0303] The device activates the camera, and the user takes a photo of their homework. After taking the photo, the device displays a preview of the image for the user to review.

[0304] Step 2:

[0305] The user taps the "Send" button, and the device sends the homework image data to the server, where encryption protocols are used to ensure the data is secure.

[0306] Step 3:

[0307] The server receives the image data sent from the user terminal, and then temporarily stores the data for further processing.

[0308] Step 4:

[0309] The server starts OCR and extracts text information from the received image data. After the OCR process is complete, the extracted text data is stored in an internal data store.

[0310] Step 5:

[0311] The server analyzes the text data to identify the homework questions and answers. Specifically, it uses natural language processing (NLP) algorithms to analyze the structure of the text and distinguish between the questions and answers.

[0312] Step 6:

[0313] The server uses an AI model to determine whether the extracted answer is correct or incorrect. The AI ​​model makes highly accurate judgments based on pre-trained data.

[0314] Step 7:

[0315] The server generates feedback for each question. If the answer is correct, it generates a message such as "That's right, well done!", and if the answer is incorrect, it generates feedback including specific points for improvement such as "That's wrong. Please review this."

[0316] Step 8:

[0317] The server generates feedback and sends it to the user's device, collecting the feedback into a single response message before sending it.

[0318] Step 9:

[0319] The terminal receives the feedback from the server and displays it on the user interface, and the user confirms the displayed feedback.

[0320] Step 10:

[0321] If a user has a question about their homework, they type the question in text format into the terminal and send it to the server, for example, "Why is this answer correct?"

[0322] Step 11:

[0323] The server analyzes the received question and uses AI models to generate an appropriate answer, based on appropriate references and explanations to help the user understand the answer.

[0324] Step 12:

[0325] The server generates a response and sends it to the user's device. The user receives the response and reviews it to further understand it.

[0326] Step 13:

[0327] The device activates the camera to capture the user's facial expression data and sends it to the server.

[0328] Step 14:

[0329] The server analyzes the facial expression data received by the emotion engine and identifies the user's emotional state, such as "smiling," "confusion," or "concentration."

[0330] Step 15:

[0331] The server adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it might send an encouraging message like, "It might be a little difficult, but please try again."

[0332] Step 16:

[0333] When a user desires online tutoring, they input the desired date and time of tutoring into their terminal and send the reservation information to the server. For example, they reserve the following Monday at 2:00 PM.

[0334] Step 17:

[0335] The server receives the reservation information, stores it in a reservation database, and generates and sends a reminder notification to the device when the reservation date and time approaches.

[0336] Step 18:

[0337] When the reserved time arrives, the server generates a link to connect the online tutor and the user's device via video chat and notifies the device. The device launches the video chat application, and the user joins the online tutoring session.

[0338] Step 19:

[0339] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor can instruct the user to provide instruction that will help the user relax.

[0340] Step 20:

[0341] After the session ends, the server collects the usage history and stores it for billing purposes. At the end of the month or when the session ends, the server calculates the charges based on the user's usage history and sends the billing information.

[0342] Example 2

[0343] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0344] Conventional homework support systems have limited functionality for automatically checking homework answers and providing feedback. Furthermore, most feedback is one-way and does not take the user's emotional state into account, making it difficult to provide effective learning support. Furthermore, the complexity of reserving and implementing online tutoring sessions makes it difficult to provide an efficient learning support environment. It is necessary to solve these issues and provide more effective, personalized homework support to users.

[0345] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: a means for the terminal to receive image data of homework; a means for the server to convert the received image data into text data using optical character recognition technology; a means for the server to analyze the converted text data and determine the question and answer; a means for the server to generate feedback based on the determination result; a means for the server to send the generated feedback to the terminal; a means for the terminal to acquire user facial expression data and send it to the server; and a means for the server to identify the user's emotion using emotion recognition technology and adjust the feedback based on the identification result. This enables efficient confirmation of homework and provision of feedback, support for learning comprehension, and advanced responses according to the user's emotions.

[0346] A "terminal" is a device operated by a user, and is used to take pictures of image data for homework, input questions, and make reservations for online tutoring.

[0347] A "server" is a computer system that receives, processes, and stores data sent from a terminal, and is a device that has functions such as optical character recognition of image data, answer determination, and feedback generation.

[0348] "Image data" is digital data including homework photos and other visual information that is taken by the terminal and sent to the server.

[0349] "Optical Character Recognition (OCR)" is a technology that converts handwritten or printed characters in image data into digital text.

[0350] "Text data" refers to data that represents character information in digital form, converted using optical character recognition technology.

[0351] The "means for determining the answer" is a mechanism by which the server analyzes the text data, identifies the type of question, and evaluates the accuracy of the answer.

[0352] "Feedback" is information provided to the user, including an evaluation of the correctness of the answer, reference explanations, and points for improvement.

[0353] "Emotion recognition technology" is a technology that analyzes a user's facial expression data and identifies the user's emotional state (for example, smiling, confused, focused, etc.).

[0354] "Online tutoring" refers to video chat and other remote instruction formats for educational assistance delivered in real time over the internet.

[0355] This invention is a system for effectively supporting children with their homework, implemented in a form where a server, terminals, and users work together. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide more advanced support. This system has the following functions: checking homework answers, providing feedback, question answering, online tutoring, and emotion recognition.

[0356] Homework answers and feedback

[0357] A means of receiving image data for homework

[0358] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system.

[0359] Optical character recognition and text conversion methods

[0360] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. OCR technology can convert handwritten mathematical formulas and sentences into text with high accuracy. Specifically, software such as Tesseract OCR is used.

[0361] Means of analyzing and judging answers

[0362] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, an AI model (e.g., Python + scikit-learn) is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[0363] Feedback generation and transmission methods

[0364] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Good job! Try your best on the next question" is generated.

[0365] Question and Answer Function

[0366] How questions are received and analyzed

[0367] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[0368] Means of generating and providing appropriate answers

[0369] The server analyzes the received question and generates an appropriate answer using an AI model (e.g., GPT-3). The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[0370] Online Tutoring

[0371] A means of accepting and managing tutoring bookings

[0372] The device accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database, typically an SQL or NoSQL database.

[0373] A way to connect online at a specified time

[0374] When the reservation time arrives, the server connects the online tutor and the user's device using a video chat tool (e.g., Zoom API). For example, if a user reserves English tutoring at 2 p.m., the server generates a video chat link at that time and notifies the device.

[0375] Emotion recognition and feedback regulation

[0376] Emotion recognition engine configuration

[0377] The device acquires the user's facial expression data and sends it to a server. The emotion engine analyzes this data and identifies the user's emotions. For example, a camera can capture a user's facial expression while they are doing their homework and recognize their emotional state, such as "smiling," "confused," or "concentrated." Specifically, emotion recognition software such as Microsoft Azure Cognitive Services is used.

[0378] A means of regulating emotion-based feedback

[0379] The server automatically adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it can send an encouraging message like, "It might be a little difficult, but please try again."

[0380] Emotion-based online tutoring adjustment measures

[0381] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor may recommend teaching that will help the user relax.

[0382] System usage examples

[0383] Checking math homework answers

[0384] A child takes a photo of a math homework page with their smartphone and uploads it to the system. The server uses OCR to convert the content into text, and AI determines the answer. Appropriate feedback is generated and displayed on the device.

[0385] English Questions and Answers

[0386] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses AI to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[0387] Online tutoring

[0388] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[0389] Feedback adjustment using emotion recognition

[0390] While a child is doing their homework, the device's camera captures facial expression data and sends it to a server. The server then uses an emotion engine to analyze the user's emotional state and adjusts the feedback accordingly. For example, if a child is confused, the system will send an encouraging message.

[0391] In this way, the system of the present invention is capable of efficiently checking homework and providing feedback, supporting learning comprehension, smoothly operating online tutoring, and providing advanced emotion-based feedback and guidance.

[0392] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0393] Step 1:

[0394] The device receives image data of the homework. The user takes a photo of the homework with their smartphone and saves the image data on the device. The input is the image file taken by the user, and the output is the image data saved on the device. The device sends this image data to the server.

[0395] Step 2:

[0396] The server receives the image data. The server receives the image data sent from the terminal. The input is the image file sent from the terminal, and the output is the image data saved on the server. The server saves this image data in temporary storage.

[0397] Step 3:

[0398] The server converts image data into text data using optical character recognition technology (OCR). Specifically, the server launches an application program (e.g., Tesseract OCR) and performs OCR on the image data. The input is an image file stored on the server, and the output is text data converted by OCR. Handwritten mathematical formulas and sentences are converted into digital text.

[0399] Step 4:

[0400] The server analyzes the converted text data. The server reads the text data and identifies the type of problem (e.g., mathematical formula or word problem). The input is the text data generated by OCR, and the output is the analysis result. The server performs appropriate processing depending on the type of problem identified.

[0401] Step 5:

[0402] The server uses an AI model to determine the answer. Specifically, the server uses a program (e.g., Python + scikit-learn) to analyze the text data and determine the accuracy of the answer. The input is the analyzed text data, and the output is the result of determining whether the answer is correct. For example, it determines whether the answer "3x + 5 = 11" is correct as "x = 2."

[0403] Step 6:

[0404] The server generates the feedback. The feedback is created based on the result of the answer's correctness. Specifically, the server generates the feedback text by combining template messages and reference explanations. The input is the result of the answer's correctness, and the output is the generated feedback text. For example, it generates feedback such as "That's correct! Excellent! Good luck on the next question."

[0405] Step 7:

[0406] The server sends the generated feedback to the terminal. The generated feedback is sent to the user's terminal. The input is the generated feedback sentence, and the output is the feedback displayed on the terminal. The user checks the feedback on the terminal.

[0407] Step 8:

[0408] The device acquires the user's facial expression data and sends it to the server. For example, the device's camera captures the user's facial expression while they are doing their homework, and sends the data to the server. The input is the facial expression image captured by the camera, and the output is the facial expression data sent to the server.

[0409] Step 9:

[0410] The server uses emotion recognition technology to identify the user's emotions. Specifically, the server uses emotion recognition software (e.g., Microsoft Azure Cognitive Services) to analyze facial expression data and identify the user's emotional state. The input is facial expression data, and the output is the identified emotional state. Recognized emotions include "smiling," "confusion," and "concentration."

[0411] Step 10:

[0412] The server adjusts the feedback based on the emotion identification results. The content and expression of the feedback are automatically adjusted according to the recognized emotional state. The input is the identified emotional state and the previously generated feedback, and the output is the adjusted feedback text. For example, if the user is confused, it adds an encouraging message such as "It might be a little difficult, but please try again."

[0413] Step 11:

[0414] The terminal accepts the user's question and sends it to the server. The user enters a question in text format, and the terminal sends the question to the server. The input is the text format question entered by the user, and the output is the question data sent to the server.

[0415] Step 12:

[0416] The server analyzes the received question and generates an appropriate answer using an AI model. Specifically, the server analyzes the question using an AI model (e.g., GPT-3) and generates an appropriate answer. The input is the question data, and the output is the generated answer. For example, in response to the question "Why is x 2?", the server generates an answer such as "From 3x + 5 = 11, first subtract 5 to make 3x = 6, then divide by 3 to make x = 2."

[0417] Step 13:

[0418] The server sends the generated answer to the terminal. The server sends the generated answer to the user's terminal. The input is the generated answer, and the output is the answer sent to the terminal.

[0419] Step 14:

[0420] The terminal sends online tutoring reservation information to the server. The user inputs the desired date and time of tutoring, and the terminal sends that information to the server. The input is the reservation information, and the output is the reservation data sent to the server.

[0421] Step 15:

[0422] The server manages reservations based on the reservation information. Specifically, the server saves the reservation information in a reservation database and manages the reservations. The input is reservation data, and the output is managed reservation information.

[0423] Step 16:

[0424] When the reservation time arrives, the server connects the online tutor and the user's device using a video chat tool. Specifically, the server generates a video chat link and notifies the device. The input is the reservation information and the video chat link, and the output is a video chat connection between the user and the tutor. For example, the connection can be made at the specified time using a video chat tool (e.g., Zoom API).

[0425] (Application example 2)

[0426] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0427] Currently, in children's homework support systems, it is important not only to judge whether an answer is correct or incorrect, but also to adjust feedback based on the user's emotions, which conventional systems have not been able to do adequately. Furthermore, in brick-and-mortar educational services, it is difficult to provide individual support in real time, resulting in an unsatisfactory user experience. Therefore, the present invention aims to provide more effective homework support and improve the quality of brick-and-mortar educational services by recognizing the user's emotions and adjusting feedback.

[0428] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0429] In this invention, the server includes means for receiving image data of homework, means for converting the received image data into text data using optical character recognition technology, means for analyzing the converted text data and determining the question and answer, means for generating feedback based on the determination result, means for transmitting the generated feedback, means for acquiring facial expression data of the user using a camera, means for analyzing the facial expression data using an emotion analysis engine and identifying the user's emotion, and means for adjusting the feedback content based on the identified emotion. This allows for more effective homework support by recognizing the emotion from the user's facial expression and providing feedback accordingly, thereby enabling the quality of educational services at physical stores to be improved.

[0430] "Homework image data" refers to digital image files of what a child has written as homework, captured with a camera.

[0431] "Means for receiving" refers to a communication means for transferring image data from a terminal to a server.

[0432] "Optical character recognition technology" is a technology that recognizes character information in an image and converts it into digital text.

[0433] "Text data" means digital data containing textual information that has been converted using optical character recognition technology.

[0434] "Means for analyzing" refers to means for processing the converted text data and determining the questions and answers.

[0435] A "means of assessment" is a means for evaluating whether the answer to the question is correct.

[0436] "Feedback" refers to information, including advice and corrections, generated based on analysis and judgment results.

[0437] "Means for sending" refers to communication means for transferring the generated feedback to a user terminal.

[0438] "Camera" refers to an image capture device for capturing a user's facial expression.

[0439] "Facial expression data" refers to image data of the user's face captured by a camera.

[0440] An "emotion analysis engine" is software that has an algorithm that analyzes facial expression data and identifies emotions.

[0441] "Means for identifying" refers to means for classifying the emotional state of a user using a sentiment analysis engine.

[0442] The "means for adjusting the feedback content" is a means for changing the content or expression of the feedback based on the identified emotion.

[0443] This invention is a system for effectively supporting children with their homework, and is implemented in a form in which a server, a terminal, and a user cooperate with each other. The system has functions for checking homework answers, providing feedback, question answering, online tutoring, and emotion recognition. Specific embodiments of the system are described below.

[0444] Homework answers and feedback

[0445] A means of receiving image data for homework

[0446] The device takes a photo of the homework and sends the image data to the server. The device can be a tablet or smartphone, and the user can take a photo of the homework page using the camera.

[0447] Optical character recognition and text conversion methods

[0448] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. Tesseract is used for the OCR technology, which enables handwritten mathematical formulas and sentences to be converted into text with high accuracy.

[0449] Means of analyzing and judging answers

[0450] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, a generative AI model is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[0451] Feedback generation and transmission methods

[0452] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Good job! Try your best on the next question" is generated.

[0453] Question and Answer Function

[0454] How questions are received and analyzed

[0455] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[0456] Means of generating and providing appropriate answers

[0457] The server analyzes the received question and generates an appropriate answer using a generative AI model. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[0458] Online Tutoring

[0459] A means of accepting and managing tutoring bookings

[0460] The terminal accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database.

[0461] A way to connect online at a specified time

[0462] When the reserved time arrives, the server connects the online tutor with the user's device via video chat, etc. For example, if a user reserves English tutoring at 2:00 p.m., the server generates a video chat link at that time and notifies the device.

[0463] Emotion recognition and feedback regulation

[0464] Emotion recognition engine configuration

[0465] The device acquires the user's facial expression data and sends it to the server. The emotion engine analyzes this data and identifies the user's emotions. For example, a camera can capture a user's facial expression while they are doing their homework and recognize their emotional state, such as "smiling," "confused," or "concentrated."

[0466] A means of regulating emotion-based feedback

[0467] The server automatically adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it can send an encouraging message like, "It might be a little difficult, but please try again."

[0468] Emotion-based online tutoring adjustment measures

[0469] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor may recommend teaching that will help the user relax.

[0470] System usage examples

[0471] Checking math homework answers

[0472] A child takes a photo of a math homework page on a tablet and uploads it to the system. The server uses OCR to convert the content into text, and a generative AI model determines the answer. Appropriate feedback is generated and displayed on the device.

[0473] English Questions and Answers

[0474] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses a generative AI model to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[0475] Online tutoring

[0476] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[0477] Feedback adjustment using emotion recognition

[0478] While a child is doing their homework, the device's camera captures facial expression data and sends it to a server. The server then uses an emotion engine to analyze the user's emotional state and adjusts the feedback accordingly. For example, if a child is confused, the system will send an encouraging message.

[0479] Examples of prompt statements

[0480] Please take a photo of your homework problem and submit it. The system will analyze your answer and provide detailed feedback on whether your answer is correct or incorrect. To proceed, press the "Next" button.

[0481] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0482] Step 1:

[0483] Take a picture of your homework and send it

[0484] Subject: User (child)

[0485] explanation:

[0486] The user takes a photo of their homework using the camera on their tablet or smartphone, and the captured image data is sent to the server via the application.

[0487] Input: Homework image (taken by user)

[0488] Output: Image data sent to the server

[0489] Step 2:

[0490] Converts received image data into text data using optical character recognition technology

[0491] Subject: Server

[0492] explanation:

[0493] The server converts the image data of the homework received into text data using Tesseract OCR, analyzing the text information in the image and converting it into digital text.

[0494] Input: Received image data

[0495] Output: Converted text data

[0496] Step 3:

[0497] Analyze text data and determine questions and answers

[0498] Subject: Server

[0499] explanation:

[0500] The server analyzes the converted text data using a generative AI model. It identifies the type of problem (e.g., mathematical formula, word problem, etc.) and determines whether the answer is correct. For example, if the answer to the analyzed formula "3x + 5 = 11" is "x = 2," it is determined to be correct.

[0501] Input: Converted text data

[0502] Output: Judgment result (correct or incorrect)

[0503] Step 4:

[0504] Generate and send feedback based on the results

[0505] Subject: Server

[0506] explanation:

[0507] The server generates feedback based on the results. The feedback includes whether the answer was correct or incorrect, as well as helpful explanations and suggestions for improvement. The generated feedback is sent to the device. For example, a message such as "That's the correct answer! Try your best on the next question" is generated.

[0508] Input: Judgment result

[0509] Output: The generated feedback (message)

[0510] Step 5:

[0511] Acquires the user's facial expression data and sends it to the server

[0512] Subject: Device

[0513] explanation:

[0514] The device's camera is used to capture the user's facial expression data, which is then sent to a server for analysis by an emotion recognition engine.

[0515] Input: User's facial expression data (captured by camera)

[0516] Output: Facial expression data sent to the server

[0517] Step 6:

[0518] Facial expression data is analyzed using an emotion analysis engine to identify emotions.

[0519] Subject: Server

[0520] explanation:

[0521] The server analyzes the received facial expression data using an emotion analysis engine. The emotion analysis engine identifies the user's emotional state (e.g., smiling, confused, focused, etc.). For example, the emotion "confused" is identified from the facial expression data.

[0522] Input: Submitted facial expression data

[0523] Output: Identified emotional state

[0524] Step 7:

[0525] Tailor your feedback based on identified emotions

[0526] Subject: Server

[0527] explanation:

[0528] The server adjusts the feedback based on the user's emotional state. For example, if the user is confused, it might send an encouraging message like, "It might be a little difficult, but please try again." The adjusted feedback is then sent to the device.

[0529] Input: Identified emotional state

[0530] Output: Adjusted feedback (message)

[0531] Step 8:

[0532] Display feedback on your device

[0533] Subject: Device

[0534] explanation:

[0535] The device displays the feedback received from the server to the user, who can use the displayed feedback to deepen their understanding of the homework. For example, a message such as "It may be a little difficult, but please try again" may be displayed.

[0536] Input: Feedback sent by the server

[0537] Output: Feedback (messages) displayed on the terminal

[0538] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0539] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0540] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0541] [Second embodiment]

[0542] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0543] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0544] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0545] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0546] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0547] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0548] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0549] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0550] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0551] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0552] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0553] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0554] The present invention is a system for effectively supporting children with their homework, and is implemented in a form in which a server, terminals, and users cooperate with each other. This system has functions for checking homework answers, providing feedback, answering questions, and online tutoring, and detailed embodiments of the system are described below.

[0555] Homework answers and feedback

[0556] A means of receiving image data for homework

[0557] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system.

[0558] Optical character recognition and text conversion methods

[0559] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. OCR technology can convert handwritten mathematical formulas and sentences into text with high accuracy.

[0560] Means of analyzing and judging answers

[0561] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, an AI model is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[0562] Feedback generation and transmission methods

[0563] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Well done! Try your best on the next question" is generated.

[0564] Question and Answer Function

[0565] How questions are received and analyzed

[0566] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[0567] Means of generating and providing appropriate answers

[0568] The server analyzes the received question using an AI model and generates an appropriate answer. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[0569] Online Tutoring

[0570] A means of accepting and managing tutoring bookings

[0571] The terminal accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database.

[0572] A way to connect online at a specified time

[0573] When the reserved time arrives, the server connects the online tutor with the user's device via video chat, etc. For example, if a user reserves English tutoring at 2:00 p.m., the server generates a video chat link at that time and notifies the device.

[0574] System usage examples

[0575] Checking math homework answers

[0576] A child takes a photo of a math homework page with their smartphone and uploads it to the system. The server uses OCR to convert the content into text, and AI determines the answer. Appropriate feedback is generated and displayed on the device.

[0577] English Questions and Answers

[0578] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses AI to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[0579] Online tutoring

[0580] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[0581] The above describes a specific embodiment of the present invention, which enables efficient confirmation of homework and provision of feedback, support for learning comprehension, and smooth operation of online tutoring.

[0582] The processing flow will be explained below.

[0583] Step 1:

[0584] The device activates the camera, and the user takes a photo of their homework. After taking the photo, the device displays a preview of the image for the user to review.

[0585] Step 2:

[0586] The user taps the "Send" button, and the device sends the homework image data to the server, where encryption protocols are used to ensure the data is secure.

[0587] Step 3:

[0588] The server receives the image data sent from the user terminal, and then temporarily stores the data for further processing.

[0589] Step 4:

[0590] The server starts OCR and extracts text information from the received image data. After the OCR process is complete, the extracted text data is stored in an internal data store.

[0591] Step 5:

[0592] The server analyzes the text data to identify the homework questions and answers. Specifically, it uses natural language processing (NLP) algorithms to analyze the structure of the text and distinguish between the questions and answers.

[0593] Step 6:

[0594] The server uses an AI model to determine whether the extracted answer is correct or incorrect. The AI ​​model makes highly accurate judgments based on pre-trained data.

[0595] Step 7:

[0596] The server generates feedback for each question. If the answer is correct, it generates a message such as "That's right, well done!", and if the answer is incorrect, it generates feedback including specific points for improvement such as "That's wrong. Please review this."

[0597] Step 8:

[0598] The server generates feedback and sends it to the user's device, collecting the feedback into a single response message before sending it.

[0599] Step 9:

[0600] The terminal receives the feedback from the server and displays it on the user interface, and the user confirms the displayed feedback.

[0601] Step 10:

[0602] If a user has a question about their homework, they can enter the question in text format into the terminal and send it to the server. For example, they can enter a question such as "Why is this answer correct?"

[0603] Step 11:

[0604] The server analyzes the received question and uses AI models to generate an appropriate answer, based on appropriate references and explanations to help the user understand the answer.

[0605] Step 12:

[0606] The server generates a response and sends it to the user's device. The user receives the response and reviews it to further understand it.

[0607] Step 13:

[0608] When a user desires online tutoring, they input the desired date and time of tutoring into their terminal and send the reservation information to the server. For example, they reserve the following Monday at 2:00 PM.

[0609] Step 14:

[0610] The server receives the reservation information, stores it in a reservation database, and generates and sends a reminder notification to the device when the reservation date and time approaches.

[0611] Step 15:

[0612] When the reserved time arrives, the server generates a link to connect the online tutor and the user's device via video chat and notifies the device. The device launches the video chat application, and the user joins the online tutoring session.

[0613] Step 16:

[0614] After the session ends, the server collects the usage history and stores it for billing purposes. At the end of the month or when the session ends, the server calculates the charges based on the user's usage history and sends the billing information.

[0615] Example 1

[0616] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0617] Currently, there is no integrated system for supporting children's homework by checking answers, understanding answers, answering questions, and providing online tutoring. This means that children do not receive appropriate feedback or learning support, resulting in a decrease in learning effectiveness. Furthermore, manually checking homework is a heavy burden for parents and teachers, and it also hinders children's independent learning.

[0618] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0619] In this invention, the server includes means for receiving image data of homework, means for converting the received image data into text data using optical character recognition technology, means for analyzing the converted text data and determining the problem and answer, means for generating feedback based on the determination results, means for transmitting the generated feedback, means for analyzing the question content and generating an appropriate answer, means for providing the appropriate answer to the user, means for accepting online tutoring reservation information, means for managing reservations, means for connecting the online tutor with the user at a specified time, and means for analyzing the answer and generating feedback using a generative AI model. This enables efficient and integrated support for children's homework. Specifically, this system realizes automatic homework answer checking, provision of appropriate feedback, immediate response to questions, and effective online tutoring, thereby improving learning effectiveness.

[0620] "Homework image data" is an image file of a homework page, which is taken by the user using a terminal and sent to the system.

[0621] "Optical character recognition technology" is a technology that extracts character information from an image and converts it into text data, and is also known as OCR (Optical Character Recognition).

[0622] "Text data" refers to character string data converted using optical character recognition technology, and includes homework content such as mathematical formulas and sentences.

[0623] "Analysis" is the process of identifying the type of question based on text data and evaluating the accuracy of the answer.

[0624] An "answer" is a response to a question included in a homework assignment, and is determined by analysis.

[0625] "Feedback" refers to messages and explanations generated based on the analysis results, and is learning support information provided to the user.

[0626] "Question content" refers to the question or request for explanation that the user inputs to the system in text format.

[0627] An "answer" is appropriate information or explanation provided in response to a user's question, and is generated by the system.

[0628] "Online tutoring" is a method of learning support in which a user connects with an online tutor via video chat or other means at a designated time and receives instruction in real time.

[0629] "Reservation information" refers to the date and time of the online tutoring session desired by the user and other detailed information, which is accepted and managed by the system.

[0630] A "generative AI model" is a machine learning model trained on large datasets and used to analyze answers, provide feedback, and generate answers to questions.

[0631] A "prompt sentence" is an input sentence given to a generative AI model, and contains information that will form the basis for the generated answer and analysis.

[0632] This invention is a system for effectively supporting children with their homework, and is implemented in a form where a server, terminals, and users work together. This system has the functions of checking homework answers, providing feedback, answering questions, and online tutoring.

[0633] Homework answers and feedback

[0634] Receiving image data for homework

[0635] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system. In this case, the device sends the captured image file to an API endpoint (e.g., / upload / homework) that sends the image file to the server.

[0636] Optical Character Recognition and Text Conversion

[0637] The server uses the received image data to perform optical character recognition (OCR) and converts the text information in the image into text data. Software such as Google Cloud Vision API and Tesseract OCR can be used. This technology can convert handwritten mathematical formulas and sentences into text with high accuracy.

[0638] Analysis and evaluation of answers

[0639] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, a generative AI model (e.g., OpenAI's GPT-4) is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[0640] Feedback Generation and Submission

[0641] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Well done! Try your best on the next question" is generated.

[0642] Question and Answer Function

[0643] Receiving and analyzing questions

[0644] The device accepts a question from the user in text format and sends it to the server. For example, the user might ask, "Why does x equal 2?" The device then sends the question to an API endpoint (e.g., / submit / question) that sends the question to the server.

[0645] Generate and provide appropriate answers

[0646] The server analyzes the received question using a generative AI model and generates an appropriate answer. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[0647] Online Tutoring

[0648] Accepting and managing tutoring appointments

[0649] The device accepts the user's desired date and time for online tutoring and sends the reservation information to the server. The server manages this information and stores it in a reservation database. Specifically, the device sends the reservation information to the API endpoint (e.g., / schedule / tutoring) that sends the information to the server.

[0650] Connect online at a specified time

[0651] When the reserved time arrives, the server connects the online tutor and the device using video chat (e.g., Zoom or Google Meet). For example, if a user reserves English tutoring at 2 p.m., the server generates a video chat link at that time and notifies the device.

[0652] Component examples and prompt sentence examples

[0653] Checking math homework answers

[0654] A user takes a photo of a math homework page with their smartphone and uploads it to the system. The server converts the content into text using OCR, and a generative AI model determines the answer. Appropriate feedback is generated and displayed on the device.

[0655] English Questions and Answers

[0656] The user enters a question such as "What does this sentence mean?" and sends it to the system. The server uses a generative AI model to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays "This sentence means..."

[0657] Examples of applying prompt statements

[0658] An example prompt might be: "The user asks, 'Why does 6 ÷ 2 = 3?' Please provide a clear explanation for answering this question."

[0659] These detailed explanations of embodiments of the present invention enable efficient checking of children's homework and providing feedback, support for learning comprehension, and smooth operation of online tutoring.

[0660] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0661] Step 1:

[0662] Receiving image data for homework

[0663] The device takes a photo of the homework. The user uses the device's camera to take a photo of the homework page and taps the "upload button" to send the image data to the system.

[0664] Input: Image data of homework taken by the user

[0665] Output: Image data sent to the server

[0666] Specific operation: The image file taken by the device is sent to the API endpoint (e.g., / upload / homework) that sends the image file to the server.

[0667] Step 2:

[0668] Optical Character Recognition and Text Conversion

[0669] The server retrieves the received image data and extracts text information from the image using optical character recognition (OCR) technology.

[0670] Input: Homework image data sent from the device

[0671] Output: Converted text data

[0672] Specific operation: The server uses the Google Cloud Vision API and Tesseract OCR to convert handwritten and printed character information in the image into text data.

[0673] Step 3:

[0674] Analysis and evaluation of answers

[0675] The server analyzes the converted text data to determine the type of question and sends prompts to the AI ​​model to generate and determine the answer.

[0676] Input: Converted text data (e.g. "3x + 5 = 11")

[0677] Output: Answer result (e.g. "x = 2")

[0678] Specific operation: The server sends a prompt to the generative AI model, such as "Solve this equation '3x + 5 = 11'," and analyzes the returned answer to determine its accuracy.

[0679] Step 4:

[0680] Feedback Generation and Submission

[0681] The server generates feedback based on the correctness of the answer and sends it to the user terminal.

[0682] Input: Answer result

[0683] Output: Generated feedback

[0684] Specific operation: The server uses the generative AI model to create a message such as "Good job! Try your best on the next question," and notifies the device.

[0685] Step 5:

[0686] Receiving and analyzing questions

[0687] The terminal accepts questions from the user in text format and sends them to the system.

[0688] Input: A question typed by the user (e.g., "Why is x equal to 2?")

[0689] Output: The question sent to the server

[0690] Specific operation: The device sends the question to the API endpoint (e.g., / submit / question) that sends the question to the server.

[0691] Step 6:

[0692] Generate and provide appropriate answers

[0693] The server analyzes the received question using a generative AI model, generates an appropriate answer, and sends it to the user's device.

[0694] Input: The question sent to the server

[0695] Output: The generated answer (e.g., "3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2")

[0696] Specific operation: The server sends a prompt to the generative AI model saying, "Please answer the user's question 'Why is 6 ÷ 2 = 3?'" and sends the answer to the user's device.

[0697] Step 7:

[0698] Accepting and managing tutoring appointments

[0699] The terminal accepts the user's desired date and time for online tutoring and transmits the reservation information to the server.

[0700] Input: Reservation information entered by the user (date, time, etc.)

[0701] Output: Reservation information sent to the server

[0702] Specific operation: The device sends the reservation information to the API endpoint (e.g., / schedule / tutoring) that sends the reservation information to the server.

[0703] Step 8:

[0704] Connect online at a specified time

[0705] When the reserved time arrives, the server arranges for the online tutor and the user terminal to be connected using video chat or the like.

[0706] Input: Reservation information and current time

[0707] Output: Video chat connection link

[0708] Specific operation: The server generates a video chat link using an API such as Zoom or Google Meet, and then connects by notifying the device at the specified time.

[0709] (Application example 1)

[0710] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0711] Supporting students with their daily homework is extremely important in today's educational environment. However, quickly checking homework answers, providing accurate feedback, and immediately answering students' questions places a heavy burden on educators. Furthermore, scheduling and managing online tutoring sessions is time-consuming, requiring efficient operation. Furthermore, the systems and applications needed to effectively implement this support are insufficient, making it difficult to provide consistent learning support.

[0712] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0713] In this invention, the server includes means for receiving image data of homework, means for converting the received image data into text data using optical character recognition technology, means for analyzing the converted text data and determining the problem and answer, means for generating feedback based on the determination result, means for transmitting the generated feedback, means for accepting questions about homework, means for analyzing the content of the questions and generating appropriate answers, means for providing the generated answers to the user, means for accepting online tutoring reservation information, means for managing reservations based on the reservation information, means for connecting the user with an online tutor at a specified time, means for installing a learning support application on a tablet or smartphone, means for taking and uploading image data of the homework using a camera function of the tablet or smartphone, means for providing the generated feedback to the student in an easy-to-understand manner using natural language processing technology, means for training an AI using structured data and evaluating the accuracy of the answers, and means for supporting online and offline tutoring. This enables fast and accurate homework answer checking, feedback provision, question and answering, and efficient operation of online tutoring in learning environments.

[0714] The "means for receiving image data of homework" is a function that allows a user to take a photo of their homework and send the image data to a server.

[0715] "Means for converting into text data using optical character recognition technology" refers to a function that converts character information in received image data into text data using optical character recognition technology (OCR).

[0716] The "means for analyzing the converted text data and determining the questions and answers" is a function that can automatically determine the homework questions and answers by analyzing the text data converted by OCR.

[0717] The "means for generating feedback based on the assessment result" is a function that automatically generates appropriate feedback based on whether the answer is correct or incorrect.

[0718] The "means for transmitting the generated feedback" is a function for transmitting the generated feedback to the user's terminal and displaying it.

[0719] The "means for accepting questions about homework" is a function that allows a user to input questions about homework in text format and transmit them to the server.

[0720] "Means for analyzing the question content and generating an appropriate answer" is a function that analyzes the received question and generates an appropriate answer using AI.

[0721] The "means for providing the generated answer to the user" is a function for transmitting the generated answer to the user's terminal and displaying it.

[0722] The "means for accepting online tutoring reservation information" is a function that allows a user to input online tutoring reservation information and transmit that information to the server.

[0723] The "means for managing reservations based on reservation information" is a function for managing reservation data based on received reservation information.

[0724] The "means for connecting an online tutor with a user at a designated time" is a function that connects an online tutor with a user at the reserved time and allows them to make a video call or chat.

[0725] The "means for installing the learning support application on a tablet or smartphone" is a function that allows the learning support application to be installed on the user's tablet or smartphone.

[0726] "Means for taking and uploading image data of homework using the camera function of a tablet or smartphone" refers to a function that allows a user to take a photo of their homework using the camera of a tablet or smartphone and upload the image to a server.

[0727] "Means for providing the generated feedback to the student in an easy-to-understand format using natural language processing technology" is a function for providing the generated feedback to the student in an easily understandable format using natural language processing technology.

[0728] "Means for training AI using structured data and evaluating the accuracy of answers" refers to a function that uses structured data to train an AI model and then uses that model to evaluate the accuracy of answers.

[0729] "Means for supporting online and offline tutoring" refers to functions for supporting both online and offline tutoring.

[0730] The present invention provides a system for effectively supporting children in their homework, and is configured so that a server, a terminal, and a user work together. A specific embodiment of the system will be described below.

[0731] Homework answers and feedback

[0732] A means of receiving image data for homework

[0733] The user takes a photo of their homework using the camera function of their smartphone or tablet and sends the image data to the server. The device used is a general smartphone or tablet. The photo is uploaded using, for example, a mobile application.

[0734] Optical character recognition and text conversion methods

[0735] The server then performs optical character recognition (OCR) on the received image data, converting the character information in the image into text data. This process uses OCR technology such as Google Cloud Vision API.

[0736] Means of analyzing and judging answers

[0737] The server analyzes the converted text data and determines the homework problem and answer. The analysis uses Azure Machine Learning services to train an AI model. For example, it determines that the answer "x = 2" is correct when "3x + 5 = 11" is given.

[0738] Feedback generation and transmission methods

[0739] The server generates feedback based on the results and sends it to the user's device. The feedback includes not only whether the answer was correct or incorrect, but also detailed explanations and suggestions for improvement. The generated feedback is provided in an easy-to-understand manner using natural language processing technology (e.g., Google Cloud Natural Language API).

[0740] Question and Answer Function

[0741] How questions are received and analyzed

[0742] Users type in a homework question and send it to the server via their device, for example, "Why is x equal to 2?"

[0743] Means of generating and providing appropriate answers

[0744] The server analyzes the received question and generates an appropriate answer using AI (e.g., GPT-3) and provides it to the user. For example, the generated answer might be, "First, subtract 5 from 3x + 5 = 11 to get 3x = 6, then divide by 3 to get x = 2."

[0745] Online Tutoring

[0746] A means of accepting and managing tutoring bookings

[0747] The user inputs the desired date and time for online tutoring through the application, and the server receives and manages this information. Reservation management systems used include Microsoft Bookings.

[0748] A way to connect online at a specified time

[0749] When the scheduled time arrives, the server connects the user with the online tutor using video chat (e.g., Zoom API).

[0750] Specific examples

[0751] For example, after a student at a cram school completes their homework, they can take a photo of the page with their smartphone and upload it through an application. This data is then analyzed using optical character recognition on the server, and feedback is provided instantly. If a student asks a question about a problem they can't solve, the AI ​​can analyze it and provide an immediate answer. Furthermore, when parents schedule online tutoring, a video chat will automatically connect at the specified time.

[0752] Prompt Sentence Examples

[0753] Question: "Why does 3x + 5 = 11 result in x = 2?"

[0754] Prompt: "This student wants to know why 3x + 5 = 11 means x = 2. Please provide a detailed explanation."

[0755] Answer: "Let's first rearrange the equation 3x + 5 = 11. First, subtract 5 from both sides to get 3x = 6. Then, divide both sides by 3 to get x = 2. This will give you the answer."

[0756] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0757] Step 1:

[0758] Users take photos of their homework using the camera function of their smartphone or tablet and upload the images to the server through the application. The input is the image data of the homework, and the output is the image data sent to the server.

[0759] Step 2:

[0760] The server performs optical character recognition (OCR) on the received image data and converts the text information in the image into text data. Using the Google Cloud Vision API, the input for this step is image data and the output is text data.

[0761] Step 3:

[0762] The server uses the Azure Machine Learning service to analyze the text data obtained by OCR and automatically determine the homework questions and answers. The input for this step is the text data, and the output is the analysis result, which indicates whether the answer is correct or not.

[0763] Step 4:

[0764] The server generates feedback based on the results of the assessment and provides the generated feedback to the user in an easy-to-understand manner using natural language processing technology (Google Cloud Natural Language API). The input of this step is the correct or incorrect answer, and the output is a feedback message.

[0765] Step 5:

[0766] The user types in a text question about their homework and sends it to the server through the application. The input is the text question data, and the output is the transmission of the question data to the server.

[0767] Step 6:

[0768] The server analyzes the received question and generates an appropriate answer using an AI model such as GPT-3. The input of this step is the question data, and the output is the generated answer data.

[0769] Step 7:

[0770] The server sends the generated answer to the user's terminal and displays it. The input of this step is the generated answer data, and the output is the answer message displayed on the user's terminal.

[0771] Step 8:

[0772] A user inputs reservation information for online tutoring through the application and sends it to the server. The input is reservation information data, and the output is reservation information sent to the server.

[0773] Step 9:

[0774] The server accepts the reservation information and manages it using the reservation management system (Microsoft Bookings). The input of this step is reservation information data, and the output is reservation management data.

[0775] Step 10:

[0776] When the reservation time arrives, the server generates a video call link (e.g., Zoom API) to connect the user with the online tutor. The input of this step is the reservation information and the current time, and the output is the generated video call link.

[0777] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0778] The present invention is a system for effectively supporting children with their homework, implemented in a form in which a server, terminals, and users cooperate with each other. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide more advanced support. This system has the functions of checking homework answers, providing feedback, question answering, online tutoring, and emotion recognition, and detailed embodiments of the system are described below.

[0779] Homework answers and feedback

[0780] A means of receiving image data for homework

[0781] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system.

[0782] Optical character recognition and text conversion methods

[0783] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. OCR technology can convert handwritten mathematical formulas and sentences into text with high accuracy.

[0784] Means of analyzing and judging answers

[0785] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, an AI model is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[0786] Feedback generation and transmission methods

[0787] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Good job! Try your best on the next question" is generated.

[0788] Question and Answer Function

[0789] How questions are received and analyzed

[0790] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[0791] Means of generating and providing appropriate answers

[0792] The server analyzes the received question and generates an appropriate answer using an AI model. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[0793] Online Tutoring

[0794] A means of accepting and managing tutoring bookings

[0795] The terminal accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database.

[0796] A way to connect online at a specified time

[0797] When the reserved time arrives, the server connects the online tutor with the user's device via video chat, etc. For example, if a user reserves English tutoring at 2:00 p.m., the server generates a video chat link at that time and notifies the device.

[0798] Emotion recognition and feedback regulation

[0799] Emotion recognition engine configuration

[0800] The device acquires the user's facial expression data and sends it to the server. The emotion engine analyzes this data and identifies the user's emotions. For example, a camera can capture a user's facial expression while they are doing their homework and recognize their emotional state, such as "smiling," "confused," or "concentrated."

[0801] A means of regulating emotion-based feedback

[0802] The server automatically adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it can send an encouraging message like, "It might be a little difficult, but please try again."

[0803] Emotion-based online tutoring adjustment measures

[0804] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor may recommend teaching that will help the user relax.

[0805] System usage examples

[0806] Checking math homework answers

[0807] A child takes a photo of a math homework page with their smartphone and uploads it to the system. The server uses OCR to convert the content into text, and AI determines the answer. Appropriate feedback is generated and displayed on the device.

[0808] English Questions and Answers

[0809] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses AI to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[0810] Online tutoring

[0811] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[0812] Feedback adjustment using emotion recognition

[0813] While a child is doing their homework, the device's camera captures facial expression data and sends it to a server. The server then uses an emotion engine to analyze the user's emotional state and adjusts the feedback accordingly. For example, if a child is confused, the system will send an encouraging message.

[0814] The above describes specific embodiments of the present invention, which enable efficient homework confirmation and feedback, support for learning comprehension, smooth operation of online tutoring, and advanced emotion-based feedback and guidance.

[0815] The processing flow will be explained below.

[0816] Step 1:

[0817] The device activates the camera, and the user takes a photo of their homework. After taking the photo, the device displays a preview of the image for the user to review.

[0818] Step 2:

[0819] The user taps the "Send" button, and the device sends the homework image data to the server, where encryption protocols are used to ensure the data is secure.

[0820] Step 3:

[0821] The server receives the image data sent from the user terminal, and then temporarily stores the data for further processing.

[0822] Step 4:

[0823] The server starts OCR and extracts text information from the received image data. After the OCR process is complete, the extracted text data is stored in an internal data store.

[0824] Step 5:

[0825] The server analyzes the text data to identify the homework questions and answers. Specifically, it uses natural language processing (NLP) algorithms to analyze the structure of the text and distinguish between the questions and answers.

[0826] Step 6:

[0827] The server uses an AI model to determine whether the extracted answer is correct or incorrect. The AI ​​model makes highly accurate judgments based on pre-trained data.

[0828] Step 7:

[0829] The server generates feedback for each question. If the answer is correct, it generates a message such as "That's right, well done!", and if the answer is incorrect, it generates feedback including specific points for improvement such as "That's wrong. Please review this."

[0830] Step 8:

[0831] The server generates feedback and sends it to the user's device, collecting the feedback into a single response message before sending it.

[0832] Step 9:

[0833] The terminal receives the feedback from the server and displays it on the user interface, and the user confirms the displayed feedback.

[0834] Step 10:

[0835] If a user has a question about their homework, they type the question in text format into the terminal and send it to the server, for example, "Why is this answer correct?"

[0836] Step 11:

[0837] The server analyzes the received question and uses AI models to generate an appropriate answer, based on appropriate references and explanations to help the user understand the answer.

[0838] Step 12:

[0839] The server generates a response and sends it to the user's device. The user receives the response and reviews it to further understand it.

[0840] Step 13:

[0841] The device activates the camera to capture the user's facial expression data and sends it to the server.

[0842] Step 14:

[0843] The server analyzes the facial expression data received by the emotion engine and identifies the user's emotional state, such as "smiling," "confusion," or "concentration."

[0844] Step 15:

[0845] The server adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it might send an encouraging message like, "It might be a little difficult, but please try again."

[0846] Step 16:

[0847] When a user desires online tutoring, they input the desired date and time of tutoring into their terminal and send the reservation information to the server. For example, they reserve the following Monday at 2:00 PM.

[0848] Step 17:

[0849] The server receives the reservation information, stores it in a reservation database, and generates and sends a reminder notification to the device when the reservation date and time approaches.

[0850] Step 18:

[0851] When the reserved time arrives, the server generates a link to connect the online tutor and the user's device via video chat and notifies the device. The device launches the video chat application, and the user joins the online tutoring session.

[0852] Step 19:

[0853] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor can instruct the user to provide instruction that will help the user relax.

[0854] Step 20:

[0855] After the session ends, the server collects the usage history and stores it for billing purposes. At the end of the month or when the session ends, the server calculates the charges based on the user's usage history and sends the billing information.

[0856] Example 2

[0857] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0858] Conventional homework support systems have limited functionality for automatically checking homework answers and providing feedback. Furthermore, most feedback is one-way and does not take the user's emotional state into account, making it difficult to provide effective learning support. Furthermore, the complexity of reserving and implementing online tutoring sessions makes it difficult to provide an efficient learning support environment. It is necessary to solve these issues and provide more effective, personalized homework support to users.

[0859] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: a means for the terminal to receive image data of homework; a means for the server to convert the received image data into text data using optical character recognition technology; a means for the server to analyze the converted text data and determine the question and answer; a means for the server to generate feedback based on the determination result; a means for the server to send the generated feedback to the terminal; a means for the terminal to acquire user facial expression data and send it to the server; and a means for the server to identify the user's emotion using emotion recognition technology and adjust the feedback based on the identification result. This enables efficient confirmation of homework and provision of feedback, support for learning comprehension, and advanced responses according to the user's emotions.

[0860] A "terminal" is a device operated by a user, and is used to take pictures of image data for homework, input questions, and make reservations for online tutoring.

[0861] A "server" is a computer system that receives, processes, and stores data sent from a terminal, and is a device that has functions such as optical character recognition of image data, answer determination, and feedback generation.

[0862] "Image data" is digital data including homework photos and other visual information that is taken by the terminal and sent to the server.

[0863] "Optical Character Recognition (OCR)" is a technology that converts handwritten or printed characters in image data into digital text.

[0864] "Text data" refers to data that represents character information in digital form, converted using optical character recognition technology.

[0865] The "means for determining the answer" is a mechanism by which the server analyzes the text data, identifies the type of question, and evaluates the accuracy of the answer.

[0866] "Feedback" is information provided to the user, including an evaluation of the correctness of the answer, reference explanations, and points for improvement.

[0867] "Emotion recognition technology" is a technology that analyzes a user's facial expression data and identifies the user's emotional state (for example, smiling, confused, focused, etc.).

[0868] "Online tutoring" refers to video chat and other remote instruction formats for educational assistance delivered in real time over the internet.

[0869] This invention is a system for effectively supporting children with their homework, implemented in a form where a server, terminals, and users work together. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide more advanced support. This system has the following functions: checking homework answers, providing feedback, question answering, online tutoring, and emotion recognition.

[0870] Homework answers and feedback

[0871] A means of receiving image data for homework

[0872] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system.

[0873] Optical character recognition and text conversion methods

[0874] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. OCR technology can convert handwritten mathematical formulas and sentences into text with high accuracy. Specifically, software such as Tesseract OCR is used.

[0875] Means of analyzing and judging answers

[0876] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, an AI model (e.g., Python + scikit-learn) is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[0877] Feedback generation and transmission methods

[0878] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Good job! Try your best on the next question" is generated.

[0879] Question and Answer Function

[0880] How questions are received and analyzed

[0881] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[0882] Means of generating and providing appropriate answers

[0883] The server analyzes the received question and generates an appropriate answer using an AI model (e.g., GPT-3). The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[0884] Online Tutoring

[0885] A means of accepting and managing tutoring bookings

[0886] The device accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database, typically an SQL or NoSQL database.

[0887] A way to connect online at a specified time

[0888] When the reservation time arrives, the server connects the online tutor and the user's device using a video chat tool (e.g., Zoom API). For example, if a user reserves English tutoring at 2 p.m., the server generates a video chat link at that time and notifies the device.

[0889] Emotion recognition and feedback regulation

[0890] Emotion recognition engine configuration

[0891] The device acquires the user's facial expression data and sends it to a server. The emotion engine analyzes this data and identifies the user's emotions. For example, a camera can capture a user's facial expression while they are doing their homework and recognize their emotional state, such as "smiling," "confused," or "concentrated." Specifically, emotion recognition software such as Microsoft Azure Cognitive Services is used.

[0892] A means of regulating emotion-based feedback

[0893] The server automatically adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it can send an encouraging message like, "It might be a little difficult, but please try again."

[0894] Emotion-based online tutoring adjustment measures

[0895] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor may recommend teaching that will help the user relax.

[0896] System usage examples

[0897] Checking math homework answers

[0898] A child takes a photo of a math homework page with their smartphone and uploads it to the system. The server uses OCR to convert the content into text, and AI determines the answer. Appropriate feedback is generated and displayed on the device.

[0899] English Questions and Answers

[0900] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses AI to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[0901] Online tutoring

[0902] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[0903] Feedback adjustment using emotion recognition

[0904] While a child is doing their homework, the device's camera captures facial expression data and sends it to a server. The server then uses an emotion engine to analyze the user's emotional state and adjusts the feedback accordingly. For example, if a child is confused, the system will send an encouraging message.

[0905] In this way, the system of the present invention is capable of efficiently checking homework and providing feedback, supporting learning comprehension, smoothly operating online tutoring, and providing advanced emotion-based feedback and guidance.

[0906] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0907] Step 1:

[0908] The device receives image data of the homework. The user takes a photo of the homework with their smartphone and saves the image data on the device. The input is the image file taken by the user, and the output is the image data saved on the device. The device sends this image data to the server.

[0909] Step 2:

[0910] The server receives the image data. The server receives the image data sent from the terminal. The input is the image file sent from the terminal, and the output is the image data saved on the server. The server saves this image data in temporary storage.

[0911] Step 3:

[0912] The server converts image data into text data using optical character recognition technology (OCR). Specifically, the server launches an application program (e.g., Tesseract OCR) and performs OCR on the image data. The input is an image file stored on the server, and the output is text data converted by OCR. Handwritten mathematical formulas and sentences are converted into digital text.

[0913] Step 4:

[0914] The server analyzes the converted text data. The server reads the text data and identifies the type of problem (e.g., mathematical formula or word problem). The input is the text data generated by OCR, and the output is the analysis result. The server performs appropriate processing depending on the type of problem identified.

[0915] Step 5:

[0916] The server uses an AI model to determine the answer. Specifically, the server uses a program (e.g., Python + scikit-learn) to analyze the text data and determine the accuracy of the answer. The input is the analyzed text data, and the output is the result of determining whether the answer is correct. For example, it determines whether the answer "3x + 5 = 11" is correct as "x = 2."

[0917] Step 6:

[0918] The server generates the feedback. The feedback is created based on the result of the answer's correctness. Specifically, the server generates the feedback text by combining template messages and reference explanations. The input is the result of the answer's correctness, and the output is the generated feedback text. For example, it generates feedback such as "That's correct! Excellent! Good luck on the next question."

[0919] Step 7:

[0920] The server sends the generated feedback to the terminal. The generated feedback is sent to the user's terminal. The input is the generated feedback sentence, and the output is the feedback displayed on the terminal. The user checks the feedback on the terminal.

[0921] Step 8:

[0922] The device acquires the user's facial expression data and sends it to the server. For example, the device's camera captures the user's facial expression while they are doing their homework, and sends the data to the server. The input is the facial expression image captured by the camera, and the output is the facial expression data sent to the server.

[0923] Step 9:

[0924] The server uses emotion recognition technology to identify the user's emotions. Specifically, the server uses emotion recognition software (e.g., Microsoft Azure Cognitive Services) to analyze facial expression data and identify the user's emotional state. The input is facial expression data, and the output is the identified emotional state. Recognized emotions include "smiling," "confusion," and "concentration."

[0925] Step 10:

[0926] The server adjusts the feedback based on the emotion identification results. The content and expression of the feedback are automatically adjusted according to the recognized emotional state. The input is the identified emotional state and the previously generated feedback, and the output is the adjusted feedback text. For example, if the user is confused, it adds an encouraging message such as "It might be a little difficult, but please try again."

[0927] Step 11:

[0928] The terminal accepts the user's question and sends it to the server. The user enters a question in text format, and the terminal sends the question to the server. The input is the text format question entered by the user, and the output is the question data sent to the server.

[0929] Step 12:

[0930] The server analyzes the received question and generates an appropriate answer using an AI model. Specifically, the server analyzes the question using an AI model (e.g., GPT-3) and generates an appropriate answer. The input is the question data, and the output is the generated answer. For example, in response to the question "Why is x 2?", the server generates an answer such as "From 3x + 5 = 11, first subtract 5 to make 3x = 6, then divide by 3 to make x = 2."

[0931] Step 13:

[0932] The server sends the generated answer to the terminal. The server sends the generated answer to the user's terminal. The input is the generated answer, and the output is the answer sent to the terminal.

[0933] Step 14:

[0934] The terminal sends online tutoring reservation information to the server. The user inputs the desired date and time of tutoring, and the terminal sends that information to the server. The input is the reservation information, and the output is the reservation data sent to the server.

[0935] Step 15:

[0936] The server manages reservations based on the reservation information. Specifically, the server saves the reservation information in a reservation database and manages the reservations. The input is reservation data, and the output is managed reservation information.

[0937] Step 16:

[0938] When the reservation time arrives, the server connects the online tutor and the user's device using a video chat tool. Specifically, the server generates a video chat link and notifies the device. The input is the reservation information and the video chat link, and the output is a video chat connection between the user and the tutor. For example, the connection can be made at the specified time using a video chat tool (e.g., Zoom API).

[0939] (Application example 2)

[0940] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0941] Currently, in children's homework support systems, it is important not only to judge whether an answer is correct or incorrect, but also to adjust feedback based on the user's emotions, which conventional systems have not been able to do adequately. Furthermore, in brick-and-mortar educational services, it is difficult to provide individual support in real time, resulting in an unsatisfactory user experience. Therefore, the present invention aims to provide more effective homework support and improve the quality of brick-and-mortar educational services by recognizing the user's emotions and adjusting feedback.

[0942] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0943] In this invention, the server includes means for receiving image data of homework, means for converting the received image data into text data using optical character recognition technology, means for analyzing the converted text data and determining the question and answer, means for generating feedback based on the determination result, means for transmitting the generated feedback, means for acquiring facial expression data of the user using a camera, means for analyzing the facial expression data using an emotion analysis engine and identifying the user's emotion, and means for adjusting the feedback content based on the identified emotion. This allows for more effective homework support by recognizing the emotion from the user's facial expression and providing feedback accordingly, thereby enabling the quality of educational services at physical stores to be improved.

[0944] "Homework image data" refers to digital image files of what a child has written as homework, captured with a camera.

[0945] "Means for receiving" refers to a communication means for transferring image data from a terminal to a server.

[0946] "Optical character recognition technology" is a technology that recognizes character information in an image and converts it into digital text.

[0947] "Text data" means digital data containing textual information that has been converted using optical character recognition technology.

[0948] "Means for analyzing" refers to means for processing the converted text data and determining the questions and answers.

[0949] A "means of assessment" is a means for evaluating whether the answer to the question is correct.

[0950] "Feedback" refers to information, including advice and corrections, generated based on analysis and judgment results.

[0951] "Means for sending" refers to communication means for transferring the generated feedback to a user terminal.

[0952] "Camera" refers to an image capture device for capturing a user's facial expression.

[0953] "Facial expression data" refers to image data of the user's face captured by a camera.

[0954] An "emotion analysis engine" is software that has an algorithm that analyzes facial expression data and identifies emotions.

[0955] "Means for identifying" refers to means for classifying the emotional state of a user using a sentiment analysis engine.

[0956] The "means for adjusting the feedback content" is a means for changing the content or expression of the feedback based on the identified emotion.

[0957] This invention is a system for effectively supporting children with their homework, and is implemented in a form in which a server, a terminal, and a user cooperate with each other. The system has functions for checking homework answers, providing feedback, question answering, online tutoring, and emotion recognition. Specific embodiments of the system are described below.

[0958] Homework answers and feedback

[0959] A means of receiving image data for homework

[0960] The device takes a photo of the homework and sends the image data to the server. The device can be a tablet or smartphone, and the user can take a photo of the homework page using the camera.

[0961] Optical character recognition and text conversion methods

[0962] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. Tesseract is used for the OCR technology, which enables handwritten mathematical formulas and sentences to be converted into text with high accuracy.

[0963] Means of analyzing and judging answers

[0964] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, a generative AI model is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[0965] Feedback generation and transmission methods

[0966] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Good job! Try your best on the next question" is generated.

[0967] Question and Answer Function

[0968] How questions are received and analyzed

[0969] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[0970] Means of generating and providing appropriate answers

[0971] The server analyzes the received question and generates an appropriate answer using a generative AI model. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[0972] Online Tutoring

[0973] A means of accepting and managing tutoring bookings

[0974] The terminal accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database.

[0975] A way to connect online at a specified time

[0976] When the reserved time arrives, the server connects the online tutor with the user's device via video chat, etc. For example, if a user reserves English tutoring at 2:00 p.m., the server generates a video chat link at that time and notifies the device.

[0977] Emotion recognition and feedback regulation

[0978] Emotion recognition engine configuration

[0979] The device acquires the user's facial expression data and sends it to the server. The emotion engine analyzes this data and identifies the user's emotions. For example, a camera can capture a user's facial expression while they are doing their homework and recognize their emotional state, such as "smiling," "confused," or "concentrated."

[0980] A means of regulating emotion-based feedback

[0981] The server automatically adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it can send an encouraging message like, "It might be a little difficult, but please try again."

[0982] Emotion-based online tutoring adjustment measures

[0983] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor may recommend teaching that will help the user relax.

[0984] System usage examples

[0985] Checking math homework answers

[0986] A child takes a photo of a math homework page on a tablet and uploads it to the system. The server uses OCR to convert the content into text, and a generative AI model determines the answer. Appropriate feedback is generated and displayed on the device.

[0987] English Questions and Answers

[0988] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses a generative AI model to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[0989] Online tutoring

[0990] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[0991] Feedback adjustment using emotion recognition

[0992] While a child is doing their homework, the device's camera captures facial expression data and sends it to a server. The server then uses an emotion engine to analyze the user's emotional state and adjusts the feedback accordingly. For example, if a child is confused, the system will send an encouraging message.

[0993] Examples of prompt statements

[0994] Please take a photo of your homework problem and submit it. The system will analyze your answer and provide detailed feedback on whether your answer is correct or incorrect. To proceed, press the "Next" button.

[0995] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0996] Step 1:

[0997] Take a picture of your homework and send it

[0998] Subject: User (child)

[0999] explanation:

[1000] The user takes a photo of their homework using the camera on their tablet or smartphone, and the captured image data is sent to the server via the application.

[1001] Input: Homework image (taken by user)

[1002] Output: Image data sent to the server

[1003] Step 2:

[1004] Converts received image data into text data using optical character recognition technology

[1005] Subject: Server

[1006] explanation:

[1007] The server converts the image data of the homework received into text data using Tesseract OCR, analyzing the text information in the image and converting it into digital text.

[1008] Input: Received image data

[1009] Output: Converted text data

[1010] Step 3:

[1011] Analyze text data and determine questions and answers

[1012] Subject: Server

[1013] explanation:

[1014] The server analyzes the converted text data using a generative AI model. It identifies the type of problem (e.g., mathematical formula, word problem, etc.) and determines whether the answer is correct. For example, if the answer to the analyzed formula "3x + 5 = 11" is "x = 2," it is determined to be correct.

[1015] Input: Converted text data

[1016] Output: Judgment result (correct or incorrect)

[1017] Step 4:

[1018] Generate and send feedback based on the results

[1019] Subject: Server

[1020] explanation:

[1021] The server generates feedback based on the results. The feedback includes whether the answer was correct or incorrect, as well as helpful explanations and suggestions for improvement. The generated feedback is sent to the device. For example, a message such as "That's the correct answer! Try your best on the next question" is generated.

[1022] Input: Judgment result

[1023] Output: The generated feedback (message)

[1024] Step 5:

[1025] Acquires the user's facial expression data and sends it to the server

[1026] Subject: Device

[1027] explanation:

[1028] The device's camera is used to capture the user's facial expression data, which is then sent to a server for analysis by an emotion recognition engine.

[1029] Input: User's facial expression data (captured by camera)

[1030] Output: Facial expression data sent to the server

[1031] Step 6:

[1032] Facial expression data is analyzed using an emotion analysis engine to identify emotions.

[1033] Subject: Server

[1034] explanation:

[1035] The server analyzes the received facial expression data using an emotion analysis engine. The emotion analysis engine identifies the user's emotional state (e.g., smiling, confused, focused, etc.). For example, the emotion "confused" is identified from the facial expression data.

[1036] Input: Submitted facial expression data

[1037] Output: Identified emotional state

[1038] Step 7:

[1039] Tailor your feedback based on identified emotions

[1040] Subject: Server

[1041] explanation:

[1042] The server adjusts the feedback based on the user's emotional state. For example, if the user is confused, it might send an encouraging message like, "It might be a little difficult, but please try again." The adjusted feedback is then sent to the device.

[1043] Input: Identified emotional state

[1044] Output: Adjusted feedback (message)

[1045] Step 8:

[1046] Display feedback on your device

[1047] Subject: Device

[1048] explanation:

[1049] The device displays the feedback received from the server to the user, who can use the displayed feedback to deepen their understanding of the homework. For example, a message such as "It may be a little difficult, but please try again" may be displayed.

[1050] Input: Feedback sent by the server

[1051] Output: Feedback (messages) displayed on the terminal

[1052] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1053] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1054] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1055] [Third embodiment]

[1056] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1057] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1058] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1059] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1060] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1061] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1062] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1063] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1064] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1065] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1066] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1067] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1068] The present invention is a system for effectively supporting children with their homework, and is implemented in a form in which a server, terminals, and users cooperate with each other. This system has functions for checking homework answers, providing feedback, answering questions, and online tutoring, and detailed embodiments of the system are described below.

[1069] Homework answers and feedback

[1070] A means of receiving image data for homework

[1071] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system.

[1072] Optical character recognition and text conversion methods

[1073] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. OCR technology can convert handwritten mathematical formulas and sentences into text with high accuracy.

[1074] Means of analyzing and judging answers

[1075] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, an AI model is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[1076] Feedback generation and transmission methods

[1077] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Well done! Try your best on the next question" is generated.

[1078] Question and Answer Function

[1079] How questions are received and analyzed

[1080] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[1081] Means of generating and providing appropriate answers

[1082] The server analyzes the received question using an AI model and generates an appropriate answer. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[1083] Online Tutoring

[1084] A means of accepting and managing tutoring bookings

[1085] The terminal accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database.

[1086] A way to connect online at a specified time

[1087] When the reserved time arrives, the server connects the online tutor with the user's device via video chat, etc. For example, if a user reserves English tutoring at 2:00 p.m., the server generates a video chat link at that time and notifies the device.

[1088] System usage examples

[1089] Checking math homework answers

[1090] A child takes a photo of a math homework page with their smartphone and uploads it to the system. The server uses OCR to convert the content into text, and AI determines the answer. Appropriate feedback is generated and displayed on the device.

[1091] English Questions and Answers

[1092] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses AI to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[1093] Online tutoring

[1094] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[1095] The above describes a specific embodiment of the present invention, which enables efficient confirmation of homework and provision of feedback, support for learning comprehension, and smooth operation of online tutoring.

[1096] The processing flow will be explained below.

[1097] Step 1:

[1098] The device activates the camera, and the user takes a photo of their homework. After taking the photo, the device displays a preview of the image for the user to review.

[1099] Step 2:

[1100] The user taps the "Send" button, and the device sends the homework image data to the server, where encryption protocols are used to ensure the data is secure.

[1101] Step 3:

[1102] The server receives the image data sent from the user terminal, and then temporarily stores the data for further processing.

[1103] Step 4:

[1104] The server starts OCR and extracts text information from the received image data. After the OCR process is complete, the extracted text data is stored in an internal data store.

[1105] Step 5:

[1106] The server analyzes the text data to identify the homework questions and answers. Specifically, it uses natural language processing (NLP) algorithms to analyze the structure of the text and distinguish between the questions and answers.

[1107] Step 6:

[1108] The server uses an AI model to determine whether the extracted answer is correct or incorrect. The AI ​​model makes highly accurate judgments based on pre-trained data.

[1109] Step 7:

[1110] The server generates feedback for each question. If the answer is correct, it generates a message such as "That's right, well done!", and if the answer is incorrect, it generates feedback including specific points for improvement such as "That's wrong. Please review this."

[1111] Step 8:

[1112] The server generates feedback and sends it to the user's device, collecting the feedback into a single response message before sending it.

[1113] Step 9:

[1114] The terminal receives the feedback from the server and displays it on the user interface, and the user confirms the displayed feedback.

[1115] Step 10:

[1116] If a user has a question about their homework, they can enter the question in text format into the terminal and send it to the server. For example, they can enter a question such as "Why is this answer correct?"

[1117] Step 11:

[1118] The server analyzes the received question and uses AI models to generate an appropriate answer, based on appropriate references and explanations to help the user understand the answer.

[1119] Step 12:

[1120] The server generates a response and sends it to the user's device. The user receives the response and reviews it to further understand it.

[1121] Step 13:

[1122] When a user desires online tutoring, they input the desired date and time of tutoring into their terminal and send the reservation information to the server. For example, they reserve the following Monday at 2:00 PM.

[1123] Step 14:

[1124] The server receives the reservation information, stores it in a reservation database, and generates and sends a reminder notification to the device when the reservation date and time approaches.

[1125] Step 15:

[1126] When the reserved time arrives, the server generates a link to connect the online tutor and the user's device via video chat and notifies the device. The device launches the video chat application, and the user joins the online tutoring session.

[1127] Step 16:

[1128] After the session ends, the server collects the usage history and stores it for billing purposes. At the end of the month or when the session ends, the server calculates the charges based on the user's usage history and sends the billing information.

[1129] Example 1

[1130] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1131] Currently, there is no integrated system for supporting children's homework by checking answers, understanding answers, answering questions, and providing online tutoring. This means that children do not receive appropriate feedback or learning support, resulting in a decrease in learning effectiveness. Furthermore, manually checking homework is a heavy burden for parents and teachers, and it also hinders children's independent learning.

[1132] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1133] In this invention, the server includes means for receiving image data of homework, means for converting the received image data into text data using optical character recognition technology, means for analyzing the converted text data and determining the problem and answer, means for generating feedback based on the determination results, means for transmitting the generated feedback, means for analyzing the question content and generating an appropriate answer, means for providing the appropriate answer to the user, means for accepting online tutoring reservation information, means for managing reservations, means for connecting the online tutor with the user at a specified time, and means for analyzing the answer and generating feedback using a generative AI model. This enables efficient and integrated support for children's homework. Specifically, this system realizes automatic homework answer checking, provision of appropriate feedback, immediate response to questions, and effective online tutoring, thereby improving learning effectiveness.

[1134] "Homework image data" is an image file of a homework page, which is taken by the user using a terminal and sent to the system.

[1135] "Optical character recognition technology" is a technology that extracts character information from an image and converts it into text data, and is also known as OCR (Optical Character Recognition).

[1136] "Text data" refers to character string data converted using optical character recognition technology, and includes homework content such as mathematical formulas and sentences.

[1137] "Analysis" is the process of identifying the type of question based on text data and evaluating the accuracy of the answer.

[1138] An "answer" is a response to a question included in a homework assignment, and is determined by analysis.

[1139] "Feedback" refers to messages and explanations generated based on the analysis results, and is learning support information provided to the user.

[1140] "Question content" refers to the question or request for explanation that the user inputs to the system in text format.

[1141] An "answer" is appropriate information or explanation provided in response to a user's question, and is generated by the system.

[1142] "Online tutoring" is a method of learning support in which a user connects with an online tutor via video chat or other means at a designated time and receives instruction in real time.

[1143] "Reservation information" refers to the date and time of the online tutoring session desired by the user and other detailed information, which is accepted and managed by the system.

[1144] A "generative AI model" is a machine learning model trained on large datasets and used to analyze answers, provide feedback, and generate answers to questions.

[1145] A "prompt sentence" is an input sentence given to a generative AI model, and contains information that will form the basis for the generated answer and analysis.

[1146] This invention is a system for effectively supporting children with their homework, and is implemented in a form where a server, terminals, and users work together. This system has the functions of checking homework answers, providing feedback, answering questions, and online tutoring.

[1147] Homework answers and feedback

[1148] Receiving image data for homework

[1149] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system. In this case, the device sends the captured image file to an API endpoint (e.g., / upload / homework) that sends the image file to the server.

[1150] Optical Character Recognition and Text Conversion

[1151] The server uses the received image data to perform optical character recognition (OCR) and converts the text information in the image into text data. Software such as Google Cloud Vision API and Tesseract OCR can be used. This technology can convert handwritten mathematical formulas and sentences into text with high accuracy.

[1152] Analysis and evaluation of answers

[1153] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, a generative AI model (e.g., OpenAI's GPT-4) is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[1154] Feedback Generation and Submission

[1155] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Well done! Try your best on the next question" is generated.

[1156] Question and Answer Function

[1157] Receiving and analyzing questions

[1158] The device accepts a question from the user in text format and sends it to the server. For example, the user might ask, "Why does x equal 2?" The device then sends the question to an API endpoint (e.g., / submit / question) that sends the question to the server.

[1159] Generate and provide appropriate answers

[1160] The server analyzes the received question using a generative AI model and generates an appropriate answer. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[1161] Online Tutoring

[1162] Accepting and managing tutoring appointments

[1163] The device accepts the user's desired date and time for online tutoring and sends the reservation information to the server. The server manages this information and stores it in a reservation database. Specifically, the device sends the reservation information to the API endpoint (e.g., / schedule / tutoring) that sends the information to the server.

[1164] Connect online at a specified time

[1165] When the reserved time arrives, the server connects the online tutor and the device using video chat (e.g., Zoom or Google Meet). For example, if a user reserves English tutoring at 2 p.m., the server generates a video chat link at that time and notifies the device.

[1166] Component examples and prompt sentence examples

[1167] Checking math homework answers

[1168] A user takes a photo of a math homework page with their smartphone and uploads it to the system. The server converts the content into text using OCR, and a generative AI model determines the answer. Appropriate feedback is generated and displayed on the device.

[1169] English Questions and Answers

[1170] The user enters a question such as "What does this sentence mean?" and sends it to the system. The server uses a generative AI model to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays "This sentence means..."

[1171] Examples of applying prompt statements

[1172] An example prompt might be: "The user asks, 'Why does 6 ÷ 2 = 3?' Please provide a clear explanation for answering this question."

[1173] These detailed explanations of embodiments of the present invention enable efficient checking of children's homework and providing feedback, support for learning comprehension, and smooth operation of online tutoring.

[1174] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1175] Step 1:

[1176] Receiving image data for homework

[1177] The device takes a photo of the homework. The user uses the device's camera to take a photo of the homework page and taps the "upload button" to send the image data to the system.

[1178] Input: Image data of homework taken by the user

[1179] Output: Image data sent to the server

[1180] Specific operation: The image file taken by the device is sent to the API endpoint (e.g., / upload / homework) that sends the image file to the server.

[1181] Step 2:

[1182] Optical Character Recognition and Text Conversion

[1183] The server retrieves the received image data and extracts text information from the image using optical character recognition (OCR) technology.

[1184] Input: Homework image data sent from the device

[1185] Output: Converted text data

[1186] Specific operation: The server uses the Google Cloud Vision API and Tesseract OCR to convert handwritten and printed character information in the image into text data.

[1187] Step 3:

[1188] Analysis and evaluation of answers

[1189] The server analyzes the converted text data to determine the type of question and sends prompts to the AI ​​model to generate and determine the answer.

[1190] Input: Converted text data (e.g. "3x + 5 = 11")

[1191] Output: Answer result (e.g. "x = 2")

[1192] Specific operation: The server sends a prompt to the generative AI model, such as "Solve this equation '3x + 5 = 11'," and analyzes the returned answer to determine its accuracy.

[1193] Step 4:

[1194] Feedback Generation and Submission

[1195] The server generates feedback based on the correctness of the answer and sends it to the user terminal.

[1196] Input: Answer result

[1197] Output: Generated feedback

[1198] Specific operation: The server uses the generative AI model to create a message such as "Good job! Try your best on the next question," and notifies the device.

[1199] Step 5:

[1200] Receiving and analyzing questions

[1201] The terminal accepts questions from the user in text format and sends them to the system.

[1202] Input: A question typed by the user (e.g., "Why is x equal to 2?")

[1203] Output: The question sent to the server

[1204] Specific operation: The device sends the question to the API endpoint (e.g., / submit / question) that sends the question to the server.

[1205] Step 6:

[1206] Generate and provide appropriate answers

[1207] The server analyzes the received question using a generative AI model, generates an appropriate answer, and sends it to the user's device.

[1208] Input: The question sent to the server

[1209] Output: The generated answer (e.g., "3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2")

[1210] Specific operation: The server sends a prompt to the generative AI model saying, "Please answer the user's question 'Why is 6 ÷ 2 = 3?'" and sends the answer to the user's device.

[1211] Step 7:

[1212] Accepting and managing tutoring appointments

[1213] The terminal accepts the user's desired date and time for online tutoring and transmits the reservation information to the server.

[1214] Input: Reservation information entered by the user (date, time, etc.)

[1215] Output: Reservation information sent to the server

[1216] Specific operation: The device sends the reservation information to the API endpoint (e.g., / schedule / tutoring) that sends the reservation information to the server.

[1217] Step 8:

[1218] Connect online at a specified time

[1219] When the reserved time arrives, the server arranges for the online tutor and the user terminal to be connected using video chat or the like.

[1220] Input: Reservation information and current time

[1221] Output: Video chat connection link

[1222] Specific operation: The server generates a video chat link using an API such as Zoom or Google Meet, and then connects by notifying the device at the specified time.

[1223] (Application example 1)

[1224] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1225] Supporting students with their daily homework is extremely important in today's educational environment. However, quickly checking homework answers, providing accurate feedback, and immediately answering students' questions places a heavy burden on educators. Furthermore, scheduling and managing online tutoring sessions is time-consuming, requiring efficient operation. Furthermore, the systems and applications needed to effectively implement this support are insufficient, making it difficult to provide consistent learning support.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1227] In this invention, the server includes means for receiving image data of homework, means for converting the received image data into text data using optical character recognition technology, means for analyzing the converted text data and determining the problem and answer, means for generating feedback based on the determination result, means for transmitting the generated feedback, means for accepting questions about homework, means for analyzing the content of the questions and generating appropriate answers, means for providing the generated answers to the user, means for accepting online tutoring reservation information, means for managing reservations based on the reservation information, means for connecting the user with an online tutor at a specified time, means for installing a learning support application on a tablet or smartphone, means for taking and uploading image data of the homework using a camera function of the tablet or smartphone, means for providing the generated feedback to the student in an easy-to-understand manner using natural language processing technology, means for training an AI using structured data and evaluating the accuracy of the answers, and means for supporting online and offline tutoring. This enables fast and accurate homework answer checking, feedback provision, question and answering, and efficient operation of online tutoring in learning environments.

[1228] The "means for receiving image data of homework" is a function that allows a user to take a photo of their homework and send the image data to a server.

[1229] "Means for converting into text data using optical character recognition technology" refers to a function that converts character information in received image data into text data using optical character recognition technology (OCR).

[1230] The "means for analyzing the converted text data and determining the questions and answers" is a function that can automatically determine the homework questions and answers by analyzing the text data converted by OCR.

[1231] The "means for generating feedback based on the assessment result" is a function that automatically generates appropriate feedback based on whether the answer is correct or incorrect.

[1232] The "means for transmitting the generated feedback" is a function for transmitting the generated feedback to the user's terminal and displaying it.

[1233] The "means for accepting questions about homework" is a function that allows a user to input questions about homework in text format and transmit them to the server.

[1234] "Means for analyzing the question content and generating an appropriate answer" is a function that analyzes the received question and generates an appropriate answer using AI.

[1235] The "means for providing the generated answer to the user" is a function for transmitting the generated answer to the user's terminal and displaying it.

[1236] The "means for accepting online tutoring reservation information" is a function that allows a user to input online tutoring reservation information and transmit that information to the server.

[1237] The "means for managing reservations based on reservation information" is a function for managing reservation data based on received reservation information.

[1238] The "means for connecting an online tutor with a user at a designated time" is a function that connects an online tutor with a user at the reserved time and allows them to make a video call or chat.

[1239] The "means for installing the learning support application on a tablet or smartphone" is a function that allows the learning support application to be installed on the user's tablet or smartphone.

[1240] "Means for taking and uploading image data of homework using the camera function of a tablet or smartphone" refers to a function that allows a user to take a photo of their homework using the camera of a tablet or smartphone and upload the image to a server.

[1241] "Means for providing the generated feedback to the student in an easy-to-understand format using natural language processing technology" is a function for providing the generated feedback to the student in an easily understandable format using natural language processing technology.

[1242] "Means for training AI using structured data and evaluating the accuracy of answers" refers to a function that uses structured data to train an AI model and then uses that model to evaluate the accuracy of answers.

[1243] "Means for supporting online and offline tutoring" refers to functions for supporting both online and offline tutoring.

[1244] The present invention provides a system for effectively supporting children in their homework, and is configured so that a server, a terminal, and a user work together. A specific embodiment of the system will be described below.

[1245] Homework answers and feedback

[1246] A means of receiving image data for homework

[1247] The user takes a photo of their homework using the camera function of their smartphone or tablet and sends the image data to the server. The device used is a general smartphone or tablet. The photo is uploaded using, for example, a mobile application.

[1248] Optical character recognition and text conversion methods

[1249] The server then performs optical character recognition (OCR) on the received image data, converting the character information in the image into text data. This process uses OCR technology such as Google Cloud Vision API.

[1250] Means of analyzing and judging answers

[1251] The server analyzes the converted text data and determines the homework problem and answer. The analysis uses Azure Machine Learning services to train an AI model. For example, it determines that the answer "x = 2" is correct when "3x + 5 = 11" is given.

[1252] Feedback generation and transmission methods

[1253] The server generates feedback based on the results and sends it to the user's device. The feedback includes not only whether the answer was correct or incorrect, but also detailed explanations and suggestions for improvement. The generated feedback is provided in an easy-to-understand manner using natural language processing technology (e.g., Google Cloud Natural Language API).

[1254] Question and Answer Function

[1255] How questions are received and analyzed

[1256] Users type in a homework question and send it to the server via their device, for example, "Why is x equal to 2?"

[1257] Means of generating and providing appropriate answers

[1258] The server analyzes the received question and generates an appropriate answer using AI (e.g., GPT-3) and provides it to the user. For example, the generated answer might be, "First, subtract 5 from 3x + 5 = 11 to get 3x = 6, then divide by 3 to get x = 2."

[1259] Online Tutoring

[1260] A means of accepting and managing tutoring bookings

[1261] The user inputs the desired date and time for online tutoring through the application, and the server receives and manages this information. Reservation management systems used include Microsoft Bookings.

[1262] A way to connect online at a specified time

[1263] When the scheduled time arrives, the server connects the user with the online tutor using video chat (e.g., Zoom API).

[1264] Specific examples

[1265] For example, after a student at a cram school completes their homework, they can take a photo of the page with their smartphone and upload it through an application. This data is then analyzed using optical character recognition on the server, and feedback is provided instantly. If a student asks a question about a problem they can't solve, the AI ​​can analyze it and provide an immediate answer. Furthermore, when parents schedule online tutoring, a video chat will automatically connect at the specified time.

[1266] Prompt Sentence Examples

[1267] Question: "Why does 3x + 5 = 11 result in x = 2?"

[1268] Prompt: "This student wants to know why 3x + 5 = 11 means x = 2. Please provide a detailed explanation."

[1269] Answer: "Let's first rearrange the equation 3x + 5 = 11. First, subtract 5 from both sides to get 3x = 6. Then, divide both sides by 3 to get x = 2. This will give you the answer."

[1270] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1271] Step 1:

[1272] Users take photos of their homework using the camera function of their smartphone or tablet and upload the images to the server through the application. The input is the image data of the homework, and the output is the image data sent to the server.

[1273] Step 2:

[1274] The server performs optical character recognition (OCR) on the received image data and converts the text information in the image into text data. Using the Google Cloud Vision API, the input for this step is image data and the output is text data.

[1275] Step 3:

[1276] The server uses the Azure Machine Learning service to analyze the text data obtained by OCR and automatically determine the homework questions and answers. The input for this step is the text data, and the output is the analysis result, which indicates whether the answer is correct or not.

[1277] Step 4:

[1278] The server generates feedback based on the results of the assessment and provides the generated feedback to the user in an easy-to-understand manner using natural language processing technology (Google Cloud Natural Language API). The input of this step is the correct or incorrect answer, and the output is a feedback message.

[1279] Step 5:

[1280] The user types in a text question about their homework and sends it to the server through the application. The input is the text question data, and the output is the transmission of the question data to the server.

[1281] Step 6:

[1282] The server analyzes the received question and generates an appropriate answer using an AI model such as GPT-3. The input of this step is the question data, and the output is the generated answer data.

[1283] Step 7:

[1284] The server sends the generated answer to the user's terminal and displays it. The input of this step is the generated answer data, and the output is the answer message displayed on the user's terminal.

[1285] Step 8:

[1286] A user inputs reservation information for online tutoring through the application and sends it to the server. The input is reservation information data, and the output is reservation information sent to the server.

[1287] Step 9:

[1288] The server accepts the reservation information and manages it using the reservation management system (Microsoft Bookings). The input of this step is reservation information data, and the output is reservation management data.

[1289] Step 10:

[1290] When the reservation time arrives, the server generates a video call link (e.g., Zoom API) to connect the user with the online tutor. The input of this step is the reservation information and the current time, and the output is the generated video call link.

[1291] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1292] The present invention is a system for effectively supporting children with their homework, implemented in a form in which a server, terminals, and users cooperate with each other. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide more advanced support. This system has the functions of checking homework answers, providing feedback, question answering, online tutoring, and emotion recognition, and detailed embodiments of the system are described below.

[1293] Homework answers and feedback

[1294] A means of receiving image data for homework

[1295] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system.

[1296] Optical character recognition and text conversion methods

[1297] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. OCR technology can convert handwritten mathematical formulas and sentences into text with high accuracy.

[1298] Means of analyzing and judging answers

[1299] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, an AI model is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[1300] Feedback generation and transmission methods

[1301] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Good job! Try your best on the next question" is generated.

[1302] Question and Answer Function

[1303] How questions are received and analyzed

[1304] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[1305] Means of generating and providing appropriate answers

[1306] The server analyzes the received question and generates an appropriate answer using an AI model. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[1307] Online Tutoring

[1308] A means of accepting and managing tutoring bookings

[1309] The terminal accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database.

[1310] A way to connect online at a specified time

[1311] When the reserved time arrives, the server connects the online tutor with the user's device via video chat, etc. For example, if a user reserves English tutoring at 2:00 p.m., the server generates a video chat link at that time and notifies the device.

[1312] Emotion recognition and feedback regulation

[1313] Emotion recognition engine configuration

[1314] The device acquires the user's facial expression data and sends it to the server. The emotion engine analyzes this data and identifies the user's emotions. For example, a camera can capture a user's facial expression while they are doing their homework and recognize their emotional state, such as "smiling," "confused," or "concentrated."

[1315] A means of regulating emotion-based feedback

[1316] The server automatically adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it can send an encouraging message like, "It might be a little difficult, but please try again."

[1317] Emotion-based online tutoring adjustment measures

[1318] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor may recommend teaching that will help the user relax.

[1319] System usage examples

[1320] Checking math homework answers

[1321] A child takes a photo of a math homework page with their smartphone and uploads it to the system. The server uses OCR to convert the content into text, and AI determines the answer. Appropriate feedback is generated and displayed on the device.

[1322] English Questions and Answers

[1323] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses AI to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[1324] Online tutoring

[1325] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[1326] Feedback adjustment using emotion recognition

[1327] While a child is doing their homework, the device's camera captures facial expression data and sends it to a server. The server then uses an emotion engine to analyze the user's emotional state and adjusts the feedback accordingly. For example, if a child is confused, the system will send an encouraging message.

[1328] The above describes specific embodiments of the present invention, which enable efficient homework confirmation and feedback, support for learning comprehension, smooth operation of online tutoring, and advanced emotion-based feedback and guidance.

[1329] The processing flow will be explained below.

[1330] Step 1:

[1331] The device activates the camera, and the user takes a photo of their homework. After taking the photo, the device displays a preview of the image for the user to review.

[1332] Step 2:

[1333] The user taps the "Send" button, and the device sends the homework image data to the server, where encryption protocols are used to ensure the data is secure.

[1334] Step 3:

[1335] The server receives the image data sent from the user terminal, and then temporarily stores the data for further processing.

[1336] Step 4:

[1337] The server starts OCR and extracts text information from the received image data. After the OCR process is complete, the extracted text data is stored in an internal data store.

[1338] Step 5:

[1339] The server analyzes the text data to identify the homework questions and answers. Specifically, it uses natural language processing (NLP) algorithms to analyze the structure of the text and distinguish between the questions and answers.

[1340] Step 6:

[1341] The server uses an AI model to determine whether the extracted answer is correct or incorrect. The AI ​​model makes highly accurate judgments based on pre-trained data.

[1342] Step 7:

[1343] The server generates feedback for each question. If the answer is correct, it generates a message such as "That's right, well done!", and if the answer is incorrect, it generates feedback including specific points for improvement such as "That's wrong. Please review this."

[1344] Step 8:

[1345] The server generates feedback and sends it to the user's device, collecting the feedback into a single response message before sending it.

[1346] Step 9:

[1347] The terminal receives the feedback from the server and displays it on the user interface, and the user confirms the displayed feedback.

[1348] Step 10:

[1349] If a user has a question about their homework, they type the question in text format into the terminal and send it to the server, for example, "Why is this answer correct?"

[1350] Step 11:

[1351] The server analyzes the received question and uses AI models to generate an appropriate answer, based on appropriate references and explanations to help the user understand the answer.

[1352] Step 12:

[1353] The server generates a response and sends it to the user's device. The user receives the response and reviews it to further understand it.

[1354] Step 13:

[1355] The device activates the camera to capture the user's facial expression data and sends it to the server.

[1356] Step 14:

[1357] The server analyzes the facial expression data received by the emotion engine and identifies the user's emotional state, such as "smiling," "confusion," or "concentration."

[1358] Step 15:

[1359] The server adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it might send an encouraging message like, "It might be a little difficult, but please try again."

[1360] Step 16:

[1361] When a user desires online tutoring, they input the desired date and time of tutoring into their terminal and send the reservation information to the server. For example, they reserve the following Monday at 2:00 PM.

[1362] Step 17:

[1363] The server receives the reservation information, stores it in a reservation database, and generates and sends a reminder notification to the device when the reservation date and time approaches.

[1364] Step 18:

[1365] When the reserved time arrives, the server generates a link to connect the online tutor and the user's device via video chat and notifies the device. The device launches the video chat application, and the user joins the online tutoring session.

[1366] Step 19:

[1367] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor can instruct the user to provide instruction that will help the user relax.

[1368] Step 20:

[1369] After the session ends, the server collects the usage history and stores it for billing purposes. At the end of the month or when the session ends, the server calculates the charges based on the user's usage history and sends the billing information.

[1370] Example 2

[1371] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1372] Conventional homework support systems have limited functionality for automatically checking homework answers and providing feedback. Furthermore, most feedback is one-way and does not take the user's emotional state into account, making it difficult to provide effective learning support. Furthermore, the complexity of reserving and implementing online tutoring sessions makes it difficult to provide an efficient learning support environment. It is necessary to solve these issues and provide more effective, personalized homework support to users.

[1373] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: a means for the terminal to receive image data of homework; a means for the server to convert the received image data into text data using optical character recognition technology; a means for the server to analyze the converted text data and determine the question and answer; a means for the server to generate feedback based on the determination result; a means for the server to send the generated feedback to the terminal; a means for the terminal to acquire user facial expression data and send it to the server; and a means for the server to identify the user's emotion using emotion recognition technology and adjust the feedback based on the identification result. This enables efficient confirmation of homework and provision of feedback, support for learning comprehension, and advanced responses according to the user's emotions.

[1374] A "terminal" is a device operated by a user, and is used to take pictures of image data for homework, input questions, and make reservations for online tutoring.

[1375] A "server" is a computer system that receives, processes, and stores data sent from a terminal, and is a device that has functions such as optical character recognition of image data, answer determination, and feedback generation.

[1376] "Image data" is digital data including homework photos and other visual information that is taken by the terminal and sent to the server.

[1377] "Optical Character Recognition (OCR)" is a technology that converts handwritten or printed characters in image data into digital text.

[1378] "Text data" refers to data that represents character information in digital form, converted using optical character recognition technology.

[1379] The "means for determining the answer" is a mechanism by which the server analyzes the text data, identifies the type of question, and evaluates the accuracy of the answer.

[1380] "Feedback" is information provided to the user, including an evaluation of the correctness of the answer, reference explanations, and points for improvement.

[1381] "Emotion recognition technology" is a technology that analyzes a user's facial expression data and identifies the user's emotional state (for example, smiling, confused, focused, etc.).

[1382] "Online tutoring" refers to video chat and other remote instruction formats for educational assistance delivered in real time over the internet.

[1383] This invention is a system for effectively supporting children with their homework, implemented in a form where a server, terminals, and users work together. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide more advanced support. This system has the following functions: checking homework answers, providing feedback, question answering, online tutoring, and emotion recognition.

[1384] Homework answers and feedback

[1385] A means of receiving image data for homework

[1386] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system.

[1387] Optical character recognition and text conversion methods

[1388] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. OCR technology can convert handwritten mathematical formulas and sentences into text with high accuracy. Specifically, software such as Tesseract OCR is used.

[1389] Means of analyzing and judging answers

[1390] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, an AI model (e.g., Python + scikit-learn) is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[1391] Feedback generation and transmission methods

[1392] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Good job! Try your best on the next question" is generated.

[1393] Question and Answer Function

[1394] How questions are received and analyzed

[1395] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[1396] Means of generating and providing appropriate answers

[1397] The server analyzes the received question and generates an appropriate answer using an AI model (e.g., GPT-3). The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[1398] Online Tutoring

[1399] A means of accepting and managing tutoring bookings

[1400] The device accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database, typically an SQL or NoSQL database.

[1401] A way to connect online at a specified time

[1402] When the reservation time arrives, the server connects the online tutor and the user's device using a video chat tool (e.g., Zoom API). For example, if a user reserves English tutoring at 2 p.m., the server generates a video chat link at that time and notifies the device.

[1403] Emotion recognition and feedback regulation

[1404] Emotion recognition engine configuration

[1405] The device acquires the user's facial expression data and sends it to a server. The emotion engine analyzes this data and identifies the user's emotions. For example, a camera can capture a user's facial expression while they are doing their homework and recognize their emotional state, such as "smiling," "confused," or "concentrated." Specifically, emotion recognition software such as Microsoft Azure Cognitive Services is used.

[1406] A means of regulating emotion-based feedback

[1407] The server automatically adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it can send an encouraging message like, "It might be a little difficult, but please try again."

[1408] Emotion-based online tutoring adjustment measures

[1409] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor may recommend teaching that will help the user relax.

[1410] System usage examples

[1411] Checking math homework answers

[1412] A child takes a photo of a math homework page with their smartphone and uploads it to the system. The server uses OCR to convert the content into text, and AI determines the answer. Appropriate feedback is generated and displayed on the device.

[1413] English Questions and Answers

[1414] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses AI to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[1415] Online tutoring

[1416] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[1417] Feedback adjustment using emotion recognition

[1418] While a child is doing their homework, the device's camera captures facial expression data and sends it to a server. The server then uses an emotion engine to analyze the user's emotional state and adjusts the feedback accordingly. For example, if a child is confused, the system will send an encouraging message.

[1419] In this way, the system of the present invention is capable of efficiently checking homework and providing feedback, supporting learning comprehension, smoothly operating online tutoring, and providing advanced emotion-based feedback and guidance.

[1420] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1421] Step 1:

[1422] The device receives image data of the homework. The user takes a photo of the homework with their smartphone and saves the image data on the device. The input is the image file taken by the user, and the output is the image data saved on the device. The device sends this image data to the server.

[1423] Step 2:

[1424] The server receives the image data. The server receives the image data sent from the terminal. The input is the image file sent from the terminal, and the output is the image data saved on the server. The server saves this image data in temporary storage.

[1425] Step 3:

[1426] The server converts image data into text data using optical character recognition technology (OCR). Specifically, the server launches an application program (e.g., Tesseract OCR) and performs OCR on the image data. The input is an image file stored on the server, and the output is text data converted by OCR. Handwritten mathematical formulas and sentences are converted into digital text.

[1427] Step 4:

[1428] The server analyzes the converted text data. The server reads the text data and identifies the type of problem (e.g., mathematical formula or word problem). The input is the text data generated by OCR, and the output is the analysis result. The server performs appropriate processing depending on the type of problem identified.

[1429] Step 5:

[1430] The server uses an AI model to determine the answer. Specifically, the server uses a program (e.g., Python + scikit-learn) to analyze the text data and determine the accuracy of the answer. The input is the analyzed text data, and the output is the result of determining whether the answer is correct. For example, it determines whether the answer "3x + 5 = 11" is correct as "x = 2."

[1431] Step 6:

[1432] The server generates the feedback. The feedback is created based on the result of the answer's correctness. Specifically, the server generates the feedback text by combining template messages and reference explanations. The input is the result of the answer's correctness, and the output is the generated feedback text. For example, it generates feedback such as "That's correct! Excellent! Good luck on the next question."

[1433] Step 7:

[1434] The server sends the generated feedback to the terminal. The generated feedback is sent to the user's terminal. The input is the generated feedback sentence, and the output is the feedback displayed on the terminal. The user checks the feedback on the terminal.

[1435] Step 8:

[1436] The device acquires the user's facial expression data and sends it to the server. For example, the device's camera captures the user's facial expression while they are doing their homework, and sends the data to the server. The input is the facial expression image captured by the camera, and the output is the facial expression data sent to the server.

[1437] Step 9:

[1438] The server uses emotion recognition technology to identify the user's emotions. Specifically, the server uses emotion recognition software (e.g., Microsoft Azure Cognitive Services) to analyze facial expression data and identify the user's emotional state. The input is facial expression data, and the output is the identified emotional state. Recognized emotions include "smiling," "confusion," and "concentration."

[1439] Step 10:

[1440] The server adjusts the feedback based on the emotion identification results. The content and expression of the feedback are automatically adjusted according to the recognized emotional state. The input is the identified emotional state and the previously generated feedback, and the output is the adjusted feedback text. For example, if the user is confused, it adds an encouraging message such as "It might be a little difficult, but please try again."

[1441] Step 11:

[1442] The terminal accepts the user's question and sends it to the server. The user enters a question in text format, and the terminal sends the question to the server. The input is the text format question entered by the user, and the output is the question data sent to the server.

[1443] Step 12:

[1444] The server analyzes the received question and generates an appropriate answer using an AI model. Specifically, the server analyzes the question using an AI model (e.g., GPT-3) and generates an appropriate answer. The input is the question data, and the output is the generated answer. For example, in response to the question "Why is x 2?", the server generates an answer such as "From 3x + 5 = 11, first subtract 5 to make 3x = 6, then divide by 3 to make x = 2."

[1445] Step 13:

[1446] The server sends the generated answer to the terminal. The server sends the generated answer to the user's terminal. The input is the generated answer, and the output is the answer sent to the terminal.

[1447] Step 14:

[1448] The terminal sends online tutoring reservation information to the server. The user inputs the desired date and time of tutoring, and the terminal sends that information to the server. The input is the reservation information, and the output is the reservation data sent to the server.

[1449] Step 15:

[1450] The server manages reservations based on the reservation information. Specifically, the server saves the reservation information in a reservation database and manages the reservations. The input is reservation data, and the output is managed reservation information.

[1451] Step 16:

[1452] When the reservation time arrives, the server connects the online tutor and the user's device using a video chat tool. Specifically, the server generates a video chat link and notifies the device. The input is the reservation information and the video chat link, and the output is a video chat connection between the user and the tutor. For example, the connection can be made at the specified time using a video chat tool (e.g., Zoom API).

[1453] (Application example 2)

[1454] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1455] Currently, in children's homework support systems, it is important not only to judge whether an answer is correct or incorrect, but also to adjust feedback based on the user's emotions, which conventional systems have not been able to do adequately. Furthermore, in brick-and-mortar educational services, it is difficult to provide individual support in real time, resulting in an unsatisfactory user experience. Therefore, the present invention aims to provide more effective homework support and improve the quality of brick-and-mortar educational services by recognizing the user's emotions and adjusting feedback.

[1456] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1457] In this invention, the server includes means for receiving image data of homework, means for converting the received image data into text data using optical character recognition technology, means for analyzing the converted text data and determining the question and answer, means for generating feedback based on the determination result, means for transmitting the generated feedback, means for acquiring facial expression data of the user using a camera, means for analyzing the facial expression data using an emotion analysis engine and identifying the user's emotion, and means for adjusting the feedback content based on the identified emotion. This allows for more effective homework support by recognizing the emotion from the user's facial expression and providing feedback accordingly, thereby enabling the quality of educational services at physical stores to be improved.

[1458] "Homework image data" refers to digital image files of what a child has written as homework, captured with a camera.

[1459] "Means for receiving" refers to a communication means for transferring image data from a terminal to a server.

[1460] "Optical character recognition technology" is a technology that recognizes character information in an image and converts it into digital text.

[1461] "Text data" means digital data containing textual information that has been converted using optical character recognition technology.

[1462] "Means for analyzing" refers to means for processing the converted text data and determining the questions and answers.

[1463] A "means of assessment" is a means for evaluating whether the answer to the question is correct.

[1464] "Feedback" refers to information, including advice and corrections, generated based on analysis and judgment results.

[1465] "Means for sending" refers to communication means for transferring the generated feedback to a user terminal.

[1466] "Camera" refers to an image capture device for capturing a user's facial expression.

[1467] "Facial expression data" refers to image data of the user's face captured by a camera.

[1468] An "emotion analysis engine" is software that has an algorithm that analyzes facial expression data and identifies emotions.

[1469] "Means for identifying" refers to means for classifying the emotional state of a user using a sentiment analysis engine.

[1470] The "means for adjusting the feedback content" is a means for changing the content or expression of the feedback based on the identified emotion.

[1471] This invention is a system for effectively supporting children with their homework, and is implemented in a form in which a server, a terminal, and a user cooperate with each other. The system has functions for checking homework answers, providing feedback, question answering, online tutoring, and emotion recognition. Specific embodiments of the system are described below.

[1472] Homework answers and feedback

[1473] A means of receiving image data for homework

[1474] The device takes a photo of the homework and sends the image data to the server. The device can be a tablet or smartphone, and the user can take a photo of the homework page using the camera.

[1475] Optical character recognition and text conversion methods

[1476] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. Tesseract is used for the OCR technology, which enables handwritten mathematical formulas and sentences to be converted into text with high accuracy.

[1477] Means of analyzing and judging answers

[1478] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, a generative AI model is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[1479] Feedback generation and transmission methods

[1480] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Good job! Try your best on the next question" is generated.

[1481] Question and Answer Function

[1482] How questions are received and analyzed

[1483] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[1484] Means of generating and providing appropriate answers

[1485] The server analyzes the received question and generates an appropriate answer using a generative AI model. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[1486] Online Tutoring

[1487] A means of accepting and managing tutoring bookings

[1488] The terminal accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database.

[1489] A way to connect online at a specified time

[1490] When the reserved time arrives, the server connects the online tutor with the user's device via video chat, etc. For example, if a user reserves English tutoring at 2:00 p.m., the server generates a video chat link at that time and notifies the device.

[1491] Emotion recognition and feedback regulation

[1492] Emotion recognition engine configuration

[1493] The device acquires the user's facial expression data and sends it to the server. The emotion engine analyzes this data and identifies the user's emotions. For example, a camera can capture a user's facial expression while they are doing their homework and recognize their emotional state, such as "smiling," "confused," or "concentrated."

[1494] A means of regulating emotion-based feedback

[1495] The server automatically adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it can send an encouraging message like, "It might be a little difficult, but please try again."

[1496] Emotion-based online tutoring adjustment measures

[1497] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor may recommend teaching that will help the user relax.

[1498] System usage examples

[1499] Checking math homework answers

[1500] A child takes a photo of a math homework page on a tablet and uploads it to the system. The server uses OCR to convert the content into text, and a generative AI model determines the answer. Appropriate feedback is generated and displayed on the device.

[1501] English Questions and Answers

[1502] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses a generative AI model to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[1503] Online tutoring

[1504] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[1505] Feedback adjustment using emotion recognition

[1506] While a child is doing their homework, the device's camera captures facial expression data and sends it to a server. The server then uses an emotion engine to analyze the user's emotional state and adjusts the feedback accordingly. For example, if a child is confused, the system will send an encouraging message.

[1507] Examples of prompt statements

[1508] Please take a photo of your homework problem and submit it. The system will analyze your answer and provide detailed feedback on whether your answer is correct or incorrect. To proceed, press the "Next" button.

[1509] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1510] Step 1:

[1511] Take a picture of your homework and send it

[1512] Subject: User (child)

[1513] explanation:

[1514] The user takes a photo of their homework using the camera on their tablet or smartphone, and the captured image data is sent to the server via the application.

[1515] Input: Homework image (taken by user)

[1516] Output: Image data sent to the server

[1517] Step 2:

[1518] Converts received image data into text data using optical character recognition technology

[1519] Subject: Server

[1520] explanation:

[1521] The server converts the image data of the homework received into text data using Tesseract OCR, analyzing the text information in the image and converting it into digital text.

[1522] Input: Received image data

[1523] Output: Converted text data

[1524] Step 3:

[1525] Analyze text data and determine questions and answers

[1526] Subject: Server

[1527] explanation:

[1528] The server analyzes the converted text data using a generative AI model. It identifies the type of problem (e.g., mathematical formula, word problem, etc.) and determines whether the answer is correct. For example, if the answer to the analyzed formula "3x + 5 = 11" is "x = 2," it is determined to be correct.

[1529] Input: Converted text data

[1530] Output: Judgment result (correct or incorrect)

[1531] Step 4:

[1532] Generate and send feedback based on the results

[1533] Subject: Server

[1534] explanation:

[1535] The server generates feedback based on the results. The feedback includes whether the answer was correct or incorrect, as well as helpful explanations and suggestions for improvement. The generated feedback is sent to the device. For example, a message such as "That's the correct answer! Try your best on the next question" is generated.

[1536] Input: Judgment result

[1537] Output: The generated feedback (message)

[1538] Step 5:

[1539] Acquires the user's facial expression data and sends it to the server

[1540] Subject: Device

[1541] explanation:

[1542] The device's camera is used to capture the user's facial expression data, which is then sent to a server for analysis by an emotion recognition engine.

[1543] Input: User's facial expression data (captured by camera)

[1544] Output: Facial expression data sent to the server

[1545] Step 6:

[1546] Facial expression data is analyzed using an emotion analysis engine to identify emotions.

[1547] Subject: Server

[1548] explanation:

[1549] The server analyzes the received facial expression data using an emotion analysis engine. The emotion analysis engine identifies the user's emotional state (e.g., smiling, confused, focused, etc.). For example, the emotion "confused" is identified from the facial expression data.

[1550] Input: Submitted facial expression data

[1551] Output: Identified emotional state

[1552] Step 7:

[1553] Tailor your feedback based on identified emotions

[1554] Subject: Server

[1555] explanation:

[1556] The server adjusts the feedback based on the user's emotional state. For example, if the user is confused, it might send an encouraging message like, "It might be a little difficult, but please try again." The adjusted feedback is then sent to the device.

[1557] Input: Identified emotional state

[1558] Output: Adjusted feedback (message)

[1559] Step 8:

[1560] Display feedback on your device

[1561] Subject: Device

[1562] explanation:

[1563] The device displays the feedback received from the server to the user, who can use the displayed feedback to deepen their understanding of the homework. For example, a message such as "It may be a little difficult, but please try again" may be displayed.

[1564] Input: Feedback sent by the server

[1565] Output: Feedback (messages) displayed on the terminal

[1566] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1567] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1568] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1569] [Fourth embodiment]

[1570] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1571] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1572] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1573] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1574] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1575] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1576] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1577] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1578] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1579] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1580] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1581] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1582] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1583] The present invention is a system for effectively supporting children with their homework, and is implemented in a form in which a server, terminals, and users cooperate with each other. This system has functions for checking homework answers, providing feedback, answering questions, and online tutoring, and detailed embodiments of the system are described below.

[1584] Homework answers and feedback

[1585] A means of receiving image data for homework

[1586] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system.

[1587] Optical character recognition and text conversion methods

[1588] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. OCR technology can convert handwritten mathematical formulas and sentences into text with high accuracy.

[1589] Means of analyzing and judging answers

[1590] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, an AI model is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[1591] Feedback generation and transmission methods

[1592] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Well done! Try your best on the next question" is generated.

[1593] Question and Answer Function

[1594] How questions are received and analyzed

[1595] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[1596] Means of generating and providing appropriate answers

[1597] The server analyzes the received question using an AI model and generates an appropriate answer. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[1598] Online Tutoring

[1599] A means of accepting and managing tutoring bookings

[1600] The terminal accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database.

[1601] A way to connect online at a specified time

[1602] When the reserved time arrives, the server connects the online tutor with the user's device via video chat, etc. For example, if a user reserves English tutoring at 2:00 p.m., the server generates a video chat link at that time and notifies the device.

[1603] System usage examples

[1604] Checking math homework answers

[1605] A child takes a photo of a math homework page with their smartphone and uploads it to the system. The server uses OCR to convert the content into text, and AI determines the answer. Appropriate feedback is generated and displayed on the device.

[1606] English Questions and Answers

[1607] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses AI to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[1608] Online tutoring

[1609] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[1610] The above describes a specific embodiment of the present invention, which enables efficient confirmation of homework and provision of feedback, support for learning comprehension, and smooth operation of online tutoring.

[1611] The processing flow will be explained below.

[1612] Step 1:

[1613] The device activates the camera, and the user takes a photo of their homework. After taking the photo, the device displays a preview of the image for the user to review.

[1614] Step 2:

[1615] The user taps the "Send" button, and the device sends the homework image data to the server, where encryption protocols are used to ensure the data is secure.

[1616] Step 3:

[1617] The server receives the image data sent from the user terminal, and then temporarily stores the data for further processing.

[1618] Step 4:

[1619] The server starts OCR and extracts text information from the received image data. After the OCR process is complete, the extracted text data is stored in an internal data store.

[1620] Step 5:

[1621] The server analyzes the text data to identify the homework questions and answers. Specifically, it uses natural language processing (NLP) algorithms to analyze the structure of the text and distinguish between the questions and answers.

[1622] Step 6:

[1623] The server uses an AI model to determine whether the extracted answer is correct or incorrect. The AI ​​model makes highly accurate judgments based on pre-trained data.

[1624] Step 7:

[1625] The server generates feedback for each question. If the answer is correct, it generates a message such as "That's right, well done!", and if the answer is incorrect, it generates feedback including specific points for improvement such as "That's wrong. Please review this."

[1626] Step 8:

[1627] The server generates feedback and sends it to the user's device, collecting the feedback into a single response message before sending it.

[1628] Step 9:

[1629] The terminal receives the feedback from the server and displays it on the user interface, and the user confirms the displayed feedback.

[1630] Step 10:

[1631] If a user has a question about their homework, they can enter the question in text format into the terminal and send it to the server. For example, they can enter a question such as "Why is this answer correct?"

[1632] Step 11:

[1633] The server analyzes the received question and uses AI models to generate an appropriate answer, based on appropriate references and explanations to help the user understand the answer.

[1634] Step 12:

[1635] The server generates a response and sends it to the user's device. The user receives the response and reviews it to further understand it.

[1636] Step 13:

[1637] When a user desires online tutoring, they input the desired date and time of tutoring into their terminal and send the reservation information to the server. For example, they reserve the following Monday at 2:00 PM.

[1638] Step 14:

[1639] The server receives the reservation information, stores it in a reservation database, and generates and sends a reminder notification to the device when the reservation date and time approaches.

[1640] Step 15:

[1641] When the reserved time arrives, the server generates a link to connect the online tutor and the user's device via video chat and notifies the device. The device launches the video chat application, and the user joins the online tutoring session.

[1642] Step 16:

[1643] After the session ends, the server collects the usage history and stores it for billing purposes. At the end of the month or when the session ends, the server calculates the charges based on the user's usage history and sends the billing information.

[1644] Example 1

[1645] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1646] Currently, there is no integrated system for supporting children's homework by checking answers, understanding answers, answering questions, and providing online tutoring. This means that children do not receive appropriate feedback or learning support, resulting in a decrease in learning effectiveness. Furthermore, manually checking homework is a heavy burden for parents and teachers, and it also hinders children's independent learning.

[1647] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1648] In this invention, the server includes means for receiving image data of homework, means for converting the received image data into text data using optical character recognition technology, means for analyzing the converted text data and determining the problem and answer, means for generating feedback based on the determination results, means for transmitting the generated feedback, means for analyzing the question content and generating an appropriate answer, means for providing the appropriate answer to the user, means for accepting online tutoring reservation information, means for managing reservations, means for connecting the online tutor with the user at a specified time, and means for analyzing the answer and generating feedback using a generative AI model. This enables efficient and integrated support for children's homework. Specifically, this system realizes automatic homework answer checking, provision of appropriate feedback, immediate response to questions, and effective online tutoring, thereby improving learning effectiveness.

[1649] "Homework image data" is an image file of a homework page, which is taken by the user using a terminal and sent to the system.

[1650] "Optical character recognition technology" is a technology that extracts character information from an image and converts it into text data, and is also known as OCR (Optical Character Recognition).

[1651] "Text data" refers to character string data converted using optical character recognition technology, and includes homework content such as mathematical formulas and sentences.

[1652] "Analysis" is the process of identifying the type of question based on text data and evaluating the accuracy of the answer.

[1653] An "answer" is a response to a question included in a homework assignment, and is determined by analysis.

[1654] "Feedback" refers to messages and explanations generated based on the analysis results, and is learning support information provided to the user.

[1655] "Question content" refers to the question or request for explanation that the user inputs to the system in text format.

[1656] An "answer" is appropriate information or explanation provided in response to a user's question, and is generated by the system.

[1657] "Online tutoring" is a method of learning support in which a user connects with an online tutor via video chat or other means at a designated time and receives instruction in real time.

[1658] "Reservation information" refers to the date and time of the online tutoring session desired by the user and other detailed information, which is accepted and managed by the system.

[1659] A "generative AI model" is a machine learning model trained on large datasets and used to analyze answers, provide feedback, and generate answers to questions.

[1660] A "prompt sentence" is an input sentence given to a generative AI model, and contains information that will form the basis for the generated answer and analysis.

[1661] This invention is a system for effectively supporting children with their homework, and is implemented in a form where a server, terminals, and users work together. This system has the functions of checking homework answers, providing feedback, answering questions, and online tutoring.

[1662] Homework answers and feedback

[1663] Receiving image data for homework

[1664] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system. In this case, the device sends the captured image file to an API endpoint (e.g., / upload / homework) that sends the image file to the server.

[1665] Optical Character Recognition and Text Conversion

[1666] The server uses the received image data to perform optical character recognition (OCR) and converts the text information in the image into text data. Software such as Google Cloud Vision API and Tesseract OCR can be used. This technology can convert handwritten mathematical formulas and sentences into text with high accuracy.

[1667] Analysis and evaluation of answers

[1668] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, a generative AI model (e.g., OpenAI's GPT-4) is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[1669] Feedback Generation and Submission

[1670] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Well done! Try your best on the next question" is generated.

[1671] Question and Answer Function

[1672] Receiving and analyzing questions

[1673] The device accepts a question from the user in text format and sends it to the server. For example, the user might ask, "Why does x equal 2?" The device then sends the question to an API endpoint (e.g., / submit / question) that sends the question to the server.

[1674] Generate and provide appropriate answers

[1675] The server analyzes the received question using a generative AI model and generates an appropriate answer. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[1676] Online Tutoring

[1677] Accepting and managing tutoring appointments

[1678] The device accepts the user's desired date and time for online tutoring and sends the reservation information to the server. The server manages this information and stores it in a reservation database. Specifically, the device sends the reservation information to the API endpoint (e.g., / schedule / tutoring) that sends the information to the server.

[1679] Connect online at a specified time

[1680] When the reserved time arrives, the server connects the online tutor and the device using video chat (e.g., Zoom or Google Meet). For example, if a user reserves English tutoring at 2 p.m., the server generates a video chat link at that time and notifies the device.

[1681] Component examples and prompt sentence examples

[1682] Checking math homework answers

[1683] A user takes a photo of a math homework page with their smartphone and uploads it to the system. The server converts the content into text using OCR, and a generative AI model determines the answer. Appropriate feedback is generated and displayed on the device.

[1684] English Questions and Answers

[1685] The user enters a question such as "What does this sentence mean?" and sends it to the system. The server uses a generative AI model to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays "This sentence means..."

[1686] Examples of applying prompt statements

[1687] An example prompt might be: "The user asks, 'Why does 6 ÷ 2 = 3?' Please provide a clear explanation for answering this question."

[1688] These detailed explanations of embodiments of the present invention enable efficient checking of children's homework and providing feedback, support for learning comprehension, and smooth operation of online tutoring.

[1689] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1690] Step 1:

[1691] Receiving image data for homework

[1692] The device takes a photo of the homework. The user uses the device's camera to take a photo of the homework page and taps the "upload button" to send the image data to the system.

[1693] Input: Image data of homework taken by the user

[1694] Output: Image data sent to the server

[1695] Specific operation: The image file taken by the device is sent to the API endpoint (e.g., / upload / homework) that sends the image file to the server.

[1696] Step 2:

[1697] Optical Character Recognition and Text Conversion

[1698] The server retrieves the received image data and extracts text information from the image using optical character recognition (OCR) technology.

[1699] Input: Homework image data sent from the device

[1700] Output: Converted text data

[1701] Specific operation: The server uses the Google Cloud Vision API and Tesseract OCR to convert handwritten and printed character information in the image into text data.

[1702] Step 3:

[1703] Analysis and evaluation of answers

[1704] The server analyzes the converted text data to determine the type of question and sends prompts to the AI ​​model to generate and determine the answer.

[1705] Input: Converted text data (e.g. "3x + 5 = 11")

[1706] Output: Answer result (e.g. "x = 2")

[1707] Specific operation: The server sends a prompt to the generative AI model, such as "Solve this equation '3x + 5 = 11'," and analyzes the returned answer to determine its accuracy.

[1708] Step 4:

[1709] Feedback Generation and Submission

[1710] The server generates feedback based on the correctness of the answer and sends it to the user terminal.

[1711] Input: Answer result

[1712] Output: Generated feedback

[1713] Specific operation: The server uses the generative AI model to create a message such as "Good job! Try your best on the next question," and notifies the device.

[1714] Step 5:

[1715] Receiving and analyzing questions

[1716] The terminal accepts questions from the user in text format and sends them to the system.

[1717] Input: A question typed by the user (e.g., "Why is x equal to 2?")

[1718] Output: The question sent to the server

[1719] Specific operation: The device sends the question to the API endpoint (e.g., / submit / question) that sends the question to the server.

[1720] Step 6:

[1721] Generate and provide appropriate answers

[1722] The server analyzes the received question using a generative AI model, generates an appropriate answer, and sends it to the user's device.

[1723] Input: The question sent to the server

[1724] Output: The generated answer (e.g., "3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2")

[1725] Specific operation: The server sends a prompt to the generative AI model saying, "Please answer the user's question 'Why is 6 ÷ 2 = 3?'" and sends the answer to the user's device.

[1726] Step 7:

[1727] Accepting and managing tutoring appointments

[1728] The terminal accepts the user's desired date and time for online tutoring and transmits the reservation information to the server.

[1729] Input: Reservation information entered by the user (date, time, etc.)

[1730] Output: Reservation information sent to the server

[1731] Specific operation: The device sends the reservation information to the API endpoint (e.g., / schedule / tutoring) that sends the reservation information to the server.

[1732] Step 8:

[1733] Connect online at a specified time

[1734] When the reserved time arrives, the server arranges for the online tutor and the user terminal to be connected using video chat or the like.

[1735] Input: Reservation information and current time

[1736] Output: Video chat connection link

[1737] Specific operation: The server generates a video chat link using an API such as Zoom or Google Meet, and then connects by notifying the device at the specified time.

[1738] (Application example 1)

[1739] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1740] Supporting students with their daily homework is extremely important in today's educational environment. However, quickly checking homework answers, providing accurate feedback, and immediately answering students' questions places a heavy burden on educators. Furthermore, scheduling and managing online tutoring sessions is time-consuming, requiring efficient operation. Furthermore, the systems and applications needed to effectively implement this support are insufficient, making it difficult to provide consistent learning support.

[1741] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1742] In this invention, the server includes means for receiving image data of homework, means for converting the received image data into text data using optical character recognition technology, means for analyzing the converted text data and determining the problem and answer, means for generating feedback based on the determination result, means for transmitting the generated feedback, means for accepting questions about homework, means for analyzing the content of the questions and generating appropriate answers, means for providing the generated answers to the user, means for accepting online tutoring reservation information, means for managing reservations based on the reservation information, means for connecting the user with an online tutor at a specified time, means for installing a learning support application on a tablet or smartphone, means for taking and uploading image data of the homework using a camera function of the tablet or smartphone, means for providing the generated feedback to the student in an easy-to-understand manner using natural language processing technology, means for training an AI using structured data and evaluating the accuracy of the answers, and means for supporting online and offline tutoring. This enables fast and accurate homework answer checking, feedback provision, question and answering, and efficient operation of online tutoring in learning environments.

[1743] The "means for receiving image data of homework" is a function that allows a user to take a photo of their homework and send the image data to a server.

[1744] "Means for converting into text data using optical character recognition technology" refers to a function that converts character information in received image data into text data using optical character recognition technology (OCR).

[1745] The "means for analyzing the converted text data and determining the questions and answers" is a function that can automatically determine the homework questions and answers by analyzing the text data converted by OCR.

[1746] The "means for generating feedback based on the assessment result" is a function that automatically generates appropriate feedback based on whether the answer is correct or incorrect.

[1747] The "means for transmitting the generated feedback" is a function for transmitting the generated feedback to the user's terminal and displaying it.

[1748] The "means for accepting questions about homework" is a function that allows a user to input questions about homework in text format and transmit them to the server.

[1749] "Means for analyzing the question content and generating an appropriate answer" is a function that analyzes the received question and generates an appropriate answer using AI.

[1750] The "means for providing the generated answer to the user" is a function for transmitting the generated answer to the user's terminal and displaying it.

[1751] The "means for accepting online tutoring reservation information" is a function that allows a user to input online tutoring reservation information and transmit that information to the server.

[1752] The "means for managing reservations based on reservation information" is a function for managing reservation data based on received reservation information.

[1753] The "means for connecting an online tutor with a user at a designated time" is a function that connects an online tutor with a user at the reserved time and allows them to make a video call or chat.

[1754] The "means for installing the learning support application on a tablet or smartphone" is a function that allows the learning support application to be installed on the user's tablet or smartphone.

[1755] "Means for taking and uploading image data of homework using the camera function of a tablet or smartphone" refers to a function that allows a user to take a photo of their homework using the camera of a tablet or smartphone and upload the image to a server.

[1756] "Means for providing the generated feedback to the student in an easy-to-understand format using natural language processing technology" is a function for providing the generated feedback to the student in an easily understandable format using natural language processing technology.

[1757] "Means for training AI using structured data and evaluating the accuracy of answers" refers to a function that uses structured data to train an AI model and then uses that model to evaluate the accuracy of answers.

[1758] "Means for supporting online and offline tutoring" refers to functions for supporting both online and offline tutoring.

[1759] The present invention provides a system for effectively supporting children in their homework, and is configured so that a server, a terminal, and a user work together. A specific embodiment of the system will be described below.

[1760] Homework answers and feedback

[1761] A means of receiving image data for homework

[1762] The user takes a photo of their homework using the camera function of their smartphone or tablet and sends the image data to the server. The device used is a general smartphone or tablet. The photo is uploaded using, for example, a mobile application.

[1763] Optical character recognition and text conversion methods

[1764] The server then performs optical character recognition (OCR) on the received image data, converting the character information in the image into text data. This process uses OCR technology such as Google Cloud Vision API.

[1765] Means of analyzing and judging answers

[1766] The server analyzes the converted text data and determines the homework problem and answer. The analysis uses Azure Machine Learning services to train an AI model. For example, it determines that the answer "x = 2" is correct when "3x + 5 = 11" is given.

[1767] Feedback generation and transmission methods

[1768] The server generates feedback based on the results and sends it to the user's device. The feedback includes not only whether the answer was correct or incorrect, but also detailed explanations and suggestions for improvement. The generated feedback is provided in an easy-to-understand manner using natural language processing technology (e.g., Google Cloud Natural Language API).

[1769] Question and Answer Function

[1770] How questions are received and analyzed

[1771] Users type in a homework question and send it to the server via their device, for example, "Why is x equal to 2?"

[1772] Means of generating and providing appropriate answers

[1773] The server analyzes the received question and generates an appropriate answer using AI (e.g., GPT-3) and provides it to the user. For example, the generated answer might be, "First, subtract 5 from 3x + 5 = 11 to get 3x = 6, then divide by 3 to get x = 2."

[1774] Online Tutoring

[1775] A means of accepting and managing tutoring bookings

[1776] The user inputs the desired date and time for online tutoring through the application, and the server receives and manages this information. Reservation management systems used include Microsoft Bookings.

[1777] A way to connect online at a specified time

[1778] When the scheduled time arrives, the server connects the user with the online tutor using video chat (e.g., Zoom API).

[1779] Specific examples

[1780] For example, after a student at a cram school completes their homework, they can take a photo of the page with their smartphone and upload it through an application. This data is then analyzed using optical character recognition on the server, and feedback is provided instantly. If a student asks a question about a problem they can't solve, the AI ​​can analyze it and provide an immediate answer. Furthermore, when parents schedule online tutoring, a video chat will automatically connect at the specified time.

[1781] Prompt Sentence Examples

[1782] Question: "Why does 3x + 5 = 11 result in x = 2?"

[1783] Prompt: "This student wants to know why 3x + 5 = 11 means x = 2. Please provide a detailed explanation."

[1784] Answer: "Let's first rearrange the equation 3x + 5 = 11. First, subtract 5 from both sides to get 3x = 6. Then, divide both sides by 3 to get x = 2. This will give you the answer."

[1785] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1786] Step 1:

[1787] Users take photos of their homework using the camera function of their smartphone or tablet and upload the images to the server through the application. The input is the image data of the homework, and the output is the image data sent to the server.

[1788] Step 2:

[1789] The server performs optical character recognition (OCR) on the received image data and converts the text information in the image into text data. Using the Google Cloud Vision API, the input for this step is image data and the output is text data.

[1790] Step 3:

[1791] The server uses the Azure Machine Learning service to analyze the text data obtained by OCR and automatically determine the homework questions and answers. The input for this step is the text data, and the output is the analysis result, which indicates whether the answer is correct or not.

[1792] Step 4:

[1793] The server generates feedback based on the results of the assessment and provides the generated feedback to the user in an easy-to-understand manner using natural language processing technology (Google Cloud Natural Language API). The input of this step is the correct or incorrect answer, and the output is a feedback message.

[1794] Step 5:

[1795] The user types in a text question about their homework and sends it to the server through the application. The input is the text question data, and the output is the transmission of the question data to the server.

[1796] Step 6:

[1797] The server analyzes the received question and generates an appropriate answer using an AI model such as GPT-3. The input of this step is the question data, and the output is the generated answer data.

[1798] Step 7:

[1799] The server sends the generated answer to the user's terminal and displays it. The input of this step is the generated answer data, and the output is the answer message displayed on the user's terminal.

[1800] Step 8:

[1801] A user inputs reservation information for online tutoring through the application and sends it to the server. The input is reservation information data, and the output is reservation information sent to the server.

[1802] Step 9:

[1803] The server accepts the reservation information and manages it using the reservation management system (Microsoft Bookings). The input of this step is reservation information data, and the output is reservation management data.

[1804] Step 10:

[1805] When the reservation time arrives, the server generates a video call link (e.g., Zoom API) to connect the user with the online tutor. The input of this step is the reservation information and the current time, and the output is the generated video call link.

[1806] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1807] The present invention is a system for effectively supporting children with their homework, implemented in a form in which a server, terminals, and users cooperate with each other. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide more advanced support. This system has the functions of checking homework answers, providing feedback, question answering, online tutoring, and emotion recognition, and detailed embodiments of the system are described below.

[1808] Homework answers and feedback

[1809] A means of receiving image data for homework

[1810] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system.

[1811] Optical character recognition and text conversion methods

[1812] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. OCR technology can convert handwritten mathematical formulas and sentences into text with high accuracy.

[1813] Means of analyzing and judging answers

[1814] The server analyzes the converted text data, identifies the type of problem (e.g., mathematical formula, word problem, etc.), and determines the answer. At this time, an AI model is introduced to evaluate the accuracy of the answer. For example, for the problem "3x + 5 = 11," the answer "x = 2" is determined to be correct.

[1815] Feedback generation and transmission methods

[1816] The server generates feedback based on the results and sends it to the device. The feedback includes not only whether the answer was correct or incorrect, but also reference explanations and areas for improvement. For example, a message such as "Good job! Try your best on the next question" is generated.

[1817] Question and Answer Function

[1818] How questions are received and analyzed

[1819] The device accepts questions from the user in text format and sends them to the server. For example, a child might ask, "Why does x equal 2?"

[1820] Means of generating and providing appropriate answers

[1821] The server analyzes the received question and generates an appropriate answer using an AI model. The generated answer is sent to the device and provided to the user. For example, an explanation such as "From 3x + 5 = 11, first subtract 5 to get 3x = 6, then divide by 3 to get x = 2" is provided.

[1822] Online Tutoring

[1823] A means of accepting and managing tutoring bookings

[1824] The terminal accepts the user's desired date and time for online tutoring and sends the reservation information to the server, which manages this information and stores it in a reservation database.

[1825] A way to connect online at a specified time

[1826] When the reserved time arrives, the server connects the online tutor with the user's device via video chat, etc. For example, if a user reserves English tutoring at 2:00 p.m., the server generates a video chat link at that time and notifies the device.

[1827] Emotion recognition and feedback regulation

[1828] Emotion recognition engine configuration

[1829] The device acquires the user's facial expression data and sends it to the server. The emotion engine analyzes this data and identifies the user's emotions. For example, a camera can capture a user's facial expression while they are doing their homework and recognize their emotional state, such as "smiling," "confused," or "concentrated."

[1830] A means of regulating emotion-based feedback

[1831] The server automatically adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it can send an encouraging message like, "It might be a little difficult, but please try again."

[1832] Emotion-based online tutoring adjustment measures

[1833] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor may recommend teaching that will help the user relax.

[1834] System usage examples

[1835] Checking math homework answers

[1836] A child takes a photo of a math homework page with their smartphone and uploads it to the system. The server uses OCR to convert the content into text, and AI determines the answer. Appropriate feedback is generated and displayed on the device.

[1837] English Questions and Answers

[1838] A child types a question, such as "What does this sentence mean?", and sends it to the system. The server uses AI to analyze the question, generates an appropriate answer, and sends it to the device. The device then displays, "This sentence means..."

[1839] Online tutoring

[1840] Parents can schedule online tutoring for their children, and at the designated time, the server connects the online tutor to the device via video chat, allowing children to receive instruction in real time, improving learning effectiveness.

[1841] Feedback adjustment using emotion recognition

[1842] While a child is doing their homework, the device's camera captures facial expression data and sends it to a server. The server then uses an emotion engine to analyze the user's emotional state and adjusts the feedback accordingly. For example, if a child is confused, the system will send an encouraging message.

[1843] The above describes specific embodiments of the present invention, which enable efficient homework confirmation and feedback, support for learning comprehension, smooth operation of online tutoring, and advanced emotion-based feedback and guidance.

[1844] The processing flow will be explained below.

[1845] Step 1:

[1846] The device activates the camera, and the user takes a photo of their homework. After taking the photo, the device displays a preview of the image for the user to review.

[1847] Step 2:

[1848] The user taps the "Send" button, and the device sends the homework image data to the server, where encryption protocols are used to ensure the data is secure.

[1849] Step 3:

[1850] The server receives the image data sent from the user terminal, and then temporarily stores the data for further processing.

[1851] Step 4:

[1852] The server starts OCR and extracts text information from the received image data. After the OCR process is complete, the extracted text data is stored in an internal data store.

[1853] Step 5:

[1854] The server analyzes the text data to identify the homework questions and answers. Specifically, it uses natural language processing (NLP) algorithms to analyze the structure of the text and distinguish between the questions and answers.

[1855] Step 6:

[1856] The server uses an AI model to determine whether the extracted answer is correct or incorrect. The AI ​​model makes highly accurate judgments based on pre-trained data.

[1857] Step 7:

[1858] The server generates feedback for each question. If the answer is correct, it generates a message such as "That's right, well done!", and if the answer is incorrect, it generates feedback including specific points for improvement such as "That's wrong. Please review this."

[1859] Step 8:

[1860] The server generates feedback and sends it to the user's device, collecting the feedback into a single response message before sending it.

[1861] Step 9:

[1862] The terminal receives the feedback from the server and displays it on the user interface, and the user confirms the displayed feedback.

[1863] Step 10:

[1864] If a user has a question about their homework, they type the question in text format into the terminal and send it to the server, for example, "Why is this answer correct?"

[1865] Step 11:

[1866] The server analyzes the received question and uses AI models to generate an appropriate answer, based on appropriate references and explanations to help the user understand the answer.

[1867] Step 12:

[1868] The server generates a response and sends it to the user's device. The user receives the response and reviews it to further understand it.

[1869] Step 13:

[1870] The device activates the camera to capture the user's facial expression data and sends it to the server.

[1871] Step 14:

[1872] The server analyzes the facial expression data received by the emotion engine and identifies the user's emotional state, such as "smiling," "confusion," or "concentration."

[1873] Step 15:

[1874] The server adjusts the content and expression of the feedback based on the user's emotional state. For example, if the user is confused, it might send an encouraging message like, "It might be a little difficult, but please try again."

[1875] Step 16:

[1876] When a user desires online tutoring, they input the desired date and time of tutoring into their terminal and send the reservation information to the server. For example, they reserve the following Monday at 2:00 PM.

[1877] Step 17:

[1878] The server receives the reservation information, stores it in a reservation database, and generates and sends a reminder notification to the device when the reservation date and time approaches.

[1879] Step 18:

[1880] When the reserved time arrives, the server generates a link to connect the online tutor and the user's device via video chat and notifies the device. The device launches the video chat application, and the user joins the online tutoring session.

[1881] Step 19:

[1882] The server uses the emotion engine data to inform the online tutor of the user's emotional state and adjust the teaching method accordingly. For example, if the user is nervous, the tutor can instruct the user to provide instruction that will help the user relax.

[1883] Step 20:

[1884] After the session ends, the server collects the usage history and stores it for billing purposes. At the end of the month or when the session ends, the server calculates the charges based on the user's usage history and sends the billing information.

[1885] Example 2

[1886] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1887] Conventional homework support systems have limited functionality for automatically checking homework answers and providing feedback. Furthermore, most feedback is one-way and does not take the user's emotional state into account, making it difficult to provide effective learning support. Furthermore, the complexity of reserving and implementing online tutoring sessions makes it difficult to provide an efficient learning support environment. It is necessary to solve these issues and provide more effective, personalized homework support to users.

[1888] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: a means for the terminal to receive image data of homework; a means for the server to convert the received image data into text data using optical character recognition technology; a means for the server to analyze the converted text data and determine the question and answer; a means for the server to generate feedback based on the determination result; a means for the server to send the generated feedback to the terminal; a means for the terminal to acquire user facial expression data and send it to the server; and a means for the server to identify the user's emotion using emotion recognition technology and adjust the feedback based on the identification result. This enables efficient confirmation of homework and provision of feedback, support for learning comprehension, and advanced responses according to the user's emotions.

[1889] A "terminal" is a device operated by a user, and is used to take pictures of image data for homework, input questions, and make reservations for online tutoring.

[1890] A "server" is a computer system that receives, processes, and stores data sent from a terminal, and is a device that has functions such as optical character recognition of image data, answer determination, and feedback generation.

[1891] "Image data" is digital data including homework photos and other visual information that is taken by the terminal and sent to the server.

[1892] "Optical Character Recognition (OCR)" is a technology that converts handwritten or printed characters in image data into digital text.

[1893] "Text data" refers to data that represents character information in digital form, converted using optical character recognition technology.

[1894] The "means for determining the answer" is a mechanism by which the server analyzes the text data, identifies the type of question, and evaluates the accuracy of the answer.

[1895] "Feedback" is information provided to the user, including an evaluation of the correctness of the answer, reference explanations, and points for improvement.

[1896] "Emotion recognition technology" is a technology that analyzes a user's facial expression data and identifies the user's emotional state (for example, smiling, confused, focused, etc.).

[1897] "Online tutoring" refers to video chat and other remote instruction formats for educational assistance delivered in real time over the internet.

[1898] This invention is a system for effectively supporting children with their homework, implemented in a form where a server, terminals, and users work together. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide more advanced support. This system has the following functions: checking homework answers, providing feedback, question answering, online tutoring, and emotion recognition.

[1899] Homework answers and feedback

[1900] A means of receiving image data for homework

[1901] The device takes a photo of the homework and sends the image data to the server. For example, a user takes a photo of a math homework page with their smartphone and uploads it to the system.

[1902] Optical character recognition and text conversion methods

[1903] The server uses the received image data to perform optical character recognition (OCR) and converts the character information in the image into text data. OCR technology can convert handwritten mathematical formulas and sentences into text with high accuracy. Specifically, software such as Tesseract OCR is...

Claims

1. means for receiving image data of homework; means for converting the received image data into text data using optical character recognition technology; means for analyzing the converted text data and determining the questions and answers; means for generating feedback based on the determination result; a means for transmitting the generated feedback; A system including:

2. a means for receiving questions regarding homework; A means for analyzing the content of the question and generating an appropriate answer; means for providing the generated answer to the user; The system of claim 1 , comprising:

3. A means for accepting online tutoring reservation information; A means for managing reservations based on the reservation information; a means for connecting the user with an online tutor at a designated time; The system of claim 1 , comprising:

Images