System

Abstract

A system is provided.SOLUTION: A system includes a means for photographing an image of a homework assignment, a means for transmitting the photographed image to a server, a means for converting the image into text data in the server, a means for analyzing a problem and scoring by an AI model generated based on the converted text data, a means for transmitting a scoring result from the server to a terminal, and a means for displaying the scoring result in the terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, parents of elementary school students spend a lot of time and effort grading their children's homework. This burden is particularly problematic for working parents. Furthermore, the difficulty of providing prompt and accurate feedback can lead to a decline in children's learning effectiveness. Furthermore, parents face the problem of being unable to provide appropriate educational support to their children because it is difficult to maintain accuracy and consistency in grading. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that includes a means for capturing images of homework assignments and transmitting the captured images to a server, a means for converting the images to text data on the server, a means for analyzing and grading the questions using a generative AI model based on the converted text data, a means for transmitting the grading results from the server to a terminal, and a means for displaying the grading results on the terminal. This reduces the burden on parents and enables the provision of prompt and accurate feedback. Specifically, optical character recognition (OCR) technology is used to convert the text data, and the generative AI model identifies the type of question and performs analysis according to the type, achieving highly accurate grading.

[0006] "Means for taking images of homework" refers to a function for taking images of homework written on paper or question papers using a device such as a smartphone or tablet.

[0007] The "means for transmitting the captured image to the server" is a communication function for uploading the homework image acquired by the terminal to the server via the Internet.

[0008] A "server" is a computer system that receives data sent from multiple terminals and processes and analyzes that data.

[0009] The "means for converting images into text data" is a function for converting image data received by the server into text information using optical character recognition (OCR) technology.

[0010] A "generative AI model" is an artificial intelligence model that uses machine learning and deep learning techniques to analyze questions from input data and determine whether the answer is correct.

[0011] The "means for analyzing questions and scoring" is a function that uses a generative AI model to analyze text data, determine whether the answers to individual questions are accurate, and calculate the scoring results.

[0012] The "means for transmitting the scoring results from the server to the terminal" is a communication function for returning the scoring results analyzed by the server to the terminal.

[0013] The "means for displaying the scoring results on the terminal" is a function for visually displaying the received scoring results to the user on the screen of the terminal.

[0014] Optical character recognition (OCR) technology is a technology that automatically reads letters and numbers in an image and converts them into electronic text. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system that automatically grades homework assignments by taking pictures of them and uploading them. The system aims to analyze the content of the homework assignments and grade them quickly and accurately, thereby reducing the burden on parents. The program processing of this system is explained below in natural language.

[0037] System Overview

[0038] The system includes the following elements:

[0039] 1. How to take pictures of your homework

[0040] 2. Means of sending the captured image to the server

[0041] 3. A method for converting images into text data on the server

[0042] 4. A method for analyzing questions and scoring them using a generative AI model based on the converted text data

[0043] 5. Means for sending the scoring results from the server to the device

[0044] 6. Means for displaying the scoring results on the device

[0045] Description of the Examples

[0046] 1. User operations

[0047] Users take a photo of their homework using a device such as a smartphone or tablet. Specifically, they launch a camera app or a dedicated app and take a clear photo of the homework paper. Once the photo is taken, the user taps the "Upload Photo" button.

[0048] 2. Terminal Processing

[0049] The device temporarily stores the captured image of the homework, adds it to an upload list, and transmits the image to a server over the Internet using an appropriate communication protocol, such as an HTTP request, to transmit the image.

[0050] 3. Server Processing

[0051] The server receives image data sent from the device. The received image data is first converted into text data using optical character recognition (OCR) technology. For example, if the math problem "3 + 2 = ?" is included, it is recognized as text data.

[0052] 4. Analysis and scoring using generative AI models

[0053] The server passes the text data converted by OCR to the generative AI model. The generative AI model determines the type of question (math, Japanese, science, etc.) and performs an analysis according to the type. In this example, the generative AI model analyzes the answer as "3 + 2 = 5" and determines that this is correct. If there is also a question such as "4 - 1 = ?", this is analyzed in the same way, and the AI ​​model determines that the correct answer is "4 - 1 = 3."

[0054] 5. Sending the score results

[0055] The server generates the scoring results, which are then formatted into an easy-to-read format such as JSON. The server then sends a response containing the scoring results to the device.

[0056] 6. Displaying the terminal

[0057] The terminal analyzes the scoring results received from the server and displays them appropriately on the user interface. For example, it displays "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)." In this way, the user can receive prompt and accurate feedback.

[0058] Specific operation example

[0059] For example, if a third-grade elementary school student has two math homework problems, "3 + 2 = ?" and "4 - 1 = ?", the user can take a photo of them and receive automatic grading results through the process described above.

[0060] Therefore, this system significantly reduces the burden on users (parents) and enables quick and accurate homework grading.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The user takes a picture of the homework.

[0064] The user takes a photo of the homework paper using the camera on their device (smartphone or tablet) using the camera function built into the application or the camera that is standard on the device.

[0065] Step 2:

[0066] The device temporarily saves the captured image.

[0067] The device temporarily stores images taken by the user and then adds them to the upload list.

[0068] Step 3:

[0069] A user uploads an image to the server.

[0070] The user taps the "upload photo" button displayed on the terminal application to send the captured image to the server.

[0071] Step 4:

[0072] The terminal transmits the image data to the server.

[0073] The device sends the captured image of the homework to a server over the Internet using an appropriate communication protocol, such as an HTTP request.

[0074] Step 5:

[0075] The server receives the image data.

[0076] The server receives the image data sent from the device, using the SSL / TLS protocol to ensure communication security.

[0077] Step 6:

[0078] The server converts the image data into text data.

[0079] The server converts the received image data into text data using OCR (Optical Character Recognition) technology. For example, an image of "3 + 2 = ?" is converted into the string "3 + 2 = ?".

[0080] Step 7:

[0081] The server passes the text data to a generative AI model for analysis.

[0082] The server passes the OCR-converted text data to the generative AI model, which determines the type of question and analyzes it using the appropriate algorithm.

[0083] Step 8:

[0084] A generative AI model analyzes the answers to the questions and scores them.

[0085] The generative AI model checks the accuracy of answers based on the input text data and scores them. For example, it recognizes "3 + 2 = 5" as the correct answer and calculates the score.

[0086] Step 9:

[0087] The server formats the scores.

[0088] The server formats the scoring results obtained from the generative AI model into an easy-to-read format such as JSON, for example, "Question 1: Correct (score 5 / 5)."

[0089] Step 10:

[0090] The server sends the scoring results to the terminal.

[0091] The server sends the formatted score results to the terminal, and the terminal checks whether it is ready to receive them.

[0092] Step 11:

[0093] The terminal receives the scoring results.

[0094] The device receives the scoring results sent from the server, including data in JSON format.

[0095] Step 12:

[0096] The terminal displays the scoring results on the user interface.

[0097] The device analyzes the received score results and displays them appropriately on the user interface, for example, in the format "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)."

[0098] These steps allow users to receive homework grading results quickly and accurately, significantly reducing the burden on parents.

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] In the past, homework had to be graded manually by parents, which was time-consuming and labor-intensive, placing a heavy burden on parents. Furthermore, manual grading can be inaccurate, making it difficult to accurately evaluate a child's learning progress. Therefore, there was a need for a system that could grade homework quickly and accurately, reducing the burden on parents.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes a means for converting a captured image into text data, a means for analyzing and grading questions using a generative AI model based on the converted text data, and a means for transmitting the grading results from the server to the terminal. This makes it possible to automatically analyze the questions and quickly and accurately grade them simply by capturing an image of the homework and transmitting it to the server.

[0104] "Terminal" refers to a mobile information terminal such as a smartphone or tablet operated by a user.

[0105] A "server" refers to a computer system on a network that receives and processes data sent from a terminal.

[0106] "Photography" refers to the act of capturing an image using a camera or other image capture device.

[0107] "Text data" refers to a data format that includes character information such as letters and numbers.

[0108] Optical character recognition (OCR) refers to the technology that reads characters from an image and converts them into text data.

[0109] A "generative AI model" refers to a model that has been trained using artificial intelligence to automatically perform a specific task.

[0110] "Analysis" refers to the process of examining data or information in detail to understand its meaning and patterns.

[0111] "Grading" refers to the act of evaluating the correctness of answers and assignments and assigning points.

[0112] "Response" refers to response data sent from the server to the terminal.

[0113] "JSON" is an abbreviation for JavaScript (registered trademark) Object Notation and refers to a lightweight data exchange format.

[0114] The present invention is a system that automatically grades homework assignments by taking pictures of them and uploading them. The system aims to analyze the content of the homework assignments and grade them quickly and accurately, thereby reducing the burden on parents. The program processing of this system is explained below.

[0115] Hardware and software used

[0116] Devices: Smartphones, tablets

[0117] This allows users to use the camera function to take a picture of their homework paper and add the image to the upload list.

[0118] Server: Cloud Server

[0119] The received data is managed and processed using cloud services such as AWS (registered trademark) and Google (registered trademark).

[0120] software:

[0121] Camera app: Software that allows users to take pictures of their homework.

[0122] OCR engine: Extracts text data from images using, for example, Google Cloud Vision API.

[0123] Generative AI models, such as OpenAI® GPT-3®, are used to analyze and score questions.

[0124] What the system does

[0125] 1. User takes a photo of their homework on their device:

[0126] The user launches the device's camera app and takes a clear photo of the homework paper. After taking the photo, the user taps the "Upload Photo" button to add the image to the upload list.

[0127] 2. Sending captured images to the server:

[0128] The terminal temporarily stores the captured image and transmits the image data to the server via an HTTP request.

[0129] 3. OCR processing on the server:

[0130] The server converts the received image data into text data using optical character recognition (OCR) technology. For example, if an image containing the question "3 + 2 = ?" is sent, OCR technology converts it into text data as "3 + 2 = ?"

[0131] 4. Analysis and scoring by generative AI model:

[0132] The server then passes the converted text data to a generative AI model for analysis and scoring. The generative AI model automatically determines the type of question (math, Japanese, science, etc.) and generates the appropriate answer. For example, the answer to the math question "3 + 2 = ?" is "5."

[0133] 5. Sending the score:

[0134] The server formats the analysis results from the generative AI model into an easy-to-read format such as JSON and sends it to the terminal.

[0135] 6. Viewing the score on your device:

[0136] The device analyzes the received score results and displays them on the user interface, for example, in the form of "Question 1: Correct (Score 5 / 5)" or "Question 2: Correct (Score 5 / 5)."

[0137] Prompt Sentence Examples

[0138] An example of a specific prompt is as follows:

[0139] "Please answer the following math problem: 3 + 2 = ?"

[0140] In this way, the system according to the present invention can quickly and accurately grade homework, significantly reducing the burden on parents.

[0141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0142] Step 1:

[0143] The user launches the camera app or a dedicated app on their device (smartphone or tablet) and takes a picture of the homework question paper. When taking the picture, they adjust the camera so that the entire question paper is clearly visible. Once the picture is taken, the user taps the "Upload Photo" button. This action adds the captured image to the upload list on the device.

[0144] Input: Homework question paper

[0145] Output: Image file of the photographed homework

[0146] Step 2:

[0147] The device temporarily stores the captured image of the homework and prepares it for transmission to a server via the Internet. The image data is sent to the server using an appropriate communication protocol, such as an HTTP request. The image is saved in a standard image format, such as JPEG.

[0148] Input: Captured image file

[0149] Output: HTTP request sent to the server

[0150] Step 3:

[0151] The server receives the image data sent from the device. After receiving it, the server converts the image into text data using optical character recognition (OCR) technology. For example, if the image contains the math problem "3 + 2 = ?", OCR processing is performed so that this problem is recognized as the text data "3 + 2 = ?". The OCR engine used is Google Cloud Vision API or similar.

[0152] Input: Received image data

[0153] Output: Converted text data

[0154] Step 4:

[0155] The server passes the OCR-converted text data to a generative AI model, which analyzes and scores the questions. The generative AI model determines the type of question (math, Japanese, science, etc.) and creates an appropriate prompt to generate an answer. For example, for the text "3 + 2 = ?", the AI ​​model generates the prompt "Please answer the following math problem: 3 + 2 = ?", and outputs the answer "5."

[0156] Input: Converted text data

[0157] Output: Scoring results

[0158] Step 5:

[0159] The server formats the generated scoring results into an easy-to-read format such as JSON and generates a response to the device. This response includes the correct answer and score for each question. For example, it generates JSON data such as "{'Question 1': 'Correct', 'Score': '5 / 5'}".

[0160] Input: Scoring results

[0161] Output: Formatted response data (JSON format)

[0162] Step 6:

[0163] The terminal analyzes the response of the score received from the server and displays it on the user interface. Specifically, it visually presents the score result to the user in the form of "Question 1: Correct (score 5 / 5)."

[0164] Input: Response data received from the server

[0165] Output: The grading results that are displayed to the user

[0166] This allows the system to provide fast and accurate automatic grading by simply allowing users to take a photo of their homework and submit it.

[0167] (Application example 1)

[0168] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0169] Conventional homework grading systems require parents or teachers to manually check homework, which is time-consuming and labor-intensive. Additionally, the use of smart devices creates operational hassles and the convenience of these devices is not fully utilized. The present invention aims to solve these issues by providing a system that can efficiently capture and review homework using smart glasses.

[0170] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0171] In this invention, the server includes a means for capturing an image of the homework, a means for transmitting the captured image to the server, a means for converting the image into text data in the server, a means for analyzing and grading the questions using a generative AI model based on the converted text data, a means for transmitting the grading results from the server to the terminal, a means for displaying the grading results on the terminal, and a means for capturing and checking the image of the homework using smart glasses, thereby enabling teachers and parents to quickly and accurately grade homework and check the results in real time.

[0172] "Means for capturing images of homework" refers to the device a user uses to capture images of their homework papers, such as a smartphone or a camera on smart glasses.

[0173] "Means for transmitting captured images to a server" refers to a function for transferring captured image data to a server via the Internet.

[0174] "Means for converting images into text data on a server" refers to a method for recognizing characters in an image on a server and extracting the character information as text data, and primarily uses optical character recognition (OCR) technology.

[0175] "Method of analyzing questions using a generative AI model based on converted text data and scoring" refers to a method in which text data converted by OCR is input into a generative AI model, which then analyzes the answers to the questions, determines whether they are correct, and scores them.

[0176] "Means for transmitting the scoring results from the server to the terminal" refers to a communication means for transmitting the scoring results on the server to the client device.

[0177] "Means for displaying the scoring results on the terminal" refers to a user interface for visually presenting the scoring results to the user on the client device.

[0178] "Means for taking and reviewing images of homework using smart glasses" refers to a function for taking a photo of a homework sheet using smart glasses and reviewing it on the screen.

[0179] The present invention is a system for automatically marking homework assignments by taking images of the assignments and uploading them to a server. This system mainly includes the following elements:

[0180] System Overview

[0181] 1. How to take a picture of your homework:

[0182] The smart glasses are used to take images of the homework. Specifically, the smart glasses' camera is used to take a clear image of the student's entire homework paper. This also includes a function to check and take a photo at hand if necessary.

[0183] 2. Means for sending captured images to the server:

[0184] The captured image is sent to a server using the smart glasses' internet connection function. Image data is transferred to the server using a communication protocol such as an HTTP request.

[0185] 3. On the server, convert the image to text data:

[0186] The server receives the captured image data and converts the character information in the image into text data using OCR technology (for example, Tesseract OCR).

[0187] 4. A method for analyzing questions and scoring them using a generative AI model based on the converted text data:

[0188] The text data converted by OCR is passed to a generative AI model (e.g., OpenAI GPT) to analyze and grade the homework questions. The generative AI model identifies the type of question (math, Japanese, science, etc.) and provides the appropriate answer.

[0189] 5. How to send the score results from the server to the device:

[0190] The server formats the analyzed and scored results and sends them to the client terminal in an easy-to-read format such as JSON.

[0191] 6. How to display the results on the device:

[0192] The client terminal (e.g., smart glasses or a smartphone) displays the received scoring results on a user interface, allowing the results to be viewed in real time.

[0193] Specific operation example

[0194] For example, if a third-grade student's math homework includes the problems "3 + 2 = ?" and "4 - 1 = ?", the teacher puts on the smart glasses and takes a photo of the homework paper. The image data is then automatically sent to a server, where the character information is converted into text data using OCR. When the text data "3 + 2 = ?" and "4 - 1 = ?" are input into the generative AI model, the AI ​​model will answer "5" and "3" respectively and determine whether they are correct.

[0195] Hardware and software used

[0196] Smart glasses (e.g., Google Glass (registered trademark))

[0197] Server (e.g. AWS)

[0198] OCR technology (e.g. Tesseract OCR)

[0199] Generative AI models (e.g., OpenAI GPT)

[0200] Prompt Sentence Examples

[0201] Homework character recognition results (math problems):

[0202] "Analyze the following questions and determine whether they are correct or incorrect: 3 + 2 = ?, 4 - 1 = ?."

[0203] In this way, the present invention allows teachers and parents to quickly and accurately grade homework and check the results in real time, significantly reducing the time and effort required for traditional manual grading and providing efficient educational support.

[0204] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0205] Step 1:

[0206] The user uses the smart glasses to take a clear photo of their homework paper. The inputs are the user's instruction to take a photo and the image of the homework paper captured by the camera. The output is the captured image data, which is temporarily stored locally. During this process, the smart glasses' camera function and the photo application are running.

[0207] Step 2:

[0208] The device (smart glasses) sends the captured image to a server via the Internet. The input is the image data generated in step 1. The output is the image data transferred to the server using an HTTP request. An appropriate communication protocol (e.g. HTTP) is used in this process.

[0209] Step 3:

[0210] The server receives the image data and uses optical character recognition (OCR) technology to recognize the text in the image and convert it into text data. The input is the image data transferred to the server. The output is the converted text data generated by OCR. Character recognition software such as Tesseract OCR is used in this process.

[0211] Step 4:

[0212] The server inputs the text data converted by OCR into a generative AI model to analyze and score the questions. The input is the text data generated by OCR. The output is the analysis results (answers and scoring information) generated by the generative AI model. In this process, a generative AI model (e.g., OpenAI GPT) is used to analyze the data based on prompts such as "Analyze the following questions and determine whether they are correct or incorrect: 3 + 2 = ?, 4 - 1 = ?."

[0213] Step 5:

[0214] The server formats the scoring results generated by the generative AI model and sends them to the terminal. The input is the analysis results of the generative AI model. The output is the scoring results formatted in an easy-to-read format such as JSON and sent to the client terminal. During this process, data is formatted and transferred using a communication protocol (e.g., HTTP).

[0215] Step 6:

[0216] The device (smart glasses or smartphone) displays the scoring results received from the server on a user interface. The input is the scoring result data sent from the server. The output is the scoring results displayed on the user interface, providing visual feedback to the user. This process uses UI components and display logic for display.

[0217] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0218] This invention is a system that automatically grades homework assignments by taking and uploading images, and also analyzes the user's emotions and provides feedback. The system aims to analyze the content of homework assignments to grade them quickly and accurately, reducing the burden on parents and recognizing the user's emotions to improve the quality of education.

[0219] System Overview

[0220] The system includes the following elements:

[0221] 1. How to take pictures of your homework

[0222] 2. Means of sending the captured image to the server

[0223] 3. A method for converting images into text data on the server

[0224] 4. A method for analyzing questions and scoring them using a generative AI model based on the converted text data

[0225] 5. Means for sending the scoring results from the server to the device

[0226] 6. Means for displaying the scoring results on the device

[0227] 7. Emotion engine that analyzes the user's facial expressions and recognizes their emotions

[0228] 8. Ability to record emotions recognized by the emotion engine as feedback

[0229] 9. A feature that provides interactive advice based on emotional data to improve learning progress and motivation

[0230] Description of the Examples

[0231] 1. User operations

[0232] Users take a photo of their homework using a device such as a smartphone or tablet. Specifically, they launch a camera app or a dedicated app and take a clear photo of the homework paper. Once the photo is taken, the user taps the "Upload Photo" button.

[0233] 2. Terminal Processing

[0234] The device temporarily stores the captured image, adds it to the upload list, and transmits it to a server over the Internet using an appropriate communication protocol, such as an HTTP request.

[0235] 3. Server Processing

[0236] The server receives image data sent from the device. The received image data is first converted into text data using optical character recognition (OCR) technology. For example, if the math problem "3 + 2 = ?" is included, it is recognized as text data.

[0237] 4. Analysis and scoring using generative AI models

[0238] The server passes the text data converted by OCR to the generative AI model. The generative AI model determines the type of question (math, Japanese, science, etc.), analyzes the question using the appropriate algorithm depending on the type, checks the accuracy of the answer, and scores it. For example, it recognizes "3 + 2 = 5" as the correct answer and calculates the score.

[0239] 5. Sending the score results

[0240] The server generates the scoring results, which are then formatted into an easy-to-read format such as JSON. The server then sends a response containing the scoring results to the device.

[0241] 6. Displaying the terminal

[0242] The device analyzes the score results received from the server and displays them appropriately on the user interface, for example, in the format "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)."

[0243] 7. Analysis by Emotion Engine

[0244] The device captures the user's facial expressions through a camera and sends them to the emotion engine. The emotion engine analyzes the user's facial expression data and recognizes their emotions. For example, if the user is smiling, it is recognized as "happy," and if they are frowning, it is recognized as "unhappy."

[0245] 8. Recording Emotional Data

[0246] The emotion engine records the user's recognized emotions as feedback data, such as their reaction to the results of their homework grading or their feelings about their learning progress.

[0247] 9. Interactive Advice

[0248] The emotion engine provides interactive advice to users based on the recorded emotion data. For example, if a user expresses dissatisfaction with their homework results, it displays a motivational message such as, "If you try a little harder, you'll do better next time."

[0249] Specific operation example

[0250] For example, if a third-grade elementary school student has two math homework problems, "3 + 2 = ?" and "4 - 1 = ?", the user can take a photo of the problem and receive the automatic scoring results through the process described above. After the scoring results are returned, the device sends the user's facial expression to the emotion engine for analysis. If the system recognizes the user's facial expression as "happy," it will provide feedback such as "Good job!"

[0251] Therefore, this system not only enables users to receive homework grading results quickly and accurately, but also improves learning progress and motivation.

[0252] The processing flow will be explained below.

[0253] Step 1:

[0254] The user takes a picture of the homework.

[0255] The user takes a picture of the entire homework paper using the camera of the device (smartphone or tablet). For example, the user takes a clear picture of the homework page using the camera function built into the application.

[0256] Step 2:

[0257] The device temporarily saves the captured image.

[0258] The terminal stores the images taken by the user in temporary memory and then adds them to the upload list.

[0259] Step 3:

[0260] A user uploads an image to the server.

[0261] The user taps the "upload photo" button in the application on the device to send the captured image to the server.

[0262] Step 4:

[0263] The terminal transmits the image data to the server.

[0264] The device sends the captured image of the homework to a server over the Internet using an appropriate communication protocol, such as an HTTP request.

[0265] Step 5:

[0266] The server receives the image data.

[0267] The server receives the image data sent from the device. This communication uses the SSL / TLS protocol to ensure security.

[0268] Step 6:

[0269] The server converts the image data into text data.

[0270] The server converts the received image data into text data using OCR (optical character recognition) technology. For example, the handwritten mathematical formula "3 + 2 = ?" in the image is converted into the string "3 + 2 = ?".

[0271] Step 7:

[0272] The server passes the text data to a generative AI model for analysis.

[0273] The server then passes the OCR-converted text data to the generative AI model, which then analyzes the question content. The generative AI model then identifies the type of question and performs the analysis using the appropriate algorithm.

[0274] Step 8:

[0275] A generative AI model analyzes the answers to the questions and scores them.

[0276] The generative AI model analyzes the answers based on the input text data, determines whether they are correct, and calculates the score. For example, "3 + 2 = 5" is recognized as the correct answer, and the score is calculated.

[0277] Step 9:

[0278] The server formats the scores.

[0279] The server converts the scoring results obtained from the generative AI model into an easy-to-read format such as JSON, for example, "Question 1: Correct (score 5 / 5)."

[0280] Step 10:

[0281] The server sends the scoring results to the terminal.

[0282] The server sends the formatted score to the device, using an appropriate protocol so that the device can receive it.

[0283] Step 11:

[0284] The terminal receives the scoring results.

[0285] The terminal receives the scoring results sent from the server and analyzes the data in JSON format, etc.

[0286] Step 12:

[0287] The terminal displays the scoring results on a user interface.

[0288] The device will then display the received scoring results appropriately on the user interface, such as "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)."

[0289] Step 13:

[0290] The device captures the user's facial expressions.

[0291] The device captures the user's facial expressions in real time via a camera, capturing their facial expressions while they are checking their homework results.

[0292] Step 14:

[0293] The device sends the facial expression data to the emotion engine.

[0294] The device analyzes the captured facial expression data and sends it to the emotion engine. This data includes key facial features of the user (e.g., eye shape, mouth movement, etc.).

[0295] Step 15:

[0296] The emotion engine analyzes the user's emotions.

[0297] The emotion engine analyzes the user's emotions based on the facial expression data sent. For example, a smile is judged to be "happy," and a frown is judged to be "unhappy."

[0298] Step 16:

[0299] The emotion engine records the emotion data.

[0300] The emotion engine recognizes and records the user's emotions as feedback data, including their reactions to the results of their homework grading and their feelings about their learning progress.

[0301] Step 17:

[0302] An emotion engine provides interactive advice.

[0303] The emotion engine provides interactive advice to users based on the recorded emotion data. For example, if a user expresses dissatisfaction with the results of their homework, it will display an encouraging message such as "Try harder next time!"

[0304] These steps allow users to not only receive the results of their homework assignments quickly and accurately, but also to properly analyze their feelings about the results and obtain feedback to improve their motivation to study.

[0305] Example 2

[0306] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0307] Traditional homework grading and learning progress management relies heavily on manual work, placing a heavy burden on parents and teachers in particular. Additionally, it is difficult to properly grasp changes in children's emotions as they learn, and there is a lack of feedback to maintain and improve their motivation. The purpose of this invention is to solve these problems by analyzing users' emotions and providing interactive feedback in addition to automatically grading homework.

[0308] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting images of homework into text data, a means for analyzing and grading questions using a generative model based on the converted text data, and a means for transmitting the grading results to the user interface device. This makes it possible to automatically grade homework and analyze the user's emotions to provide feedback.

[0309] "Means for taking images of homework" refers to a device equipped with a camera or a dedicated application that allows a user to take images of their homework.

[0310] The term "means for transmitting a captured image to an information processing device" refers to a protocol or a communication interface for transmitting captured image data to an information processing device using a network.

[0311] "Means for converting an image into text data in an information processing device" refers to software or hardware for converting image data into text information using optical character recognition technology.

[0312] "Means for analyzing questions using a generative model based on converted text data and grading" refers to an algorithm and processing system that uses a generative AI model to analyze text data, determine the answer to each question, and grade the answers.

[0313] The term "means for transmitting the scoring results from the information processing device to the user interface device" refers to a communication means for transmitting the analysis and scoring results to the user interface device via a network.

[0314] The term "means for displaying the scoring results in the user interface device" refers to a display screen and GUI (Graphical User Interface) for displaying the scoring results on the user interface device.

[0315] "Means for capturing a user's facial expression and analyzing emotions" refers to a device and software for capturing a user's facial expression data using a camera and sending it to an emotion analysis engine for analysis.

[0316] "Means for recording analyzed emotional data" refers to a system for analyzing a user's emotional data and then recording it in a database or storage.

[0317] "Means for providing interactive advice based on recorded emotional data" refers to scripts or algorithms that utilize the recorded emotional data to provide appropriate feedback or advice to the user.

[0318] This invention is a system that automatically grades homework assignments by taking and uploading images, and also analyzes the user's emotions and provides feedback. This system aims to analyze the content of homework assignments to grade them quickly and accurately, reducing the burden on parents and teachers, and improving the quality of education by recognizing the user's emotions.

[0319] Hardware and software used

[0320] 1. Hardware:

[0321] Device: A user device such as a smartphone or tablet that allows the user to take and upload images of their homework.

[0322] Server: A central information processing device that can analyze and score submitted images and generate feedback.

[0323] Camera: A camera built into the device that can capture the user's facial expressions.

[0324] 2. Software:

[0325] Camera app or dedicated app: An application used on a device that provides the ability to take and upload images of homework.

[0326] Optical Character Recognition (OCR) technology, such as Google Cloud Vision API, which converts images of homework assignments into text data.

[0327] A generative AI model, such as OpenAI GPT-4®, that analyzes the converted text data and scores the questions.

[0328] Emotion engine: For example, Microsoft® Azure® Emotion API, which analyzes the user's facial expression data and recognizes their emotions.

[0329] Overview of program processing

[0330] User operations

[0331] Users open a camera app or a dedicated app on their smartphone or tablet and take a photo of their homework paper. Once they have taken the photo, they tap the "Upload Photo" button.

[0332] Terminal handling

[0333] The device temporarily stores the captured image and transmits it to a server via the Internet. For example, it generates an HTTP request and sends a POST request to the server's upload endpoint.

[0334] Server reception and image analysis

[0335] The server receives the image data sent from the device and converts it into text data using optical character recognition (OCR) technology. For example, if the data contains the math problem "3 + 2 = ?", it will recognize this as text data.

[0336] Analysis using generative AI models

[0337] The server passes the text data obtained by OCR to a generative AI model, which analyzes the questions and scores them. An example of a prompt is "Analyze the following text to determine the answer to each question and score it: '3 + 2 = ?'".

[0338] Formatting and sending the graded results

[0339] The server converts the scoring results returned by the generative AI model into JSON format and sends it to the device. For example, it converts it into the following format: {"Question 1": {"Answer": "5", "Score": 5}, "Question 2": {"Answer": "3", "Score": 5}}.

[0340] Display results on device and capture facial expressions

[0341] The device analyzes the JSON formatted scoring results received from the server and displays them on the user interface. For example, it displays "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)." It also captures the user's facial expressions with a camera and sends them to the emotion engine.

[0342] Emotion analysis and feedback using an emotion engine

[0343] The emotion engine analyzes the user's facial expression data and recognizes their emotions. For example, if the user is smiling, it is recognized as "happy," and if they are frowning, it is recognized as "unhappy." The analyzed emotion data is recorded, and based on the recorded data, it provides interactive advice such as "You did a great job!"

[0344] Specific operation example

[0345] For example, if a third-grade elementary school student has two math homework problems, "3 + 2 = ?" and "4 - 1 = ?", the user can take a photo of them and upload them to the system. The image is converted into text on the server, and an AI model recognizes "3 + 2 = 5" and "4 - 1 = 3" as the correct answers, awarding points for each. At the same time as the results are displayed on the device, the user's facial expressions are analyzed by an emotion engine, and feedback such as "You did a great job!" is provided.

[0346] This allows the system to not only provide fast and accurate homework grading, but also improve users' learning progress and motivation.

[0347] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0348] Step 1: User takes a picture of their homework

[0349] Users take pictures of their homework using a smartphone or tablet camera app or a dedicated app. The input is a physical copy of the homework, and the output is a digital image file stored on the device. For example, a user takes a picture of a third-grade math homework sheet.

[0350] Step 2: The device sends the image to the server

[0351] The device temporarily stores the captured image file and sends it to a server via the Internet. The input is the captured digital image file, and the output is the image data sent to the server. Specifically, the device generates an HTTP request and POSTs the image data to an endpoint.

[0352] Step 3: The server receives the image data

[0353] The server receives image data sent from the user's device and saves it in local storage. The input is the received image data, and the output is the saved image file. For example, the received "Homework_20231012.jpg" is saved.

[0354] Step 4: The server OCRs the image

[0355] The server converts the saved image file into text data using OCR technology. The input is the saved image file, and the output is the recognized text data. For example, the Google Cloud Vision API is used to extract the text "3 + 2 = ?" and "4 - 1 = ?".

[0356] Step 5: The server analyzes the text using the generated AI model

[0357] The server passes the text data obtained by OCR processing to a generative AI model, which analyzes the questions and scores them. The input is the text data and the analysis prompt, and the output is the scoring results. Specifically, the generative AI model (e.g., OpenAI GPT-4) analyzes the prompt, "Analyze the following text, determine the answer to each question, and score it: '3 + 2 = ?'."

[0358] Step 6: The server formats and sends the results

[0359] The server formats the scoring results returned by the generative AI model into JSON format and sends it to the device. The input is the scoring results from the generative AI model, and the output is formatted JSON data. For example, it is formatted as follows: {"Question 1": {"Answer": "5", "Score": 5}, "Question 2": {"Answer": "3", "Score": 5}}.

[0360] Step 7: Your device will display the results

[0361] The terminal parses the JSON data received from the server and displays it in the user interface in an appropriate format. The input is JSON data, and the output is the displayed scoring result. Specifically, "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)" are displayed on the screen.

[0362] Step 8: The device captures the user's facial expression

[0363] The device captures the user's facial expression with a camera while checking the scoring result. The input is the user's facial expression, and the output is the captured image data. For example, the device captures the user's face using a front camera.

[0364] Step 9: The server analyzes facial expressions using the emotion engine

[0365] The server sends the captured facial expression data to the emotion engine for analysis. The input is facial image data, and the output is the emotion analysis results. For example, Microsoft Azure Emotion API is used to recognize emotions such as "happy" or "unhappy."

[0366] Step 10: The server records the emotion data and sends the feedback

[0367] The server records the recognized emotion data and provides feedback to the user. The input is the emotion analysis result, and the output is a feedback message. For example, a message such as "Good job!" is generated and sent to the device.

[0368] The above is the flow of specific processing steps. At each step, we have explained in detail how input data is processed and calculated to generate output data.

[0369] (Application example 2)

[0370] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0371] Currently, in many families, parents are burdened with the task of checking and grading their children's homework. It is also often difficult for children to maintain their motivation to study through homework. Furthermore, there is a lack of feedback that is sensitive to children's emotions, making it difficult to provide effective learning support.

[0372] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0373] In this invention, the server includes means for taking images of homework, means for transmitting the taken images to the server, means for converting the images to text data in the server, means for analyzing and grading the questions using a generative AI model based on the converted text data, means for transmitting the grading results from the server to the terminal, means for displaying the grading results on the terminal, means for capturing the user's facial expression with a camera and recognizing their emotions, means for recording the recognized emotions as feedback, and means for presenting interactive advice based on the emotion data. This makes it possible to efficiently support children's learning through automatic homework grading and emotion analysis, reduce the burden on parents, and increase children's motivation to learn.

[0374] "Means for taking pictures of homework" refers to a method in which a user takes a picture of the content written on a homework paper using a camera on a device such as a smartphone or tablet.

[0375] The "means for transmitting the captured images to the server" refers to a communication protocol or software that uploads the captured image data of the homework to a remote server via the Internet.

[0376] The "means of converting images to text data" is the process of converting the text information in the homework images into digital text using optical character recognition (OCR) technology.

[0377] The "means of analyzing questions and scoring using a generative AI model" refers to an artificial intelligence model that analyzes text data generated by OCR and scores questions by comparing actual answers with theoretically correct answers.

[0378] "Means for transmitting scoring results from the server to the terminal" refers to a system for returning the scoring result data from the server to the user's device (such as a smartphone or tablet) once scoring is complete.

[0379] The "means for displaying the scoring results on the terminal" refers to software or a device for displaying the scoring results received from the server as a user interface.

[0380] "Means for capturing a user's facial expression via a camera and recognizing emotions" refers to algorithms or software for capturing a user's facial expression via a camera and analyzing emotions from the facial expression data.

[0381] The "means for recording the recognized emotions as feedback" is a system for recording the analyzed emotion data in a database or the like and using it as feedback to the user.

[0382] The "means for presenting interactive advice based on emotional data" is a function for providing advice and feedback to the user according to the actual situation based on the recorded emotional data.

[0383] This invention shows a specific embodiment of a system that automatically grades homework by taking and uploading images, and further analyzes the user's emotions and provides feedback.

[0384] The entire system mainly consists of a terminal such as a smartphone, a server, and an emotion analysis engine.

[0385] First, the user takes a photo of the homework paper using the device's camera. The device is equipped with appropriate image capture functions to help the user capture the entire homework. The captured image is temporarily stored on the device and then sent to a remote server over the Internet. This process uses communication protocols such as HTTP requests.

[0386] The server converts the received image data into text using optical character recognition (OCR) technology. This text data includes homework questions and answers. The server uses software such as Tesseract OCR to extract text information from the image.

[0387] Next, a generative AI model is used to analyze and score the extracted text data. OpenAI's GPT-4 is used as the generative AI model. This model first determines the type of question (e.g., math, Japanese, science, etc.), and then evaluates the correctness of the answer using an appropriate algorithm depending on the type, and scores the answer. For example, it recognizes that the correct answer to the question "3 + 2 = ?" is "5," and awards points if the user's answer is correct.

[0388] The server organizes the scoring results as structured data, such as JSON format, and sends them to the device. The device then analyzes the received scoring results and displays them on the user interface. Specifically, it displays something like "Question 1: Correct (score 5 / 5)."

[0389] Furthermore, the device captures the user's facial expression with a camera and sends the data to an emotion analysis engine. The emotion analysis engine uses the FER (Facial Expression Recognition) library to analyze the user's facial expression and recognize their emotion. In this process, if the user is smiling, it is determined to be "happy," and if they are frowning, it is determined to be "unhappy."

[0390] The analyzed emotional data is recorded in a database, and interactive advice is generated to improve the user's learning progress and motivation. For example, if a user expresses dissatisfaction with their homework results, a message such as "If you try a little harder, you'll get a better result next time" is provided.

[0391] (Examples of specific examples and prompts)

[0392] Specifically, for a third-grade elementary school student's math homework, if there are two problems, "3 + 2 = ?" and "4 - 1 = ?", the user takes a photo of the problem and receives the automatic scoring results through the process described above. After the scoring results are returned, the device sends the user's facial expression to the emotion engine for analysis. If the system recognizes the user's facial expression as "happy," it will provide feedback such as "Good job!"

[0393] An example prompt is:

[0394] "Please grade the following homework: 3 + 2 = ?"

[0395] This invention not only allows users to receive the results of homework grading quickly and accurately, but also improves learning progress and motivation.

[0396] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0397] Step 1:

[0398] The user takes a photo of their homework paper with the device's camera. The input is the homework paper, and the output is the captured image file. This image file is temporarily saved on the device. The specific operation of the device is to launch the camera app or a dedicated app, and the user presses the shutter button to capture the image.

[0399] Step 2:

[0400] The device sends the captured image to the server. The input is the captured image file, and the output is the image data sent to the server. The device uploads the image data via the Internet using an HTTP request. Specifically, it issues a POST request to upload the file to the specified server URL.

[0401] Step 3:

[0402] The server converts the image data it receives into text data using OCR technology. The input is the image data sent to the server, and the output is text data. The server uses Tesseract OCR to extract the character information in the image as text. Specifically, it reads the image file and performs character recognition to generate text data.

[0403] Step 4:

[0404] The server uses a generative AI model to analyze the text data and grade the homework questions. The input is text data extracted by OCR, and the output is the graded results. The server calls a generative AI model (for example, OpenAI's GPT-4), creates a prompt, and requests grading. Specifically, the text data is sent to the generative AI model as a prompt in the format "Please grade the following homework: [text data]". The generative AI model returns a text containing the score as the analysis result.

[0405] Step 5:

[0406] The server sends the scoring results to the device. The input is the scoring result text received from the generative AI model, and the output is the scoring result sent to the device. The server formats the scoring results as structured data such as JSON format and sends them to the device as an HTTP response.

[0407] Step 6:

[0408] The terminal displays the marking results it has received. The input is the JSON format marking result data received from the server, and the output is the marking result displayed on the user interface. The terminal parses the JSON data and displays the marking results in a format that is easy for the user to understand. Specifically, it displays the results in a format such as "Question 1: Correct (score 5 / 5)."

[0409] Step 7:

[0410] The device captures the user's facial expression with a camera and recognizes the emotion. The input is the user's facial expression image captured by the device's camera, and the output is the recognized emotion data. The device uses the FER library to analyze the facial expression image and identify the user's emotion. Specifically, the device passes the captured image to the FER library and obtains the emotion data.

[0411] Step 8:

[0412] The server records the recognized emotions as feedback data. The input is the emotion data obtained by emotion analysis, and the output is the emotion data recorded in the database. The server saves the emotion data in the database and manages it together with the user's learning history.

[0413] Step 9:

[0414] The device presents interactive advice based on emotional data. The input is the recorded emotional data and the scoring results, and the output is an advice message displayed on the user interface. The device analyzes the emotional data and the scoring results, and generates and displays appropriate advice and feedback. Specifically, it displays encouraging messages such as, "If you try a little harder, you'll get better results next time."

[0415] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0416] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0417] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0418] [Second embodiment]

[0419] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0420] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0421] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0422] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0423] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0425] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0426] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0427] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0428] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0429] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0430] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0431] This invention is a system that automatically grades homework assignments by taking pictures of them and uploading them. The system aims to analyze the content of the homework assignments and grade them quickly and accurately, thereby reducing the burden on parents. The program processing of this system is explained below in natural language.

[0432] System Overview

[0433] The system includes the following elements:

[0434] 1. How to take pictures of your homework

[0435] 2. Means of sending the captured image to the server

[0436] 3. A method for converting images into text data on the server

[0437] 4. A method for analyzing questions and scoring them using a generative AI model based on the converted text data

[0438] 5. Means for sending the scoring results from the server to the device

[0439] 6. Means for displaying the scoring results on the device

[0440] Description of the Examples

[0441] 1. User operations

[0442] Users take a photo of their homework using a device such as a smartphone or tablet. Specifically, they launch a camera app or a dedicated app and take a clear photo of the homework paper. Once the photo is taken, the user taps the "Upload Photo" button.

[0443] 2. Terminal Processing

[0444] The device temporarily stores the captured image of the homework, adds it to an upload list, and transmits the image to a server over the Internet using an appropriate communication protocol, such as an HTTP request, to transmit the image.

[0445] 3. Server Processing

[0446] The server receives image data sent from the device. The received image data is first converted into text data using optical character recognition (OCR) technology. For example, if the math problem "3 + 2 = ?" is included, it is recognized as text data.

[0447] 4. Analysis and scoring using generative AI models

[0448] The server passes the text data converted by OCR to the generative AI model. The generative AI model determines the type of question (math, Japanese, science, etc.) and performs an analysis according to the type. In this example, the generative AI model analyzes the answer as "3 + 2 = 5" and determines that this is correct. If there is also a question such as "4 - 1 = ?", this is analyzed in the same way, and the AI ​​model determines that the correct answer is "4 - 1 = 3."

[0449] 5. Sending the score results

[0450] The server generates the scoring results, which are then formatted into an easy-to-read format such as JSON. The server then sends a response containing the scoring results to the device.

[0451] 6. Displaying the terminal

[0452] The terminal analyzes the scoring results received from the server and displays them appropriately on the user interface. For example, it displays "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)." In this way, the user can receive prompt and accurate feedback.

[0453] Specific operation example

[0454] For example, if a third-grade elementary school student has two math homework problems, "3 + 2 = ?" and "4 - 1 = ?", the user can take a photo of them and receive automatic grading results through the process described above.

[0455] Therefore, this system significantly reduces the burden on users (parents) and enables quick and accurate homework grading.

[0456] The processing flow will be explained below.

[0457] Step 1:

[0458] The user takes a picture of the homework.

[0459] The user takes a photo of the homework paper using the camera on their device (smartphone or tablet) using the camera function built into the application or the camera that is standard on the device.

[0460] Step 2:

[0461] The device temporarily saves the captured image.

[0462] The device temporarily stores images taken by the user and then adds them to the upload list.

[0463] Step 3:

[0464] A user uploads an image to the server.

[0465] The user taps the "upload photo" button displayed on the terminal application to send the captured image to the server.

[0466] Step 4:

[0467] The terminal transmits the image data to the server.

[0468] The device sends the captured image of the homework to a server over the Internet using an appropriate communication protocol, such as an HTTP request.

[0469] Step 5:

[0470] The server receives the image data.

[0471] The server receives the image data sent from the device, using the SSL / TLS protocol to ensure communication security.

[0472] Step 6:

[0473] The server converts the image data into text data.

[0474] The server converts the received image data into text data using OCR (Optical Character Recognition) technology. For example, an image of "3 + 2 = ?" is converted into the string "3 + 2 = ?".

[0475] Step 7:

[0476] The server passes the text data to a generative AI model for analysis.

[0477] The server passes the OCR-converted text data to the generative AI model, which determines the type of question and analyzes it using the appropriate algorithm.

[0478] Step 8:

[0479] A generative AI model analyzes the answers to the questions and scores them.

[0480] The generative AI model checks the accuracy of answers based on the input text data and scores them. For example, it recognizes "3 + 2 = 5" as the correct answer and calculates the score.

[0481] Step 9:

[0482] The server formats the scores.

[0483] The server formats the scoring results obtained from the generative AI model into an easy-to-read format such as JSON, for example, "Question 1: Correct (score 5 / 5)."

[0484] Step 10:

[0485] The server sends the scoring results to the terminal.

[0486] The server sends the formatted score results to the terminal, and the terminal checks whether it is ready to receive them.

[0487] Step 11:

[0488] The terminal receives the scoring results.

[0489] The device receives the scoring results sent from the server, including data in JSON format.

[0490] Step 12:

[0491] The terminal displays the scoring results on the user interface.

[0492] The device analyzes the received score results and displays them appropriately on the user interface, for example, in the format "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)."

[0493] These steps allow users to receive homework grading results quickly and accurately, significantly reducing the burden on parents.

[0494] Example 1

[0495] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0496] In the past, homework had to be graded manually by parents, which was time-consuming and labor-intensive, placing a heavy burden on parents. Furthermore, manual grading can be inaccurate, making it difficult to accurately evaluate a child's learning progress. Therefore, there was a need for a system that could grade homework quickly and accurately, reducing the burden on parents.

[0497] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0498] In this invention, the server includes a means for converting a captured image into text data, a means for analyzing and grading questions using a generative AI model based on the converted text data, and a means for transmitting the grading results from the server to the terminal. This makes it possible to automatically analyze the questions and quickly and accurately grade them simply by capturing an image of the homework and transmitting it to the server.

[0499] "Terminal" refers to a mobile information terminal such as a smartphone or tablet operated by a user.

[0500] A "server" refers to a computer system on a network that receives and processes data sent from a terminal.

[0501] "Photography" refers to the act of capturing an image using a camera or other image capture device.

[0502] "Text data" refers to a data format that includes character information such as letters and numbers.

[0503] Optical character recognition (OCR) refers to the technology that reads characters from an image and converts them into text data.

[0504] A "generative AI model" refers to a model that has been trained using artificial intelligence to automatically perform a specific task.

[0505] "Analysis" refers to the process of examining data or information in detail to understand its meaning and patterns.

[0506] "Grading" refers to the act of evaluating the correctness of answers and assignments and assigning points.

[0507] "Response" refers to response data sent from the server to the terminal.

[0508] "JSON" stands for JavaScript Object Notation and refers to a lightweight data exchange format.

[0509] The present invention is a system that automatically grades homework assignments by taking pictures of them and uploading them. The system aims to analyze the content of the homework assignments and grade them quickly and accurately, thereby reducing the burden on parents. The program processing of this system is explained below.

[0510] Hardware and software used

[0511] Devices: Smartphones, tablets

[0512] This allows users to use the camera function to take a picture of their homework paper and add the image to the upload list.

[0513] Server: Cloud Server

[0514] The received data is managed and processed using cloud services such as AWS and Google Cloud.

[0515] software:

[0516] Camera app: Software that allows users to take pictures of their homework.

[0517] OCR engine: Extracts text data from images using, for example, Google Cloud Vision API.

[0518] Generative AI models, such as OpenAI GPT-3, are used to analyze and score questions.

[0519] What the system does

[0520] 1. User takes a photo of their homework on their device:

[0521] The user launches the device's camera app and takes a clear photo of the homework paper. After taking the photo, the user taps the "Upload Photo" button to add the image to the upload list.

[0522] 2. Sending captured images to the server:

[0523] The terminal temporarily stores the captured image and transmits the image data to the server via an HTTP request.

[0524] 3. OCR processing on the server:

[0525] The server converts the received image data into text data using optical character recognition (OCR) technology. For example, if an image containing the question "3 + 2 = ?" is sent, OCR technology converts it into text data as "3 + 2 = ?"

[0526] 4. Analysis and scoring by generative AI model:

[0527] The server then passes the converted text data to a generative AI model for analysis and scoring. The generative AI model automatically determines the type of question (math, Japanese, science, etc.) and generates the appropriate answer. For example, the answer to the math question "3 + 2 = ?" is "5."

[0528] 5. Sending the score:

[0529] The server formats the analysis results from the generative AI model into an easy-to-read format such as JSON and sends it to the terminal.

[0530] 6. Viewing the score on your device:

[0531] The device analyzes the received score results and displays them on the user interface, for example, in the form of "Question 1: Correct (Score 5 / 5)" or "Question 2: Correct (Score 5 / 5)."

[0532] Prompt Sentence Examples

[0533] An example of a specific prompt is as follows:

[0534] "Please answer the following math problem: 3 + 2 = ?"

[0535] In this way, the system according to the present invention can quickly and accurately grade homework, significantly reducing the burden on parents.

[0536] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0537] Step 1:

[0538] The user launches the camera app or a dedicated app on their device (smartphone or tablet) and takes a picture of the homework question paper. When taking the picture, they adjust the camera so that the entire question paper is clearly visible. Once the picture is taken, the user taps the "Upload Photo" button. This action adds the captured image to the upload list on the device.

[0539] Input: Homework question paper

[0540] Output: Image file of the photographed homework

[0541] Step 2:

[0542] The device temporarily stores the captured image of the homework and prepares it for transmission to a server via the Internet. The image data is sent to the server using an appropriate communication protocol, such as an HTTP request. The image is saved in a standard image format, such as JPEG.

[0543] Input: Captured image file

[0544] Output: HTTP request sent to the server

[0545] Step 3:

[0546] The server receives the image data sent from the device. After receiving it, the server converts the image into text data using optical character recognition (OCR) technology. For example, if the image contains the math problem "3 + 2 = ?", OCR processing is performed so that this problem is recognized as the text data "3 + 2 = ?". The OCR engine used is Google Cloud Vision API or similar.

[0547] Input: Received image data

[0548] Output: Converted text data

[0549] Step 4:

[0550] The server passes the OCR-converted text data to a generative AI model, which analyzes and scores the questions. The generative AI model determines the type of question (math, Japanese, science, etc.) and creates an appropriate prompt to generate an answer. For example, for the text "3 + 2 = ?", the AI ​​model generates the prompt "Please answer the following math problem: 3 + 2 = ?", and outputs the answer "5."

[0551] Input: Converted text data

[0552] Output: Scoring results

[0553] Step 5:

[0554] The server formats the generated scoring results into an easy-to-read format such as JSON and generates a response to the device. This response includes the correct answer and score for each question. For example, it generates JSON data such as "{'Question 1': 'Correct', 'Score': '5 / 5'}".

[0555] Input: Scoring results

[0556] Output: Formatted response data (JSON format)

[0557] Step 6:

[0558] The terminal analyzes the response of the score received from the server and displays it on the user interface. Specifically, it visually presents the score result to the user in the form of "Question 1: Correct (score 5 / 5)."

[0559] Input: Response data received from the server

[0560] Output: The grading results that are displayed to the user

[0561] This allows the system to provide fast and accurate automatic grading by simply allowing users to take a photo of their homework and submit it.

[0562] (Application example 1)

[0563] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0564] Conventional homework grading systems require parents or teachers to manually check homework, which is time-consuming and labor-intensive. Additionally, the use of smart devices creates operational hassles and the convenience of these devices is not fully utilized. The present invention aims to solve these issues by providing a system that can efficiently capture and review homework using smart glasses.

[0565] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0566] In this invention, the server includes a means for capturing an image of the homework, a means for transmitting the captured image to the server, a means for converting the image into text data in the server, a means for analyzing and grading the questions using a generative AI model based on the converted text data, a means for transmitting the grading results from the server to the terminal, a means for displaying the grading results on the terminal, and a means for capturing and checking the image of the homework using smart glasses, thereby enabling teachers and parents to quickly and accurately grade homework and check the results in real time.

[0567] "Means for capturing images of homework" refers to the device a user uses to capture images of their homework papers, such as a smartphone or a camera on smart glasses.

[0568] "Means for transmitting captured images to a server" refers to a function for transferring captured image data to a server via the Internet.

[0569] "Means for converting images into text data on a server" refers to a method for recognizing characters in an image on a server and extracting the character information as text data, and primarily uses optical character recognition (OCR) technology.

[0570] "Method of analyzing questions using a generative AI model based on converted text data and scoring" refers to a method in which text data converted by OCR is input into a generative AI model, which then analyzes the answers to the questions, determines whether they are correct, and scores them.

[0571] "Means for transmitting the scoring results from the server to the terminal" refers to a communication means for transmitting the scoring results on the server to the client device.

[0572] "Means for displaying the scoring results on the terminal" refers to a user interface for visually presenting the scoring results to the user on the client device.

[0573] "Means for taking and reviewing images of homework using smart glasses" refers to a function for taking a photo of a homework sheet using smart glasses and reviewing it on the screen.

[0574] The present invention is a system for automatically marking homework assignments by taking images of the assignments and uploading them to a server. This system mainly includes the following elements:

[0575] System Overview

[0576] 1. How to take a picture of your homework:

[0577] The smart glasses are used to take images of the homework. Specifically, the smart glasses' camera is used to take a clear image of the student's entire homework paper. This also includes a function to check and take a photo at hand if necessary.

[0578] 2. Means for sending captured images to the server:

[0579] The captured image is sent to a server using the smart glasses' internet connection function. Image data is transferred to the server using a communication protocol such as an HTTP request.

[0580] 3. On the server, convert the image to text data:

[0581] The server receives the captured image data and converts the character information in the image into text data using OCR technology (for example, Tesseract OCR).

[0582] 4. A method for analyzing questions and scoring them using a generative AI model based on the converted text data:

[0583] The text data converted by OCR is passed to a generative AI model (e.g., OpenAI GPT) to analyze and grade the homework questions. The generative AI model identifies the type of question (math, Japanese, science, etc.) and provides the appropriate answer.

[0584] 5. How to send the score results from the server to the device:

[0585] The server formats the analyzed and scored results and sends them to the client terminal in an easy-to-read format such as JSON.

[0586] 6. How to display the results on the device:

[0587] The client terminal (e.g., smart glasses or a smartphone) displays the received scoring results on a user interface, allowing the results to be viewed in real time.

[0588] Specific operation example

[0589] For example, if a third-grade student's math homework includes the problems "3 + 2 = ?" and "4 - 1 = ?", the teacher puts on the smart glasses and takes a photo of the homework paper. The image data is then automatically sent to a server, where the character information is converted into text data using OCR. When the text data "3 + 2 = ?" and "4 - 1 = ?" are input into the generative AI model, the AI ​​model will answer "5" and "3" respectively and determine whether they are correct.

[0590] Hardware and software used

[0591] Smart glasses (e.g. Google Glass)

[0592] Server (e.g. AWS)

[0593] OCR technology (e.g. Tesseract OCR)

[0594] Generative AI models (e.g., OpenAI GPT)

[0595] Prompt Sentence Examples

[0596] Homework character recognition results (math problems):

[0597] "Analyze the following questions and determine whether they are correct or incorrect: 3 + 2 = ?, 4 - 1 = ?."

[0598] In this way, the present invention allows teachers and parents to quickly and accurately grade homework and check the results in real time, significantly reducing the time and effort required for traditional manual grading and providing efficient educational support.

[0599] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0600] Step 1:

[0601] The user uses the smart glasses to take a clear photo of their homework paper. The inputs are the user's instruction to take a photo and the image of the homework paper captured by the camera. The output is the captured image data, which is temporarily stored locally. During this process, the smart glasses' camera function and the photo application are running.

[0602] Step 2:

[0603] The device (smart glasses) sends the captured image to a server via the Internet. The input is the image data generated in step 1. The output is the image data transferred to the server using an HTTP request. An appropriate communication protocol (e.g. HTTP) is used in this process.

[0604] Step 3:

[0605] The server receives the image data and uses optical character recognition (OCR) technology to recognize the text in the image and convert it into text data. The input is the image data transferred to the server. The output is the converted text data generated by OCR. Character recognition software such as Tesseract OCR is used in this process.

[0606] Step 4:

[0607] The server inputs the text data converted by OCR into a generative AI model to analyze and score the questions. The input is the text data generated by OCR. The output is the analysis results (answers and scoring information) generated by the generative AI model. In this process, a generative AI model (e.g., OpenAI GPT) is used to analyze the data based on prompts such as "Analyze the following questions and determine whether they are correct or incorrect: 3 + 2 = ?, 4 - 1 = ?."

[0608] Step 5:

[0609] The server formats the scoring results generated by the generative AI model and sends them to the terminal. The input is the analysis results of the generative AI model. The output is the scoring results formatted in an easy-to-read format such as JSON and sent to the client terminal. During this process, data is formatted and transferred using a communication protocol (e.g., HTTP).

[0610] Step 6:

[0611] The device (smart glasses or smartphone) displays the scoring results received from the server on a user interface. The input is the scoring result data sent from the server. The output is the scoring results displayed on the user interface, providing visual feedback to the user. This process uses UI components and display logic for display.

[0612] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0613] This invention is a system that automatically grades homework assignments by taking and uploading images, and also analyzes the user's emotions and provides feedback. The system aims to analyze the content of homework assignments to grade them quickly and accurately, reducing the burden on parents and recognizing the user's emotions to improve the quality of education.

[0614] System Overview

[0615] The system includes the following elements:

[0616] 1. How to take pictures of your homework

[0617] 2. Means of sending the captured image to the server

[0618] 3. A method for converting images into text data on the server

[0619] 4. A method for analyzing questions and scoring them using a generative AI model based on the converted text data

[0620] 5. Means for sending the scoring results from the server to the device

[0621] 6. Means for displaying the scoring results on the device

[0622] 7. Emotion engine that analyzes the user's facial expressions and recognizes their emotions

[0623] 8. Ability to record emotions recognized by the emotion engine as feedback

[0624] 9. A feature that provides interactive advice based on emotional data to improve learning progress and motivation

[0625] Description of the Examples

[0626] 1. User operations

[0627] Users take a photo of their homework using a device such as a smartphone or tablet. Specifically, they launch a camera app or a dedicated app and take a clear photo of the homework paper. Once the photo is taken, the user taps the "Upload Photo" button.

[0628] 2. Terminal Processing

[0629] The device temporarily stores the captured image, adds it to the upload list, and transmits it to a server over the Internet using an appropriate communication protocol, such as an HTTP request.

[0630] 3. Server Processing

[0631] The server receives image data sent from the device. The received image data is first converted into text data using optical character recognition (OCR) technology. For example, if the math problem "3 + 2 = ?" is included, it is recognized as text data.

[0632] 4. Analysis and scoring using generative AI models

[0633] The server passes the text data converted by OCR to the generative AI model. The generative AI model determines the type of question (math, Japanese, science, etc.), analyzes the question using the appropriate algorithm depending on the type, checks the accuracy of the answer, and scores it. For example, it recognizes "3 + 2 = 5" as the correct answer and calculates the score.

[0634] 5. Sending the score results

[0635] The server generates the scoring results, which are then formatted into an easy-to-read format such as JSON. The server then sends a response containing the scoring results to the device.

[0636] 6. Displaying the terminal

[0637] The device analyzes the score results received from the server and displays them appropriately on the user interface, for example, in the format "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)."

[0638] 7. Analysis by Emotion Engine

[0639] The device captures the user's facial expressions through a camera and sends them to the emotion engine. The emotion engine analyzes the user's facial expression data and recognizes their emotions. For example, if the user is smiling, it is recognized as "happy," and if they are frowning, it is recognized as "unhappy."

[0640] 8. Recording Emotional Data

[0641] The emotion engine records the user's recognized emotions as feedback data, such as their reaction to the results of their homework grading or their feelings about their learning progress.

[0642] 9. Interactive Advice

[0643] The emotion engine provides interactive advice to users based on the recorded emotion data. For example, if a user expresses dissatisfaction with their homework results, it displays a motivational message such as, "If you try a little harder, you'll do better next time."

[0644] Specific operation example

[0645] For example, if a third-grade elementary school student has two math homework problems, "3 + 2 = ?" and "4 - 1 = ?", the user can take a photo of the problem and receive the automatic scoring results through the process described above. After the scoring results are returned, the device sends the user's facial expression to the emotion engine for analysis. If the system recognizes the user's facial expression as "happy," it will provide feedback such as "Good job!"

[0646] Therefore, this system not only enables users to receive homework grading results quickly and accurately, but also improves learning progress and motivation.

[0647] The processing flow will be explained below.

[0648] Step 1:

[0649] The user takes a picture of the homework.

[0650] The user takes a picture of the entire homework paper using the camera of the device (smartphone or tablet). For example, the user takes a clear picture of the homework page using the camera function built into the application.

[0651] Step 2:

[0652] The device temporarily saves the captured image.

[0653] The terminal stores the images taken by the user in temporary memory and then adds them to the upload list.

[0654] Step 3:

[0655] A user uploads an image to the server.

[0656] The user taps the "upload photo" button in the application on the device to send the captured image to the server.

[0657] Step 4:

[0658] The terminal transmits the image data to the server.

[0659] The device sends the captured image of the homework to a server over the Internet using an appropriate communication protocol, such as an HTTP request.

[0660] Step 5:

[0661] The server receives the image data.

[0662] The server receives the image data sent from the device. This communication uses the SSL / TLS protocol to ensure security.

[0663] Step 6:

[0664] The server converts the image data into text data.

[0665] The server converts the received image data into text data using OCR (optical character recognition) technology. For example, the handwritten mathematical formula "3 + 2 = ?" in the image is converted into the string "3 + 2 = ?".

[0666] Step 7:

[0667] The server passes the text data to a generative AI model for analysis.

[0668] The server then passes the OCR-converted text data to the generative AI model, which then analyzes the question content. The generative AI model then identifies the type of question and performs the analysis using the appropriate algorithm.

[0669] Step 8:

[0670] A generative AI model analyzes the answers to the questions and scores them.

[0671] The generative AI model analyzes the answers based on the input text data, determines whether they are correct, and calculates the score. For example, "3 + 2 = 5" is recognized as the correct answer, and the score is calculated.

[0672] Step 9:

[0673] The server formats the scores.

[0674] The server converts the scoring results obtained from the generative AI model into an easy-to-read format such as JSON, for example, "Question 1: Correct (score 5 / 5)."

[0675] Step 10:

[0676] The server sends the scoring results to the terminal.

[0677] The server sends the formatted score to the device, using an appropriate protocol so that the device can receive it.

[0678] Step 11:

[0679] The terminal receives the scoring results.

[0680] The terminal receives the scoring results sent from the server and analyzes the data in JSON format, etc.

[0681] Step 12:

[0682] The terminal displays the scoring results on a user interface.

[0683] The device will then display the received scoring results appropriately on the user interface, such as "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)."

[0684] Step 13:

[0685] The device captures the user's facial expressions.

[0686] The device captures the user's facial expressions in real time via a camera, capturing their facial expressions while they are checking their homework results.

[0687] Step 14:

[0688] The device sends the facial expression data to the emotion engine.

[0689] The device analyzes the captured facial expression data and sends it to the emotion engine. This data includes key facial features of the user (e.g., eye shape, mouth movement, etc.).

[0690] Step 15:

[0691] The emotion engine analyzes the user's emotions.

[0692] The emotion engine analyzes the user's emotions based on the facial expression data sent. For example, a smile is judged to be "happy," and a frown is judged to be "unhappy."

[0693] Step 16:

[0694] The emotion engine records the emotion data.

[0695] The emotion engine recognizes and records the user's emotions as feedback data, including their reactions to the results of their homework grading and their feelings about their learning progress.

[0696] Step 17:

[0697] An emotion engine provides interactive advice.

[0698] The emotion engine provides interactive advice to users based on the recorded emotion data. For example, if a user expresses dissatisfaction with the results of their homework, it will display an encouraging message such as "Try harder next time!"

[0699] These steps allow users to not only receive the results of their homework assignments quickly and accurately, but also to properly analyze their feelings about the results and obtain feedback to improve their motivation to study.

[0700] Example 2

[0701] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0702] Traditional homework grading and learning progress management relies heavily on manual work, placing a heavy burden on parents and teachers in particular. Additionally, it is difficult to properly grasp changes in children's emotions as they learn, and there is a lack of feedback to maintain and improve their motivation. The purpose of this invention is to solve these problems by analyzing users' emotions and providing interactive feedback in addition to automatically grading homework.

[0703] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting images of homework into text data, a means for analyzing and grading questions using a generative model based on the converted text data, and a means for transmitting the grading results to the user interface device. This makes it possible to automatically grade homework and analyze the user's emotions to provide feedback.

[0704] "Means for taking images of homework" refers to a device equipped with a camera or a dedicated application that allows a user to take images of their homework.

[0705] The term "means for transmitting a captured image to an information processing device" refers to a protocol or a communication interface for transmitting captured image data to an information processing device using a network.

[0706] "Means for converting an image into text data in an information processing device" refers to software or hardware for converting image data into text information using optical character recognition technology.

[0707] "Means for analyzing questions using a generative model based on converted text data and grading" refers to an algorithm and processing system that uses a generative AI model to analyze text data, determine the answer to each question, and grade the answers.

[0708] The term "means for transmitting the scoring results from the information processing device to the user interface device" refers to a communication means for transmitting the analysis and scoring results to the user interface device via a network.

[0709] The term "means for displaying the scoring results in the user interface device" refers to a display screen and GUI (Graphical User Interface) for displaying the scoring results on the user interface device.

[0710] "Means for capturing a user's facial expression and analyzing emotions" refers to a device and software for capturing a user's facial expression data using a camera and sending it to an emotion analysis engine for analysis.

[0711] "Means for recording analyzed emotional data" refers to a system for analyzing a user's emotional data and then recording it in a database or storage.

[0712] "Means for providing interactive advice based on recorded emotional data" refers to scripts or algorithms that utilize the recorded emotional data to provide appropriate feedback or advice to the user.

[0713] This invention is a system that automatically grades homework assignments by taking and uploading images, and also analyzes the user's emotions and provides feedback. This system aims to analyze the content of homework assignments to grade them quickly and accurately, reducing the burden on parents and teachers, and improving the quality of education by recognizing the user's emotions.

[0714] Hardware and software used

[0715] 1. Hardware:

[0716] Device: A user device such as a smartphone or tablet that allows the user to take and upload images of their homework.

[0717] Server: A central information processing device that can analyze and score submitted images and generate feedback.

[0718] Camera: A camera built into the device that can capture the user's facial expressions.

[0719] 2. Software:

[0720] Camera app or dedicated app: An application used on a device that provides the ability to take and upload images of homework.

[0721] Optical Character Recognition (OCR) technology, such as Google Cloud Vision API, which converts images of homework assignments into text data.

[0722] A generative AI model, such as OpenAI GPT-4, that analyzes the converted text data and scores the questions.

[0723] Emotion engine: For example, Microsoft Azure Emotion API, which analyzes the user's facial expression data and recognizes their emotions.

[0724] Overview of program processing

[0725] User operations

[0726] Users open a camera app or a dedicated app on their smartphone or tablet and take a photo of their homework paper. Once they have taken the photo, they tap the "Upload Photo" button.

[0727] Terminal handling

[0728] The device temporarily stores the captured image and transmits it to a server via the Internet. For example, it generates an HTTP request and sends a POST request to the server's upload endpoint.

[0729] Server reception and image analysis

[0730] The server receives the image data sent from the device and converts it into text data using optical character recognition (OCR) technology. For example, if the data contains the math problem "3 + 2 = ?", it will recognize this as text data.

[0731] Analysis using generative AI models

[0732] The server passes the text data obtained by OCR to a generative AI model, which analyzes the questions and scores them. An example of a prompt is "Analyze the following text to determine the answer to each question and score it: '3 + 2 = ?'".

[0733] Formatting and sending the graded results

[0734] The server converts the scoring results returned by the generative AI model into JSON format and sends it to the device. For example, it converts it into the following format: {"Question 1": {"Answer": "5", "Score": 5}, "Question 2": {"Answer": "3", "Score": 5}}.

[0735] Display results on device and capture facial expressions

[0736] The device analyzes the JSON formatted scoring results received from the server and displays them on the user interface. For example, it displays "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)." It also captures the user's facial expressions with a camera and sends them to the emotion engine.

[0737] Emotion analysis and feedback using an emotion engine

[0738] The emotion engine analyzes the user's facial expression data and recognizes their emotions. For example, if the user is smiling, it is recognized as "happy," and if they are frowning, it is recognized as "unhappy." The analyzed emotion data is recorded, and based on the recorded data, it provides interactive advice such as "You did a great job!"

[0739] Specific operation example

[0740] For example, if a third-grade elementary school student has two math homework problems, "3 + 2 = ?" and "4 - 1 = ?", the user can take a photo of them and upload them to the system. The image is converted into text on the server, and an AI model recognizes "3 + 2 = 5" and "4 - 1 = 3" as the correct answers, awarding points for each. At the same time as the results are displayed on the device, the user's facial expressions are analyzed by an emotion engine, and feedback such as "You did a great job!" is provided.

[0741] This allows the system to not only provide fast and accurate homework grading, but also improve users' learning progress and motivation.

[0742] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0743] Step 1: User takes a picture of their homework

[0744] Users take pictures of their homework using a smartphone or tablet camera app or a dedicated app. The input is a physical copy of the homework, and the output is a digital image file stored on the device. For example, a user takes a picture of a third-grade math homework sheet.

[0745] Step 2: The device sends the image to the server

[0746] The device temporarily stores the captured image file and sends it to a server via the Internet. The input is the captured digital image file, and the output is the image data sent to the server. Specifically, the device generates an HTTP request and POSTs the image data to an endpoint.

[0747] Step 3: The server receives the image data

[0748] The server receives image data sent from the user's device and saves it in local storage. The input is the received image data, and the output is the saved image file. For example, the received "Homework_20231012.jpg" is saved.

[0749] Step 4: The server OCRs the image

[0750] The server converts the saved image file into text data using OCR technology. The input is the saved image file, and the output is the recognized text data. For example, the Google Cloud Vision API is used to extract the text "3 + 2 = ?" and "4 - 1 = ?".

[0751] Step 5: The server analyzes the text using the generated AI model

[0752] The server passes the text data obtained by OCR processing to a generative AI model, which analyzes the questions and scores them. The input is the text data and the analysis prompt, and the output is the scoring results. Specifically, the generative AI model (e.g., OpenAI GPT-4) analyzes the prompt, "Analyze the following text, determine the answer to each question, and score it: '3 + 2 = ?'."

[0753] Step 6: The server formats and sends the results

[0754] The server formats the scoring results returned by the generative AI model into JSON format and sends it to the device. The input is the scoring results from the generative AI model, and the output is formatted JSON data. For example, it is formatted as follows: {"Question 1": {"Answer": "5", "Score": 5}, "Question 2": {"Answer": "3", "Score": 5}}.

[0755] Step 7: Your device will display the results

[0756] The terminal parses the JSON data received from the server and displays it in the user interface in an appropriate format. The input is JSON data, and the output is the displayed scoring result. Specifically, "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)" are displayed on the screen.

[0757] Step 8: The device captures the user's facial expression

[0758] The device captures the user's facial expression with a camera while checking the scoring result. The input is the user's facial expression, and the output is the captured image data. For example, the device captures the user's face using a front camera.

[0759] Step 9: The server analyzes facial expressions using the emotion engine

[0760] The server sends the captured facial expression data to the emotion engine for analysis. The input is facial image data, and the output is the emotion analysis results. For example, Microsoft Azure Emotion API is used to recognize emotions such as "happy" or "unhappy."

[0761] Step 10: The server records the emotion data and sends the feedback

[0762] The server records the recognized emotion data and provides feedback to the user. The input is the emotion analysis result, and the output is a feedback message. For example, a message such as "Good job!" is generated and sent to the device.

[0763] The above is the flow of specific processing steps. At each step, we have explained in detail how input data is processed and calculated to generate output data.

[0764] (Application example 2)

[0765] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0766] Currently, in many families, parents are burdened with the task of checking and grading their children's homework. It is also often difficult for children to maintain their motivation to study through homework. Furthermore, there is a lack of feedback that is sensitive to children's emotions, making it difficult to provide effective learning support.

[0767] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0768] In this invention, the server includes means for taking images of homework, means for transmitting the taken images to the server, means for converting the images to text data in the server, means for analyzing and grading the questions using a generative AI model based on the converted text data, means for transmitting the grading results from the server to the terminal, means for displaying the grading results on the terminal, means for capturing the user's facial expression with a camera and recognizing their emotions, means for recording the recognized emotions as feedback, and means for presenting interactive advice based on the emotion data. This makes it possible to efficiently support children's learning through automatic homework grading and emotion analysis, reduce the burden on parents, and increase children's motivation to learn.

[0769] "Means for taking pictures of homework" refers to a method in which a user takes a picture of the content written on a homework paper using a camera on a device such as a smartphone or tablet.

[0770] The "means for transmitting the captured images to the server" refers to a communication protocol or software that uploads the captured image data of the homework to a remote server via the Internet.

[0771] The "means of converting images to text data" is the process of converting the text information in the homework images into digital text using optical character recognition (OCR) technology.

[0772] The "means of analyzing questions and scoring using a generative AI model" refers to an artificial intelligence model that analyzes text data generated by OCR and scores questions by comparing actual answers with theoretically correct answers.

[0773] "Means for transmitting scoring results from the server to the terminal" refers to a system for returning the scoring result data from the server to the user's device (such as a smartphone or tablet) once scoring is complete.

[0774] The "means for displaying the scoring results on the terminal" refers to software or a device for displaying the scoring results received from the server as a user interface.

[0775] "Means for capturing a user's facial expression via a camera and recognizing emotions" refers to algorithms or software for capturing a user's facial expression via a camera and analyzing emotions from the facial expression data.

[0776] The "means for recording the recognized emotions as feedback" is a system for recording the analyzed emotion data in a database or the like and using it as feedback to the user.

[0777] The "means for presenting interactive advice based on emotional data" is a function for providing advice and feedback to the user according to the actual situation based on the recorded emotional data.

[0778] This invention shows a specific embodiment of a system that automatically grades homework by taking and uploading images, and further analyzes the user's emotions and provides feedback.

[0779] The entire system mainly consists of a terminal such as a smartphone, a server, and an emotion analysis engine.

[0780] First, the user takes a photo of the homework paper using the device's camera. The device is equipped with appropriate image capture functions to help the user capture the entire homework. The captured image is temporarily stored on the device and then sent to a remote server over the Internet. This process uses communication protocols such as HTTP requests.

[0781] The server converts the received image data into text using optical character recognition (OCR) technology. This text data includes homework questions and answers. The server uses software such as Tesseract OCR to extract text information from the image.

[0782] Next, a generative AI model is used to analyze and score the extracted text data. OpenAI's GPT-4 is used as the generative AI model. This model first determines the type of question (e.g., math, Japanese, science, etc.), and then evaluates the correctness of the answer using an appropriate algorithm depending on the type, and scores the answer. For example, it recognizes that the correct answer to the question "3 + 2 = ?" is "5," and awards points if the user's answer is correct.

[0783] The server organizes the scoring results as structured data, such as JSON format, and sends them to the device. The device then analyzes the received scoring results and displays them on the user interface. Specifically, it displays something like "Question 1: Correct (score 5 / 5)."

[0784] Furthermore, the device captures the user's facial expression with a camera and sends the data to an emotion analysis engine. The emotion analysis engine uses the FER (Facial Expression Recognition) library to analyze the user's facial expression and recognize their emotion. In this process, if the user is smiling, it is determined to be "happy," and if they are frowning, it is determined to be "unhappy."

[0785] The analyzed emotional data is recorded in a database, and interactive advice is generated to improve the user's learning progress and motivation. For example, if a user expresses dissatisfaction with their homework results, a message such as "If you try a little harder, you'll get a better result next time" is provided.

[0786] (Examples of specific examples and prompts)

[0787] Specifically, for a third-grade elementary school student's math homework, if there are two problems, "3 + 2 = ?" and "4 - 1 = ?", the user takes a photo of the problem and receives the automatic scoring results through the process described above. After the scoring results are returned, the device sends the user's facial expression to the emotion engine for analysis. If the system recognizes the user's facial expression as "happy," it will provide feedback such as "Good job!"

[0788] An example prompt is:

[0789] "Please grade the following homework: 3 + 2 = ?"

[0790] This invention not only allows users to receive the results of homework grading quickly and accurately, but also improves learning progress and motivation.

[0791] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0792] Step 1:

[0793] The user takes a photo of their homework paper with the device's camera. The input is the homework paper, and the output is the captured image file. This image file is temporarily saved on the device. The specific operation of the device is to launch the camera app or a dedicated app, and the user presses the shutter button to capture the image.

[0794] Step 2:

[0795] The device sends the captured image to the server. The input is the captured image file, and the output is the image data sent to the server. The device uploads the image data via the Internet using an HTTP request. Specifically, it issues a POST request to upload the file to the specified server URL.

[0796] Step 3:

[0797] The server converts the image data it receives into text data using OCR technology. The input is the image data sent to the server, and the output is text data. The server uses Tesseract OCR to extract the character information in the image as text. Specifically, it reads the image file and performs character recognition to generate text data.

[0798] Step 4:

[0799] The server uses a generative AI model to analyze the text data and grade the homework questions. The input is text data extracted by OCR, and the output is the graded results. The server calls a generative AI model (for example, OpenAI's GPT-4), creates a prompt, and requests grading. Specifically, the text data is sent to the generative AI model as a prompt in the format "Please grade the following homework: [text data]". The generative AI model returns a text containing the score as the analysis result.

[0800] Step 5:

[0801] The server sends the scoring results to the device. The input is the scoring result text received from the generative AI model, and the output is the scoring result sent to the device. The server formats the scoring results as structured data such as JSON format and sends them to the device as an HTTP response.

[0802] Step 6:

[0803] The terminal displays the marking results it has received. The input is the JSON format marking result data received from the server, and the output is the marking result displayed on the user interface. The terminal parses the JSON data and displays the marking results in a format that is easy for the user to understand. Specifically, it displays the results in a format such as "Question 1: Correct (score 5 / 5)."

[0804] Step 7:

[0805] The device captures the user's facial expression with a camera and recognizes the emotion. The input is the user's facial expression image captured by the device's camera, and the output is the recognized emotion data. The device uses the FER library to analyze the facial expression image and identify the user's emotion. Specifically, the device passes the captured image to the FER library and obtains the emotion data.

[0806] Step 8:

[0807] The server records the recognized emotions as feedback data. The input is the emotion data obtained by emotion analysis, and the output is the emotion data recorded in the database. The server saves the emotion data in the database and manages it together with the user's learning history.

[0808] Step 9:

[0809] The device presents interactive advice based on emotional data. The input is the recorded emotional data and the scoring results, and the output is an advice message displayed on the user interface. The device analyzes the emotional data and the scoring results, and generates and displays appropriate advice and feedback. Specifically, it displays encouraging messages such as, "If you try a little harder, you'll get better results next time."

[0810] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0811] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0812] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0813] [Third embodiment]

[0814] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0815] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0816] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0817] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0818] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0819] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0820] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0821] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0822] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0823] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0824] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0825] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0826] This invention is a system that automatically grades homework assignments by taking pictures of them and uploading them. The system aims to analyze the content of the homework assignments and grade them quickly and accurately, thereby reducing the burden on parents. The program processing of this system is explained below in natural language.

[0827] System Overview

[0828] The system includes the following elements:

[0829] 1. How to take pictures of your homework

[0830] 2. Means of sending the captured image to the server

[0831] 3. A method for converting images into text data on the server

[0832] 4. A method for analyzing questions and scoring them using a generative AI model based on the converted text data

[0833] 5. Means for sending the scoring results from the server to the device

[0834] 6. Means for displaying the scoring results on the device

[0835] Description of the Examples

[0836] 1. User operations

[0837] Users take a photo of their homework using a device such as a smartphone or tablet. Specifically, they launch a camera app or a dedicated app and take a clear photo of the homework paper. Once the photo is taken, the user taps the "Upload Photo" button.

[0838] 2. Terminal Processing

[0839] The device temporarily stores the captured image of the homework, adds it to an upload list, and transmits the image to a server over the Internet using an appropriate communication protocol, such as an HTTP request, to transmit the image.

[0840] 3. Server Processing

[0841] The server receives image data sent from the device. The received image data is first converted into text data using optical character recognition (OCR) technology. For example, if the math problem "3 + 2 = ?" is included, it is recognized as text data.

[0842] 4. Analysis and scoring using generative AI models

[0843] The server passes the text data converted by OCR to the generative AI model. The generative AI model determines the type of question (math, Japanese, science, etc.) and performs an analysis according to the type. In this example, the generative AI model analyzes the answer as "3 + 2 = 5" and determines that this is correct. If there is also a question such as "4 - 1 = ?", this is analyzed in the same way, and the AI ​​model determines that the correct answer is "4 - 1 = 3."

[0844] 5. Sending the score results

[0845] The server generates the scoring results, which are then formatted into an easy-to-read format such as JSON. The server then sends a response containing the scoring results to the device.

[0846] 6. Displaying the terminal

[0847] The terminal analyzes the scoring results received from the server and displays them appropriately on the user interface. For example, it displays "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)." In this way, the user can receive prompt and accurate feedback.

[0848] Specific operation example

[0849] For example, if a third-grade elementary school student has two math homework problems, "3 + 2 = ?" and "4 - 1 = ?", the user can take a photo of them and receive automatic grading results through the process described above.

[0850] Therefore, this system significantly reduces the burden on users (parents) and enables quick and accurate homework grading.

[0851] The processing flow will be explained below.

[0852] Step 1:

[0853] The user takes a picture of the homework.

[0854] The user takes a photo of the homework paper using the camera on their device (smartphone or tablet) using the camera function built into the application or the camera that is standard on the device.

[0855] Step 2:

[0856] The device temporarily saves the captured image.

[0857] The device temporarily stores images taken by the user and then adds them to the upload list.

[0858] Step 3:

[0859] A user uploads an image to the server.

[0860] The user taps the "upload photo" button displayed on the terminal application to send the captured image to the server.

[0861] Step 4:

[0862] The terminal transmits the image data to the server.

[0863] The device sends the captured image of the homework to a server over the Internet using an appropriate communication protocol, such as an HTTP request.

[0864] Step 5:

[0865] The server receives the image data.

[0866] The server receives the image data sent from the device, using the SSL / TLS protocol to ensure communication security.

[0867] Step 6:

[0868] The server converts the image data into text data.

[0869] The server converts the received image data into text data using OCR (Optical Character Recognition) technology. For example, an image of "3 + 2 = ?" is converted into the string "3 + 2 = ?".

[0870] Step 7:

[0871] The server passes the text data to a generative AI model for analysis.

[0872] The server passes the OCR-converted text data to the generative AI model, which determines the type of question and analyzes it using the appropriate algorithm.

[0873] Step 8:

[0874] A generative AI model analyzes the answers to the questions and scores them.

[0875] The generative AI model checks the accuracy of answers based on the input text data and scores them. For example, it recognizes "3 + 2 = 5" as the correct answer and calculates the score.

[0876] Step 9:

[0877] The server formats the scores.

[0878] The server formats the scoring results obtained from the generative AI model into an easy-to-read format such as JSON, for example, "Question 1: Correct (score 5 / 5)."

[0879] Step 10:

[0880] The server sends the scoring results to the terminal.

[0881] The server sends the formatted score results to the terminal, and the terminal checks whether it is ready to receive them.

[0882] Step 11:

[0883] The terminal receives the scoring results.

[0884] The device receives the scoring results sent from the server, including data in JSON format.

[0885] Step 12:

[0886] The terminal displays the scoring results on the user interface.

[0887] The device analyzes the received score results and displays them appropriately on the user interface, for example, in the format "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)."

[0888] These steps allow users to receive homework grading results quickly and accurately, significantly reducing the burden on parents.

[0889] Example 1

[0890] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0891] In the past, homework had to be graded manually by parents, which was time-consuming and labor-intensive, placing a heavy burden on parents. Furthermore, manual grading can be inaccurate, making it difficult to accurately evaluate a child's learning progress. Therefore, there was a need for a system that could grade homework quickly and accurately, reducing the burden on parents.

[0892] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0893] In this invention, the server includes a means for converting a captured image into text data, a means for analyzing and grading questions using a generative AI model based on the converted text data, and a means for transmitting the grading results from the server to the terminal. This makes it possible to automatically analyze the questions and quickly and accurately grade them simply by capturing an image of the homework and transmitting it to the server.

[0894] "Terminal" refers to a mobile information terminal such as a smartphone or tablet operated by a user.

[0895] A "server" refers to a computer system on a network that receives and processes data sent from a terminal.

[0896] "Photography" refers to the act of capturing an image using a camera or other image capture device.

[0897] "Text data" refers to a data format that includes character information such as letters and numbers.

[0898] Optical character recognition (OCR) refers to the technology that reads characters from an image and converts them into text data.

[0899] A "generative AI model" refers to a model that has been trained using artificial intelligence to automatically perform a specific task.

[0900] "Analysis" refers to the process of examining data or information in detail to understand its meaning and patterns.

[0901] "Grading" refers to the act of evaluating the correctness of answers and assignments and assigning points.

[0902] "Response" refers to response data sent from the server to the terminal.

[0903] "JSON" stands for JavaScript Object Notation and refers to a lightweight data exchange format.

[0904] The present invention is a system that automatically grades homework assignments by taking pictures of them and uploading them. The system aims to analyze the content of the homework assignments and grade them quickly and accurately, thereby reducing the burden on parents. The program processing of this system is explained below.

[0905] Hardware and software used

[0906] Devices: Smartphones, tablets

[0907] This allows users to use the camera function to take a picture of their homework paper and add the image to the upload list.

[0908] Server: Cloud Server

[0909] The received data is managed and processed using cloud services such as AWS and Google Cloud.

[0910] software:

[0911] Camera app: Software that allows users to take pictures of their homework.

[0912] OCR engine: Extracts text data from images using, for example, Google Cloud Vision API.

[0913] Generative AI models, such as OpenAI GPT-3, are used to analyze and score questions.

[0914] What the system does

[0915] 1. User takes a photo of their homework on their device:

[0916] The user launches the device's camera app and takes a clear photo of the homework paper. After taking the photo, the user taps the "Upload Photo" button to add the image to the upload list.

[0917] 2. Sending captured images to the server:

[0918] The terminal temporarily stores the captured image and transmits the image data to the server via an HTTP request.

[0919] 3. OCR processing on the server:

[0920] The server converts the received image data into text data using optical character recognition (OCR) technology. For example, if an image containing the question "3 + 2 = ?" is sent, OCR technology converts it into text data as "3 + 2 = ?"

[0921] 4. Analysis and scoring by generative AI model:

[0922] The server then passes the converted text data to a generative AI model for analysis and scoring. The generative AI model automatically determines the type of question (math, Japanese, science, etc.) and generates the appropriate answer. For example, the answer to the math question "3 + 2 = ?" is "5."

[0923] 5. Sending the score:

[0924] The server formats the analysis results from the generative AI model into an easy-to-read format such as JSON and sends it to the terminal.

[0925] 6. Viewing the score on your device:

[0926] The device analyzes the received score results and displays them on the user interface, for example, in the form of "Question 1: Correct (Score 5 / 5)" or "Question 2: Correct (Score 5 / 5)."

[0927] Prompt Sentence Examples

[0928] An example of a specific prompt is as follows:

[0929] "Please answer the following math problem: 3 + 2 = ?"

[0930] In this way, the system according to the present invention can quickly and accurately grade homework, significantly reducing the burden on parents.

[0931] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0932] Step 1:

[0933] The user launches the camera app or a dedicated app on their device (smartphone or tablet) and takes a picture of the homework question paper. When taking the picture, they adjust the camera so that the entire question paper is clearly visible. Once the picture is taken, the user taps the "Upload Photo" button. This action adds the captured image to the upload list on the device.

[0934] Input: Homework question paper

[0935] Output: Image file of the photographed homework

[0936] Step 2:

[0937] The device temporarily stores the captured image of the homework and prepares it for transmission to a server via the Internet. The image data is sent to the server using an appropriate communication protocol, such as an HTTP request. The image is saved in a standard image format, such as JPEG.

[0938] Input: Captured image file

[0939] Output: HTTP request sent to the server

[0940] Step 3:

[0941] The server receives the image data sent from the device. After receiving it, the server converts the image into text data using optical character recognition (OCR) technology. For example, if the image contains the math problem "3 + 2 = ?", OCR processing is performed so that this problem is recognized as the text data "3 + 2 = ?". The OCR engine used is Google Cloud Vision API or similar.

[0942] Input: Received image data

[0943] Output: Converted text data

[0944] Step 4:

[0945] The server passes the OCR-converted text data to a generative AI model, which analyzes and scores the questions. The generative AI model determines the type of question (math, Japanese, science, etc.) and creates an appropriate prompt to generate an answer. For example, for the text "3 + 2 = ?", the AI ​​model generates the prompt "Please answer the following math problem: 3 + 2 = ?", and outputs the answer "5."

[0946] Input: Converted text data

[0947] Output: Scoring results

[0948] Step 5:

[0949] The server formats the generated scoring results into an easy-to-read format such as JSON and generates a response to the device. This response includes the correct answer and score for each question. For example, it generates JSON data such as "{'Question 1': 'Correct', 'Score': '5 / 5'}".

[0950] Input: Scoring results

[0951] Output: Formatted response data (JSON format)

[0952] Step 6:

[0953] The terminal analyzes the response of the score received from the server and displays it on the user interface. Specifically, it visually presents the score result to the user in the form of "Question 1: Correct (score 5 / 5)."

[0954] Input: Response data received from the server

[0955] Output: The grading results that are displayed to the user

[0956] This allows the system to provide fast and accurate automatic grading by simply allowing users to take a photo of their homework and submit it.

[0957] (Application example 1)

[0958] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0959] Conventional homework grading systems require parents or teachers to manually check homework, which is time-consuming and labor-intensive. Additionally, the use of smart devices creates operational hassles and the convenience of these devices is not fully utilized. The present invention aims to solve these issues by providing a system that can efficiently capture and review homework using smart glasses.

[0960] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0961] In this invention, the server includes a means for capturing an image of the homework, a means for transmitting the captured image to the server, a means for converting the image into text data in the server, a means for analyzing and grading the questions using a generative AI model based on the converted text data, a means for transmitting the grading results from the server to the terminal, a means for displaying the grading results on the terminal, and a means for capturing and checking the image of the homework using smart glasses, thereby enabling teachers and parents to quickly and accurately grade homework and check the results in real time.

[0962] "Means for capturing images of homework" refers to the device a user uses to capture images of their homework papers, such as a smartphone or a camera on smart glasses.

[0963] "Means for transmitting captured images to a server" refers to a function for transferring captured image data to a server via the Internet.

[0964] "Means for converting images into text data on a server" refers to a method for recognizing characters in an image on a server and extracting the character information as text data, and primarily uses optical character recognition (OCR) technology.

[0965] "Method of analyzing questions using a generative AI model based on converted text data and scoring" refers to a method in which text data converted by OCR is input into a generative AI model, which then analyzes the answers to the questions, determines whether they are correct, and scores them.

[0966] "Means for transmitting the scoring results from the server to the terminal" refers to a communication means for transmitting the scoring results on the server to the client device.

[0967] "Means for displaying the scoring results on the terminal" refers to a user interface for visually presenting the scoring results to the user on the client device.

[0968] "Means for taking and reviewing images of homework using smart glasses" refers to a function for taking a photo of a homework sheet using smart glasses and reviewing it on the screen.

[0969] The present invention is a system for automatically marking homework assignments by taking images of the assignments and uploading them to a server. This system mainly includes the following elements:

[0970] System Overview

[0971] 1. How to take a picture of your homework:

[0972] The smart glasses are used to take images of the homework. Specifically, the smart glasses' camera is used to take a clear image of the student's entire homework paper. This also includes a function to check and take a photo at hand if necessary.

[0973] 2. Means for sending captured images to the server:

[0974] The captured image is sent to a server using the smart glasses' internet connection function. Image data is transferred to the server using a communication protocol such as an HTTP request.

[0975] 3. On the server, convert the image to text data:

[0976] The server receives the captured image data and converts the character information in the image into text data using OCR technology (for example, Tesseract OCR).

[0977] 4. A method for analyzing questions and scoring them using a generative AI model based on the converted text data:

[0978] The text data converted by OCR is passed to a generative AI model (e.g., OpenAI GPT) to analyze and grade the homework questions. The generative AI model identifies the type of question (math, Japanese, science, etc.) and provides the appropriate answer.

[0979] 5. How to send the score results from the server to the device:

[0980] The server formats the analyzed and scored results and sends them to the client terminal in an easy-to-read format such as JSON.

[0981] 6. How to display the results on the device:

[0982] The client terminal (e.g., smart glasses or a smartphone) displays the received scoring results on a user interface, allowing the results to be viewed in real time.

[0983] Specific operation example

[0984] For example, if a third-grade student's math homework includes the problems "3 + 2 = ?" and "4 - 1 = ?", the teacher puts on the smart glasses and takes a photo of the homework paper. The image data is then automatically sent to a server, where the character information is converted into text data using OCR. When the text data "3 + 2 = ?" and "4 - 1 = ?" are input into the generative AI model, the AI ​​model will answer "5" and "3" respectively and determine whether they are correct.

[0985] Hardware and software used

[0986] Smart glasses (e.g. Google Glass)

[0987] Server (e.g. AWS)

[0988] OCR technology (e.g. Tesseract OCR)

[0989] Generative AI models (e.g., OpenAI GPT)

[0990] Prompt Sentence Examples

[0991] Homework character recognition results (math problems):

[0992] "Analyze the following questions and determine whether they are correct or incorrect: 3 + 2 = ?, 4 - 1 = ?."

[0993] In this way, the present invention allows teachers and parents to quickly and accurately grade homework and check the results in real time, significantly reducing the time and effort required for traditional manual grading and providing efficient educational support.

[0994] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0995] Step 1:

[0996] The user uses the smart glasses to take a clear photo of their homework paper. The inputs are the user's instruction to take a photo and the image of the homework paper captured by the camera. The output is the captured image data, which is temporarily stored locally. During this process, the smart glasses' camera function and the photo application are running.

[0997] Step 2:

[0998] The device (smart glasses) sends the captured image to a server via the Internet. The input is the image data generated in step 1. The output is the image data transferred to the server using an HTTP request. An appropriate communication protocol (e.g. HTTP) is used in this process.

[0999] Step 3:

[1000] The server receives the image data and uses optical character recognition (OCR) technology to recognize the text in the image and convert it into text data. The input is the image data transferred to the server. The output is the converted text data generated by OCR. Character recognition software such as Tesseract OCR is used in this process.

[1001] Step 4:

[1002] The server inputs the text data converted by OCR into a generative AI model to analyze and score the questions. The input is the text data generated by OCR. The output is the analysis results (answers and scoring information) generated by the generative AI model. In this process, a generative AI model (e.g., OpenAI GPT) is used to analyze the data based on prompts such as "Analyze the following questions and determine whether they are correct or incorrect: 3 + 2 = ?, 4 - 1 = ?."

[1003] Step 5:

[1004] The server formats the scoring results generated by the generative AI model and sends them to the terminal. The input is the analysis results of the generative AI model. The output is the scoring results formatted in an easy-to-read format such as JSON and sent to the client terminal. During this process, data is formatted and transferred using a communication protocol (e.g., HTTP).

[1005] Step 6:

[1006] The device (smart glasses or smartphone) displays the scoring results received from the server on a user interface. The input is the scoring result data sent from the server. The output is the scoring results displayed on the user interface, providing visual feedback to the user. This process uses UI components and display logic for display.

[1007] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1008] This invention is a system that automatically grades homework assignments by taking and uploading images, and also analyzes the user's emotions and provides feedback. The system aims to analyze the content of homework assignments to grade them quickly and accurately, reducing the burden on parents and recognizing the user's emotions to improve the quality of education.

[1009] System Overview

[1010] The system includes the following elements:

[1011] 1. How to take pictures of your homework

[1012] 2. Means of sending the captured image to the server

[1013] 3. A method for converting images into text data on the server

[1014] 4. A method for analyzing questions and scoring them using a generative AI model based on the converted text data

[1015] 5. Means for sending the scoring results from the server to the device

[1016] 6. Means for displaying the scoring results on the device

[1017] 7. Emotion engine that analyzes the user's facial expressions and recognizes their emotions

[1018] 8. Ability to record emotions recognized by the emotion engine as feedback

[1019] 9. A feature that provides interactive advice based on emotional data to improve learning progress and motivation

[1020] Description of the Examples

[1021] 1. User operations

[1022] Users take a photo of their homework using a device such as a smartphone or tablet. Specifically, they launch a camera app or a dedicated app and take a clear photo of the homework paper. Once the photo is taken, the user taps the "Upload Photo" button.

[1023] 2. Terminal Processing

[1024] The device temporarily stores the captured image, adds it to the upload list, and transmits it to a server over the Internet using an appropriate communication protocol, such as an HTTP request.

[1025] 3. Server Processing

[1026] The server receives image data sent from the device. The received image data is first converted into text data using optical character recognition (OCR) technology. For example, if the math problem "3 + 2 = ?" is included, it is recognized as text data.

[1027] 4. Analysis and scoring using generative AI models

[1028] The server passes the text data converted by OCR to the generative AI model. The generative AI model determines the type of question (math, Japanese, science, etc.), analyzes the question using the appropriate algorithm depending on the type, checks the accuracy of the answer, and scores it. For example, it recognizes "3 + 2 = 5" as the correct answer and calculates the score.

[1029] 5. Sending the score results

[1030] The server generates the scoring results, which are then formatted into an easy-to-read format such as JSON. The server then sends a response containing the scoring results to the device.

[1031] 6. Displaying the terminal

[1032] The device analyzes the score results received from the server and displays them appropriately on the user interface, for example, in the format "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)."

[1033] 7. Analysis by Emotion Engine

[1034] The device captures the user's facial expressions through a camera and sends them to the emotion engine. The emotion engine analyzes the user's facial expression data and recognizes their emotions. For example, if the user is smiling, it is recognized as "happy," and if they are frowning, it is recognized as "unhappy."

[1035] 8. Recording Emotional Data

[1036] The emotion engine records the user's recognized emotions as feedback data, such as their reaction to the results of their homework grading or their feelings about their learning progress.

[1037] 9. Interactive Advice

[1038] The emotion engine provides interactive advice to users based on the recorded emotion data. For example, if a user expresses dissatisfaction with their homework results, it displays a motivational message such as, "If you try a little harder, you'll do better next time."

[1039] Specific operation example

[1040] For example, if a third-grade elementary school student has two math homework problems, "3 + 2 = ?" and "4 - 1 = ?", the user can take a photo of the problem and receive the automatic scoring results through the process described above. After the scoring results are returned, the device sends the user's facial expression to the emotion engine for analysis. If the system recognizes the user's facial expression as "happy," it will provide feedback such as "Good job!"

[1041] Therefore, this system not only enables users to receive homework grading results quickly and accurately, but also improves learning progress and motivation.

[1042] The processing flow will be explained below.

[1043] Step 1:

[1044] The user takes a picture of the homework.

[1045] The user takes a picture of the entire homework paper using the camera of the device (smartphone or tablet). For example, the user takes a clear picture of the homework page using the camera function built into the application.

[1046] Step 2:

[1047] The device temporarily saves the captured image.

[1048] The terminal stores the images taken by the user in temporary memory and then adds them to the upload list.

[1049] Step 3:

[1050] A user uploads an image to the server.

[1051] The user taps the "upload photo" button in the application on the device to send the captured image to the server.

[1052] Step 4:

[1053] The terminal transmits the image data to the server.

[1054] The device sends the captured image of the homework to a server over the Internet using an appropriate communication protocol, such as an HTTP request.

[1055] Step 5:

[1056] The server receives the image data.

[1057] The server receives the image data sent from the device. This communication uses the SSL / TLS protocol to ensure security.

[1058] Step 6:

[1059] The server converts the image data into text data.

[1060] The server converts the received image data into text data using OCR (optical character recognition) technology. For example, the handwritten mathematical formula "3 + 2 = ?" in the image is converted into the string "3 + 2 = ?".

[1061] Step 7:

[1062] The server passes the text data to a generative AI model for analysis.

[1063] The server then passes the OCR-converted text data to the generative AI model, which then analyzes the question content. The generative AI model then identifies the type of question and performs the analysis using the appropriate algorithm.

[1064] Step 8:

[1065] A generative AI model analyzes the answers to the questions and scores them.

[1066] The generative AI model analyzes the answers based on the input text data, determines whether they are correct, and calculates the score. For example, "3 + 2 = 5" is recognized as the correct answer, and the score is calculated.

[1067] Step 9:

[1068] The server formats the scores.

[1069] The server converts the scoring results obtained from the generative AI model into an easy-to-read format such as JSON, for example, "Question 1: Correct (score 5 / 5)."

[1070] Step 10:

[1071] The server sends the scoring results to the terminal.

[1072] The server sends the formatted score to the device, using an appropriate protocol so that the device can receive it.

[1073] Step 11:

[1074] The terminal receives the scoring results.

[1075] The terminal receives the scoring results sent from the server and analyzes the data in JSON format, etc.

[1076] Step 12:

[1077] The terminal displays the scoring results on a user interface.

[1078] The device will then display the received scoring results appropriately on the user interface, such as "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)."

[1079] Step 13:

[1080] The device captures the user's facial expressions.

[1081] The device captures the user's facial expressions in real time via a camera, capturing their facial expressions while they are checking their homework results.

[1082] Step 14:

[1083] The device sends the facial expression data to the emotion engine.

[1084] The device analyzes the captured facial expression data and sends it to the emotion engine. This data includes key facial features of the user (e.g., eye shape, mouth movement, etc.).

[1085] Step 15:

[1086] The emotion engine analyzes the user's emotions.

[1087] The emotion engine analyzes the user's emotions based on the facial expression data sent. For example, a smile is judged to be "happy," and a frown is judged to be "unhappy."

[1088] Step 16:

[1089] The emotion engine records the emotion data.

[1090] The emotion engine recognizes and records the user's emotions as feedback data, including their reactions to the results of their homework grading and their feelings about their learning progress.

[1091] Step 17:

[1092] An emotion engine provides interactive advice.

[1093] The emotion engine provides interactive advice to users based on the recorded emotion data. For example, if a user expresses dissatisfaction with the results of their homework, it will display an encouraging message such as "Try harder next time!"

[1094] These steps allow users to not only receive the results of their homework assignments quickly and accurately, but also to properly analyze their feelings about the results and obtain feedback to improve their motivation to study.

[1095] Example 2

[1096] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1097] Traditional homework grading and learning progress management relies heavily on manual work, placing a heavy burden on parents and teachers in particular. Additionally, it is difficult to properly grasp changes in children's emotions as they learn, and there is a lack of feedback to maintain and improve their motivation. The purpose of this invention is to solve these problems by analyzing users' emotions and providing interactive feedback in addition to automatically grading homework.

[1098] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting images of homework into text data, a means for analyzing and grading questions using a generative model based on the converted text data, and a means for transmitting the grading results to the user interface device. This makes it possible to automatically grade homework and analyze the user's emotions to provide feedback.

[1099] "Means for taking images of homework" refers to a device equipped with a camera or a dedicated application that allows a user to take images of their homework.

[1100] The term "means for transmitting a captured image to an information processing device" refers to a protocol or a communication interface for transmitting captured image data to an information processing device using a network.

[1101] "Means for converting an image into text data in an information processing device" refers to software or hardware for converting image data into text information using optical character recognition technology.

[1102] "Means for analyzing questions using a generative model based on converted text data and grading" refers to an algorithm and processing system that uses a generative AI model to analyze text data, determine the answer to each question, and grade the answers.

[1103] The term "means for transmitting the scoring results from the information processing device to the user interface device" refers to a communication means for transmitting the analysis and scoring results to the user interface device via a network.

[1104] The term "means for displaying the scoring results in the user interface device" refers to a display screen and GUI (Graphical User Interface) for displaying the scoring results on the user interface device.

[1105] "Means for capturing a user's facial expression and analyzing emotions" refers to a device and software for capturing a user's facial expression data using a camera and sending it to an emotion analysis engine for analysis.

[1106] "Means for recording analyzed emotional data" refers to a system for analyzing a user's emotional data and then recording it in a database or storage.

[1107] "Means for providing interactive advice based on recorded emotional data" refers to scripts or algorithms that utilize the recorded emotional data to provide appropriate feedback or advice to the user.

[1108] This invention is a system that automatically grades homework assignments by taking and uploading images, and also analyzes the user's emotions and provides feedback. This system aims to analyze the content of homework assignments to grade them quickly and accurately, reducing the burden on parents and teachers, and improving the quality of education by recognizing the user's emotions.

[1109] Hardware and software used

[1110] 1. Hardware:

[1111] Device: A user device such as a smartphone or tablet that allows the user to take and upload images of their homework.

[1112] Server: A central information processing device that can analyze and score submitted images and generate feedback.

[1113] Camera: A camera built into the device that can capture the user's facial expressions.

[1114] 2. Software:

[1115] Camera app or dedicated app: An application used on a device that provides the ability to take and upload images of homework.

[1116] Optical Character Recognition (OCR) technology, such as Google Cloud Vision API, which converts images of homework assignments into text data.

[1117] A generative AI model, such as OpenAI GPT-4, that analyzes the converted text data and scores the questions.

[1118] Emotion engine: For example, Microsoft Azure Emotion API, which analyzes the user's facial expression data and recognizes their emotions.

[1119] Overview of program processing

[1120] User operations

[1121] Users open a camera app or a dedicated app on their smartphone or tablet and take a photo of their homework paper. Once they have taken the photo, they tap the "Upload Photo" button.

[1122] Terminal handling

[1123] The device temporarily stores the captured image and transmits it to a server via the Internet. For example, it generates an HTTP request and sends a POST request to the server's upload endpoint.

[1124] Server reception and image analysis

[1125] The server receives the image data sent from the device and converts it into text data using optical character recognition (OCR) technology. For example, if the data contains the math problem "3 + 2 = ?", it will recognize this as text data.

[1126] Analysis using generative AI models

[1127] The server passes the text data obtained by OCR to a generative AI model, which analyzes the questions and scores them. An example of a prompt is "Analyze the following text to determine the answer to each question and score it: '3 + 2 = ?'".

[1128] Formatting and sending the graded results

[1129] The server converts the scoring results returned by the generative AI model into JSON format and sends it to the device. For example, it converts it into the following format: {"Question 1": {"Answer": "5", "Score": 5}, "Question 2": {"Answer": "3", "Score": 5}}.

[1130] Display results on device and capture facial expressions

[1131] The device analyzes the JSON formatted scoring results received from the server and displays them on the user interface. For example, it displays "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)." It also captures the user's facial expressions with a camera and sends them to the emotion engine.

[1132] Emotion analysis and feedback using an emotion engine

[1133] The emotion engine analyzes the user's facial expression data and recognizes their emotions. For example, if the user is smiling, it is recognized as "happy," and if they are frowning, it is recognized as "unhappy." The analyzed emotion data is recorded, and based on the recorded data, it provides interactive advice such as "You did a great job!"

[1134] Specific operation example

[1135] For example, if a third-grade elementary school student has two math homework problems, "3 + 2 = ?" and "4 - 1 = ?", the user can take a photo of them and upload them to the system. The image is converted into text on the server, and an AI model recognizes "3 + 2 = 5" and "4 - 1 = 3" as the correct answers, awarding points for each. At the same time as the results are displayed on the device, the user's facial expressions are analyzed by an emotion engine, and feedback such as "You did a great job!" is provided.

[1136] This allows the system to not only provide fast and accurate homework grading, but also improve users' learning progress and motivation.

[1137] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1138] Step 1: User takes a picture of their homework

[1139] Users take pictures of their homework using a smartphone or tablet camera app or a dedicated app. The input is a physical copy of the homework, and the output is a digital image file stored on the device. For example, a user takes a picture of a third-grade math homework sheet.

[1140] Step 2: The device sends the image to the server

[1141] The device temporarily stores the captured image file and sends it to a server via the Internet. The input is the captured digital image file, and the output is the image data sent to the server. Specifically, the device generates an HTTP request and POSTs the image data to an endpoint.

[1142] Step 3: The server receives the image data

[1143] The server receives image data sent from the user's device and saves it in local storage. The input is the received image data, and the output is the saved image file. For example, the received "Homework_20231012.jpg" is saved.

[1144] Step 4: The server OCRs the image

[1145] The server converts the saved image file into text data using OCR technology. The input is the saved image file, and the output is the recognized text data. For example, the Google Cloud Vision API is used to extract the text "3 + 2 = ?" and "4 - 1 = ?".

[1146] Step 5: The server analyzes the text using the generated AI model

[1147] The server passes the text data obtained by OCR processing to a generative AI model, which analyzes the questions and scores them. The input is the text data and the analysis prompt, and the output is the scoring results. Specifically, the generative AI model (e.g., OpenAI GPT-4) analyzes the prompt, "Analyze the following text, determine the answer to each question, and score it: '3 + 2 = ?'."

[1148] Step 6: The server formats and sends the results

[1149] The server formats the scoring results returned by the generative AI model into JSON format and sends it to the device. The input is the scoring results from the generative AI model, and the output is formatted JSON data. For example, it is formatted as follows: {"Question 1": {"Answer": "5", "Score": 5}, "Question 2": {"Answer": "3", "Score": 5}}.

[1150] Step 7: Your device will display the results

[1151] The terminal parses the JSON data received from the server and displays it in the user interface in an appropriate format. The input is JSON data, and the output is the displayed scoring result. Specifically, "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)" are displayed on the screen.

[1152] Step 8: The device captures the user's facial expression

[1153] The device captures the user's facial expression with a camera while checking the scoring result. The input is the user's facial expression, and the output is the captured image data. For example, the device captures the user's face using a front camera.

[1154] Step 9: The server analyzes facial expressions using the emotion engine

[1155] The server sends the captured facial expression data to the emotion engine for analysis. The input is facial image data, and the output is the emotion analysis results. For example, Microsoft Azure Emotion API is used to recognize emotions such as "happy" or "unhappy."

[1156] Step 10: The server records the emotion data and sends the feedback

[1157] The server records the recognized emotion data and provides feedback to the user. The input is the emotion analysis result, and the output is a feedback message. For example, a message such as "Good job!" is generated and sent to the device.

[1158] The above is the flow of specific processing steps. At each step, we have explained in detail how input data is processed and calculated to generate output data.

[1159] (Application example 2)

[1160] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1161] Currently, in many families, parents are burdened with the task of checking and grading their children's homework. It is also often difficult for children to maintain their motivation to study through homework. Furthermore, there is a lack of feedback that is sensitive to children's emotions, making it difficult to provide effective learning support.

[1162] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1163] In this invention, the server includes means for taking images of homework, means for transmitting the taken images to the server, means for converting the images to text data in the server, means for analyzing and grading the questions using a generative AI model based on the converted text data, means for transmitting the grading results from the server to the terminal, means for displaying the grading results on the terminal, means for capturing the user's facial expression with a camera and recognizing their emotions, means for recording the recognized emotions as feedback, and means for presenting interactive advice based on the emotion data. This makes it possible to efficiently support children's learning through automatic homework grading and emotion analysis, reduce the burden on parents, and increase children's motivation to learn.

[1164] "Means for taking pictures of homework" refers to a method in which a user takes a picture of the content written on a homework paper using a camera on a device such as a smartphone or tablet.

[1165] The "means for transmitting the captured images to the server" refers to a communication protocol or software that uploads the captured image data of the homework to a remote server via the Internet.

[1166] The "means of converting images to text data" is the process of converting the text information in the homework images into digital text using optical character recognition (OCR) technology.

[1167] The "means of analyzing questions and scoring using a generative AI model" refers to an artificial intelligence model that analyzes text data generated by OCR and scores questions by comparing actual answers with theoretically correct answers.

[1168] "Means for transmitting scoring results from the server to the terminal" refers to a system for returning the scoring result data from the server to the user's device (such as a smartphone or tablet) once scoring is complete.

[1169] The "means for displaying the scoring results on the terminal" refers to software or a device for displaying the scoring results received from the server as a user interface.

[1170] "Means for capturing a user's facial expression via a camera and recognizing emotions" refers to algorithms or software for capturing a user's facial expression via a camera and analyzing emotions from the facial expression data.

[1171] The "means for recording the recognized emotions as feedback" is a system for recording the analyzed emotion data in a database or the like and using it as feedback to the user.

[1172] The "means for presenting interactive advice based on emotional data" is a function for providing advice and feedback to the user according to the actual situation based on the recorded emotional data.

[1173] This invention shows a specific embodiment of a system that automatically grades homework by taking and uploading images, and further analyzes the user's emotions and provides feedback.

[1174] The entire system mainly consists of a terminal such as a smartphone, a server, and an emotion analysis engine.

[1175] First, the user takes a photo of the homework paper using the device's camera. The device is equipped with appropriate image capture functions to help the user capture the entire homework. The captured image is temporarily stored on the device and then sent to a remote server over the Internet. This process uses communication protocols such as HTTP requests.

[1176] The server converts the received image data into text using optical character recognition (OCR) technology. This text data includes homework questions and answers. The server uses software such as Tesseract OCR to extract text information from the image.

[1177] Next, a generative AI model is used to analyze and score the extracted text data. OpenAI's GPT-4 is used as the generative AI model. This model first determines the type of question (e.g., math, Japanese, science, etc.), and then evaluates the correctness of the answer using an appropriate algorithm depending on the type, and scores the answer. For example, it recognizes that the correct answer to the question "3 + 2 = ?" is "5," and awards points if the user's answer is correct.

[1178] The server organizes the scoring results as structured data, such as JSON format, and sends them to the device. The device then analyzes the received scoring results and displays them on the user interface. Specifically, it displays something like "Question 1: Correct (score 5 / 5)."

[1179] Furthermore, the device captures the user's facial expression with a camera and sends the data to an emotion analysis engine. The emotion analysis engine uses the FER (Facial Expression Recognition) library to analyze the user's facial expression and recognize their emotion. In this process, if the user is smiling, it is determined to be "happy," and if they are frowning, it is determined to be "unhappy."

[1180] The analyzed emotional data is recorded in a database, and interactive advice is generated to improve the user's learning progress and motivation. For example, if a user expresses dissatisfaction with their homework results, a message such as "If you try a little harder, you'll get a better result next time" is provided.

[1181] (Examples of specific examples and prompts)

[1182] Specifically, for a third-grade elementary school student's math homework, if there are two problems, "3 + 2 = ?" and "4 - 1 = ?", the user takes a photo of the problem and receives the automatic scoring results through the process described above. After the scoring results are returned, the device sends the user's facial expression to the emotion engine for analysis. If the system recognizes the user's facial expression as "happy," it will provide feedback such as "Good job!"

[1183] An example prompt is:

[1184] "Please grade the following homework: 3 + 2 = ?"

[1185] This invention not only allows users to receive the results of homework grading quickly and accurately, but also improves learning progress and motivation.

[1186] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1187] Step 1:

[1188] The user takes a photo of their homework paper with the device's camera. The input is the homework paper, and the output is the captured image file. This image file is temporarily saved on the device. The specific operation of the device is to launch the camera app or a dedicated app, and the user presses the shutter button to capture the image.

[1189] Step 2:

[1190] The device sends the captured image to the server. The input is the captured image file, and the output is the image data sent to the server. The device uploads the image data via the Internet using an HTTP request. Specifically, it issues a POST request to upload the file to the specified server URL.

[1191] Step 3:

[1192] The server converts the image data it receives into text data using OCR technology. The input is the image data sent to the server, and the output is text data. The server uses Tesseract OCR to extract the character information in the image as text. Specifically, it reads the image file and performs character recognition to generate text data.

[1193] Step 4:

[1194] The server uses a generative AI model to analyze the text data and grade the homework questions. The input is text data extracted by OCR, and the output is the graded results. The server calls a generative AI model (for example, OpenAI's GPT-4), creates a prompt, and requests grading. Specifically, the text data is sent to the generative AI model as a prompt in the format "Please grade the following homework: [text data]". The generative AI model returns a text containing the score as the analysis result.

[1195] Step 5:

[1196] The server sends the scoring results to the device. The input is the scoring result text received from the generative AI model, and the output is the scoring result sent to the device. The server formats the scoring results as structured data such as JSON format and sends them to the device as an HTTP response.

[1197] Step 6:

[1198] The terminal displays the marking results it has received. The input is the JSON format marking result data received from the server, and the output is the marking result displayed on the user interface. The terminal parses the JSON data and displays the marking results in a format that is easy for the user to understand. Specifically, it displays the results in a format such as "Question 1: Correct (score 5 / 5)."

[1199] Step 7:

[1200] The device captures the user's facial expression with a camera and recognizes the emotion. The input is the user's facial expression image captured by the device's camera, and the output is the recognized emotion data. The device uses the FER library to analyze the facial expression image and identify the user's emotion. Specifically, the device passes the captured image to the FER library and obtains the emotion data.

[1201] Step 8:

[1202] The server records the recognized emotions as feedback data. The input is the emotion data obtained by emotion analysis, and the output is the emotion data recorded in the database. The server saves the emotion data in the database and manages it together with the user's learning history.

[1203] Step 9:

[1204] The device presents interactive advice based on emotional data. The input is the recorded emotional data and the scoring results, and the output is an advice message displayed on the user interface. The device analyzes the emotional data and the scoring results, and generates and displays appropriate advice and feedback. Specifically, it displays encouraging messages such as, "If you try a little harder, you'll get better results next time."

[1205] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1206] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1207] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1208] [Fourth embodiment]

[1209] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1210] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1211] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1212] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1213] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1214] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1215] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1216] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1217] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1218] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1219] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1220] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1221] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1222] This invention is a system that automatically grades homework assignments by taking pictures of them and uploading them. The system aims to analyze the content of the homework assignments and grade them quickly and accurately, thereby reducing the burden on parents. The program processing of this system is explained below in natural language.

[1223] System Overview

[1224] The system includes the following elements:

[1225] 1. How to take pictures of your homework

[1226] 2. Means of sending the captured image to the server

[1227] 3. A method for converting images into text data on the server

[1228] 4. A method for analyzing questions and scoring them using a generative AI model based on the converted text data

[1229] 5. Means for sending the scoring results from the server to the device

[1230] 6. Means for displaying the scoring results on the device

[1231] Description of the Examples

[1232] 1. User operations

[1233] Users take a photo of their homework using a device such as a smartphone or tablet. Specifically, they launch a camera app or a dedicated app and take a clear photo of the homework paper. Once the photo is taken, the user taps the "Upload Photo" button.

[1234] 2. Terminal Processing

[1235] The device temporarily stores the captured image of the homework, adds it to an upload list, and transmits the image to a server over the Internet using an appropriate communication protocol, such as an HTTP request, to transmit the image.

[1236] 3. Server Processing

[1237] The server receives image data sent from the device. The received image data is first converted into text data using optical character recognition (OCR) technology. For example, if the math problem "3 + 2 = ?" is included, it is recognized as text data.

[1238] 4. Analysis and scoring using generative AI models

[1239] The server passes the text data converted by OCR to the generative AI model. The generative AI model determines the type of question (math, Japanese, science, etc.) and performs an analysis according to the type. In this example, the generative AI model analyzes the answer as "3 + 2 = 5" and determines that this is correct. If there is also a question such as "4 - 1 = ?", this is analyzed in the same way, and the AI ​​model determines that the correct answer is "4 - 1 = 3."

[1240] 5. Sending the score results

[1241] The server generates the scoring results, which are then formatted into an easy-to-read format such as JSON. The server then sends a response containing the scoring results to the device.

[1242] 6. Displaying the terminal

[1243] The terminal analyzes the scoring results received from the server and displays them appropriately on the user interface. For example, it displays "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)." In this way, the user can receive prompt and accurate feedback.

[1244] Specific operation example

[1245] For example, if a third-grade elementary school student has two math homework problems, "3 + 2 = ?" and "4 - 1 = ?", the user can take a photo of them and receive automatic grading results through the process described above.

[1246] Therefore, this system significantly reduces the burden on users (parents) and enables quick and accurate homework grading.

[1247] The processing flow will be explained below.

[1248] Step 1:

[1249] The user takes a picture of the homework.

[1250] The user takes a photo of the homework paper using the camera on their device (smartphone or tablet) using the camera function built into the application or the camera that is standard on the device.

[1251] Step 2:

[1252] The device temporarily saves the captured image.

[1253] The device temporarily stores images taken by the user and then adds them to the upload list.

[1254] Step 3:

[1255] A user uploads an image to the server.

[1256] The user taps the "upload photo" button displayed on the terminal application to send the captured image to the server.

[1257] Step 4:

[1258] The terminal transmits the image data to the server.

[1259] The device sends the captured image of the homework to a server over the Internet using an appropriate communication protocol, such as an HTTP request.

[1260] Step 5:

[1261] The server receives the image data.

[1262] The server receives the image data sent from the device, using the SSL / TLS protocol to ensure communication security.

[1263] Step 6:

[1264] The server converts the image data into text data.

[1265] The server converts the received image data into text data using OCR (Optical Character Recognition) technology. For example, an image of "3 + 2 = ?" is converted into the string "3 + 2 = ?".

[1266] Step 7:

[1267] The server passes the text data to a generative AI model for analysis.

[1268] The server passes the OCR-converted text data to the generative AI model, which determines the type of question and analyzes it using the appropriate algorithm.

[1269] Step 8:

[1270] A generative AI model analyzes the answers to the questions and scores them.

[1271] The generative AI model checks the accuracy of answers based on the input text data and scores them. For example, it recognizes "3 + 2 = 5" as the correct answer and calculates the score.

[1272] Step 9:

[1273] The server formats the scores.

[1274] The server formats the scoring results obtained from the generative AI model into an easy-to-read format such as JSON, for example, "Question 1: Correct (score 5 / 5)."

[1275] Step 10:

[1276] The server sends the scoring results to the terminal.

[1277] The server sends the formatted score results to the terminal, and the terminal checks whether it is ready to receive them.

[1278] Step 11:

[1279] The terminal receives the scoring results.

[1280] The device receives the scoring results sent from the server, including data in JSON format.

[1281] Step 12:

[1282] The terminal displays the scoring results on the user interface.

[1283] The device analyzes the received score results and displays them appropriately on the user interface, for example, in the format "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)."

[1284] These steps allow users to receive homework grading results quickly and accurately, significantly reducing the burden on parents.

[1285] Example 1

[1286] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1287] In the past, homework had to be graded manually by parents, which was time-consuming and labor-intensive, placing a heavy burden on parents. Furthermore, manual grading can be inaccurate, making it difficult to accurately evaluate a child's learning progress. Therefore, there was a need for a system that could grade homework quickly and accurately, reducing the burden on parents.

[1288] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1289] In this invention, the server includes a means for converting a captured image into text data, a means for analyzing and grading questions using a generative AI model based on the converted text data, and a means for transmitting the grading results from the server to the terminal. This makes it possible to automatically analyze the questions and quickly and accurately grade them simply by capturing an image of the homework and transmitting it to the server.

[1290] "Terminal" refers to a mobile information terminal such as a smartphone or tablet operated by a user.

[1291] A "server" refers to a computer system on a network that receives and processes data sent from a terminal.

[1292] "Photography" refers to the act of capturing an image using a camera or other image capture device.

[1293] "Text data" refers to a data format that includes character information such as letters and numbers.

[1294] Optical character recognition (OCR) refers to the technology that reads characters from an image and converts them into text data.

[1295] A "generative AI model" refers to a model that has been trained using artificial intelligence to automatically perform a specific task.

[1296] "Analysis" refers to the process of examining data or information in detail to understand its meaning and patterns.

[1297] "Grading" refers to the act of evaluating the correctness of answers and assignments and assigning points.

[1298] "Response" refers to response data sent from the server to the terminal.

[1299] "JSON" stands for JavaScript Object Notation and refers to a lightweight data exchange format.

[1300] The present invention is a system that automatically grades homework assignments by taking pictures of them and uploading them. The system aims to analyze the content of the homework assignments and grade them quickly and accurately, thereby reducing the burden on parents. The program processing of this system is explained below.

[1301] Hardware and software used

[1302] Devices: Smartphones, tablets

[1303] This allows users to use the camera function to take a picture of their homework paper and add the image to the upload list.

[1304] Server: Cloud Server

[1305] The received data is managed and processed using cloud services such as AWS and Google Cloud.

[1306] software:

[1307] Camera app: Software that allows users to take pictures of their homework.

[1308] OCR engine: Extracts text data from images using, for example, Google Cloud Vision API.

[1309] Generative AI models, such as OpenAI GPT-3, are used to analyze and score questions.

[1310] What the system does

[1311] 1. User takes a photo of their homework on their device:

[1312] The user launches the device's camera app and takes a clear photo of the homework paper. After taking the photo, the user taps the "Upload Photo" button to add the image to the upload list.

[1313] 2. Sending captured images to the server:

[1314] The terminal temporarily stores the captured image and transmits the image data to the server via an HTTP request.

[1315] 3. OCR processing on the server:

[1316] The server converts the received image data into text data using optical character recognition (OCR) technology. For example, if an image containing the question "3 + 2 = ?" is sent, OCR technology converts it into text data as "3 + 2 = ?"

[1317] 4. Analysis and scoring by generative AI model:

[1318] The server then passes the converted text data to a generative AI model for analysis and scoring. The generative AI model automatically determines the type of question (math, Japanese, science, etc.) and generates the appropriate answer. For example, the answer to the math question "3 + 2 = ?" is "5."

[1319] 5. Sending the score:

[1320] The server formats the analysis results from the generative AI model into an easy-to-read format such as JSON and sends it to the terminal.

[1321] 6. Viewing the score on your device:

[1322] The device analyzes the received score results and displays them on the user interface, for example, in the form of "Question 1: Correct (Score 5 / 5)" or "Question 2: Correct (Score 5 / 5)."

[1323] Prompt Sentence Examples

[1324] An example of a specific prompt is as follows:

[1325] "Please answer the following math problem: 3 + 2 = ?"

[1326] In this way, the system according to the present invention can quickly and accurately grade homework, significantly reducing the burden on parents.

[1327] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1328] Step 1:

[1329] The user launches the camera app or a dedicated app on their device (smartphone or tablet) and takes a picture of the homework question paper. When taking the picture, they adjust the camera so that the entire question paper is clearly visible. Once the picture is taken, the user taps the "Upload Photo" button. This action adds the captured image to the upload list on the device.

[1330] Input: Homework question paper

[1331] Output: Image file of the photographed homework

[1332] Step 2:

[1333] The device temporarily stores the captured image of the homework and prepares it for transmission to a server via the Internet. The image data is sent to the server using an appropriate communication protocol, such as an HTTP request. The image is saved in a standard image format, such as JPEG.

[1334] Input: Captured image file

[1335] Output: HTTP request sent to the server

[1336] Step 3:

[1337] The server receives the image data sent from the device. After receiving it, the server converts the image into text data using optical character recognition (OCR) technology. For example, if the image contains the math problem "3 + 2 = ?", OCR processing is performed so that this problem is recognized as the text data "3 + 2 = ?". The OCR engine used is Google Cloud Vision API or similar.

[1338] Input: Received image data

[1339] Output: Converted text data

[1340] Step 4:

[1341] The server passes the OCR-converted text data to a generative AI model, which analyzes and scores the questions. The generative AI model determines the type of question (math, Japanese, science, etc.) and creates an appropriate prompt to generate an answer. For example, for the text "3 + 2 = ?", the AI ​​model generates the prompt "Please answer the following math problem: 3 + 2 = ?", and outputs the answer "5."

[1342] Input: Converted text data

[1343] Output: Scoring results

[1344] Step 5:

[1345] The server formats the generated scoring results into an easy-to-read format such as JSON and generates a response to the device. This response includes the correct answer and score for each question. For example, it generates JSON data such as "{'Question 1': 'Correct', 'Score': '5 / 5'}".

[1346] Input: Scoring results

[1347] Output: Formatted response data (JSON format)

[1348] Step 6:

[1349] The terminal analyzes the response of the score received from the server and displays it on the user interface. Specifically, it visually presents the score result to the user in the form of "Question 1: Correct (score 5 / 5)."

[1350] Input: Response data received from the server

[1351] Output: The grading results that are displayed to the user

[1352] This allows the system to provide fast and accurate automatic grading by simply allowing users to take a photo of their homework and submit it.

[1353] (Application example 1)

[1354] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1355] Conventional homework grading systems require parents or teachers to manually check homework, which is time-consuming and labor-intensive. Additionally, the use of smart devices creates operational hassles and the convenience of these devices is not fully utilized. The present invention aims to solve these issues by providing a system that can efficiently capture and review homework using smart glasses.

[1356] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1357] In this invention, the server includes a means for capturing an image of the homework, a means for transmitting the captured image to the server, a means for converting the image into text data in the server, a means for analyzing and grading the questions using a generative AI model based on the converted text data, a means for transmitting the grading results from the server to the terminal, a means for displaying the grading results on the terminal, and a means for capturing and checking the image of the homework using smart glasses, thereby enabling teachers and parents to quickly and accurately grade homework and check the results in real time.

[1358] "Means for capturing images of homework" refers to the device a user uses to capture images of their homework papers, such as a smartphone or a camera on smart glasses.

[1359] "Means for transmitting captured images to a server" refers to a function for transferring captured image data to a server via the Internet.

[1360] "Means for converting images into text data on a server" refers to a method for recognizing characters in an image on a server and extracting the character information as text data, and primarily uses optical character recognition (OCR) technology.

[1361] "Method of analyzing questions using a generative AI model based on converted text data and scoring" refers to a method in which text data converted by OCR is input into a generative AI model, which then analyzes the answers to the questions, determines whether they are correct, and scores them.

[1362] "Means for transmitting the scoring results from the server to the terminal" refers to a communication means for transmitting the scoring results on the server to the client device.

[1363] "Means for displaying the scoring results on the terminal" refers to a user interface for visually presenting the scoring results to the user on the client device.

[1364] "Means for taking and reviewing images of homework using smart glasses" refers to a function for taking a photo of a homework sheet using smart glasses and reviewing it on the screen.

[1365] The present invention is a system for automatically marking homework assignments by taking images of the assignments and uploading them to a server. This system mainly includes the following elements:

[1366] System Overview

[1367] 1. How to take a picture of your homework:

[1368] The smart glasses are used to take images of the homework. Specifically, the smart glasses' camera is used to take a clear image of the student's entire homework paper. This also includes a function to check and take a photo at hand if necessary.

[1369] 2. Means for sending captured images to the server:

[1370] The captured image is sent to a server using the smart glasses' internet connection function. Image data is transferred to the server using a communication protocol such as an HTTP request.

[1371] 3. On the server, convert the image to text data:

[1372] The server receives the captured image data and converts the character information in the image into text data using OCR technology (for example, Tesseract OCR).

[1373] 4. A method for analyzing questions and scoring them using a generative AI model based on the converted text data:

[1374] The text data converted by OCR is passed to a generative AI model (e.g., OpenAI GPT) to analyze and grade the homework questions. The generative AI model identifies the type of question (math, Japanese, science, etc.) and provides the appropriate answer.

[1375] 5. How to send the score results from the server to the device:

[1376] The server formats the analyzed and scored results and sends them to the client terminal in an easy-to-read format such as JSON.

[1377] 6. How to display the results on the device:

[1378] The client terminal (e.g., smart glasses or a smartphone) displays the received scoring results on a user interface, allowing the results to be viewed in real time.

[1379] Specific operation example

[1380] For example, if a third-grade student's math homework includes the problems "3 + 2 = ?" and "4 - 1 = ?", the teacher puts on the smart glasses and takes a photo of the homework paper. The image data is then automatically sent to a server, where the character information is converted into text data using OCR. When the text data "3 + 2 = ?" and "4 - 1 = ?" are input into the generative AI model, the AI ​​model will answer "5" and "3" respectively and determine whether they are correct.

[1381] Hardware and software used

[1382] Smart glasses (e.g. Google Glass)

[1383] Server (e.g. AWS)

[1384] OCR technology (e.g. Tesseract OCR)

[1385] Generative AI models (e.g., OpenAI GPT)

[1386] Prompt Sentence Examples

[1387] Homework character recognition results (math problems):

[1388] "Analyze the following questions and determine whether they are correct or incorrect: 3 + 2 = ?, 4 - 1 = ?."

[1389] In this way, the present invention allows teachers and parents to quickly and accurately grade homework and check the results in real time, significantly reducing the time and effort required for traditional manual grading and providing efficient educational support.

[1390] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1391] Step 1:

[1392] The user uses the smart glasses to take a clear photo of their homework paper. The inputs are the user's instruction to take a photo and the image of the homework paper captured by the camera. The output is the captured image data, which is temporarily stored locally. During this process, the smart glasses' camera function and the photo application are running.

[1393] Step 2:

[1394] The device (smart glasses) sends the captured image to a server via the Internet. The input is the image data generated in step 1. The output is the image data transferred to the server using an HTTP request. An appropriate communication protocol (e.g. HTTP) is used in this process.

[1395] Step 3:

[1396] The server receives the image data and uses optical character recognition (OCR) technology to recognize the text in the image and convert it into text data. The input is the image data transferred to the server. The output is the converted text data generated by OCR. Character recognition software such as Tesseract OCR is used in this process.

[1397] Step 4:

[1398] The server inputs the text data converted by OCR into a generative AI model to analyze and score the questions. The input is the text data generated by OCR. The output is the analysis results (answers and scoring information) generated by the generative AI model. In this process, a generative AI model (e.g., OpenAI GPT) is used to analyze the data based on prompts such as "Analyze the following questions and determine whether they are correct or incorrect: 3 + 2 = ?, 4 - 1 = ?."

[1399] Step 5:

[1400] The server formats the scoring results generated by the generative AI model and sends them to the terminal. The input is the analysis results of the generative AI model. The output is the scoring results formatted in an easy-to-read format such as JSON and sent to the client terminal. During this process, data is formatted and transferred using a communication protocol (e.g., HTTP).

[1401] Step 6:

[1402] The device (smart glasses or smartphone) displays the scoring results received from the server on a user interface. The input is the scoring result data sent from the server. The output is the scoring results displayed on the user interface, providing visual feedback to the user. This process uses UI components and display logic for display.

[1403] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1404] This invention is a system that automatically grades homework assignments by taking and uploading images, and also analyzes the user's emotions and provides feedback. The system aims to analyze the content of homework assignments to grade them quickly and accurately, reducing the burden on parents and recognizing the user's emotions to improve the quality of education.

[1405] System Overview

[1406] The system includes the following elements:

[1407] 1. How to take pictures of your homework

[1408] 2. Means of sending the captured image to the server

[1409] 3. A method for converting images into text data on the server

[1410] 4. A method for analyzing questions and scoring them using a generative AI model based on the converted text data

[1411] 5. Means for sending the scoring results from the server to the device

[1412] 6. Means for displaying the scoring results on the device

[1413] 7. Emotion engine that analyzes the user's facial expressions and recognizes their emotions

[1414] 8. Ability to record emotions recognized by the emotion engine as feedback

[1415] 9. A feature that provides interactive advice based on emotional data to improve learning progress and motivation

[1416] Description of the Examples

[1417] 1. User operations

[1418] Users take a photo of their homework using a device such as a smartphone or tablet. Specifically, they launch a camera app or a dedicated app and take a clear photo of the homework paper. Once the photo is taken, the user taps the "Upload Photo" button.

[1419] 2. Terminal Processing

[1420] The device temporarily stores the captured image, adds it to the upload list, and transmits it to a server over the Internet using an appropriate communication protocol, such as an HTTP request.

[1421] 3. Server Processing

[1422] The server receives image data sent from the device. The received image data is first converted into text data using optical character recognition (OCR) technology. For example, if the math problem "3 + 2 = ?" is included, it is recognized as text data.

[1423] 4. Analysis and scoring using generative AI models

[1424] The server passes the text data converted by OCR to the generative AI model. The generative AI model determines the type of question (math, Japanese, science, etc.), analyzes the question using the appropriate algorithm depending on the type, checks the accuracy of the answer, and scores it. For example, it recognizes "3 + 2 = 5" as the correct answer and calculates the score.

[1425] 5. Sending the score results

[1426] The server generates the scoring results, which are then formatted into an easy-to-read format such as JSON. The server then sends a response containing the scoring results to the device.

[1427] 6. Displaying the terminal

[1428] The device analyzes the score results received from the server and displays them appropriately on the user interface, for example, in the format "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)."

[1429] 7. Analysis by Emotion Engine

[1430] The device captures the user's facial expressions through a camera and sends them to the emotion engine. The emotion engine analyzes the user's facial expression data and recognizes their emotions. For example, if the user is smiling, it is recognized as "happy," and if they are frowning, it is recognized as "unhappy."

[1431] 8. Recording Emotional Data

[1432] The emotion engine records the user's recognized emotions as feedback data, such as their reaction to the results of their homework grading or their feelings about their learning progress.

[1433] 9. Interactive Advice

[1434] The emotion engine provides interactive advice to users based on the recorded emotion data. For example, if a user expresses dissatisfaction with their homework results, it displays a motivational message such as, "If you try a little harder, you'll do better next time."

[1435] Specific operation example

[1436] For example, if a third-grade elementary school student has two math homework problems, "3 + 2 = ?" and "4 - 1 = ?", the user can take a photo of the problem and receive the automatic scoring results through the process described above. After the scoring results are returned, the device sends the user's facial expression to the emotion engine for analysis. If the system recognizes the user's facial expression as "happy," it will provide feedback such as "Good job!"

[1437] Therefore, this system not only enables users to receive homework grading results quickly and accurately, but also improves learning progress and motivation.

[1438] The processing flow will be explained below.

[1439] Step 1:

[1440] The user takes a picture of the homework.

[1441] The user takes a picture of the entire homework paper using the camera of the device (smartphone or tablet). For example, the user takes a clear picture of the homework page using the camera function built into the application.

[1442] Step 2:

[1443] The device temporarily saves the captured image.

[1444] The terminal stores the images taken by the user in temporary memory and then adds them to the upload list.

[1445] Step 3:

[1446] A user uploads an image to the server.

[1447] The user taps the "upload photo" button in the application on the device to send the captured image to the server.

[1448] Step 4:

[1449] The terminal transmits the image data to the server.

[1450] The device sends the captured image of the homework to a server over the Internet using an appropriate communication protocol, such as an HTTP request.

[1451] Step 5:

[1452] The server receives the image data.

[1453] The server receives the image data sent from the device. This communication uses the SSL / TLS protocol to ensure security.

[1454] Step 6:

[1455] The server converts the image data into text data.

[1456] The server converts the received image data into text data using OCR (optical character recognition) technology. For example, the handwritten mathematical formula "3 + 2 = ?" in the image is converted into the string "3 + 2 = ?".

[1457] Step 7:

[1458] The server passes the text data to a generative AI model for analysis.

[1459] The server then passes the OCR-converted text data to the generative AI model, which then analyzes the question content. The generative AI model then identifies the type of question and performs the analysis using the appropriate algorithm.

[1460] Step 8:

[1461] A generative AI model analyzes the answers to the questions and scores them.

[1462] The generative AI model analyzes the answers based on the input text data, determines whether they are correct, and calculates the score. For example, "3 + 2 = 5" is recognized as the correct answer, and the score is calculated.

[1463] Step 9:

[1464] The server formats the scores.

[1465] The server converts the scoring results obtained from the generative AI model into an easy-to-read format such as JSON, for example, "Question 1: Correct (score 5 / 5)."

[1466] Step 10:

[1467] The server sends the scoring results to the terminal.

[1468] The server sends the formatted score to the device, using an appropriate protocol so that the device can receive it.

[1469] Step 11:

[1470] The terminal receives the scoring results.

[1471] The terminal receives the scoring results sent from the server and analyzes the data in JSON format, etc.

[1472] Step 12:

[1473] The terminal displays the scoring results on a user interface.

[1474] The device will then display the received scoring results appropriately on the user interface, such as "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)."

[1475] Step 13:

[1476] The device captures the user's facial expressions.

[1477] The device captures the user's facial expressions in real time via a camera, capturing their facial expressions while they are checking their homework results.

[1478] Step 14:

[1479] The device sends the facial expression data to the emotion engine.

[1480] The device analyzes the captured facial expression data and sends it to the emotion engine. This data includes key facial features of the user (e.g., eye shape, mouth movement, etc.).

[1481] Step 15:

[1482] The emotion engine analyzes the user's emotions.

[1483] The emotion engine analyzes the user's emotions based on the facial expression data sent. For example, a smile is judged to be "happy," and a frown is judged to be "unhappy."

[1484] Step 16:

[1485] The emotion engine records the emotion data.

[1486] The emotion engine recognizes and records the user's emotions as feedback data, including their reactions to the results of their homework grading and their feelings about their learning progress.

[1487] Step 17:

[1488] An emotion engine provides interactive advice.

[1489] The emotion engine provides interactive advice to users based on the recorded emotion data. For example, if a user expresses dissatisfaction with the results of their homework, it will display an encouraging message such as "Try harder next time!"

[1490] These steps allow users to not only receive the results of their homework assignments quickly and accurately, but also to properly analyze their feelings about the results and obtain feedback to improve their motivation to study.

[1491] Example 2

[1492] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1493] Traditional homework grading and learning progress management relies heavily on manual work, placing a heavy burden on parents and teachers in particular. Additionally, it is difficult to properly grasp changes in children's emotions as they learn, and there is a lack of feedback to maintain and improve their motivation. The purpose of this invention is to solve these problems by analyzing users' emotions and providing interactive feedback in addition to automatically grading homework.

[1494] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting images of homework into text data, a means for analyzing and grading questions using a generative model based on the converted text data, and a means for transmitting the grading results to the user interface device. This makes it possible to automatically grade homework and analyze the user's emotions to provide feedback.

[1495] "Means for taking images of homework" refers to a device equipped with a camera or a dedicated application that allows a user to take images of their homework.

[1496] The term "means for transmitting a captured image to an information processing device" refers to a protocol or a communication interface for transmitting captured image data to an information processing device using a network.

[1497] "Means for converting an image into text data in an information processing device" refers to software or hardware for converting image data into text information using optical character recognition technology.

[1498] "Means for analyzing questions using a generative model based on converted text data and grading" refers to an algorithm and processing system that uses a generative AI model to analyze text data, determine the answer to each question, and grade the answers.

[1499] The term "means for transmitting the scoring results from the information processing device to the user interface device" refers to a communication means for transmitting the analysis and scoring results to the user interface device via a network.

[1500] The term "means for displaying the scoring results in the user interface device" refers to a display screen and GUI (Graphical User Interface) for displaying the scoring results on the user interface device.

[1501] "Means for capturing a user's facial expression and analyzing emotions" refers to a device and software for capturing a user's facial expression data using a camera and sending it to an emotion analysis engine for analysis.

[1502] "Means for recording analyzed emotional data" refers to a system for analyzing a user's emotional data and then recording it in a database or storage.

[1503] "Means for providing interactive advice based on recorded emotional data" refers to scripts or algorithms that utilize the recorded emotional data to provide appropriate feedback or advice to the user.

[1504] This invention is a system that automatically grades homework assignments by taking and uploading images, and also analyzes the user's emotions and provides feedback. This system aims to analyze the content of homework assignments to grade them quickly and accurately, reducing the burden on parents and teachers, and improving the quality of education by recognizing the user's emotions.

[1505] Hardware and software used

[1506] 1. Hardware:

[1507] Device: A user device such as a smartphone or tablet that allows the user to take and upload images of their homework.

[1508] Server: A central information processing device that can analyze and score submitted images and generate feedback.

[1509] Camera: A camera built into the device that can capture the user's facial expressions.

[1510] 2. Software:

[1511] Camera app or dedicated app: An application used on a device that provides the ability to take and upload images of homework.

[1512] Optical Character Recognition (OCR) technology, such as Google Cloud Vision API, which converts images of homework assignments into text data.

[1513] A generative AI model, such as OpenAI GPT-4, that analyzes the converted text data and scores the questions.

[1514] Emotion engine: For example, Microsoft Azure Emotion API, which analyzes the user's facial expression data and recognizes their emotions.

[1515] Overview of program processing

[1516] User operations

[1517] Users open a camera app or a dedicated app on their smartphone or tablet and take a photo of their homework paper. Once they have taken the photo, they tap the "Upload Photo" button.

[1518] Terminal handling

[1519] The device temporarily stores the captured image and transmits it to a server via the Internet. For example, it generates an HTTP request and sends a POST request to the server's upload endpoint.

[1520] Server reception and image analysis

[1521] The server receives the image data sent from the device and converts it into text data using optical character recognition (OCR) technology. For example, if the data contains the math problem "3 + 2 = ?", it will recognize this as text data.

[1522] Analysis using generative AI models

[1523] The server passes the text data obtained by OCR to a generative AI model, which analyzes the questions and scores them. An example of a prompt is "Analyze the following text to determine the answer to each question and score it: '3 + 2 = ?'".

[1524] Formatting and sending the graded results

[1525] The server converts the scoring results returned by the generative AI model into JSON format and sends it to the device. For example, it converts it into the following format: {"Question 1": {"Answer": "5", "Score": 5}, "Question 2": {"Answer": "3", "Score": 5}}.

[1526] Display results on device and capture facial expressions

[1527] The device analyzes the JSON formatted scoring results received from the server and displays them on the user interface. For example, it displays "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)." It also captures the user's facial expressions with a camera and sends them to the emotion engine.

[1528] Emotion analysis and feedback using an emotion engine

[1529] The emotion engine analyzes the user's facial expression data and recognizes their emotions. For example, if the user is smiling, it is recognized as "happy," and if they are frowning, it is recognized as "unhappy." The analyzed emotion data is recorded, and based on the recorded data, it provides interactive advice such as "You did a great job!"

[1530] Specific operation example

[1531] For example, if a third-grade elementary school student has two math homework problems, "3 + 2 = ?" and "4 - 1 = ?", the user can take a photo of them and upload them to the system. The image is converted into text on the server, and an AI model recognizes "3 + 2 = 5" and "4 - 1 = 3" as the correct answers, awarding points for each. At the same time as the results are displayed on the device, the user's facial expressions are analyzed by an emotion engine, and feedback such as "You did a great job!" is provided.

[1532] This allows the system to not only provide fast and accurate homework grading, but also improve users' learning progress and motivation.

[1533] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1534] Step 1: User takes a picture of their homework

[1535] Users take pictures of their homework using a smartphone or tablet camera app or a dedicated app. The input is a physical copy of the homework, and the output is a digital image file stored on the device. For example, a user takes a picture of a third-grade math homework sheet.

[1536] Step 2: The device sends the image to the server

[1537] The device temporarily stores the captured image file and sends it to a server via the Internet. The input is the captured digital image file, and the output is the image data sent to the server. Specifically, the device generates an HTTP request and POSTs the image data to an endpoint.

[1538] Step 3: The server receives the image data

[1539] The server receives image data sent from the user's device and saves it in local storage. The input is the received image data, and the output is the saved image file. For example, the received "Homework_20231012.jpg" is saved.

[1540] Step 4: The server OCRs the image

[1541] The server converts the saved image file into text data using OCR technology. The input is the saved image file, and the output is the recognized text data. For example, the Google Cloud Vision API is used to extract the text "3 + 2 = ?" and "4 - 1 = ?".

[1542] Step 5: The server analyzes the text using the generated AI model

[1543] The server passes the text data obtained by OCR processing to a generative AI model, which analyzes the questions and scores them. The input is the text data and the analysis prompt, and the output is the scoring results. Specifically, the generative AI model (e.g., OpenAI GPT-4) analyzes the prompt, "Analyze the following text, determine the answer to each question, and score it: '3 + 2 = ?'."

[1544] Step 6: The server formats and sends the results

[1545] The server formats the scoring results returned by the generative AI model into JSON format and sends it to the device. The input is the scoring results from the generative AI model, and the output is formatted JSON data. For example, it is formatted as follows: {"Question 1": {"Answer": "5", "Score": 5}, "Question 2": {"Answer": "3", "Score": 5}}.

[1546] Step 7: Your device will display the results

[1547] The terminal parses the JSON data received from the server and displays it in the user interface in an appropriate format. The input is JSON data, and the output is the displayed scoring result. Specifically, "Question 1: Correct (score 5 / 5)" and "Question 2: Correct (score 5 / 5)" are displayed on the screen.

[1548] Step 8: The device captures the user's facial expression

[1549] The device captures the user's facial expression with a camera while checking the scoring result. The input is the user's facial expression, and the output is the captured image data. For example, the device captures the user's face using a front camera.

[1550] Step 9: The server analyzes facial expressions using the emotion engine

[1551] The server sends the captured facial expression data to the emotion engine for analysis. The input is facial image data, and the output is the emotion analysis results. For example, Microsoft Azure Emotion API is used to recognize emotions such as "happy" or "unhappy."

[1552] Step 10: The server records the emotion data and sends the feedback

[1553] The server records the recognized emotion data and provides feedback to the user. The input is the emotion analysis result, and the output is a feedback message. For example, a message such as "Good job!" is generated and sent to the device.

[1554] The above is the flow of specific processing steps. At each step, we have explained in detail how input data is processed and calculated to generate output data.

[1555] (Application example 2)

[1556] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1557] Currently, in many families, parents are burdened with the task of checking and grading their children's homework. It is also often difficult for children to maintain their motivation to study through homework. Furthermore, there is a lack of feedback that is sensitive to children's emotions, making it difficult to provide effective learning support.

[1558] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1559] In this invention, the server includes means for taking images of homework, means for transmitting the taken images to the server, means for converting the images to text data in the server, means for analyzing and grading the questions using a generative AI model based on the converted text data, means for transmitting the grading results from the server to the terminal, means for displaying the grading results on the terminal, means for capturing the user's facial expression with a camera and recognizing their emotions, means for recording the recognized emotions as feedback, and means for presenting interactive advice based on the emotion data. This makes it possible to efficiently support children's learning through automatic homework grading and emotion analysis, reduce the burden on parents, and increase children's motivation to learn.

[1560] "Means for taking pictures of homework" refers to a method in which a user takes a picture of the content written on a homework paper using a camera on a device such as a smartphone or tablet.

[1561] The "means for transmitting the captured images to the server" refers to a communication protocol or software that uploads the captured image data of the homework to a remote server via the Internet.

[1562] The "means of converting images to text data" is the process of converting the text information in the homework images into digital text using optical character recognition (OCR) technology.

[1563] The "means of analyzing questions and scoring using a generative AI model" refers to an artificial intelligence model that analyzes text data generated by OCR and scores questions by comparing actual answers with theoretically correct answers.

[1564] "Means for transmitting scoring results from the server to the terminal" refers to a system for returning the scoring result data from the server to the user's device (such as a smartphone or tablet) once scoring is complete.

[1565] The "means for displaying the scoring results on the terminal" refers to software or a device for displaying the scoring results received from the server as a user interface.

[1566] "Means for capturing a user's facial expression via a camera and recognizing emotions" refers to algorithms or software for capturing a user's facial expression via a camera and analyzing emotions from the facial expression data.

[1567] The "means for recording the recognized emotions as feedback" is a system for recording the analyzed emotion data in a database or the like and using it as feedback to the user.

[1568] The "means for presenting interactive advice based on emotional data" is a function for providing advice and feedback to the user according to the actual situation based on the recorded emotional data.

[1569] This invention shows a specific embodiment of a system that automatically grades homework by taking and uploading images, and further analyzes the user's emotions and provides feedback.

[1570] The entire system mainly consists of a terminal such as a smartphone, a server, and an emotion analysis engine.

[1571] First, the user takes a photo of the homework paper using the device's camera. The device is equipped with appropriate image capture functions to help the user capture the entire homework. The captured image is temporarily stored on the device and then sent to a remote server over the Internet. This process uses communication protocols such as HTTP requests.

[1572] The server converts the received image data into text using optical character recognition (OCR) technology. This text data includes homework questions and answers. The server uses software such as Tesseract OCR to extract text information from the image.

[1573] Next, a generative AI model is used to analyze and score the extracted text data. OpenAI's GPT-4 is used as the generative AI model. This model first determines the type of question (e.g., math, Japanese, science, etc.), and then evaluates the correctness of the answer using an appropriate algorithm depending on the type, and scores the answer. For example, it recognizes that the correct answer to the question "3 + 2 = ?" is "5," and awards points if the user's answer is correct.

[1574] The server organizes the scoring results as structured data, such as JSON format, and sends them to the device. The device then analyzes the received scoring results and displays them on the user interface. Specifically, it displays something like "Question 1: Correct (score 5 / 5)."

[1575] Furthermore, the device captures the user's facial expression with a camera and sends the data to an emotion analysis engine. The emotion analysis engine uses the FER (Facial Expression Recognition) library to analyze the user's facial expression and recognize their emotion. In this process, if the user is smiling, it is determined to be "happy," and if they are frowning, it is determined to be "unhappy."

[1576] The analyzed emotional data is recorded in a database, and interactive advice is generated to improve the user's learning progress and motivation. For example, if a user expresses dissatisfaction with their homework results, a message such as "If you try a little harder, you'll get a better result next time" is provided.

[1577] (Examples of specific examples and prompts)

[1578] Specifically, for a third-grade elementary school student's math homework, if there are two problems, "3 + 2 = ?" and "4 - 1 = ?", the user takes a photo of the problem and receives the automatic scoring results through the process described above. After the scoring results are returned, the device sends the user's facial expression to the emotion engine for analysis. If the system recognizes the user's facial expression as "happy," it will provide feedback such as "Good job!"

[1579] An example prompt is:

[1580] "Please grade the following homework: 3 + 2 = ?"

[1581] This invention not only allows users to receive the results of homework grading quickly and accurately, but also improves learning progress and motivation.

[1582] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1583] Step 1:

[1584] The user takes a photo of their homework paper with the device's camera. The input is the homework paper, and the output is the captured image file. This image file is temporarily saved on the device. The specific operation of the device is to launch the camera app or a dedicated app, and the user presses the shutter button to capture the image.

[1585] Step 2:

[1586] The device sends the captured image to the server. The input is the captured image file, and the output is the image data sent to the server. The device uploads the image data via the Internet using an HTTP request. Specifically, it issues a POST request to upload the file to the specified server URL.

[1587] Step 3:

[1588] The server converts the image data it receives into text data using OCR technology. The input is the image data sent to the server, and the output is text data. The server uses Tesseract OCR to extract the character information in the image as text. Specifically, it reads the image file and performs character recognition to generate text data.

[1589] Step 4:

[1590] The server uses a generative AI model to analyze the text data and grade the homework questions. The input is text data extracted by OCR, and the output is the graded results. The server calls a generative AI model (for example, OpenAI's GPT-4), creates a prompt, and requests grading. Specifically, the text data is sent to the generative AI model as a prompt in the format "Please grade the following homework: [text data]". The generative AI model returns a text containing the score as the analysis result.

[1591] Step 5:

[1592] The server sends the scoring results to the device. The input is the scoring result text received from the generative AI model, and the output is the scoring result sent to the device. The server formats the scoring results as structured data such as JSON format and sends them to the device as an HTTP response.

[1593] Step 6:

[1594] The terminal displays the marking results it has received. The input is the JSON format marking result data received from the server, and the output is the marking result displayed on the user interface. The terminal parses the JSON data and displays the marking results in a format that is easy for the user to understand. Specifically, it displays the results in a format such as "Question 1: Correct (score 5 / 5)."

[1595] Step 7:

[1596] The device captures the user's facial expression with a camera and recognizes the emotion. The input is the user's facial expression image captured by the device's camera, and the output is the recognized emotion data. The device uses the FER library to analyze the facial expression image and identify the user's emotion. Specifically, the device passes the captured image to the FER library and obtains the emotion data.

[1597] Step 8:

[1598] The server records the recognized emotions as feedback data. The input is the emotion data obtained by emotion analysis, and the output is the emotion data recorded in the database. The server saves the emotion data in the database and manages it together with the user's learning history.

[1599] Step 9:

[1600] The device presents interactive advice based on emotional data. The input is the recorded emotional data and the scoring results, and the output is an advice message displayed on the user interface. The device analyzes the emotional data and the scoring results, and generates and displays appropriate advice and feedback. Specifically, it displays encouraging messages such as, "If you try a little harder, you'll get better results next time."

[1601] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1602] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1603] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1604] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1605] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1606] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1607] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1608] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1609] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1610] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1611] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1612] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1613] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1614] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1615] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1616] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1617] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1618] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1619] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1620] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1621] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1622] The following is further disclosed regarding the above embodiment.

[1623] (Claim 1)

[1624] a means for capturing an image of the homework;

[1625] means for transmitting the captured image to a server;

[1626] a means for converting an image into text data in a server;

[1627] A means for analyzing and scoring questions using a generative AI model based on the converted text data;

[1628] A means for transmitting the scoring results from the server to the terminal;

[1629] A means for displaying the scoring results on the terminal;

[1630] A system including:

[1631] (Claim 2)

[1632] 10. The system of claim 1, wherein the means for converting into text data uses optical character recognition (OCR) technology.

[1633] (Claim 3)

[1634] The system of claim 1, wherein the generative AI model determines the type of problem and performs analysis according to the type.

[1635] "Example 1"

[1636] (Claim 1)

[1637] means for capturing an image of the homework on the terminal;

[1638] means for transmitting the captured image to a server;

[1639] a means for converting an image into text data in a server;

[1640] A means for analyzing and scoring questions using a generative AI model based on the converted text data;

[1641] A means for transmitting the scoring results from the server to the terminal;

[1642] and means for displaying the scoring results on the terminal.

[1643] (Claim 2)

[1644] 10. The system of claim 1, wherein the means for converting the image into text data uses optical character recognition (OCR) technology.

[1645] (Claim 3)

[1646] The system of claim 1, wherein the generative AI model determines the type of problem and performs analysis according to the type.

[1647] "Application Example 1"

[1648] (Claim 1)

[1649] a means for capturing an image of the homework;

[1650] means for transmitting the captured image to a server;

[1651] a means for converting an image into text data in a server;

[1652] A means for analyzing and scoring questions using a generative AI model based on the converted text data;

[1653] A means for transmitting the scoring results from the server to the terminal;

[1654] means for displaying the scoring results on the terminal;

[1655] means for capturing and reviewing images of the homework using smart glasses;

[1656] A system including:

[1657] (Claim 2)

[1658] 10. The system of claim 1, wherein the means for converting into text data uses optical character recognition (OCR) technology.

[1659] (Claim 3)

[1660] The system of claim 1, wherein the generative AI model determines the type of problem and performs analysis according to the type.

[1661] "Example 2: Combining Emotion Engines"

[1662] (Claim 1)

[1663] a means for capturing an image of the homework;

[1664] means for transmitting the captured image to an information processing device;

[1665] an information processing device, a means for converting an image into text data;

[1666] A means for analyzing questions using a generative model based on the converted text data and scoring the questions;

[1667] means for transmitting the scoring results from the information processing device to the user interface device;

[1668] means for displaying the scoring results in a user interface device;

[1669] A means for photographing a user's facial expression and analyzing emotions;

[1670] means for recording the analyzed emotion data;

[1671] A means of providing interactive advice based on recorded emotional data

[1672] A system including:

[1673] (Claim 2)

[1674] 10. The system of claim 1, wherein the means for converting into text data uses optical character recognition technology.

[1675] (Claim 3)

[1676] The system of claim 1, wherein the generative model determines the type of problem and performs analysis according to the type.

[1677] "Application example 2 when combining emotion engines"

[1678] (Claim 1)

[1679] a means for capturing an image of the homework;

[1680] means for transmitting the captured image to a server;

[1681] a means for converting an image into text data in a server;

[1682] A means for analyzing and scoring questions using a generative AI model based on the converted text data;

[1683] A means for transmitting the scoring results from the server to the terminal;

[1684] means for displaying the scoring results on the terminal;

[1685] A means for capturing a user's facial expression through a camera and recognizing the user's emotion;

[1686] a means for recording the perceived emotion as feedback;

[1687] A means of providing interactive advice based on sentiment data

[1688] A system including:

[1689] (Claim 2)

[1690] 10. The system of claim 1, wherein the means for converting into text data uses optical character recognition (OCR) technology.

[1691] (Claim 3)

[1692] The system of claim 1, wherein the generative AI model determines the type of problem and performs analysis according to the type. [Explanation of symbols]

[1693] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for capturing an image of the homework; means for transmitting the captured image to a server; a means for converting an image into text data in a server; A means for analyzing and scoring questions using a generative AI model based on the converted text data; A means for transmitting the scoring results from the server to the terminal; A means for displaying the scoring results on the terminal; A system including:

2. 2. The system of claim 1, wherein the means for converting to text data uses optical character recognition (OCR) technology.

3. The system according to claim 1, wherein the generative AI model determines the type of problem and performs analysis according to the type.

Images