System

Abstract

To provide a system.SOLUTION: A system includes: means for inputting an image containing a problem previously solved by a user through a messenger application, and analyzing the problem contained in the image; means for generating an answer to the analyzed problem using a generative AI model; means for generating an explanation for the answer by inputting a specific prompt sentence into the generative AI model; an emotion engine for recognizing the user's emotions; means for adjusting a level of details of the explanation based on the emotions; and means for providing the adjusted explanation to the user together with the problem.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional generative AI is weak in mathematics and lacks the ability to solve mathematical problems. In addition to providing answers, there is also a need for AI that can generate explanations for those answers. [Means for solving the problem]

[0005] The system of the present disclosure includes a means for inputting an image containing a problem that a user has solved in the past via a messenger application and analyzing the problem contained in the image, a means for generating an answer to the analyzed problem using a generative AI model, a means for generating an explanation of the answer by inputting a specific prompt sentence into the generative AI model, an emotion engine that recognizes the user's emotions, a means for adjusting the level of detail of the explanation based on the emotion, and a means for providing the adjusted explanation to the user together with the problem. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, a user sends a mathematical problem as an image to a messenger app BOT. The BOT analyzes the problem from the sent image and solves the analyzed problem. The answer is generated using generative AI and a conversion means in the Wolfram Language. Furthermore, an explanation for the answer is also generated by the generative AI. Specifically, when a user sends an image containing a mathematical problem to the BOT, the BOT reads the problem from the image and solves the problem. The answer is generated using generative AI and a conversion means in the Wolfram Language, and an explanation for the answer is also generated at the same time. The server may include means for inputting an image containing a problem previously solved by the user through the messenger application and analyzing the problem contained in the image.

[0029] "Example 2"

[0030] In another aspect of the present invention, a problem generation function can be added in the future. Specifically, the system can generate, for example, mathematics problems at an appropriate level based on the user's learning history and level of understanding. The server may include means for generating problems at a specific level based on the user's learning history and level of understanding of the problems. The problems may be from subjects or academic fields other than mathematics. This problem generation function allows users to solve problems tailored to their level, enabling them to study more efficiently.

[0031] The processing flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: The user sends an image containing a math problem to the messenger app bot.

[0034] Step 2: The bot analyzes the problem from the image sent. This analysis is done using image recognition technology.

[0035] Step 3: The analyzed problem is solved using generative AI and transformation tools in the Wolfram Language.

[0036] Step 4: An explanation for the answer is also generated by the generative AI and presented to the user.

[0037] "Example 2"

[0038] Step 1: The user sends an image containing a math problem to the messenger app bot.

[0039] Step 2: The bot analyzes the problem from the image sent. This analysis is done using image recognition technology.

[0040] Step 3: The analyzed problem is solved using generative AI and transformation tools in the Wolfram Language.

[0041] Step 4: An explanation for the answer is also generated by the generative AI and presented to the user.

[0042] Step 5: The system generates math problems at an appropriate level based on the user's learning history and level of understanding.

[0043] Example 1

[0044] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] Conventional mathematical problem-solving systems require users to input problems in text format, which is time-consuming, and they have the problem of not being able to handle handwritten or image-formatted problems. Furthermore, the generation of the answer and its explanation must be done separately, which makes it difficult for users to understand.

[0046] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0047] In this invention, the server includes: a means for a user to send a problem as an image to a messenger application; a means for transferring the sent image to the server; a means for the server to extract text data from the image using image analysis software; a means for inputting the extracted text data into a generative AI model to generate an answer to the problem; a conversion means for converting the generated answer into a mathematical formula or a calculation result; a means for generating an explanation for the answer using the generative AI model; and a means for returning the answer and explanation to the user. This allows users to easily solve mathematical problems in handwritten or image format and obtain both the answer and the explanation at the same time. The server may also include a means for generating an answer to the analyzed problem using the generative AI model; a means for generating an explanation for the answer by inputting a specific prompt sentence into the generative AI model; an emotion engine for recognizing the user's emotion; a means for adjusting the level of detail of the explanation based on the emotion; and a means for providing the adjusted explanation to the user together with the problem.

[0048] A "user" is any person or entity that intends to use the system to solve mathematical problems.

[0049] A "messenger application" is communication software that users use to send text and images.

[0050] An "image" is digital data that visually represents a mathematical problem.

[0051] A "server" is a computer system that receives and processes data sent by users.

[0052] "Image analysis software" is a program for extracting text data from images.

[0053] "Text data" is text information extracted by image analysis software.

[0054] A "generative AI model" is an artificial intelligence system that generates answers and explanations to questions based on input text data.

[0055] "Conversion means" refers to techniques or tools for converting generated answers into mathematical formulas or calculation results.

[0056] An "answer" is a solution to a user-submitted math problem.

[0057] An "explanation" is an explanation generated by a generative AI model to help understand the answer.

[0058] "Means for sending back" refers to the method by which the server sends back the generated answers and explanations to the user.

[0059] MODE FOR CARRYING OUT THE INVENTION

[0060] The present invention is a system that allows a user to send an image containing a mathematical problem to a messenger application, analyzes the problem from the image, and generates an answer and explanation. A specific embodiment of this system will be described below.

[0061] Hardware and software used

[0062] Server: A high-performance computer system that analyzes images, runs generative AI models, and transforms and returns data.

[0063] Device: The smartphone or PC used by the user to send images via a messenger application.

[0064] Messenger application: Communication software that allows users to send images.

[0065] Image analysis software: Extract text data from images, using Tesseract OCR as an example.

[0066] Generative AI model: As an example, we will use OpenAI's (registered trademark) GPT-4 (registered trademark) to generate answers and explanations for problems.

[0067] Conversion method: For example, we use the Wolfram Alpha API to convert the generated answers into mathematical formulas and calculation results.

[0068] System Operation

[0069] A user sends an image containing a math problem to a messenger application. The device transfers the image to a server. The server uses image analysis software to extract text data from the image. The extracted text data is input into a generative AI model to generate an answer to the problem. The generated answer is then converted into a mathematical formula or calculation result using a transformation method. The generative AI model also generates an explanation for the answer. Finally, the server returns the answer and explanation to the user.

[0070] Specific examples

[0071] For example, if a user sends an image of the equation "x^2 + 3x + 2 = 0" to a messenger application, the process would proceed as follows:

[0072] 1. The user takes a picture of the equation with their smartphone and sends it to a messenger application.

[0073] 2. The device transfers the image data to the server.

[0074] 3. The server uses Tesseract OCR to extract the text data "x^2 + 3x + 2 = 0" from the image.

[0075] 4. The server sends the prompt "Solve the equation x^2 + 3x + 2 = 0" to the generated AI model, which generates the answer "x = -1, x = -2".

[0076] 5. The server uses the Wolfram Alpha API to materialize the answer in mathematical form.

[0077] 6. The server sends the prompt "Explain the steps to solve the quadratic equation x^2 + 3x + 2 = 0" to the generated AI model and generates an explanation.

[0078] 7. The server returns the answer "x = -1, x = -2" to the user along with the explanation "This equation is a quadratic equation and can be solved using the formula. The formula gives us x = -1 and x = -2."

[0079] In this way, users can receive answers and explanations for math problems simply by submitting an image of the problem.

[0080] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0081] Step 1:

[0082] A user sends an image containing a math problem to a messenger application. Using a smartphone or PC, the user takes a photo of the handwritten equation "x^2 + 3x + 2 = 0" and sends it to the messenger application. The input is the image containing the math problem, and the output is the image sent to the messenger application.

[0083] Step 2:

[0084] The device transfers the image sent to the server. The device transfers the image data sent by the user to the server in an appropriate format (e.g., JPEG, PNG). The input is the image sent to the messenger application, and the output is the image data transferred to the server.

[0085] Step 3:

[0086] The server uses image analysis software to extract text data from the image. The server uses image analysis software (e.g., Tesseract OCR) to extract the text data "x^2 + 3x + 2 = 0" from the image. The input is the image data transferred to the server, and the output is the extracted text data.

[0087] Step 4:

[0088] The server inputs the extracted text data into a generative AI model to generate an answer to the problem. The server inputs the extracted text data into a generative AI model (e.g., OpenAI's GPT-4) and sends the prompt "Solve the equation x^2 + 3x + 2 = 0" to the generative AI model. The generative AI model generates the answer "x = -1, x = -2". The input is the extracted text data and the prompt, and the output is the generated answer.

[0089] Step 5:

[0090] The server materializes the answer using Wolfram Language transformations. The server materializes the generated answer as a formula or result using Wolfram Language transformations (e.g., Wolfram Alpha API). The input is the generated answer, and the output is the materialized formula or result.

[0091] Step 6:

[0092] The server generates an explanation for the answer using a generative AI model. The server sends the generative AI model a prompt, "Explain the steps to solve the quadratic equation x^2 + 3x + 2 = 0," and generates an explanation. The generative AI model generates an explanation, "This equation is a quadratic equation and can be solved using the solution formula. The solution formula gives x = -1 and x = -2." The input is the generated answer and prompt, and the output is the generated explanation.

[0093] Step 7:

[0094] The server returns the answer and explanation to the user. The server returns the generated answer "x = -1, x = -2" and explanation "This equation is a quadratic equation and can be solved using the solution formula. The solution formula gives x = -1 and x = -2." The input is the generated answer and explanation, and the output is the answer and explanation returned to the user.

[0095] (Application example 1)

[0096] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] Conventional math problem-solving systems require users to input questions, and it takes time to obtain the answers and explanations. In addition, the accuracy of the answers and the quality of the explanations are often not guaranteed, which can reduce the user's learning efficiency.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for a user to send a problem as an image to a messenger app BOT, means for analyzing the problem from the sent image, means for solving the analyzed problem, means for converting between a generative AI and a formula manipulation language to generate an answer, means for generating an explanation of the answer, means including an application to be installed on the smartphone, means for extracting text from an image using an image processing library, means for solving the problem using a formula manipulation language API, means for generating an explanation using a generative AI, and means for displaying the answer and explanation to the user. This allows a user to easily analyze a mathematical problem photographed with their smartphone and quickly obtain an accurate answer and high-quality explanation.

[0099] A "user" is an individual or group that uses the system to solve mathematical problems.

[0100] "Messenger App BOT" is an automatic response program that receives and analyzes mathematical problems sent by users as images.

[0101] "Means of sending as an image" refers to a method in which a user takes a photo of a math problem using a smartphone or other device and sends the image to a messenger app BOT.

[0102] The "means for analyzing problems from transmitted images" is a method for extracting text from images using an image processing library and recognizing the text as a mathematical problem.

[0103] The "means for solving the analyzed problem" is a method for solving the extracted mathematical problem using a computer algebra language API.

[0104] A "generative AI for generating solutions" is an artificial intelligence model used to generate answers to mathematical problems.

[0105] A "computer algebra language transformation means" is a method of using a computer algebra language to solve a mathematical problem.

[0106] "Means for generating explanations for answers" refers to a method for generating detailed explanations for answers using generative AI.

[0107] The "application installed on a smartphone" is software that allows users to take a photo of a math problem using their smartphone and receive the answer and explanation.

[0108] An "image processing library" is a software tool for extracting text from images.

[0109] A "computer algebra language API" is a programmatic interface used to solve mathematical problems.

[0110] "Means for generating explanations using generative AI" refers to a method for generating explanations for answers to mathematical problems using generative AI.

[0111] "Means for displaying answers and explanations to the user" refers to a method for displaying the generated answers and explanations on the user's smartphone.

[0112] The system for implementing this invention starts when a user takes a photo of a math problem using a smartphone and sends the image to a messenger app BOT. A specific embodiment of this system will be described below.

[0113] Hardware and software used

[0114] Hardware: Smartphone (with camera)

[0115] software:

[0116] Messenger app BOT

[0117] Image processing library (e.g. OpenCV)

[0118] OCR (Optical Character Recognition) libraries (e.g., pytesseract)

[0119] Mathematical processing language API (e.g. WolframAlpha API)

[0120] Generative AI models (e.g., GPT-3 (registered trademark))

[0121] Data processing and calculation

[0122] 1. Image Acquisition:

[0123] Users take a picture of a math problem using their smartphone camera and send the image to a messenger app BOT.

[0124] 2. Image Analysis:

[0125] The server converts the submitted image to grayscale using an image processing library (OpenCV) and extracts text from the image using an OCR library (pytesseract).

[0126] 3. Problem answer:

[0127] The extracted text is sent to a mathematical processing language API (WolframAlpha API), and the server obtains the answer.

[0128] 4. Description generation:

[0129] The server uses a generative AI model (GPT-3) to generate a detailed explanation for the obtained answer. Specifically, it inputs the following prompt sentence into the generative AI:

[0130] Example prompt sentence:

[0131] Math problem: x^2 + 2x + 1 = 0

[0132] Answer: x = -1

[0133] Description:

[0134] 5. Results display:

[0135] The server sends the generated answers and explanations to the user's smartphone and displays them through the application.

[0136] Specific examples

[0137] If a user photographs and submits the problem "x^2 + 2x + 1 = 0," the system operates as follows:

[0138] 1. Extract the text "x^2 + 2x + 1 = 0" from the image.

[0139] 2. Use the formula processing language API to obtain the answer "x = -1".

[0140] 3. Use generative AI to generate an explanation such as, "This equation is a quadratic equation, and using the formula for the solution, we get x = -1."

[0141] 4. Display the answer and explanation to the user.

[0142] In this way, users can easily analyze math problems photographed with their smartphones and quickly obtain accurate answers and high-quality explanations.

[0143] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0144] Step 1:

[0145] A user takes a picture of a math problem using the camera on their smartphone and sends the image to a messenger app BOT. The input is the image of the math problem that was taken, and the output is the image sent to the messenger app BOT.

[0146] Step 2:

[0147] The server converts the transmitted image to grayscale using an image processing library (OpenCV). The input is the image transmitted by the user, and the output is the image converted to grayscale. Specifically, the server reads the image using OpenCV functions and converts it to grayscale.

[0148] Step 3:

[0149] The server uses an OCR library (pytesseract) to extract text from grayscale images. The input is the image converted to grayscale, and the output is the extracted text. Specifically, it uses pytesseract's functions to read text from the image.

[0150] Step 4:

[0151] The server sends the extracted text to a mathematical expression processing language API (WolframAlpha API) and obtains the answer. The input is the extracted text, and the output is the answer obtained from the mathematical expression processing language API. Specifically, the server sends the text to the WolframAlpha API and receives the answer.

[0152] Step 5:

[0153] The server uses a generative AI model (GPT-3) to generate a detailed explanation for the obtained answer. The input is the answer obtained from the formula processing language API, and the output is the generated explanation. Specifically, the following prompt sentence is input to the generative AI:

[0154] Example prompt sentence:

[0155] Math problem: x^2 + 2x + 1 = 0

[0156] Answer: x = -1

[0157] Description:

[0158] Step 6:

[0159] The server sends the generated answers and explanations to the user's smartphone and displays them through the application. The input is the generated answers and explanations, and the output is the answers and explanations displayed on the user's smartphone. Specifically, the answers and explanations are sent through a messenger app BOT and displayed on the smartphone application.

[0160] Example 2

[0161] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0162] Conventional learning systems have difficulty generating questions based on a user's learning history and level of understanding, making it impossible to provide optimal questions for each individual user. Furthermore, efficient and accurate processing is required for analyzing questions answered by users and generating answers. This reduces the user's learning efficiency, making effective learning difficult.

[0163] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0164] In this invention, the server includes means for a user to send a question to a messenger application as an image, means for analyzing the question from the sent image, and means for answering the analyzed question, thereby making it possible to collect the user's learning history data, evaluate the user's level of understanding based on the analysis results, create prompt sentences to be input to the generative AI based on the level of understanding, generate questions using the generative AI, and provide the generated questions to the user.

[0165] "User" refers to an individual or organization that uses the system to study.

[0166] "Messenger application" refers to communication software that allows users to send and receive text and images.

[0167] "Image" refers to visual data that a user sends through a messenger application.

[0168] "Question" refers to a task or question provided for a user to answer.

[0169] "Analysis" refers to the process of reading and understanding the problem content from the transmitted image.

[0170] "Solution" refers to the answer to the question being analyzed.

[0171] "Generative AI" refers to AI technology that generates questions and answers based on a user's learning history and level of understanding.

[0172] "Programming language" refers to the computer language used by generative artificial intelligence to generate answers.

[0173] "Study history data" refers to data relating to questions that a user has answered in the past and the results of those answers.

[0174] "Analysis results" refers to information obtained after analyzing learning history data.

[0175] "Level of understanding" refers to an index that indicates how well a user can understand a question and answer it.

[0176] A "prompt sentence" refers to an instruction sentence that is input to a generative artificial intelligence when generating a problem.

[0177] "Problem generation" refers to the process by which a generative AI creates new questions based on prompts.

[0178] "Providing" refers to the act of displaying or transmitting the generated question to the user.

[0179] This invention is a system that allows a user to send a question as an image to a messenger application, analyzes the question from the image, generates an answer, and further generates new questions based on the user's learning history data. A specific embodiment of this system will be described below.

[0180] Hardware and software used

[0181] The server uses the following hardware and software:

[0182] Hardware: High-performance server machine

[0183] Software: generative AI models (e.g., GPT-4), database management systems, messenger applications

[0184] Data processing and calculation

[0185] 1. Collecting user learning history data

[0186] The server collects data on questions that users have previously solved and stores it in a database, including the type of question, the percentage of correct answers, and the time it took to solve the question.

[0187] 2. Analysis of learning history data

[0188] The server analyzes the collected learning history data, specifically, what answers the user gave to which questions and how long it took them to answer them.

[0189] 3. Assessment of understanding

[0190] The server evaluates the user's level of understanding based on the analysis results. For example, if the correct answer rate is high and the answer time is short, the server evaluates the user as "advanced," and if the correct answer rate is low and the answer time is long, the server evaluates the user as "beginner."

[0191] 4. Creating prompts for question generation

[0192] The server creates a prompt sentence to be input to the generative AI model based on the user's level of comprehension. For example, if the user's level of comprehension is evaluated as "intermediate," the server creates a prompt sentence such as "Generate intermediate level math problems."

[0193] 5. Problem generation using generative AI models

[0194] The server inputs a prompt into the generative AI model to generate a math problem. For example, based on the prompt "Generate an intermediate level math problem," the model generates the problem "Solve the following equation: 2x + 3 = 7."

[0195] 6. Providing generated questions

[0196] The server provides the generated questions to the user's device, and the user can progress with their learning by solving the questions provided.

[0197] Specific examples

[0198] Suppose the data of problems that a user has solved in the past is stored as follows:

[0199] Question 1: Correct answer, answer time 5 minutes

[0200] Question 2: Wrong answer, answer time 10 minutes

[0201] Question 3: Correct answer, answer time 7 minutes

[0202] The server analyzes this data and evaluates the user's level of understanding as "intermediate." It then feeds the following prompt to the generative AI model:

[0203] "If the user has an intermediate level of comprehension, generate intermediate level math problems."

[0204] Based on this prompt, the generative AI model generates intermediate-level math problems, such as "Solve the following equation: 2x + 3 = 7."

[0205] In this way, users can solve problems that are suited to their own level, enabling them to study more efficiently.

[0206] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0207] Step 1: Collecting user learning history data

[0208] The server collects data on questions that users have previously solved, and stores the results of questions that users have solved through the messenger application (such as correct answer rate and answer time) in a database.

[0209] Input: The result of the question answered by the user

[0210] Data processing: Data such as question type, correct answer rate, and answer time are extracted and saved in a database

[0211] Output: Learning history data stored in a database

[0212] Step 2: Analyzing learning history data

[0213] The server retrieves the user's learning history data from the database and analyzes it, specifically, analyzing what answers the user gave to which questions and how long it took them to answer.

[0214] Input: Learning history data stored in the database

[0215] Data calculation: Calculation of correct answer rate, answer time, evaluation of problem difficulty

[0216] Output: Analysis results (user correct answer rate, answer time, difficulty of questions)

[0217] Step 3: Assess understanding

[0218] The server evaluates the user's level of understanding based on the analysis results. For example, if the correct answer rate is high and the answer time is short, the server evaluates the user as "advanced," and if the correct answer rate is low and the answer time is long, the server evaluates the user as "beginner."

[0219] Input: Analysis results

[0220] Data arithmetic: Evaluation of comprehension based on correct answer rate and answer time

[0221] Output: User's level of understanding (advanced, intermediate, beginner, etc.)

[0222] Step 4: Create a prompt to generate questions

[0223] The server creates a prompt sentence to be input to the generative AI model based on the user's level of comprehension. For example, if the user's level of comprehension is evaluated as "intermediate," the server creates a prompt sentence such as "Generate intermediate level math problems."

[0224] Input: User comprehension

[0225] Data processing: Prompt generation based on comprehension

[0226] Output: A prompt to be input to the generative AI model

[0227] Step 5: Generate questions using a generative AI model

[0228] The server inputs a prompt into the generative AI model to generate a math problem. For example, based on the prompt "Generate an intermediate level math problem," the model generates the problem "Solve the following equation: 2x + 3 = 7."

[0229] Input: prompt statement

[0230] Data Computation: Problem Generation with Generative AI Models

[0231] Output: Generated math problem

[0232] Step 6: Providing generated questions

[0233] The server provides the generated questions to the user's device, and the user can progress with their learning by solving the questions provided.

[0234] Input: Generated math problem

[0235] Data processing: Convert data to fit the problem format

[0236] Output: The problem as it appears on the user's terminal

[0237] (Application example 2)

[0238] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0239] Conventional learning systems have had the problem of being unable to provide users with questions at an appropriate level based on their learning history and level of understanding, making it difficult for them to efficiently solve questions suited to their own level. Furthermore, they lacked the functionality to record the history of questions answered by users and dynamically update their learning level, making it difficult to effectively manage learning progress.

[0240] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0241] In this invention, the server includes means for a user to send a problem as an image to a messenger app, means for analyzing the problem from the sent image, means for solving the analyzed problem, means for converting between a generative AI and a computer algebra language to generate an answer, means for generating an explanation of the answer, means for generating problems of an appropriate level based on the user's learning history and level of understanding, means for presenting the generated problems to the user, and means for recording the user's answer history and updating the learning level. This allows the user to efficiently solve problems that match their learning level and effectively manage their learning progress.

[0242] "User" refers to an individual or group that uses the system to study.

[0243] A "messenger app" is a software application for sending and receiving messages such as text and images.

[0244] "Image" means a digital representation of visual information used by a user to submit a question.

[0245] A "problem" is a mathematical or other learning task for which a user seeks an answer.

[0246] "Analysis" is the process of recognizing the problematic content from the transmitted image and extracting it as digital data.

[0247] An "answer" is a correct answer to the problem being analyzed.

[0248] "Generative AI" is a system that uses artificial intelligence technology to generate answers and explanations to problems.

[0249] A "computational mathematics language" is a programming language for processing mathematical expressions on a computer.

[0250] "Explanation" is information that explains the process or method for arriving at the answer.

[0251] "Study history" is data that records questions that a user has answered in the past and their scores.

[0252] "Level of understanding" is an index that indicates how well a user understands a particular learning content.

[0253] "Level" refers to the level of difficulty according to the user's learning progress and ability.

[0254] "Presenting" is the act of displaying the generated questions and answers to the user.

[0255] "Answer history" is data that records questions that a user has answered in the past and the results of those answers.

[0256] "Study level update" is the process of dynamically changing a user's learning level based on the user's answer history.

[0257] A system for implementing this invention includes means for a user to send a question as an image to a messenger app, means for analyzing the question from the sent image, means for answering the analyzed question, means for converting between a generative AI and a mathematical processing language to generate an answer, means for generating an explanation of the answer, means for generating questions of an appropriate level based on the user's learning history and level of understanding, means for presenting the generated questions to the user, and means for recording the user's answer history and updating the learning level.

[0258] System Program Overview

[0259] The system is realized using the following hardware and software.

[0260] Hardware:

[0261] Smartphone

[0262] server

[0263] software:

[0264] Messenger app

[0265] Generative AI (e.g. GPT-3)

[0266] mathematical processing languages ​​(e.g., Wolfram Language)

[0267] JSON file (saves user data)

[0268] Processing flow

[0269] 1. User submits issue:

[0270] Users use a messenger app to send an image of a question that requires an answer.

[0271] 2. Image Analysis:

[0272] The server receives the transmitted image and uses image analysis technology to extract the problematic content as digital data.

[0273] 3. Generate answers to questions:

[0274] Using generative AI and mathematical processing languages, answers to the analyzed problems are generated.

[0275] 4. Generate explanations for answers:

[0276] Generative AI generates explanations that explain the process and methods used to arrive at the answer.

[0277] 5. Problem Generation:

[0278] The server generates new questions at an appropriate level based on the user's learning history and level of understanding.

[0279] 6. Problem Statement:

[0280] The generated questions are presented to the user through a messenger app.

[0281] 7. Record your answer history and update your learning level:

[0282] The user's answer history is recorded and their learning level is dynamically updated.

[0283] Specific examples

[0284] When a user first uses the MathMaster app, the app sets the user's level as "beginner." Each time the user solves a problem, the number of correct answers is counted, and once a certain number of correct answers are achieved, the level is updated to "intermediate" or "advanced."

[0285] Prompt Sentence Examples

[0286] Generate math problems at the appropriate level based on the user's learning history and level of understanding. The user's level can be "beginner," "intermediate," or "advanced." Provide the Python code that generates problems for each level.

[0287] In this way, the system can maximize the user's learning efficiency and effectively manage the learning progress.

[0288] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0289] Step 1:

[0290] The user sends the problem as an image to a messenger app. The user uses the messenger app on their smartphone to take a picture of the problem for which they want to answer and send it. The input is the image of the problem taken by the user, and the output is the image data sent to the server via the messenger app.

[0291] Step 2:

[0292] The server receives the image and performs image analysis. The server analyzes the received image data using an image analysis algorithm (e.g., OCR technology) and extracts the problem content as text data. The input is the image data sent by the user, and the output is the analyzed text data of the problem.

[0293] Step 3:

[0294] The server generates an answer to the analyzed problem. The server uses a generative AI (e.g., GPT-3) and a mathematical processing language (e.g., the Wolfram Language) to generate an answer based on the text data of the analyzed problem. The input is the text data of the analyzed problem, and the output is the generated answer.

[0295] Step 4:

[0296] The server generates an explanation that explains the process and method for arriving at the answer. Using generative AI, an explanation is generated that explains in detail the process and method for arriving at the answer. The input is the generated answer, and the output is an explanation of the answer.

[0297] Step 5:

[0298] The server generates new questions at an appropriate level based on the user's learning history and level of comprehension. The server references the user's learning history data and comprehension data and uses generative AI to generate new questions at an appropriate level. The input is the user's learning history data and comprehension data, and the output is the generated new questions.

[0299] Step 6:

[0300] The server presents the generated problem to the user. The generated new problem is sent to the user through a messenger app and presented to them. The input is the generated new problem, and the output is the problem presented to the user.

[0301] Step 7:

[0302] The server records the user's answer history and updates the learning level. The server records the history of questions answered by the user and dynamically updates the learning level. The input is the user's answer history data, and the output is the updated learning level.

[0303] In this way, the system can maximize the user's learning efficiency and effectively manage the learning progress.

[0304] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0305] "Example 1"

[0306] One embodiment of the present invention provides a system incorporating an emotion engine. The system includes a means for a user to send a problem to a messenger app BOT as an image, a means for analyzing the problem from the sent image, a means for solving the analyzed problem, a means for converting between a generative AI and the Wolfram Language to generate an answer, a means for generating an explanation for the answer, and an emotion engine that recognizes the user's emotions. The emotion engine recognizes the user's emotions and adjusts the difficulty of the problems based on the user's emotions. For example, if the user feels confused, the emotion engine lowers the difficulty of the problems. On the other hand, if the user feels joy or a sense of accomplishment, the emotion engine increases the difficulty of the problems. This improves the user's learning efficiency and satisfaction. The problem generation means of the server may adjust the difficulty of certain levels of problems based on the user's emotions.

[0307] "Example 2"

[0308] Furthermore, the emotion engine adjusts the level of detail in the answer explanation based on the user's emotions. For example, if the user feels confused, the emotion engine generates a more detailed explanation. On the other hand, if the user feels understood, the emotion engine generates a more concise explanation. This provides an appropriate explanation according to the user's level of understanding, enabling efficient learning.

[0309] The processing flow of each embodiment will be described below.

[0310] "Example 1"

[0311] Step 1: The user sends the problem as an image to the messenger app BOT.

[0312] Step 2: The system analyzes the problem from the image sent.

[0313] Step 3: The system answers the analyzed question.

[0314] Step 4: The system generates a solution using generative AI and the Wolfram Language.

[0315] Step 5: The system generates an explanation for the answer.

[0316] Step 6: The emotion engine recognizes the user's emotion.

[0317] Step 7: Adjust the difficulty of the questions based on the emotions recognized by the emotion engine.

[0318] "Example 2"

[0319] Step 1: The user sends the problem as an image to the messenger app BOT.

[0320] Step 2: The system analyzes the problem from the image sent.

[0321] Step 3: The system answers the analyzed question.

[0322] Step 4: The system generates a solution using generative AI and the Wolfram Language.

[0323] Step 5: The system generates an explanation for the answer.

[0324] Step 6: The emotion engine recognizes the user's emotion.

[0325] Step 7: Adjust the detail of the answer explanation based on the sentiment recognized by the sentiment engine.

[0326] Example 1

[0327] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0328] In conventional math problem-solving systems, when users submit problems as images, the problem analysis and answer generation may not be performed accurately. Furthermore, there is a lack of emotion recognition functionality to improve user learning efficiency and satisfaction, which means that the difficulty of the problem cannot be adjusted appropriately according to the user's emotions.

[0329] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0330] In this invention, the server includes means for a user to send a question as an image to a communication application, means for analyzing the question from the sent image, means for solving the analyzed question, generative artificial intelligence and mathematical language conversion means for generating an answer, means for generating an explanation for the answer, emotion analysis means for recognizing the user's emotion, and means for adjusting the difficulty of the question based on the recognized emotion. This makes it possible to accurately analyze the question from the image sent by the user and provide appropriate answers and explanations. Furthermore, adjusting the difficulty of the question according to the user's emotion can improve learning efficiency and satisfaction.

[0331] "User" refers to any person or entity that uses the system to solve mathematical problems.

[0332] "Communication application" refers to a software platform used by users to send messages and images.

[0333] "Image" refers to visual data containing a mathematical problem that a user sends through a communication application.

[0334] "Means of analysis" refers to the technology that recognizes mathematical problems from the transmitted images and extracts them as text data.

[0335] "Means of solving" refers to the technique of deriving an appropriate answer to the analyzed mathematical problem.

[0336] "Generative AI" refers to artificial intelligence technology that generates answers and explanations to mathematical problems.

[0337] "Mathematical language conversion means" refers to the technology that converts the generated answers into formats such as mathematical formulas and graphs.

[0338] "Means for generating an explanation for an answer" refers to technology that uses generative artificial intelligence to generate detailed explanations and solutions for the answer.

[0339] "Emotion analysis means" refers to technology for recognizing and analyzing a user's emotions.

[0340] The "means for adjusting the difficulty of a problem" refers to a technology for changing the difficulty of the next mathematical problem provided based on the user's emotions recognized by the emotion analysis means.

[0341] This invention is a system that allows users to send images containing mathematical problems via a communication application, analyzes the problems from the images, and generates answers and explanations. It also has the ability to recognize the user's emotions and adjust the difficulty of the problems based on those emotions.

[0342] Hardware and software used

[0343] Hardware

[0344] Server: A central processing unit for performing processes such as image analysis, answer generation, and emotion recognition.

[0345] Terminal: A device (smartphone, tablet, PC, etc.) through which a user uses a communication application.

[0346] software

[0347] Communication application: a messenger application that allows users to send images containing math problems.

[0348] OCR Engine: Use the Tesseract OCR engine to extract text from images.

[0349] Generative artificial intelligence: Using OpenAI's GPT-4 and other technologies, we generate answers and explanations for mathematical problems.

[0350] Mathematical language converters: Use the Wolfram Language to convert generated answers into mathematical formulas, graphs, and other formats.

[0351] Sentiment analysis engine: Software for recognizing and analyzing user emotions.

[0352] Data processing and calculation

[0353] Image analysis

[0354] The server receives the image submitted by the user and uses the Tesseract OCR engine to extract text from the image. For example, if the image submitted by the user contains the equation "x^2 + 3x + 2 = 0", the OCR engine will accurately extract this text.

[0355] Answer generation

[0356] The server inputs the extracted text into a generative artificial intelligence (GPT-4) to generate an answer to the problem. It then uses the Wolfram Language to convert the answer into a mathematical formula, graph, or other format. For example, the answer "x = -1 or x = -2" is generated.

[0357] Generate explanations for answers

[0358] The server uses generative artificial intelligence to generate explanations for the solution. The explanations include the solution to the problem and related concepts. For example, the server might generate an explanation like, "This equation can be solved by factorization. First, factor the equation as (x + 1)(x + 2) = 0. Then, set each factor equal to 0 to find the solution."

[0359] Emotion recognition and difficulty adjustment

[0360] The server uses an emotion analysis engine to recognize the user's emotions. It analyzes the user's messages and reactions (e.g., emojis and text) to determine whether the user is confused or happy. Based on the recognized emotions, it adjusts the difficulty of the next problem to be presented. For example, if it determines that the user is confused, it lowers the difficulty of the next problem to be presented. Conversely, if it determines that the user is feeling happy or a sense of accomplishment, it increases the difficulty of the next problem to be presented.

[0361] Specific examples

[0362] Prompt Sentence Examples

[0363] "Please answer the math problem contained in this image and provide an explanation for the answer."

[0364] Specific operation flow

[0365] 1. A user sends an image containing "x^2 + 3x + 2 = 0" through a communication application.

[0366] 2. The server receives the image and uses the Tesseract OCR engine to extract the text "x^2 + 3x + 2 = 0".

[0367] 3. The server uses generative artificial intelligence (GPT-4) and the Wolfram Language to generate the answer "x = -1 or x = -2".

[0368] 4. The server uses generative artificial intelligence to generate an explanation of the answer, including the factorization steps.

[0369] 5. The server uses an emotion analysis engine to analyze the user's emotions, and if it determines that the user is confused, it lowers the difficulty of the next question it presents.

[0370] 6. If the user answers the next question and the sentiment analysis engine recognizes the user's joy or sense of accomplishment, the difficulty of the next question provided will be increased.

[0371] This system allows users to study mathematical problems efficiently.

[0372] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0373] Step 1:

[0374] A user uses a communication application to send an image containing a math problem.

[0375] Input: An image containing a math problem taken or saved by the user.

[0376] Output: Image data sent to a server through a communication application.

[0377] Specific operation: The user uses a smartphone or PC to take a photo of a piece of paper with a math problem written on it, then uploads and sends the image to a communication application.

[0378] Step 2:

[0379] The server receives the image sent by the user.

[0380] Input: Image data sent from a communication application.

[0381] Output: Image data stored in the server's storage.

[0382] Specific operation: The server receives image data through the API of the communication application and temporarily stores it in storage.

[0383] Step 3:

[0384] The server analyzes the image to find the math problem.

[0385] Input: Image data stored on the server.

[0386] Output: Text data extracted from the image.

[0387] What it does: The server uses the Tesseract OCR engine to parse the image data and extract text containing the math problem, for example, "x^2 + 3x + 2 = 0."

[0388] Step 4:

[0389] The server answers the math questions based on the extracted text.

[0390] Input: Text data extracted by the OCR engine.

[0391] Output: Answer data generated by generative artificial intelligence.

[0392] How it works: The server inputs the extracted text into a generative artificial intelligence (GPT-4) to generate an answer to the problem. It then uses the Wolfram Language to convert the answer into a mathematical formula, graph, or other format. For example, the answer "x = -1 or x = -2" is generated.

[0393] Step 5:

[0394] The server generates an explanation for the answer.

[0395] Input: Answer data generated by generative artificial intelligence.

[0396] Output: Explanation data of the answer generated by the generative artificial intelligence.

[0397] How it works: The server uses generative artificial intelligence to generate an explanation for the solution. The explanation includes the solution to the problem and related concepts. For example, the server might generate an explanation like, "This equation can be solved by factorization. First, factor the equation as (x + 1)(x + 2) = 0. Then, set each factor equal to 0 to find the solution."

[0398] Step 6:

[0399] The server recognizes the user's emotions.

[0400] Input: User message and response data.

[0401] Output: Sentiment data recognized by the sentiment analysis engine.

[0402] What it does: The server uses a sentiment analysis engine to analyze the user's messages and reactions (e.g., emojis and text) to determine whether the user is confused or happy.

[0403] Step 7:

[0404] The difficulty of the questions is adjusted based on the emotions recognized by the server.

[0405] Input: Emotion data recognized by the sentiment analysis engine.

[0406] Output: Adjusted problem difficulty data.

[0407] Specific operation: The server adjusts the difficulty of the next problem based on the recognized emotion. For example, if it determines that the user is confused, it will lower the difficulty of the next problem. Conversely, if it determines that the user is feeling joy or a sense of accomplishment, it will increase the difficulty of the next problem.

[0408] (Application example 1)

[0409] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0410] Conventional math problem-solving systems can provide answers and explanations for problems submitted by users, but they have the problem of not being able to adjust the difficulty of the problems according to the user's learning situation or emotions. Also, when a user is confused or, conversely, when they get bored with problems that are too easy, it is difficult to provide problems of an appropriate difficulty.

[0411] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0412] In this invention, the server includes means for a user to send a question as an image to a messenger app BOT, means for analyzing the question from the sent image, means for solving the analyzed question, means for converting between a generative AI and a formula processing language to generate an answer, means for generating an explanation of the answer, means for recognizing the user's emotions, and means for adjusting the difficulty of the questions based on the recognized emotions. This makes it possible to provide questions of an appropriate level of difficulty according to the user's learning situation and emotions.

[0413] A "user" is an individual or entity that uses the system to solve mathematical problems.

[0414] A "messenger app BOT" is an automated program that receives mathematical problems sent by users as images, analyzes them, and provides answers.

[0415] "Means to send as an image" is a function that allows users to send math problems in image format to a messenger app BOT.

[0416] The "means for analyzing problems from transmitted images" is a function for extracting mathematical problems as text from received images and analyzing them.

[0417] The "means for solving the analyzed problem" is a function for generating an answer to the analyzed mathematical problem.

[0418] "Generative AI" is a system that uses artificial intelligence technology to generate answers and explanations for mathematical problems.

[0419] A "computer algebra language" is a programming language used to solve mathematical problems.

[0420] "Means for generating explanations for answers" is a function for generating detailed explanations for answers using generative AI.

[0421] The "means for recognizing the user's emotions" is a function for detecting and recognizing emotions from the user's facial expressions and actions.

[0422] The "means for adjusting the difficulty of the question based on the recognized emotion" is a function for changing the difficulty of the next question to be posed in accordance with the emotion of the user.

[0423] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[0424] System Configuration

[0425] The system includes a messenger app BOT that allows users to send math problems as images, an image analysis engine that analyzes the problems from the images, a generative AI and a mathematical processing language that solves the analyzed problems, a generative AI that generates explanations for the answers, an emotion recognition engine that recognizes the user's emotions, and a difficulty adjustment engine that adjusts the difficulty of the problems based on the recognized emotions.

[0426] Hardware and software used

[0427] Hardware: Servers, user devices (smartphones, tablets, PCs, etc.)

[0428] software:

[0429] Image analysis engine: OpenCV, Tesseract OCR

[0430] Generative AI: Hugging Face transformers library

[0431] Mathematical Computation Language: WolframAlpha API

[0432] Emotion Recognition Engine: EmotionRecognizer Library

[0433] Data processing and calculation

[0434] 1. Image analysis: Images containing mathematical problems sent from the user's device are analyzed using an image analysis engine (OpenCV and Tesseract OCR) on the server and extracted as text data.

[0435] 2. Answer generation: The extracted text data is input into a mathematical processing language (WolframAlpha API) to generate an answer.

[0436] 3. Explanation generation: Using generative AI (Hugging Face's transformers library), we generate detailed explanations for the answers.

[0437] 4. Emotion recognition: The user's facial expressions and behavior are analyzed using an emotion recognition engine (EmotionRecognizer library) to recognize the user's emotions.

[0438] 5. Difficulty Adjustment: Based on the recognized emotions, the difficulty of the next question is adjusted by the difficulty adjustment engine.

[0439] Specific examples

[0440] A user sends an image containing a math problem to a messenger app BOT using their smartphone. The server uses an image analysis engine to extract the problem as text from the image and generates an answer using a formula processing language. The generative AI generates an explanation for the answer using a prompt like the following:

[0441] Question: Solve the equation 2x + 3 = 7.

[0442] Answer: x = 2

[0443] Explain the solution.

[0444] The generated description might look something like this:

[0445] To solve the equation 2x + 3 = 7, we first subtract 3 from both sides to get 2x = 4. Then, we divide both sides by 2 to find x = 2.

[0446] Furthermore, the emotion recognition engine analyzes the user's facial expressions and behavior, and if the user is confused, it lowers the difficulty of the next question, and if the user is happy, it raises the difficulty, thus improving the user's learning efficiency and satisfaction.

[0447] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0448] Step 1:

[0449] A user sends an image containing a math problem to a messenger app BOT using a smartphone. The input is the image of the math problem taken by the user, and the output is the image data sent to the server.

[0450] Step 2:

[0451] The server analyzes the received image data using an image analysis engine (OpenCV and Tesseract OCR). The input is the transmitted image data, and the output is the text data extracted from the image. Specifically, the image is read using OpenCV, and the text in the image is extracted using Tesseract OCR.

[0452] Step 3:

[0453] The server inputs the extracted text data into a mathematical processing language (WolframAlpha API) to generate an answer. The input is the extracted text data, and the output is the answer to the mathematical problem. Specifically, the server sends the text data to the WolframAlpha API and obtains the returned answer.

[0454] Step 4:

[0455] The server uses a generative AI (Hugging Face's transformers library) to generate a detailed explanation of the answer. The input is a math problem and its answer, and the output is an explanation of the answer. Specifically, the server inputs the following prompt into the generative AI, which then generates an explanation:

[0456] Question: Solve the equation 2x + 3 = 7.

[0457] Answer: x = 2

[0458] Explain the solution.

[0459] Step 5:

[0460] The server analyzes the user's facial expressions and behavior using an emotion recognition engine (EmotionRecognizer library) to recognize the user's emotions. The input is the user's facial expressions and behavior data, and the output is the recognized emotion. Specifically, the server uses the EmotionRecognizer library to analyze the user's facial expressions and behavior and identify the emotion.

[0461] Step 6:

[0462] The server uses a difficulty adjustment engine to adjust the difficulty of the next question based on the recognized emotion. The input is the recognized emotion data, and the output is the adjusted difficulty of the question. Specifically, the difficulty is set to be lowered if the user is confused, and to be higher if the user is happy.

[0463] Step 7:

[0464] The server selects the next question based on the adjusted difficulty and provides it to the user. The input is the adjusted difficulty of the question, and the output is the next question to be asked. Specifically, the server selects a question of appropriate difficulty from the question database and sends it to the user.

[0465] Example 2

[0466] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0467] Conventional learning support systems lacked the ability to generate appropriate questions based on the user's learning history and level of understanding, which led to problems with reduced learning efficiency. They also lacked the ability to adjust the level of detail in explanations according to the user's emotional state, meaning they were unable to provide appropriate support according to the user's level of understanding.

[0468] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0469] In this invention, the server includes: means for a user to send a question as an image to a messenger application; means for analyzing the question from the sent image; means for solving the analyzed question; means for converting between a generative artificial intelligence and a computer algebra language to generate an answer; means for generating an explanation for the answer; means for collecting a user's learning history data; means for analyzing the collected learning history data; means for generating a prompt sentence based on the analysis result; means for inputting the generated prompt sentence into the generative artificial intelligence to generate a question; means for collecting a user's emotional data; means for analyzing the collected emotional data; means for adjusting the level of detail of the explanation based on the analysis result; and means for providing the question and explanation to the user. This enables appropriate questions to be generated based on the user's learning history and level of understanding, and the level of detail of the explanation to be adjusted according to the user's emotional state. The server may also include a question generation means for generating questions of a specific level based on the user's learning history and level of understanding of the question.

[0470] "User" refers to an individual who uses the system to learn.

[0471] "Messenger application" refers to software for sending and receiving messages such as text and images.

[0472] "Image" refers to digital data containing visual information.

[0473] "Question" refers to a task or question for the user to answer.

[0474] "Analysis" refers to the process of examining data in detail to understand its structure and meaning.

[0475] "Answer" refers to the correct answer to a question.

[0476] "Generative artificial intelligence" refers to algorithms and models that use natural language processing and machine learning to generate text and data.

[0477] A "computational mathematics language" refers to a programming language used to perform mathematical calculations and analysis.

[0478] "Explanation" refers to a detailed explanation of the question and answer.

[0479] "Learning history data" refers to data relating to learning activities that a user has performed in the past.

[0480] A "prompt sentence" refers to an instruction sentence to be input into generative artificial intelligence.

[0481] "Emotion data" refers to data that indicates the user's emotional state.

[0482] "Level of detail" refers to the specificity and depth of the explanation.

[0483] MODE FOR CARRYING OUT THE INVENTION

[0484] This invention provides a learning support system that generates appropriate questions based on a user's learning history and level of understanding, and adjusts the level of detail of explanations according to the user's emotional state. Specific embodiments of this system are described below.

[0485] Hardware and software used

[0486] server

[0487] The server uses the following hardware and software:

[0488] Hardware: A server machine with a powerful processor, sufficient memory, and ample storage.

[0489] Software: Python, TENSORFLOW®, OpenCV, Google® Cloud Speech-to-Text API, MySQL®

[0490] Terminal

[0491] A terminal is a device used by a user that uses the following hardware and software:

[0492] Hardware: A personal computer or smartphone with a camera, microphone, and display

[0493] Software: Messenger application

[0494] Data processing and calculation

[0495] Collecting user learning history data

[0496] The server collects learning history data such as the percentage of correct answers for questions the user has answered in the past, the time it took to answer, and the types of questions they got wrong. A database (MySQL) is used for this purpose.

[0497] Analysis of learning history data

[0498] The server analyzes the collected learning history data and evaluates the user's level of understanding. This is done using Python and TensorFlow. Specifically, it runs an algorithm that quantifies the user's level of understanding based on the user's accuracy rate and answer time.

[0499] Generate prompt statement

[0500] Based on the analysis results, the server generates a prompt sentence to be input to the generative AI model. For example, it generates the following prompt sentence:

[0501] "User is studying 8th grade math and struggled with equation problems on a recent test. Generate equation problems at the appropriate level for the user's understanding."

[0502] Problem generation using generative AI models

[0503] The server inputs the generated prompt sentence into a generative AI model (e.g., GPT-4) to generate an appropriate mathematical problem. The generative AI model generates a problem based on the prompt sentence.

[0504] Collecting user emotion data

[0505] The device collects facial and voice data from the user using a camera and microphone. While the user is solving the problem, the device captures facial expressions with the camera and records voice with the microphone.

[0506] Emotional Data Analysis

[0507] The server analyzes the collected emotion data and assesses the user's emotional state using OpenCV and the Google Cloud Speech-to-Text API. The server analyzes facial expression data to determine whether the user is confused and audio data to assess the user's tone of voice.

[0508] Adjusting the level of detail in the explanation

[0509] The server adjusts the level of detail of the explanation based on the analysis of the emotion data. If it determines that the user is confused, it generates a detailed explanation, and if it determines that the user understands, it generates a concise explanation.

[0510] Providing questions and explanations

[0511] The server provides the generated questions and the adjusted explanations to the user. The server sends the questions and explanations to the terminal, which displays them to the user.

[0512] Specific examples

[0513] For example, if the user is studying eighth-grade math, the server might input the following prompts into the generative AI model:

[0514] "User is studying 8th grade math and struggled with equation problems on a recent test. Generate equation problems at the appropriate level for the user's understanding."

[0515] Based on this prompt, the generative AI model generates the following question:

[0516] "Solve the following equation: 3x + 5 = 20"

[0517] Also, if the emotion engine determines that the user is confused by the issue, the server will provide a detailed explanation, such as:

[0518] "First, subtract 5 from both sides of the equation. Then divide what's left, 3x, by 20 minus 5. This gives you the value of x."

[0519] On the other hand, if the server determines that the user does understand, it provides a brief explanation such as:

[0520] "Solve the equation to find the value of x."

[0521] In this way, the system supports efficient learning by providing appropriate questions and explanations according to the user's level of understanding and emotional state.

[0522] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0523] Step 1:

[0524] Collecting user learning history data

[0525] The server retrieves learning history data from a database (MySQL), such as the accuracy rate of questions the user has answered in the past, the time it took to answer, and the types of questions they got wrong.

[0526] Input: User ID

[0527] Output: Learning history data

[0528] Specific operation: Every time a user logs in, the server queries and obtains the user's learning history data from the database.

[0529] Step 2:

[0530] Analysis of learning history data

[0531] The server analyzes the collected learning history data and evaluates the user's level of understanding, using Python and TensorFlow.

[0532] Input: Learning history data

[0533] Output: User's understanding rating

[0534] Specific operation: The server executes an algorithm that quantifies the user's level of understanding based on the user's accuracy rate and answer time.

[0535] Step 3:

[0536] Generate prompt statement

[0537] Based on the analysis results, the server generates a prompt sentence to be input into the generative AI model.

[0538] Input: User's comprehension rating

[0539] Output: prompt statement

[0540] Specific behavior: The server generates a prompt sentence based on the user's level of understanding, such as: "The user is studying eighth-grade math and has struggled with equation problems on recent tests. Please generate equation problems at an appropriate level based on the user's level of understanding."

[0541] Step 4:

[0542] Problem generation using generative AI models

[0543] The server inputs the generated prompt sentences into a generative AI model (e.g., GPT-4) to generate appropriate math problems.

[0544] Input: prompt statement

[0545] Output: Generated problem

[0546] Specific operation: The server sends a prompt to the generative AI model and receives the generated problem.

[0547] Step 5:

[0548] Collecting user emotion data

[0549] The device collects facial and voice data from the user using a camera and microphone.

[0550] Input: Real-time user data (facial expressions, voice)

[0551] Output: Emotion data

[0552] Specific operation: While the user is solving the problem, the device captures facial expressions with a camera and records audio with a microphone.

[0553] Step 6:

[0554] Emotional Data Analysis

[0555] The server analyzes the collected emotion data and assesses the user's emotional state using OpenCV and the Google Cloud Speech-to-Text API.

[0556] Input: Emotion data

[0557] Output: User's emotional state rating

[0558] Specific operation: The server analyzes facial expression data to determine whether the user is confused, and analyzes voice data to evaluate the user's tone of voice.

[0559] Step 7:

[0560] Adjusting the level of detail in the explanation

[0561] The server adjusts the level of detail of the commentary based on the results of analyzing the emotional data.

[0562] Input: User's emotional state rating

[0563] Output: Adjusted commentary

[0564] Specific behavior: If the server determines that the user is confused, it generates a detailed explanation, and if it determines that the user understands, it generates a concise explanation.

[0565] Step 8:

[0566] Providing questions and explanations

[0567] The server provides the generated questions and tailored explanations to the user.

[0568] Input: Generated questions, adjusted explanations

[0569] Output: The problem and explanation provided to the user

[0570] Specific operation: The server sends the problem and explanation to the terminal, which then displays it to the user.

[0571] (Application example 2)

[0572] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0573] Conventional learning systems do not generate questions based on the user's learning history or level of understanding, nor do they adjust explanations based on the user's emotions, making efficient learning difficult. In particular, they are unable to provide appropriate feedback when the user is confused or has already understood the material, which reduces the effectiveness of learning.

[0574] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0575] In this invention, the server includes means for a user to send a question as an image to a messenger app BOT, means for analyzing the question from the sent image, means for solving the analyzed question, means for converting between a generative AI and a formula processing language for generating an answer, means for generating an explanation of the answer, means for generating questions based on the user's learning history and level of understanding, and means for adjusting the level of detail of the explanation of the answer based on the user's emotions. This makes it possible to generate appropriate questions according to the user's learning history and level of understanding, and to adjust the explanation according to the user's emotions.

[0576] "User" refers to an individual who uses the system to learn.

[0577] "Messenger app bot" refers to a messaging application that has an automated response function that allows users to send images of their problems.

[0578] "Image" refers to a data file containing visual information that a User sends to a Messenger App BOT.

[0579] "Problem" refers to a mathematical or other learning task for which a user seeks an answer.

[0580] "Analysis" refers to the process of recognizing and understanding the problem content from the transmitted image.

[0581] "Answer" refers to the correct response to the analyzed problem.

[0582] "Generative AI" refers to artificial intelligence technology that generates answers and explanations to problems.

[0583] A "computational mathematics language" refers to a programming language for processing mathematical expressions on a computer.

[0584] "Explanation" refers to explanations or supplementary information for the answer.

[0585] "Study history" refers to data that records questions that a user has solved in the past and the answers they gave.

[0586] "Level of understanding" refers to an index that indicates how well the user understands the problem.

[0587] "Emotion" refers to the psychological state that a user feels when solving a problem.

[0588] "Level of detail" refers to the degree to which the content of the commentary is explained in detail.

[0589] A system for implementing this invention includes means for a user to send a question as an image to a messenger app BOT, means for analyzing the question from the sent image, means for solving the analyzed question, means for converting between a generative AI and a mathematical processing language to generate an answer, means for generating an explanation for the answer, means for generating a question based on the user's learning history and level of understanding, and means for adjusting the level of detail of the explanation for the answer based on the user's emotions.

[0590] System configuration

[0591] 1. User device: The device through which the user submits the question, such as a smartphone or tablet.

[0592] 2. Messenger app bot: A messaging application with an automated response function that allows users to send images of their problems.

[0593] 3. Server: A central processing unit for analyzing problems, generating answers, generating explanations, generating questions, and sentiment analysis.

[0594] Software used

[0595] Generative AI: Artificial intelligence technology for generating answers and explanations to problems.

[0596] Computer algebra language: A programming language for processing mathematical expressions on a computer.

[0597] Sentiment analysis engine: Software for analyzing user emotions.

[0598] Learning history management system: A system that records the user's learning history and uses it to generate questions next time.

[0599] Processing flow

[0600] 1. Sending the problem: The user sends the problem image to the messenger app BOT using their smartphone.

[0601] 2. Problem analysis: The server receives the image and analyzes the problem. Image recognition technology is used for the analysis.

[0602] 3. Answer generation: The generative AI generates an answer to the analyzed problem. It uses a mathematical processing language to derive an accurate answer.

[0603] 4. Explanation generation: Generative AI generates an explanation for the answer. The level of detail of the explanation is adjusted by a sentiment analysis engine.

[0604] 5. Question generation: New questions are generated based on the user's learning history and level of understanding.

[0605] 6. Sentiment analysis: Analyze the user's emotions in real time while answering questions and provide appropriate feedback.

[0606] Specific examples

[0607] For example, if the sentiment analysis engine determines that a user is "confused" while solving a math problem, it will provide a detailed explanation, whereas if it determines that the user "understands," it will provide a concise explanation.

[0608] Prompt Sentence Examples

[0609] Generate math problems at the appropriate level based on the user's learning history and understanding, and adjust the level of detail in the explanations depending on the user's emotions: provide detailed explanations if the user is confused, and concise explanations if the user understands.

[0610] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0611] Step 1:

[0612] The user sends the image in question to the messenger app BOT using their smartphone.

[0613] Input: A user-taken image of the problem.

[0614] Output: Image data sent to the messenger app BOT.

[0615] How it works: The user takes a picture of the problem with their smartphone camera and sends it to the messenger app BOT, which then transfers the image data to the server.

[0616] Step 2:

[0617] The server receives the transmitted image and analyzes the problem.

[0618] Input: Image data transferred from the messenger app BOT.

[0619] Output: The parsed text data of the problem.

[0620] Specific operation: The server uses image recognition technology (e.g., OCR) to extract the problem text from the image data. The extracted text data is sent to the problem analysis module.

[0621] Step 3:

[0622] Generative AI generates answers to the analyzed problems.

[0623] Input: The parsed text data of the problem.

[0624] Output: Text data of the generated answer.

[0625] Specific operation: The server uses generative AI to generate answers to the analyzed problems, and derives accurate answers using a mathematical processing language.

[0626] Step 4:

[0627] Generative AI generates explanations for the answers.

[0628] Input: The text data of the generated answer.

[0629] Output: The generated commentary text.

[0630] How it works: The server uses generative AI to generate explanations for the answers. The level of detail of the explanations is adjusted by a sentiment analysis engine.

[0631] Step 5:

[0632] New questions are generated based on the user's learning history and level of understanding.

[0633] Input: User learning history data, comprehension data.

[0634] Output: The text data of the new problem generated.

[0635] Specific operation: The server analyzes the user's learning history and level of understanding, and generates new questions at an appropriate level. The generated questions will be used for the next learning session.

[0636] Step 6:

[0637] Analyzes the user's emotions while answering in real time and provides appropriate feedback.

[0638] Input: Facial expression data, voice data, etc. while the user answers.

[0639] Output: Sentiment analysis results and appropriate feedback text data.

[0640] Specific operation: The server uses an emotion analysis engine to analyze the emotions expressed in the user's answers in real time. Based on the analysis results, it provides detailed or concise explanations.

[0641] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0642] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0643] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0644] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0645] [Second embodiment]

[0646] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0647] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0648] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure.

[0649] The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0650] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0651] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0652] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0653] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0654] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0655] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0656] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0657] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0658] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0659] "Example 1"

[0660] In one embodiment of the present invention, a user sends a math problem as an image to a messenger app BOT. The BOT analyzes the problem from the image and solves the analyzed problem. The answer is generated using generative AI and a conversion means in the Wolfram Language. Furthermore, an explanation for the answer is also generated by the generative AI. Specifically, when a user sends an image containing a math problem to the BOT, the BOT reads the problem from the image and solves the problem. The answer is generated using generative AI and a conversion means in the Wolfram Language, and an explanation for the answer is also generated at the same time. "Example 2"

[0661] In another aspect of the present invention, a problem generation function can be added in the future. Specifically, the system can generate mathematics problems at an appropriate level based on the user's learning history and level of understanding. This problem generation function allows users to solve problems tailored to their own level, enabling them to study more efficiently.

[0662] The processing flow of each embodiment will be described below.

[0663] "Example 1"

[0664] Step 1: The user sends an image containing a math problem to the messenger app bot.

[0665] Step 2: The bot analyzes the problem from the image sent. This analysis is done using image recognition technology.

[0666] Step 3: The analyzed problem is solved using generative AI and transformation tools in the Wolfram Language.

[0667] Step 4: An explanation for the answer is also generated by the generative AI and presented to the user.

[0668] "Example 2"

[0669] Step 1: The user sends an image containing a math problem to the messenger app bot.

[0670] Step 2: The bot analyzes the problem from the image sent. This analysis is done using image recognition technology.

[0671] Step 3: The analyzed problem is solved using generative AI and transformation tools in the Wolfram Language.

[0672] Step 4: An explanation for the answer is also generated by the generative AI and presented to the user.

[0673] Step 5: The system generates math problems at an appropriate level based on the user's learning history and level of understanding.

[0674] Example 1

[0675] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0676] Conventional mathematical problem-solving systems require users to input problems in text format, which is time-consuming, and they have the problem of not being able to handle handwritten or image-formatted problems. Furthermore, the generation of the answer and its explanation must be done separately, which makes it difficult for users to understand.

[0677] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0678] In this invention, the server includes means for a user to send a problem as an image to a messenger application, means for transferring the sent image to the server, means for the server to extract text data from the image using image analysis software, means for inputting the extracted text data into a generative AI model to generate an answer to the problem, means for converting the generated answer into a mathematical formula or a calculation result, means for generating an explanation for the answer using the generative AI model, and means for returning the answer and explanation to the user. This allows users to easily solve mathematical problems in handwritten or image format and obtain the answer and its explanation at the same time.

[0679] A "user" is any person or entity that intends to use the system to solve mathematical problems.

[0680] A "messenger application" is communication software that users use to send text and images.

[0681] An "image" is digital data that visually represents a mathematical problem.

[0682] A "server" is a computer system that receives and processes data sent by users.

[0683] "Image analysis software" is a program for extracting text data from images.

[0684] "Text data" is text information extracted by image analysis software.

[0685] A "generative AI model" is an artificial intelligence system that generates answers and explanations to questions based on input text data.

[0686] "Conversion means" refers to techniques or tools for converting generated answers into mathematical formulas or calculation results.

[0687] An "answer" is a solution to a user-submitted math problem.

[0688] An "explanation" is an explanation generated by a generative AI model to help understand the answer.

[0689] "Means for sending back" refers to the method by which the server sends back the generated answers and explanations to the user.

[0690] MODE FOR CARRYING OUT THE INVENTION

[0691] The present invention is a system that allows a user to send an image containing a mathematical problem to a messenger application, analyzes the problem from the image, and generates an answer and explanation. A specific embodiment of this system will be described below.

[0692] Hardware and software used

[0693] Server: A high-performance computer system that analyzes images, runs generative AI models, and transforms and returns data.

[0694] Device: The smartphone or PC used by the user to send images via a messenger application.

[0695] Messenger application: Communication software that allows users to send images.

[0696] Image analysis software: Extract text data from images, using Tesseract OCR as an example.

[0697] Generative AI model: As an example, we use OpenAI's GPT-4 to generate answers and explanations for problems.

[0698] Conversion method: For example, we use the Wolfram Alpha API to convert the generated answers into mathematical formulas and calculation results.

[0699] System Operation

[0700] A user sends an image containing a math problem to a messenger application. The device transfers the image to a server. The server uses image analysis software to extract text data from the image. The extracted text data is input into a generative AI model to generate an answer to the problem. The generated answer is then converted into a mathematical formula or calculation result using a transformation method. The generative AI model also generates an explanation for the answer. Finally, the server returns the answer and explanation to the user.

[0701] Specific examples

[0702] For example, if a user sends an image of the equation "x^2 + 3x + 2 = 0" to a messenger application, the process would proceed as follows:

[0703] 1. The user takes a picture of the equation with their smartphone and sends it to a messenger application.

[0704] 2. The device transfers the image data to the server.

[0705] 3. The server uses Tesseract OCR to extract the text data "x^2 + 3x + 2 = 0" from the image.

[0706] 4. The server sends the prompt "Solve the equation x^2 + 3x + 2 = 0" to the generated AI model, which generates the answer "x = -1, x = -2".

[0707] 5. The server uses the Wolfram Alpha API to materialize the answer in mathematical form.

[0708] 6. The server sends the prompt "Explain the steps to solve the quadratic equation x^2 + 3x + 2 = 0" to the generated AI model and generates an explanation.

[0709] 7. The server returns the answer "x = -1, x = -2" to the user along with the explanation "This equation is a quadratic equation and can be solved using the formula. The formula gives us x = -1 and x = -2."

[0710] In this way, users can receive answers and explanations for math problems simply by submitting an image of the problem.

[0711] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0712] Step 1:

[0713] A user sends an image containing a math problem to a messenger application. Using a smartphone or PC, the user takes a photo of the handwritten equation "x^2 + 3x + 2 = 0" and sends it to the messenger application. The input is the image containing the math problem, and the output is the image sent to the messenger application.

[0714] Step 2:

[0715] The device transfers the image sent to the server. The device transfers the image data sent by the user to the server in an appropriate format (e.g., JPEG, PNG). The input is the image sent to the messenger application, and the output is the image data transferred to the server.

[0716] Step 3:

[0717] The server uses image analysis software to extract text data from the image. The server uses image analysis software (e.g., Tesseract OCR) to extract the text data "x^2 + 3x + 2 = 0" from the image. The input is the image data transferred to the server, and the output is the extracted text data.

[0718] Step 4:

[0719] The server inputs the extracted text data into a generative AI model to generate an answer to the problem. The server inputs the extracted text data into a generative AI model (e.g., OpenAI's GPT-4) and sends the prompt "Solve the equation x^2 + 3x + 2 = 0" to the generative AI model. The generative AI model generates the answer "x = -1, x = -2". The input is the extracted text data and the prompt, and the output is the generated answer.

[0720] Step 5:

[0721] The server materializes the answer using Wolfram Language transformations. The server materializes the generated answer as a formula or result using Wolfram Language transformations (e.g., Wolfram Alpha API). The input is the generated answer, and the output is the materialized formula or result.

[0722] Step 6:

[0723] The server generates an explanation for the answer using a generative AI model. The server sends the generative AI model a prompt, "Explain the steps to solve the quadratic equation x^2 + 3x + 2 = 0," and generates an explanation. The generative AI model generates an explanation, "This equation is a quadratic equation and can be solved using the solution formula. The solution formula gives x = -1 and x = -2." The input is the generated answer and prompt, and the output is the generated explanation.

[0724] Step 7:

[0725] The server returns the answer and explanation to the user. The server returns the generated answer "x = -1, x = -2" and explanation "This equation is a quadratic equation and can be solved using the solution formula. The solution formula gives x = -1 and x = -2." The input is the generated answer and explanation, and the output is the answer and explanation returned to the user.

[0726] (Application example 1)

[0727] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0728] Conventional math problem-solving systems require users to input questions, and it takes time to obtain the answers and explanations. In addition, the accuracy of the answers and the quality of the explanations are often not guaranteed, which can reduce the user's learning efficiency.

[0729] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for a user to send a problem as an image to a messenger app BOT, means for analyzing the problem from the sent image, means for solving the analyzed problem, means for converting between a generative AI and a formula manipulation language to generate an answer, means for generating an explanation of the answer, means including an application to be installed on the smartphone, means for extracting text from an image using an image processing library, means for solving the problem using a formula manipulation language API, means for generating an explanation using a generative AI, and means for displaying the answer and explanation to the user. This allows a user to easily analyze a mathematical problem photographed with their smartphone and quickly obtain an accurate answer and high-quality explanation.

[0730] A "user" is an individual or group that uses the system to solve mathematical problems.

[0731] "Messenger App BOT" is an automatic response program that receives and analyzes mathematical problems sent by users as images.

[0732] "Means of sending as an image" refers to a method in which a user takes a photo of a math problem using a smartphone or other device and sends the image to a messenger app BOT.

[0733] The "means for analyzing problems from transmitted images" is a method for extracting text from images using an image processing library and recognizing the text as a mathematical problem.

[0734] The "means for solving the analyzed problem" is a method for solving the extracted mathematical problem using a computer algebra language API.

[0735] A "generative AI for generating solutions" is an artificial intelligence model used to generate answers to mathematical problems.

[0736] A "computer algebra language transformation means" is a method of using a computer algebra language to solve a mathematical problem.

[0737] "Means for generating explanations for answers" refers to a method for generating detailed explanations for answers using generative AI.

[0738] The "application installed on a smartphone" is software that allows users to take a photo of a math problem using their smartphone and receive the answer and explanation.

[0739] An "image processing library" is a software tool for extracting text from images.

[0740] A "computer algebra language API" is a programmatic interface used to solve mathematical problems.

[0741] "Means for generating explanations using generative AI" refers to a method for generating explanations for answers to mathematical problems using generative AI.

[0742] "Means for displaying answers and explanations to the user" refers to a method for displaying the generated answers and explanations on the user's smartphone.

[0743] The system for implementing this invention starts when a user takes a photo of a math problem using a smartphone and sends the image to a messenger app BOT. A specific embodiment of this system will be described below.

[0744] Hardware and software used

[0745] Hardware: Smartphone (with camera)

[0746] software:

[0747] Messenger app BOT

[0748] Image processing library (e.g. OpenCV)

[0749] OCR (Optical Character Recognition) libraries (e.g., pytesseract)

[0750] Mathematical processing language API (e.g. WolframAlpha API)

[0751] Generative AI models (e.g., GPT-3)

[0752] Data processing and calculation

[0753] 1. Image Acquisition:

[0754] Users take a picture of a math problem using their smartphone camera and send the image to a messenger app BOT.

[0755] 2. Image Analysis:

[0756] The server converts the submitted image to grayscale using an image processing library (OpenCV) and extracts text from the image using an OCR library (pytesseract).

[0757] 3. Problem answer:

[0758] The extracted text is sent to a mathematical processing language API (WolframAlpha API), and the server obtains the answer.

[0759] 4. Description generation:

[0760] The server uses a generative AI model (GPT-3) to generate a detailed explanation for the obtained answer. Specifically, it inputs the following prompt sentence into the generative AI:

[0761] Example prompt sentence:

[0762] Math problem: x^2 + 2x + 1 = 0

[0763] Answer: x = -1

[0764] Description:

[0765] 5. Results display:

[0766] The server sends the generated answers and explanations to the user's smartphone and displays them through the application.

[0767] Specific examples

[0768] If a user photographs and submits the problem "x^2 + 2x + 1 = 0," the system operates as follows:

[0769] 1. Extract the text "x^2 + 2x + 1 = 0" from the image.

[0770] 2. Use the formula processing language API to obtain the answer "x = -1".

[0771] 3. Use generative AI to generate an explanation such as, "This equation is a quadratic equation, and using the formula for the solution, we get x = -1."

[0772] 4. Display the answer and explanation to the user.

[0773] In this way, users can easily analyze math problems photographed with their smartphones and quickly obtain accurate answers and high-quality explanations.

[0774] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0775] Step 1:

[0776] A user takes a picture of a math problem using the camera on their smartphone and sends the image to a messenger app BOT. The input is the image of the math problem that was taken, and the output is the image sent to the messenger app BOT.

[0777] Step 2:

[0778] The server converts the transmitted image to grayscale using an image processing library (OpenCV). The input is the image transmitted by the user, and the output is the image converted to grayscale. Specifically, the server reads the image using OpenCV functions and converts it to grayscale.

[0779] Step 3:

[0780] The server uses an OCR library (pytesseract) to extract text from grayscale images. The input is the image converted to grayscale, and the output is the extracted text. Specifically, it uses pytesseract's functions to read text from the image.

[0781] Step 4:

[0782] The server sends the extracted text to a mathematical expression processing language API (WolframAlpha API) and obtains the answer. The input is the extracted text, and the output is the answer obtained from the mathematical expression processing language API. Specifically, the server sends the text to the WolframAlpha API and receives the answer.

[0783] Step 5:

[0784] The server uses a generative AI model (GPT-3) to generate a detailed explanation for the obtained answer. The input is the answer obtained from the formula processing language API, and the output is the generated explanation. Specifically, the following prompt sentence is input to the generative AI:

[0785] Example prompt sentence:

[0786] Math problem: x^2 + 2x + 1 = 0

[0787] Answer: x = -1

[0788] Description:

[0789] Step 6:

[0790] The server sends the generated answers and explanations to the user's smartphone and displays them through the application. The input is the generated answers and explanations, and the output is the answers and explanations displayed on the user's smartphone. Specifically, the answers and explanations are sent through a messenger app BOT and displayed on the smartphone application.

[0791] Example 2

[0792] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0793] Conventional learning systems have difficulty generating questions based on a user's learning history and level of understanding, making it impossible to provide optimal questions for each individual user. Furthermore, efficient and accurate processing is required for analyzing questions answered by users and generating answers. This reduces the user's learning efficiency, making effective learning difficult.

[0794] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0795] In this invention, the server includes means for a user to send a question to a messenger application as an image, means for analyzing the question from the sent image, and means for answering the analyzed question, thereby making it possible to collect the user's learning history data, evaluate the user's level of understanding based on the analysis results, create prompt sentences to be input to the generative AI based on the level of understanding, generate questions using the generative AI, and provide the generated questions to the user.

[0796] "User" refers to an individual or organization that uses the system to study.

[0797] "Messenger application" refers to communication software that allows users to send and receive text and images.

[0798] "Image" refers to visual data that a user sends through a messenger application.

[0799] "Question" refers to a task or question provided for a user to answer.

[0800] "Analysis" refers to the process of reading and understanding the problem content from the transmitted image.

[0801] "Solution" refers to the answer to the question being analyzed.

[0802] "Generative AI" refers to AI technology that generates questions and answers based on a user's learning history and level of understanding.

[0803] "Programming language" refers to the computer language used by generative artificial intelligence to generate answers.

[0804] "Study history data" refers to data relating to questions that a user has answered in the past and the results of those answers.

[0805] "Analysis results" refers to information obtained after analyzing learning history data.

[0806] "Level of understanding" refers to an index that indicates how well a user can understand a question and answer it.

[0807] A "prompt sentence" refers to an instruction sentence that is input to a generative artificial intelligence when generating a problem.

[0808] "Problem generation" refers to the process by which a generative AI creates new questions based on prompts.

[0809] "Providing" refers to the act of displaying or transmitting the generated question to the user.

[0810] This invention is a system that allows a user to send a question as an image to a messenger application, analyzes the question from the image, generates an answer, and further generates new questions based on the user's learning history data. A specific embodiment of this system will be described below.

[0811] Hardware and software used

[0812] The server uses the following hardware and software:

[0813] Hardware: High-performance server machine

[0814] Software: generative AI models (e.g., GPT-4), database management systems, messenger applications

[0815] Data processing and calculation

[0816] 1. Collecting user learning history data

[0817] The server collects data on questions that users have previously solved and stores it in a database, including the type of question, the percentage of correct answers, and the time it took to solve the question.

[0818] 2. Analysis of learning history data

[0819] The server analyzes the collected learning history data, specifically, what answers the user gave to which questions and how long it took them to answer them.

[0820] 3. Assessment of understanding

[0821] The server evaluates the user's level of understanding based on the analysis results. For example, if the correct answer rate is high and the answer time is short, the server evaluates the user as "advanced," and if the correct answer rate is low and the answer time is long, the server evaluates the user as "beginner."

[0822] 4. Creating prompts for question generation

[0823] The server creates a prompt sentence to be input to the generative AI model based on the user's level of comprehension. For example, if the user's level of comprehension is evaluated as "intermediate," the server creates a prompt sentence such as "Generate intermediate level math problems."

[0824] 5. Problem generation using generative AI models

[0825] The server inputs a prompt into the generative AI model to generate a math problem. For example, based on the prompt "Generate an intermediate level math problem," the model generates the problem "Solve the following equation: 2x + 3 = 7."

[0826] 6. Providing generated questions

[0827] The server provides the generated questions to the user's device, and the user can progress with their learning by solving the questions provided.

[0828] Specific examples

[0829] Suppose the data of problems that a user has solved in the past is stored as follows:

[0830] Question 1: Correct answer, answer time 5 minutes

[0831] Question 2: Wrong answer, answer time 10 minutes

[0832] Question 3: Correct answer, answer time 7 minutes

[0833] The server analyzes this data and evaluates the user's level of understanding as "intermediate." It then feeds the following prompt to the generative AI model:

[0834] "If the user has an intermediate level of comprehension, generate intermediate level math problems."

[0835] Based on this prompt, the generative AI model generates intermediate-level math problems, such as "Solve the following equation: 2x + 3 = 7."

[0836] In this way, users can solve problems that are suited to their own level, enabling them to study more efficiently.

[0837] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0838] Step 1: Collecting user learning history data

[0839] The server collects data on questions that users have previously solved, and stores the results of questions that users have solved through the messenger application (such as correct answer rate and answer time) in a database.

[0840] Input: The result of the question answered by the user

[0841] Data processing: Data such as question type, correct answer rate, and answer time are extracted and saved in a database

[0842] Output: Learning history data stored in a database

[0843] Step 2: Analyzing learning history data

[0844] The server retrieves the user's learning history data from the database and analyzes it, specifically, analyzing what answers the user gave to which questions and how long it took them to answer.

[0845] Input: Learning history data stored in the database

[0846] Data calculation: Calculation of correct answer rate, answer time, evaluation of problem difficulty

[0847] Output: Analysis results (user correct answer rate, answer time, difficulty of questions)

[0848] Step 3: Assess understanding

[0849] The server evaluates the user's level of understanding based on the analysis results. For example, if the correct answer rate is high and the answer time is short, the server evaluates the user as "advanced," and if the correct answer rate is low and the answer time is long, the server evaluates the user as "beginner."

[0850] Input: Analysis results

[0851] Data arithmetic: Evaluation of comprehension based on correct answer rate and answer time

[0852] Output: User's level of understanding (advanced, intermediate, beginner, etc.)

[0853] Step 4: Create a prompt to generate questions

[0854] The server creates a prompt sentence to be input to the generative AI model based on the user's level of comprehension. For example, if the user's level of comprehension is evaluated as "intermediate," the server creates a prompt sentence such as "Generate intermediate level math problems."

[0855] Input: User comprehension

[0856] Data processing: Prompt generation based on comprehension

[0857] Output: A prompt to be input to the generative AI model

[0858] Step 5: Generate questions using a generative AI model

[0859] The server inputs a prompt into the generative AI model to generate a math problem. For example, based on the prompt "Generate an intermediate level math problem," the model generates the problem "Solve the following equation: 2x + 3 = 7."

[0860] Input: prompt statement

[0861] Data Computation: Problem Generation with Generative AI Models

[0862] Output: Generated math problem

[0863] Step 6: Providing generated questions

[0864] The server provides the generated questions to the user's device, and the user can progress with their learning by solving the questions provided.

[0865] Input: Generated math problem

[0866] Data processing: Convert data to fit the problem format

[0867] Output: The problem as it appears on the user's terminal

[0868] (Application example 2)

[0869] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0870] Conventional learning systems have had the problem of being unable to provide users with questions at an appropriate level based on their learning history and level of understanding, making it difficult for them to efficiently solve questions suited to their own level. Furthermore, they lacked the functionality to record the history of questions answered by users and dynamically update their learning level, making it difficult to effectively manage learning progress.

[0871] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0872] In this invention, the server includes means for a user to send a problem as an image to a messenger app, means for analyzing the problem from the sent image, means for solving the analyzed problem, means for converting between a generative AI and a computer algebra language to generate an answer, means for generating an explanation of the answer, means for generating problems of an appropriate level based on the user's learning history and level of understanding, means for presenting the generated problems to the user, and means for recording the user's answer history and updating the learning level. This allows the user to efficiently solve problems that match their learning level and effectively manage their learning progress.

[0873] "User" refers to an individual or group that uses the system to study.

[0874] A "messenger app" is a software application for sending and receiving messages such as text and images.

[0875] "Image" means a digital representation of visual information used by a user to submit a question.

[0876] A "problem" is a mathematical or other learning task for which a user seeks an answer.

[0877] "Analysis" is the process of recognizing the problematic content from the transmitted image and extracting it as digital data.

[0878] An "answer" is a correct answer to the problem being analyzed.

[0879] "Generative AI" is a system that uses artificial intelligence technology to generate answers and explanations to problems.

[0880] A "computational mathematics language" is a programming language for processing mathematical expressions on a computer.

[0881] "Explanation" is information that explains the process or method for arriving at the answer.

[0882] "Study history" is data that records questions that a user has answered in the past and their scores.

[0883] "Level of understanding" is an index that indicates how well a user understands a particular learning content.

[0884] "Level" refers to the level of difficulty according to the user's learning progress and ability.

[0885] "Presenting" is the act of displaying the generated questions and answers to the user.

[0886] "Answer history" is data that records questions that a user has answered in the past and the results of those answers.

[0887] "Study level update" is the process of dynamically changing a user's learning level based on the user's answer history.

[0888] A system for implementing this invention includes means for a user to send a question as an image to a messenger app, means for analyzing the question from the sent image, means for answering the analyzed question, means for converting between a generative AI and a mathematical processing language to generate an answer, means for generating an explanation of the answer, means for generating questions of an appropriate level based on the user's learning history and level of understanding, means for presenting the generated questions to the user, and means for recording the user's answer history and updating the learning level.

[0889] System Program Overview

[0890] The system is realized using the following hardware and software.

[0891] Hardware:

[0892] Smartphone

[0893] server

[0894] software:

[0895] Messenger app

[0896] Generative AI (e.g. GPT-3)

[0897] mathematical processing languages ​​(e.g., Wolfram Language)

[0898] JSON file (saves user data)

[0899] Processing flow

[0900] 1. User submits issue:

[0901] Users use a messenger app to send an image of a question that requires an answer.

[0902] 2. Image Analysis:

[0903] The server receives the transmitted image and uses image analysis technology to extract the problematic content as digital data.

[0904] 3. Generate answers to questions:

[0905] Using generative AI and mathematical processing languages, answers to the analyzed problems are generated.

[0906] 4. Generate explanations for answers:

[0907] Generative AI generates explanations that explain the process and methods used to arrive at the answer.

[0908] 5. Problem Generation:

[0909] The server generates new questions at an appropriate level based on the user's learning history and level of understanding.

[0910] 6. Problem Statement:

[0911] The generated questions are presented to the user through a messenger app.

[0912] 7. Record your answer history and update your learning level:

[0913] The user's answer history is recorded and their learning level is dynamically updated.

[0914] Specific examples

[0915] When a user first uses the MathMaster app, the app sets the user's level as "beginner." Each time the user solves a problem, the number of correct answers is counted, and once a certain number of correct answers are achieved, the level is updated to "intermediate" or "advanced."

[0916] Prompt Sentence Examples

[0917] Generate math problems at the appropriate level based on the user's learning history and level of understanding. The user's level can be "beginner," "intermediate," or "advanced." Provide the Python code that generates problems for each level.

[0918] In this way, the system can maximize the user's learning efficiency and effectively manage the learning progress.

[0919] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0920] Step 1:

[0921] The user sends the problem as an image to a messenger app. The user uses the messenger app on their smartphone to take a picture of the problem for which they want to answer and send it. The input is the image of the problem taken by the user, and the output is the image data sent to the server via the messenger app.

[0922] Step 2:

[0923] The server receives the image and performs image analysis. The server analyzes the received image data using an image analysis algorithm (e.g., OCR technology) and extracts the problem content as text data. The input is the image data sent by the user, and the output is the analyzed text data of the problem.

[0924] Step 3:

[0925] The server generates an answer to the analyzed problem. The server uses a generative AI (e.g., GPT-3) and a mathematical processing language (e.g., the Wolfram Language) to generate an answer based on the text data of the analyzed problem. The input is the text data of the analyzed problem, and the output is the generated answer.

[0926] Step 4:

[0927] The server generates an explanation that explains the process and method for arriving at the answer. Using generative AI, an explanation is generated that explains in detail the process and method for arriving at the answer. The input is the generated answer, and the output is an explanation of the answer.

[0928] Step 5:

[0929] The server generates new questions at an appropriate level based on the user's learning history and level of comprehension. The server references the user's learning history data and comprehension data and uses generative AI to generate new questions at an appropriate level. The input is the user's learning history data and comprehension data, and the output is the generated new questions.

[0930] Step 6:

[0931] The server presents the generated problem to the user. The generated new problem is sent to the user through a messenger app and presented to them. The input is the generated new problem, and the output is the problem presented to the user.

[0932] Step 7:

[0933] The server records the user's answer history and updates the learning level. The server records the history of questions answered by the user and dynamically updates the learning level. The input is the user's answer history data, and the output is the updated learning level.

[0934] In this way, the system can maximize the user's learning efficiency and effectively manage the learning progress.

[0935] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0936] "Example 1"

[0937] One embodiment of the present invention provides a system incorporating an emotion engine. The system includes a means for a user to send a problem to a messenger app BOT as an image, a means for analyzing the problem from the sent image, a means for solving the analyzed problem, a means for converting between a generative AI and the Wolfram Language to generate an answer, a means for generating an explanation for the answer, and an emotion engine that recognizes the user's emotions. The emotion engine recognizes the user's emotions and adjusts the difficulty of the problems based on the user's emotions. For example, if the user feels confused, the emotion engine lowers the difficulty of the problems. On the other hand, if the user feels joy or a sense of accomplishment, the emotion engine increases the difficulty of the problems. This improves the user's learning efficiency and satisfaction.

[0938] "Example 2"

[0939] Additionally, the emotion engine adjusts the detail of the answer explanation based on the user's emotion.

[0940] For example, if the user feels confused, the emotion engine will generate a more detailed explanation. On the other hand, if the user feels they understand, the emotion engine will generate a more concise explanation. This allows for the provision of appropriate explanations according to the user's level of understanding, enabling efficient learning.

[0941] The processing flow of each embodiment will be described below.

[0942] "Example 1"

[0943] Step 1: The user sends the problem as an image to the messenger app BOT.

[0944] Step 2: The system analyzes the problem from the image sent.

[0945] Step 3: The system answers the analyzed question.

[0946] Step 4: The system generates a solution using generative AI and the Wolfram Language.

[0947] Step 5: The system generates an explanation for the answer.

[0948] Step 6: The emotion engine recognizes the user's emotion.

[0949] Step 7: Adjust the difficulty of the questions based on the emotions recognized by the emotion engine.

[0950] "Example 2"

[0951] Step 1: The user sends the problem as an image to the messenger app BOT.

[0952] Step 2: The system analyzes the problem from the image sent.

[0953] Step 3: The system answers the analyzed question.

[0954] Step 4: The system generates a solution using generative AI and the Wolfram Language.

[0955] Step 5: The system generates an explanation for the answer.

[0956] Step 6: The emotion engine recognizes the user's emotion.

[0957] Step 7: Adjust the detail of the answer explanation based on the sentiment recognized by the sentiment engine.

[0958] Example 1

[0959] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0960] In conventional math problem-solving systems, when users submit problems as images, the problem analysis and answer generation may not be performed accurately. Furthermore, there is a lack of emotion recognition functionality to improve user learning efficiency and satisfaction, which means that the difficulty of the problem cannot be adjusted appropriately according to the user's emotions.

[0961] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0962] In this invention, the server includes means for a user to send a question as an image to a communication application, means for analyzing the question from the sent image, means for solving the analyzed question, generative artificial intelligence and mathematical language conversion means for generating an answer, means for generating an explanation for the answer, emotion analysis means for recognizing the user's emotion, and means for adjusting the difficulty of the question based on the recognized emotion. This makes it possible to accurately analyze the question from the image sent by the user and provide appropriate answers and explanations. Furthermore, adjusting the difficulty of the question according to the user's emotion can improve learning efficiency and satisfaction.

[0963] "User" refers to any person or entity that uses the system to solve mathematical problems.

[0964] "Communication application" refers to a software platform used by users to send messages and images.

[0965] "Image" refers to visual data containing a mathematical problem that a user sends through a communication application.

[0966] "Means of analysis" refers to the technology that recognizes mathematical problems from the transmitted images and extracts them as text data.

[0967] "Means of solving" refers to the technique of deriving an appropriate answer to the analyzed mathematical problem.

[0968] "Generative AI" refers to artificial intelligence technology that generates answers and explanations to mathematical problems.

[0969] "Mathematical language conversion means" refers to the technology that converts the generated answers into formats such as mathematical formulas and graphs.

[0970] "Means for generating an explanation for an answer" refers to technology that uses generative artificial intelligence to generate detailed explanations and solutions for the answer.

[0971] "Emotion analysis means" refers to technology for recognizing and analyzing a user's emotions.

[0972] The "means for adjusting the difficulty of a problem" refers to a technology for changing the difficulty of the next mathematical problem provided based on the user's emotions recognized by the emotion analysis means.

[0973] This invention is a system that allows users to send images containing mathematical problems via a communication application, analyzes the problems from the images, and generates answers and explanations. It also has the ability to recognize the user's emotions and adjust the difficulty of the problems based on those emotions.

[0974] Hardware and software used

[0975] Hardware

[0976] Server: A central processing unit for performing processes such as image analysis, answer generation, and emotion recognition.

[0977] Terminal: A device (smartphone, tablet, PC, etc.) through which a user uses a communication application.

[0978] software

[0979] Communication application: a messenger application that allows users to send images containing math problems.

[0980] OCR Engine: Use the Tesseract OCR engine to extract text from images.

[0981] Generative artificial intelligence: Using OpenAI's GPT-4 and other technologies, we generate answers and explanations for mathematical problems.

[0982] Mathematical language converters: Use the Wolfram Language to convert generated answers into mathematical formulas, graphs, and other formats.

[0983] Sentiment analysis engine: Software for recognizing and analyzing user emotions.

[0984] Data processing and calculation

[0985] Image analysis

[0986] The server receives the image submitted by the user and uses the Tesseract OCR engine to extract text from the image. For example, if the image submitted by the user contains the equation "x^2 + 3x + 2 = 0", the OCR engine will accurately extract this text.

[0987] Answer generation

[0988] The server inputs the extracted text into a generative artificial intelligence (GPT-4) to generate an answer to the problem. It then uses the Wolfram Language to convert the answer into a mathematical formula, graph, or other format. For example, the answer "x = -1 or x = -2" is generated.

[0989] Generate explanations for answers

[0990] The server uses generative artificial intelligence to generate explanations for the solution. The explanations include the solution to the problem and related concepts. For example, the server might generate an explanation like, "This equation can be solved by factorization. First, factor the equation as (x + 1)(x + 2) = 0. Then, set each factor equal to 0 to find the solution."

[0991] Emotion recognition and difficulty adjustment

[0992] The server uses an emotion analysis engine to recognize the user's emotions. It analyzes the user's messages and reactions (e.g., emojis and text) to determine whether the user is confused or happy. Based on the recognized emotions, it adjusts the difficulty of the next problem to be presented. For example, if it determines that the user is confused, it lowers the difficulty of the next problem to be presented. Conversely, if it determines that the user is feeling happy or a sense of accomplishment, it increases the difficulty of the next problem to be presented.

[0993] Specific examples

[0994] Prompt Sentence Examples

[0995] "Please answer the math problem contained in this image and provide an explanation for the answer."

[0996] Specific operation flow

[0997] 1. A user sends an image containing "x^2 + 3x + 2 = 0" through a communication application.

[0998] 2. The server receives the image and uses the Tesseract OCR engine to extract the text "x^2 + 3x + 2 = 0".

[0999] 3. The server uses generative artificial intelligence (GPT-4) and the Wolfram Language to generate the answer "x = -1 or x = -2".

[1000] 4. The server uses generative artificial intelligence to generate an explanation of the answer, including the factorization steps.

[1001] 5. The server uses an emotion analysis engine to analyze the user's emotions, and if it determines that the user is confused, it lowers the difficulty of the next question it presents.

[1002] 6. If the user answers the next question and the sentiment analysis engine recognizes the user's joy or sense of accomplishment, the difficulty of the next question provided will be increased.

[1003] This system allows users to study mathematical problems efficiently.

[1004] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1005] Step 1:

[1006] A user uses a communication application to send an image containing a math problem.

[1007] Input: An image containing a math problem taken or saved by the user.

[1008] Output: Image data sent to a server through a communication application.

[1009] Specific operation: The user uses a smartphone or PC to take a photo of a piece of paper with a math problem written on it, then uploads and sends the image to a communication application.

[1010] Step 2:

[1011] The server receives the image sent by the user.

[1012] Input: Image data sent from a communication application.

[1013] Output: Image data stored in the server's storage.

[1014] Specific operation: The server receives image data through the API of the communication application and temporarily stores it in storage.

[1015] Step 3:

[1016] The server analyzes the image to find the math problem.

[1017] Input: Image data stored on the server.

[1018] Output: Text data extracted from the image.

[1019] What it does: The server uses the Tesseract OCR engine to parse the image data and extract text containing the math problem, for example, "x^2 + 3x + 2 = 0."

[1020] Step 4:

[1021] The server answers the math questions based on the extracted text.

[1022] Input: Text data extracted by the OCR engine.

[1023] Output: Answer data generated by generative artificial intelligence.

[1024] How it works: The server inputs the extracted text into a generative artificial intelligence (GPT-4) to generate an answer to the problem. It then uses the Wolfram Language to convert the answer into a mathematical formula, graph, or other format. For example, the answer "x = -1 or x = -2" is generated.

[1025] Step 5:

[1026] The server generates an explanation for the answer.

[1027] Input: Answer data generated by generative artificial intelligence.

[1028] Output: Explanation data of the answer generated by the generative artificial intelligence.

[1029] How it works: The server uses generative artificial intelligence to generate an explanation for the solution. The explanation includes the solution to the problem and related concepts. For example, the server might generate an explanation like, "This equation can be solved by factorization. First, factor the equation as (x + 1)(x + 2) = 0. Then, set each factor equal to 0 to find the solution."

[1030] Step 6:

[1031] The server recognizes the user's emotions.

[1032] Input: User message and response data.

[1033] Output: Sentiment data recognized by the sentiment analysis engine.

[1034] What it does: The server uses a sentiment analysis engine to analyze the user's messages and reactions (e.g., emojis and text) to determine whether the user is confused or happy.

[1035] Step 7:

[1036] The difficulty of the questions is adjusted based on the emotions recognized by the server.

[1037] Input: Emotion data recognized by the sentiment analysis engine.

[1038] Output: Adjusted problem difficulty data.

[1039] Specific operation: The server adjusts the difficulty of the next problem based on the recognized emotion. For example, if it determines that the user is confused, it will lower the difficulty of the next problem. Conversely, if it determines that the user is feeling joy or a sense of accomplishment, it will increase the difficulty of the next problem.

[1040] (Application example 1)

[1041] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1042] Conventional math problem-solving systems can provide answers and explanations for problems submitted by users, but they have the problem of not being able to adjust the difficulty of the problems according to the user's learning situation or emotions. Also, when a user is confused or, conversely, when they get bored with problems that are too easy, it is difficult to provide problems of an appropriate difficulty.

[1043] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1044] In this invention, the server includes means for a user to send a question as an image to a messenger app BOT, means for analyzing the question from the sent image, means for solving the analyzed question, means for converting between a generative AI and a formula processing language to generate an answer, means for generating an explanation of the answer, means for recognizing the user's emotions, and means for adjusting the difficulty of the questions based on the recognized emotions. This makes it possible to provide questions of an appropriate level of difficulty according to the user's learning situation and emotions.

[1045] A "user" is an individual or entity that uses the system to solve mathematical problems.

[1046] A "messenger app BOT" is an automated program that receives mathematical problems sent by users as images, analyzes them, and provides answers.

[1047] "Means to send as an image" is a function that allows users to send math problems in image format to a messenger app BOT.

[1048] The "means for analyzing problems from transmitted images" is a function for extracting mathematical problems as text from received images and analyzing them.

[1049] The "means for solving the analyzed problem" is a function for generating an answer to the analyzed mathematical problem.

[1050] "Generative AI" is a system that uses artificial intelligence technology to generate answers and explanations for mathematical problems.

[1051] A "computer algebra language" is a programming language used to solve mathematical problems.

[1052] "Means for generating explanations for answers" is a function for generating detailed explanations for answers using generative AI.

[1053] The "means for recognizing the user's emotions" is a function for detecting and recognizing emotions from the user's facial expressions and actions.

[1054] The "means for adjusting the difficulty of the question based on the recognized emotion" is a function for changing the difficulty of the next question to be posed in accordance with the emotion of the user.

[1055] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[1056] System Configuration

[1057] The system includes a messenger app BOT that allows users to send math problems as images, an image analysis engine that analyzes the problems from the images, a generative AI and a mathematical processing language that solves the analyzed problems, a generative AI that generates explanations for the answers, an emotion recognition engine that recognizes the user's emotions, and a difficulty adjustment engine that adjusts the difficulty of the problems based on the recognized emotions.

[1058] Hardware and software used

[1059] Hardware: Servers, user devices (smartphones, tablets, PCs, etc.)

[1060] software:

[1061] Image analysis engine: OpenCV, Tesseract OCR

[1062] Generative AI: Hugging Face transformers library

[1063] Mathematical Computation Language: WolframAlpha API

[1064] Emotion Recognition Engine: EmotionRecognizer Library

[1065] Data processing and calculation

[1066] 1. Image analysis: Images containing mathematical problems sent from the user's device are analyzed using an image analysis engine (OpenCV and Tesseract OCR) on the server and extracted as text data.

[1067] 2. Answer generation: The extracted text data is input into a mathematical processing language (WolframAlpha API) to generate an answer.

[1068] 3. Explanation generation: Using generative AI (Hugging Face's transformers library), we generate detailed explanations for the answers.

[1069] 4. Emotion recognition: The user's facial expressions and behavior are analyzed using an emotion recognition engine (EmotionRecognizer library) to recognize the user's emotions.

[1070] 5. Difficulty Adjustment: Based on the recognized emotions, the difficulty of the next question is adjusted by the difficulty adjustment engine.

[1071] Specific examples

[1072] A user sends an image containing a math problem to a messenger app BOT using their smartphone. The server uses an image analysis engine to extract the problem as text from the image and generates an answer using a formula processing language. The generative AI generates an explanation for the answer using a prompt like the following:

[1073] Question: Solve the equation 2x + 3 = 7.

[1074] Answer: x = 2

[1075] Explain the solution.

[1076] The generated description might look something like this:

[1077] To solve the equation 2x + 3 = 7, we first subtract 3 from both sides to get 2x = 4. Then, we divide both sides by 2 to find x = 2.

[1078] Furthermore, the emotion recognition engine analyzes the user's facial expressions and behavior, and if the user is confused, it lowers the difficulty of the next question, and if the user is happy, it raises the difficulty, thus improving the user's learning efficiency and satisfaction.

[1079] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1080] Step 1:

[1081] A user sends an image containing a math problem to a messenger app BOT using a smartphone. The input is the image of the math problem taken by the user, and the output is the image data sent to the server.

[1082] Step 2:

[1083] The server analyzes the received image data using an image analysis engine (OpenCV and Tesseract OCR). The input is the transmitted image data, and the output is the text data extracted from the image. Specifically, the image is read using OpenCV, and the text in the image is extracted using Tesseract OCR.

[1084] Step 3:

[1085] The server inputs the extracted text data into a mathematical processing language (WolframAlpha API) to generate an answer. The input is the extracted text data, and the output is the answer to the mathematical problem. Specifically, the server sends the text data to the WolframAlpha API and obtains the returned answer.

[1086] Step 4:

[1087] The server uses a generative AI (Hugging Face's transformers library) to generate a detailed explanation of the answer. The input is a math problem and its answer, and the output is an explanation of the answer. Specifically, the server inputs the following prompt into the generative AI, which then generates an explanation:

[1088] Question: Solve the equation 2x + 3 = 7.

[1089] Answer: x = 2

[1090] Explain the solution.

[1091] Step 5:

[1092] The server analyzes the user's facial expressions and behavior using an emotion recognition engine (EmotionRecognizer library) to recognize the user's emotions. The input is the user's facial expressions and behavior data, and the output is the recognized emotion. Specifically, the server uses the EmotionRecognizer library to analyze the user's facial expressions and behavior and identify the emotion.

[1093] Step 6:

[1094] The server uses a difficulty adjustment engine to adjust the difficulty of the next question based on the recognized emotion. The input is the recognized emotion data, and the output is the adjusted difficulty of the question. Specifically, the difficulty is set to be lowered if the user is confused, and to be higher if the user is happy.

[1095] Step 7:

[1096] The server selects the next question based on the adjusted difficulty and provides it to the user. The input is the adjusted difficulty of the question, and the output is the next question to be asked. Specifically, the server selects a question of appropriate difficulty from the question database and sends it to the user.

[1097] Example 2

[1098] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1099] Conventional learning support systems lacked the ability to generate appropriate questions based on the user's learning history and level of understanding, which led to problems with reduced learning efficiency. They also lacked the ability to adjust the level of detail in explanations according to the user's emotional state, meaning they were unable to provide appropriate support according to the user's level of understanding.

[1100] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1101] In this invention, the server includes means for a user to send a question as an image to a messenger application, means for analyzing the question from the sent image, means for solving the analyzed question, means for converting between a generative artificial intelligence and a computer algebra language for generating an answer, means for generating an explanation for the answer, means for collecting user learning history data, means for analyzing the collected learning history data, means for generating a prompt sentence based on the analysis result, means for inputting the generated prompt sentence into the generative artificial intelligence to generate a question, means for collecting user emotion data, means for analyzing the collected emotion data, means for adjusting the level of detail of the explanation based on the analysis result, and means for providing the question and explanation to the user. This makes it possible to generate appropriate questions based on the user's learning history and level of understanding, and to adjust the level of detail of the explanation according to the user's emotional state.

[1102] "User" refers to an individual who uses the system to learn.

[1103] "Messenger application" refers to software for sending and receiving messages such as text and images.

[1104] "Image" refers to digital data containing visual information.

[1105] "Question" refers to a task or question for the user to answer.

[1106] "Analysis" refers to the process of examining data in detail to understand its structure and meaning.

[1107] "Answer" refers to the correct answer to a question.

[1108] "Generative artificial intelligence" refers to algorithms and models that use natural language processing and machine learning to generate text and data.

[1109] A "computational mathematics language" refers to a programming language used to perform mathematical calculations and analysis.

[1110] "Explanation" refers to a detailed explanation of the question and answer.

[1111] "Learning history data" refers to data relating to learning activities that a user has performed in the past.

[1112] A "prompt sentence" refers to an instruction sentence to be input into generative artificial intelligence.

[1113] "Emotion data" refers to data that indicates the user's emotional state.

[1114] "Level of detail" refers to the specificity and depth of the explanation.

[1115] MODE FOR CARRYING OUT THE INVENTION

[1116] This invention provides a learning support system that generates appropriate questions based on a user's learning history and level of understanding, and adjusts the level of detail of explanations according to the user's emotional state. Specific embodiments of this system are described below.

[1117] Hardware and software used

[1118] server

[1119] The server uses the following hardware and software:

[1120] Hardware: A server machine with a powerful processor, sufficient memory, and ample storage.

[1121] Software: Python, TensorFlow, OpenCV, Google Cloud Speech-to-Text API, MySQL

[1122] Terminal

[1123] A terminal is a device used by a user that uses the following hardware and software:

[1124] Hardware: A personal computer or smartphone with a camera, microphone, and display

[1125] Software: Messenger application

[1126] Data processing and calculation

[1127] Collecting user learning history data

[1128] The server collects learning history data such as the percentage of correct answers for questions the user has answered in the past, the time it took to answer, and the types of questions they got wrong. A database (MySQL) is used for this purpose.

[1129] Analysis of learning history data

[1130] The server analyzes the collected learning history data and evaluates the user's level of understanding. This is done using Python and TensorFlow. Specifically, it runs an algorithm that quantifies the user's level of understanding based on the user's accuracy rate and answer time.

[1131] Generate prompt statement

[1132] Based on the analysis results, the server generates a prompt sentence to be input to the generative AI model. For example, it generates the following prompt sentence:

[1133] "User is studying 8th grade math and struggled with equation problems on a recent test. Generate equation problems at the appropriate level for the user's understanding."

[1134] Problem generation using generative AI models

[1135] The server inputs the generated prompt sentence into a generative AI model (e.g., GPT-4) to generate an appropriate mathematical problem. The generative AI model generates a problem based on the prompt sentence.

[1136] Collecting user emotion data

[1137] The device collects facial and voice data from the user using a camera and microphone. While the user is solving the problem, the device captures facial expressions with the camera and records voice with the microphone.

[1138] Emotional Data Analysis

[1139] The server analyzes the collected emotion data and assesses the user's emotional state using OpenCV and the Google Cloud Speech-to-Text API. The server analyzes facial expression data to determine whether the user is confused and audio data to assess the user's tone of voice.

[1140] Adjusting the level of detail in the explanation

[1141] The server adjusts the level of detail of the explanation based on the analysis of the emotion data. If it determines that the user is confused, it generates a detailed explanation, and if it determines that the user understands, it generates a concise explanation.

[1142] Providing questions and explanations

[1143] The server provides the generated questions and the adjusted explanations to the user. The server sends the questions and explanations to the terminal, which displays them to the user.

[1144] Specific examples

[1145] For example, if the user is studying eighth-grade math, the server might input the following prompts into the generative AI model:

[1146] "User is studying 8th grade math and struggled with equation problems on a recent test. Generate equation problems at the appropriate level for the user's understanding."

[1147] Based on this prompt, the generative AI model generates the following question:

[1148] "Solve the following equation: 3x + 5 = 20"

[1149] Also, if the emotion engine determines that the user is confused by the issue, the server will provide a detailed explanation, such as:

[1150] "First, subtract 5 from both sides of the equation. Then divide what's left, 3x, by 20 minus 5. This gives you the value of x."

[1151] On the other hand, if the server determines that the user does understand, it provides a brief explanation such as:

[1152] "Solve the equation to find the value of x."

[1153] In this way, the system supports efficient learning by providing appropriate questions and explanations according to the user's level of understanding and emotional state.

[1154] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1155] Step 1:

[1156] Collecting user learning history data

[1157] The server retrieves learning history data from a database (MySQL), such as the accuracy rate of questions the user has answered in the past, the time it took to answer, and the types of questions they got wrong.

[1158] Input: User ID

[1159] Output: Learning history data

[1160] Specific operation: Every time a user logs in, the server queries and obtains the user's learning history data from the database.

[1161] Step 2:

[1162] Analysis of learning history data

[1163] The server analyzes the collected learning history data and evaluates the user's level of understanding, using Python and TensorFlow.

[1164] Input: Learning history data

[1165] Output: User's understanding rating

[1166] Specific operation: The server executes an algorithm that quantifies the user's level of understanding based on the user's accuracy rate and answer time.

[1167] Step 3:

[1168] Generate prompt statement

[1169] Based on the analysis results, the server generates a prompt sentence to be input into the generative AI model.

[1170] Input: User's comprehension rating

[1171] Output: prompt statement

[1172] Specific behavior: The server generates a prompt sentence based on the user's level of understanding, such as: "The user is studying eighth-grade math and has struggled with equation problems on recent tests. Please generate equation problems at an appropriate level based on the user's level of understanding."

[1173] Step 4:

[1174] Problem generation using generative AI models

[1175] The server inputs the generated prompt sentences into a generative AI model (e.g., GPT-4) to generate appropriate math problems.

[1176] Input: prompt statement

[1177] Output: Generated problem

[1178] Specific operation: The server sends a prompt to the generative AI model and receives the generated problem.

[1179] Step 5:

[1180] Collecting user emotion data

[1181] The device collects facial and voice data from the user using a camera and microphone.

[1182] Input: Real-time user data (facial expressions, voice)

[1183] Output: Emotion data

[1184] Specific operation: While the user is solving the problem, the device captures facial expressions with a camera and records audio with a microphone.

[1185] Step 6:

[1186] Emotional Data Analysis

[1187] The server analyzes the collected emotion data and assesses the user's emotional state using OpenCV and the Google Cloud Speech-to-Text API.

[1188] Input: Emotion data

[1189] Output: User's emotional state rating

[1190] Specific operation: The server analyzes facial expression data to determine whether the user is confused, and analyzes voice data to evaluate the user's tone of voice.

[1191] Step 7:

[1192] Adjusting the level of detail in the explanation

[1193] The server adjusts the level of detail of the commentary based on the results of analyzing the emotional data.

[1194] Input: User's emotional state rating

[1195] Output: Adjusted commentary

[1196] Specific behavior: If the server determines that the user is confused, it generates a detailed explanation, and if it determines that the user understands, it generates a concise explanation.

[1197] Step 8:

[1198] Providing questions and explanations

[1199] The server provides the generated questions and tailored explanations to the user.

[1200] Input: Generated questions, adjusted explanations

[1201] Output: The problem and explanation provided to the user

[1202] Specific operation: The server sends the problem and explanation to the terminal, which then displays it to the user.

[1203] (Application example 2)

[1204] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1205] Conventional learning systems do not generate questions based on the user's learning history or level of understanding, nor do they adjust explanations based on the user's emotions, making efficient learning difficult. In particular, they are unable to provide appropriate feedback when the user is confused or has already understood the material, which reduces the effectiveness of learning.

[1206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1207] In this invention, the server includes means for a user to send a question as an image to a messenger app BOT, means for analyzing the question from the sent image, means for solving the analyzed question, means for converting between a generative AI and a formula processing language for generating an answer, means for generating an explanation of the answer, means for generating questions based on the user's learning history and level of understanding, and means for adjusting the level of detail of the explanation of the answer based on the user's emotions. This makes it possible to generate appropriate questions according to the user's learning history and level of understanding, and to adjust the explanation according to the user's emotions.

[1208] "User" refers to an individual who uses the system to learn.

[1209] "Messenger app bot" refers to a messaging application that has an automated response function that allows users to send images of their problems.

[1210] "Image" refers to a data file containing visual information that a User sends to a Messenger App BOT.

[1211] "Problem" refers to a mathematical or other learning task for which a user seeks an answer.

[1212] "Analysis" refers to the process of recognizing and understanding the problem content from the transmitted image.

[1213] "Answer" refers to the correct response to the analyzed problem.

[1214] "Generative AI" refers to artificial intelligence technology that generates answers and explanations to problems.

[1215] A "computational mathematics language" refers to a programming language for processing mathematical expressions on a computer.

[1216] "Explanation" refers to explanations or supplementary information for the answer.

[1217] "Study history" refers to data that records questions that a user has solved in the past and the answers they gave.

[1218] "Level of understanding" refers to an index that indicates how well the user understands the problem.

[1219] "Emotion" refers to the psychological state that a user feels when solving a problem.

[1220] "Level of detail" refers to the degree to which the content of the commentary is explained in detail.

[1221] A system for implementing this invention includes means for a user to send a question as an image to a messenger app BOT, means for analyzing the question from the sent image, means for solving the analyzed question, means for converting between a generative AI and a mathematical processing language to generate an answer, means for generating an explanation for the answer, means for generating a question based on the user's learning history and level of understanding, and means for adjusting the level of detail of the explanation for the answer based on the user's emotions.

[1222] System configuration

[1223] 1. User device: The device through which the user submits the question, such as a smartphone or tablet.

[1224] 2. Messenger app bot: A messaging application with an automated response function that allows users to send images of their problems.

[1225] 3. Server: A central processing unit for analyzing problems, generating answers, generating explanations, generating questions, and sentiment analysis.

[1226] Software used

[1227] Generative AI: Artificial intelligence technology for generating answers and explanations to problems.

[1228] Computer algebra language: A programming language for processing mathematical expressions on a computer.

[1229] Sentiment analysis engine: Software for analyzing user emotions.

[1230] Learning history management system: A system that records the user's learning history and uses it to generate questions next time.

[1231] Processing flow

[1232] 1. Sending the problem: The user sends the problem image to the messenger app BOT using their smartphone.

[1233] 2. Problem analysis: The server receives the image and analyzes the problem. Image recognition technology is used for the analysis.

[1234] 3. Answer generation: The generative AI generates an answer to the analyzed problem. It uses a mathematical processing language to derive an accurate answer.

[1235] 4. Explanation generation: Generative AI generates an explanation for the answer. The level of detail of the explanation is adjusted by a sentiment analysis engine.

[1236] 5. Question generation: New questions are generated based on the user's learning history and level of understanding.

[1237] 6. Sentiment analysis: Analyze the user's emotions in real time while answering questions and provide appropriate feedback.

[1238] Specific examples

[1239] For example, if the sentiment analysis engine determines that a user is "confused" while solving a math problem, it will provide a detailed explanation, whereas if it determines that the user "understands," it will provide a concise explanation.

[1240] Prompt Sentence Examples

[1241] Generate math problems at the appropriate level based on the user's learning history and understanding, and adjust the level of detail in the explanations depending on the user's emotions: provide detailed explanations if the user is confused, and concise explanations if the user understands.

[1242] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1243] Step 1:

[1244] The user sends the image in question to the messenger app BOT using their smartphone.

[1245] Input: A user-taken image of the problem.

[1246] Output: Image data sent to the messenger app BOT.

[1247] How it works: The user takes a picture of the problem with their smartphone camera and sends it to the messenger app BOT, which then transfers the image data to the server.

[1248] Step 2:

[1249] The server receives the transmitted image and analyzes the problem.

[1250] Input: Image data transferred from the messenger app BOT.

[1251] Output: The parsed text data of the problem.

[1252] Specific operation: The server uses image recognition technology (e.g., OCR) to extract the problem text from the image data. The extracted text data is sent to the problem analysis module.

[1253] Step 3:

[1254] Generative AI generates answers to the analyzed problems.

[1255] Input: The parsed text data of the problem.

[1256] Output: Text data of the generated answer.

[1257] Specific operation: The server uses generative AI to generate answers to the analyzed problems, and derives accurate answers using a mathematical processing language.

[1258] Step 4:

[1259] Generative AI generates explanations for the answers.

[1260] Input: The text data of the generated answer.

[1261] Output: The generated commentary text.

[1262] How it works: The server uses generative AI to generate explanations for the answers. The level of detail of the explanations is adjusted by a sentiment analysis engine.

[1263] Step 5:

[1264] New questions are generated based on the user's learning history and level of understanding.

[1265] Input: User learning history data, comprehension data.

[1266] Output: The text data of the new problem generated.

[1267] Specific operation: The server analyzes the user's learning history and level of understanding, and generates new questions at an appropriate level. The generated questions will be used for the next learning session.

[1268] Step 6:

[1269] Analyzes the user's emotions while answering in real time and provides appropriate feedback.

[1270] Input: Facial expression data, voice data, etc. while the user answers.

[1271] Output: Sentiment analysis results and appropriate feedback text data.

[1272] Specific operation: The server uses an emotion analysis engine to analyze the emotions expressed in the user's answers in real time. Based on the analysis results, it provides detailed or concise explanations.

[1273] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1274] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1275] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1276] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1277] [Third embodiment]

[1278] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1279] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1280] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1281] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1282] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1283] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1284] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1285] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1286] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1287] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1288] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1289] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1290] "Example 1"

[1291] In one embodiment of the present invention, a user sends a math problem as an image to a messenger app BOT. The BOT analyzes the problem from the image and solves the analyzed problem. The answer is generated using generative AI and a conversion means in the Wolfram Language. Furthermore, an explanation for the answer is also generated by the generative AI. Specifically, when a user sends an image containing a math problem to the BOT, the BOT reads the problem from the image and solves the problem. The answer is generated using generative AI and a conversion means in the Wolfram Language, and an explanation for the answer is also generated at the same time. "Example 2"

[1292] In another aspect of the present invention, a problem generation function can be added in the future. Specifically, the system can generate mathematics problems at an appropriate level based on the user's learning history and level of understanding. This problem generation function allows users to solve problems tailored to their own level, enabling them to study more efficiently.

[1293] The processing flow of each embodiment will be described below.

[1294] "Example 1"

[1295] Step 1: The user sends an image containing a math problem to the messenger app bot.

[1296] Step 2: The bot analyzes the problem from the image sent. This analysis is done using image recognition technology.

[1297] Step 3: The analyzed problem is solved using generative AI and transformation tools in the Wolfram Language.

[1298] Step 4: An explanation for the answer is also generated by the generative AI and presented to the user.

[1299] "Example 2"

[1300] Step 1: The user sends an image containing a math problem to the messenger app bot.

[1301] Step 2: The bot analyzes the problem from the image sent. This analysis is done using image recognition technology.

[1302] Step 3: The analyzed problem is solved using generative AI and transformation tools in the Wolfram Language.

[1303] Step 4: An explanation for the answer is also generated by the generative AI and presented to the user.

[1304] Step 5: The system generates math problems at an appropriate level based on the user's learning history and level of understanding.

[1305] Example 1

[1306] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1307] Conventional mathematical problem-solving systems require users to input problems in text format, which is time-consuming, and they have the problem of not being able to handle handwritten or image-formatted problems. Furthermore, the generation of the answer and its explanation must be done separately, which makes it difficult for users to understand.

[1308] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1309] In this invention, the server includes means for a user to send a problem as an image to a messenger application, means for transferring the sent image to the server, means for the server to extract text data from the image using image analysis software, means for inputting the extracted text data into a generative AI model to generate an answer to the problem, means for converting the generated answer into a mathematical formula or a calculation result, means for generating an explanation for the answer using the generative AI model, and means for returning the answer and explanation to the user. This allows users to easily solve mathematical problems in handwritten or image format and obtain the answer and its explanation at the same time.

[1310] A "user" is any person or entity that intends to use the system to solve mathematical problems.

[1311] A "messenger application" is communication software that users use to send text and images.

[1312] An "image" is digital data that visually represents a mathematical problem.

[1313] A "server" is a computer system that receives and processes data sent by users.

[1314] "Image analysis software" is a program for extracting text data from images.

[1315] "Text data" is text information extracted by image analysis software.

[1316] A "generative AI model" is an artificial intelligence system that generates answers and explanations to questions based on input text data.

[1317] "Conversion means" refers to techniques or tools for converting generated answers into mathematical formulas or calculation results.

[1318] An "answer" is a solution to a user-submitted math problem.

[1319] An "explanation" is an explanation generated by a generative AI model to help understand the answer.

[1320] "Means for sending back" refers to the method by which the server sends back the generated answers and explanations to the user.

[1321] MODE FOR CARRYING OUT THE INVENTION

[1322] The present invention is a system that allows a user to send an image containing a mathematical problem to a messenger application, analyzes the problem from the image, and generates an answer and explanation. A specific embodiment of this system will be described below.

[1323] Hardware and software used

[1324] Server: A high-performance computer system that analyzes images, runs generative AI models, and transforms and returns data.

[1325] Device: The smartphone or PC used by the user to send images via a messenger application.

[1326] Messenger application: Communication software that allows users to send images.

[1327] Image analysis software: Extract text data from images, using Tesseract OCR as an example.

[1328] Generative AI model: As an example, we use OpenAI's GPT-4 to generate answers and explanations for problems.

[1329] Conversion method: For example, we use the Wolfram Alpha API to convert the generated answers into mathematical formulas and calculation results.

[1330] System Operation

[1331] A user sends an image containing a math problem to a messenger application. The device transfers the image to a server. The server uses image analysis software to extract text data from the image. The extracted text data is input into a generative AI model to generate an answer to the problem. The generated answer is then converted into a mathematical formula or calculation result using a transformation method. The generative AI model also generates an explanation for the answer. Finally, the server returns the answer and explanation to the user.

[1332] Specific examples

[1333] For example, if a user sends an image of the equation "x^2 + 3x + 2 = 0" to a messenger application, the process would proceed as follows:

[1334] 1. The user takes a picture of the equation with their smartphone and sends it to a messenger application.

[1335] 2. The device transfers the image data to the server.

[1336] 3. The server uses Tesseract OCR to extract the text data "x^2 + 3x + 2 = 0" from the image.

[1337] 4. The server sends the prompt "Solve the equation x^2 + 3x + 2 = 0" to the generated AI model, which generates the answer "x = -1, x = -2".

[1338] 5. The server uses the Wolfram Alpha API to materialize the answer in mathematical form.

[1339] 6. The server sends the prompt "Explain the steps to solve the quadratic equation x^2 + 3x + 2 = 0" to the generated AI model and generates an explanation.

[1340] 7. The server returns the answer "x = -1, x = -2" to the user along with the explanation "This equation is a quadratic equation and can be solved using the formula. The formula gives us x = -1 and x = -2."

[1341] In this way, users can receive answers and explanations for math problems simply by submitting an image of the problem.

[1342] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1343] Step 1:

[1344] A user sends an image containing a math problem to a messenger application. Using a smartphone or PC, the user takes a photo of the handwritten equation "x^2 + 3x + 2 = 0" and sends it to the messenger application. The input is the image containing the math problem, and the output is the image sent to the messenger application.

[1345] Step 2:

[1346] The device transfers the image sent to the server. The device transfers the image data sent by the user to the server in an appropriate format (e.g., JPEG, PNG). The input is the image sent to the messenger application, and the output is the image data transferred to the server.

[1347] Step 3:

[1348] The server uses image analysis software to extract text data from the image. The server uses image analysis software (e.g., Tesseract OCR) to extract the text data "x^2 + 3x + 2 = 0" from the image. The input is the image data transferred to the server, and the output is the extracted text data.

[1349] Step 4:

[1350] The server inputs the extracted text data into a generative AI model to generate an answer to the problem. The server inputs the extracted text data into a generative AI model (e.g., OpenAI's GPT-4) and sends the prompt "Solve the equation x^2 + 3x + 2 = 0" to the generative AI model. The generative AI model generates the answer "x = -1, x = -2". The input is the extracted text data and the prompt, and the output is the generated answer.

[1351] Step 5:

[1352] The server materializes the answer using Wolfram Language transformations. The server materializes the generated answer as a formula or result using Wolfram Language transformations (e.g., Wolfram Alpha API). The input is the generated answer, and the output is the materialized formula or result.

[1353] Step 6:

[1354] The server generates an explanation for the answer using a generative AI model. The server sends the generative AI model a prompt, "Explain the steps to solve the quadratic equation x^2 + 3x + 2 = 0," and generates an explanation. The generative AI model generates an explanation, "This equation is a quadratic equation and can be solved using the solution formula. The solution formula gives x = -1 and x = -2." The input is the generated answer and prompt, and the output is the generated explanation.

[1355] Step 7:

[1356] The server returns the answer and explanation to the user. The server returns the generated answer "x = -1, x = -2" and explanation "This equation is a quadratic equation and can be solved using the solution formula. The solution formula gives x = -1 and x = -2." The input is the generated answer and explanation, and the output is the answer and explanation returned to the user.

[1357] (Application example 1)

[1358] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1359] Conventional math problem-solving systems require users to input questions, and it takes time to obtain the answers and explanations. In addition, the accuracy of the answers and the quality of the explanations are often not guaranteed, which can reduce the user's learning efficiency.

[1360] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for a user to send a problem as an image to a messenger app BOT, means for analyzing the problem from the sent image, means for solving the analyzed problem, means for converting between a generative AI and a formula manipulation language to generate an answer, means for generating an explanation of the answer, means including an application to be installed on the smartphone, means for extracting text from an image using an image processing library, means for solving the problem using a formula manipulation language API, means for generating an explanation using a generative AI, and means for displaying the answer and explanation to the user. This allows a user to easily analyze a mathematical problem photographed with their smartphone and quickly obtain an accurate answer and high-quality explanation.

[1361] A "user" is an individual or group that uses the system to solve mathematical problems.

[1362] "Messenger App BOT" is an automatic response program that receives and analyzes mathematical problems sent by users as images.

[1363] "Means of sending as an image" refers to a method in which a user takes a photo of a math problem using a smartphone or other device and sends the image to a messenger app BOT.

[1364] The "means for analyzing problems from transmitted images" is a method for extracting text from images using an image processing library and recognizing the text as a mathematical problem.

[1365] The "means for solving the analyzed problem" is a method for solving the extracted mathematical problem using a computer algebra language API.

[1366] A "generative AI for generating solutions" is an artificial intelligence model used to generate answers to mathematical problems.

[1367] A "computer algebra language transformation means" is a method of using a computer algebra language to solve a mathematical problem.

[1368] "Means for generating explanations for answers" refers to a method for generating detailed explanations for answers using generative AI.

[1369] The "application installed on a smartphone" is software that allows users to take a photo of a math problem using their smartphone and receive the answer and explanation.

[1370] An "image processing library" is a software tool for extracting text from images.

[1371] A "computer algebra language API" is a programmatic interface used to solve mathematical problems.

[1372] "Means for generating explanations using generative AI" refers to a method for generating explanations for answers to mathematical problems using generative AI.

[1373] "Means for displaying answers and explanations to the user" refers to a method for displaying the generated answers and explanations on the user's smartphone.

[1374] The system for implementing this invention starts when a user takes a photo of a math problem using a smartphone and sends the image to a messenger app BOT. A specific embodiment of this system will be described below.

[1375] Hardware and software used

[1376] Hardware: Smartphone (with camera)

[1377] software:

[1378] Messenger app BOT

[1379] Image processing library (e.g. OpenCV)

[1380] OCR (Optical Character Recognition) libraries (e.g., pytesseract)

[1381] Mathematical processing language API (e.g. WolframAlpha API)

[1382] Generative AI models (e.g., GPT-3)

[1383] Data processing and calculation

[1384] 1. Image Acquisition:

[1385] Users take a picture of a math problem using their smartphone camera and send the image to a messenger app BOT.

[1386] 2. Image Analysis:

[1387] The server converts the submitted image to grayscale using an image processing library (OpenCV) and extracts text from the image using an OCR library (pytesseract).

[1388] 3. Problem answer:

[1389] The extracted text is sent to a mathematical processing language API (WolframAlpha API), and the server obtains the answer.

[1390] 4. Description generation:

[1391] The server uses a generative AI model (GPT-3) to generate a detailed explanation for the obtained answer. Specifically, it inputs the following prompt sentence into the generative AI:

[1392] Example prompt sentence:

[1393] Math problem: x^2 + 2x + 1 = 0

[1394] Answer: x = -1

[1395] Description:

[1396] 5. Results display:

[1397] The server sends the generated answers and explanations to the user's smartphone and displays them through the application.

[1398] Specific examples

[1399] If a user photographs and submits the problem "x^2 + 2x + 1 = 0," the system operates as follows:

[1400] 1. Extract the text "x^2 + 2x + 1 = 0" from the image.

[1401] 2. Use the formula processing language API to obtain the answer "x = -1".

[1402] 3. Use generative AI to generate an explanation such as, "This equation is a quadratic equation, and using the formula for the solution, we get x = -1."

[1403] 4. Display the answer and explanation to the user.

[1404] In this way, users can easily analyze math problems photographed with their smartphones and quickly obtain accurate answers and high-quality explanations.

[1405] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1406] Step 1:

[1407] A user takes a picture of a math problem using the camera on their smartphone and sends the image to a messenger app BOT. The input is the image of the math problem that was taken, and the output is the image sent to the messenger app BOT.

[1408] Step 2:

[1409] The server converts the transmitted image to grayscale using an image processing library (OpenCV). The input is the image transmitted by the user, and the output is the image converted to grayscale. Specifically, the server reads the image using OpenCV functions and converts it to grayscale.

[1410] Step 3:

[1411] The server uses an OCR library (pytesseract) to extract text from grayscale images. The input is the image converted to grayscale, and the output is the extracted text. Specifically, it uses pytesseract's functions to read text from the image.

[1412] Step 4:

[1413] The server sends the extracted text to a mathematical expression processing language API (WolframAlpha API) and obtains the answer. The input is the extracted text, and the output is the answer obtained from the mathematical expression processing language API. Specifically, the server sends the text to the WolframAlpha API and receives the answer.

[1414] Step 5:

[1415] The server uses a generative AI model (GPT-3) to generate a detailed explanation for the obtained answer. The input is the answer obtained from the formula processing language API, and the output is the generated explanation. Specifically, the following prompt sentence is input to the generative AI:

[1416] Example prompt sentence:

[1417] Math problem: x^2 + 2x + 1 = 0

[1418] Answer: x = -1

[1419] Description:

[1420] Step 6:

[1421] The server sends the generated answers and explanations to the user's smartphone and displays them through the application. The input is the generated answers and explanations, and the output is the answers and explanations displayed on the user's smartphone. Specifically, the answers and explanations are sent through a messenger app BOT and displayed on the smartphone application.

[1422] Example 2

[1423] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1424] Conventional learning systems have difficulty generating questions based on a user's learning history and level of understanding, making it impossible to provide optimal questions for each individual user. Furthermore, efficient and accurate processing is required for analyzing questions answered by users and generating answers. This reduces the user's learning efficiency, making effective learning difficult.

[1425] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1426] In this invention, the server includes means for a user to send a question to a messenger application as an image, means for analyzing the question from the sent image, and means for answering the analyzed question, thereby making it possible to collect the user's learning history data, evaluate the user's level of understanding based on the analysis results, create prompt sentences to be input to the generative AI based on the level of understanding, generate questions using the generative AI, and provide the generated questions to the user.

[1427] "User" refers to an individual or organization that uses the system to study.

[1428] "Messenger application" refers to communication software that allows users to send and receive text and images.

[1429] "Image" refers to visual data that a user sends through a messenger application.

[1430] "Question" refers to a task or question provided for a user to answer.

[1431] "Analysis" refers to the process of reading and understanding the problem content from the transmitted image.

[1432] "Solution" refers to the answer to the question being analyzed.

[1433] "Generative AI" refers to AI technology that generates questions and answers based on a user's learning history and level of understanding.

[1434] "Programming language" refers to the computer language used by generative artificial intelligence to generate answers.

[1435] "Study history data" refers to data relating to questions that a user has answered in the past and the results of those answers.

[1436] "Analysis results" refers to information obtained after analyzing learning history data.

[1437] "Level of understanding" refers to an index that indicates how well a user can understand a question and answer it.

[1438] A "prompt sentence" refers to an instruction sentence that is input to a generative artificial intelligence when generating a problem.

[1439] "Problem generation" refers to the process by which a generative AI creates new questions based on prompts.

[1440] "Providing" refers to the act of displaying or transmitting the generated question to the user.

[1441] This invention is a system that allows a user to send a question as an image to a messenger application, analyzes the question from the image, generates an answer, and further generates new questions based on the user's learning history data. A specific embodiment of this system will be described below.

[1442] Hardware and software used

[1443] The server uses the following hardware and software:

[1444] Hardware: High-performance server machine

[1445] Software: generative AI models (e.g., GPT-4), database management systems, messenger applications

[1446] Data processing and calculation

[1447] 1. Collecting user learning history data

[1448] The server collects data on questions that users have previously solved and stores it in a database, including the type of question, the percentage of correct answers, and the time it took to solve the question.

[1449] 2. Analysis of learning history data

[1450] The server analyzes the collected learning history data, specifically, what answers the user gave to which questions and how long it took them to answer them.

[1451] 3. Assessment of understanding

[1452] The server evaluates the user's level of understanding based on the analysis results. For example, if the correct answer rate is high and the answer time is short, the server evaluates the user as "advanced," and if the correct answer rate is low and the answer time is long, the server evaluates the user as "beginner."

[1453] 4. Creating prompts for question generation

[1454] The server creates a prompt sentence to be input to the generative AI model based on the user's level of comprehension. For example, if the user's level of comprehension is evaluated as "intermediate," the server creates a prompt sentence such as "Generate intermediate level math problems."

[1455] 5. Problem generation using generative AI models

[1456] The server inputs a prompt into the generative AI model to generate a math problem. For example, based on the prompt "Generate an intermediate level math problem," the model generates the problem "Solve the following equation: 2x + 3 = 7."

[1457] 6. Providing generated questions

[1458] The server provides the generated questions to the user's device, and the user can progress with their learning by solving the questions provided.

[1459] Specific examples

[1460] Suppose the data of problems that a user has solved in the past is stored as follows:

[1461] Question 1: Correct answer, answer time 5 minutes

[1462] Question 2: Wrong answer, answer time 10 minutes

[1463] Question 3: Correct answer, answer time 7 minutes

[1464] The server analyzes this data and evaluates the user's level of understanding as "intermediate." It then feeds the following prompt to the generative AI model:

[1465] "If the user has an intermediate level of comprehension, generate intermediate level math problems."

[1466] Based on this prompt, the generative AI model generates intermediate-level math problems, such as "Solve the following equation: 2x + 3 = 7."

[1467] In this way, users can solve problems that are suited to their own level, enabling them to study more efficiently.

[1468] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1469] Step 1: Collecting user learning history data

[1470] The server collects data on questions that users have previously solved, and stores the results of questions that users have solved through the messenger application (such as correct answer rate and answer time) in a database.

[1471] Input: The result of the question answered by the user

[1472] Data processing: Data such as question type, correct answer rate, and answer time are extracted and saved in a database

[1473] Output: Learning history data stored in a database

[1474] Step 2: Analyzing learning history data

[1475] The server retrieves the user's learning history data from the database and analyzes it, specifically, analyzing what answers the user gave to which questions and how long it took them to answer.

[1476] Input: Learning history data stored in the database

[1477] Data calculation: Calculation of correct answer rate, answer time, evaluation of problem difficulty

[1478] Output: Analysis results (user correct answer rate, answer time, difficulty of questions)

[1479] Step 3: Assess understanding

[1480] The server evaluates the user's level of understanding based on the analysis results. For example, if the correct answer rate is high and the answer time is short, the server evaluates the user as "advanced," and if the correct answer rate is low and the answer time is long, the server evaluates the user as "beginner."

[1481] Input: Analysis results

[1482] Data arithmetic: Evaluation of comprehension based on correct answer rate and answer time

[1483] Output: User's level of understanding (advanced, intermediate, beginner, etc.)

[1484] Step 4: Create a prompt to generate questions

[1485] The server creates a prompt sentence to be input to the generative AI model based on the user's level of comprehension. For example, if the user's level of comprehension is evaluated as "intermediate," the server creates a prompt sentence such as "Generate intermediate level math problems."

[1486] Input: User comprehension

[1487] Data processing: Prompt generation based on comprehension

[1488] Output: A prompt to be input to the generative AI model

[1489] Step 5: Generate questions using a generative AI model

[1490] The server inputs a prompt into the generative AI model to generate a math problem. For example, based on the prompt "Generate an intermediate level math problem," the model generates the problem "Solve the following equation: 2x + 3 = 7."

[1491] Input: prompt statement

[1492] Data Computation: Problem Generation with Generative AI Models

[1493] Output: Generated math problem

[1494] Step 6: Providing generated questions

[1495] The server provides the generated questions to the user's device, and the user can progress with their learning by solving the questions provided.

[1496] Input: Generated math problem

[1497] Data processing: Convert data to fit the problem format

[1498] Output: The problem as it appears on the user's terminal

[1499] (Application example 2)

[1500] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1501] Conventional learning systems have had the problem of being unable to provide users with questions at an appropriate level based on their learning history and level of understanding, making it difficult for them to efficiently solve questions suited to their own level. Furthermore, they lacked the functionality to record the history of questions answered by users and dynamically update their learning level, making it difficult to effectively manage learning progress.

[1502] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1503] In this invention, the server includes means for a user to send a problem as an image to a messenger app, means for analyzing the problem from the sent image, means for solving the analyzed problem, means for converting between a generative AI and a computer algebra language to generate an answer, means for generating an explanation of the answer, means for generating problems of an appropriate level based on the user's learning history and level of understanding, means for presenting the generated problems to the user, and means for recording the user's answer history and updating the learning level. This allows the user to efficiently solve problems that match their learning level and effectively manage their learning progress.

[1504] "User" refers to an individual or group that uses the system to study.

[1505] A "messenger app" is a software application for sending and receiving messages such as text and images.

[1506] "Image" means a digital representation of visual information used by a user to submit a question.

[1507] A "problem" is a mathematical or other learning task for which a user seeks an answer.

[1508] "Analysis" is the process of recognizing the problematic content from the transmitted image and extracting it as digital data.

[1509] An "answer" is a correct answer to the problem being analyzed.

[1510] "Generative AI" is a system that uses artificial intelligence technology to generate answers and explanations to problems.

[1511] A "computational mathematics language" is a programming language for processing mathematical expressions on a computer.

[1512] "Explanation" is information that explains the process or method for arriving at the answer.

[1513] "Study history" is data that records questions that a user has answered in the past and their scores.

[1514] "Level of understanding" is an index that indicates how well a user understands a particular learning content.

[1515] "Level" refers to the level of difficulty according to the user's learning progress and ability.

[1516] "Presenting" is the act of displaying the generated questions and answers to the user.

[1517] "Answer history" is data that records questions that a user has answered in the past and the results of those answers.

[1518] "Study level update" is the process of dynamically changing a user's learning level based on the user's answer history.

[1519] A system for implementing this invention includes means for a user to send a question as an image to a messenger app, means for analyzing the question from the sent image, means for answering the analyzed question, means for converting between a generative AI and a mathematical processing language to generate an answer, means for generating an explanation of the answer, means for generating questions of an appropriate level based on the user's learning history and level of understanding, means for presenting the generated questions to the user, and means for recording the user's answer history and updating the learning level.

[1520] System Program Overview

[1521] The system is realized using the following hardware and software.

[1522] Hardware:

[1523] Smartphone

[1524] server

[1525] software:

[1526] Messenger app

[1527] Generative AI (e.g. GPT-3)

[1528] mathematical processing languages ​​(e.g., Wolfram Language)

[1529] JSON file (saves user data)

[1530] Processing flow

[1531] 1. User submits issue:

[1532] Users use a messenger app to send an image of a question that requires an answer.

[1533] 2. Image Analysis:

[1534] The server receives the transmitted image and uses image analysis technology to extract the problematic content as digital data.

[1535] 3. Generate answers to questions:

[1536] Using generative AI and mathematical processing languages, answers to the analyzed problems are generated.

[1537] 4. Generate explanations for answers:

[1538] Generative AI generates explanations that explain the process and methods used to arrive at the answer.

[1539] 5. Problem Generation:

[1540] The server generates new questions at an appropriate level based on the user's learning history and level of understanding.

[1541] 6. Problem Statement:

[1542] The generated questions are presented to the user through a messenger app.

[1543] 7. Record your answer history and update your learning level:

[1544] The user's answer history is recorded and their learning level is dynamically updated.

[1545] Specific examples

[1546] When a user first uses the MathMaster app, the app sets the user's level as "beginner." Each time the user solves a problem, the number of correct answers is counted, and once a certain number of correct answers are achieved, the level is updated to "intermediate" or "advanced."

[1547] Prompt Sentence Examples

[1548] Generate math problems at the appropriate level based on the user's learning history and level of understanding. The user's level can be "beginner," "intermediate," or "advanced." Provide the Python code that generates problems for each level.

[1549] In this way, the system can maximize the user's learning efficiency and effectively manage the learning progress.

[1550] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1551] Step 1:

[1552] The user sends the problem as an image to a messenger app. The user uses the messenger app on their smartphone to take a picture of the problem for which they want to answer and send it. The input is the image of the problem taken by the user, and the output is the image data sent to the server via the messenger app.

[1553] Step 2:

[1554] The server receives the image and performs image analysis. The server analyzes the received image data using an image analysis algorithm (e.g., OCR technology) and extracts the problem content as text data. The input is the image data sent by the user, and the output is the analyzed text data of the problem.

[1555] Step 3:

[1556] The server generates an answer to the analyzed problem. The server uses a generative AI (e.g., GPT-3) and a mathematical processing language (e.g., the Wolfram Language) to generate an answer based on the text data of the analyzed problem. The input is the text data of the analyzed problem, and the output is the generated answer.

[1557] Step 4:

[1558] The server generates an explanation that explains the process and method for arriving at the answer. Using generative AI, an explanation is generated that explains in detail the process and method for arriving at the answer. The input is the generated answer, and the output is an explanation of the answer.

[1559] Step 5:

[1560] The server generates new questions at an appropriate level based on the user's learning history and level of comprehension. The server references the user's learning history data and comprehension data and uses generative AI to generate new questions at an appropriate level. The input is the user's learning history data and comprehension data, and the output is the generated new questions.

[1561] Step 6:

[1562] The server presents the generated problem to the user. The generated new problem is sent to the user through a messenger app and presented to them. The input is the generated new problem, and the output is the problem presented to the user.

[1563] Step 7:

[1564] The server records the user's answer history and updates the learning level. The server records the history of questions answered by the user and dynamically updates the learning level. The input is the user's answer history data, and the output is the updated learning level.

[1565] In this way, the system can maximize the user's learning efficiency and effectively manage the learning progress.

[1566] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1567] "Example 1"

[1568] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system includes a means for a user to send a problem to a messenger app BOT as an image, a means for analyzing the problem from the sent image, a means for answering the analyzed problem, and a means for generating an answer.

[1569] It includes a generative AI for solving problems, a Wolfram Language conversion means, a means for generating explanations for answers, and an emotion engine that recognizes the user's emotions. The emotion engine recognizes the user's emotions and adjusts the difficulty of the problems based on those emotions. For example, if the user feels confused, the emotion engine lowers the difficulty of the problems. On the other hand, if the emotion engine senses that the user is feeling joy or a sense of accomplishment, the emotion engine increases the difficulty of the problems. This improves the user's learning efficiency and satisfaction.

[1570] "Example 2"

[1571] Furthermore, the emotion engine adjusts the level of detail in the answer explanation based on the user's emotions. For example, if the user feels confused, the emotion engine generates a more detailed explanation. On the other hand, if the user feels understood, the emotion engine generates a more concise explanation. This provides an appropriate explanation according to the user's level of understanding, enabling efficient learning.

[1572] The processing flow of each embodiment will be described below.

[1573] "Example 1"

[1574] Step 1: The user sends the problem as an image to the messenger app BOT.

[1575] Step 2: The system analyzes the problem from the image sent.

[1576] Step 3: The system answers the analyzed question.

[1577] Step 4: The system generates a solution using generative AI and the Wolfram Language.

[1578] Step 5: The system generates an explanation for the answer.

[1579] Step 6: The emotion engine recognizes the user's emotion.

[1580] Step 7: Adjust the difficulty of the questions based on the emotions recognized by the emotion engine.

[1581] "Example 2"

[1582] Step 1: The user sends the problem as an image to the messenger app BOT.

[1583] Step 2: The system analyzes the problem from the image sent.

[1584] Step 3: The system answers the analyzed question.

[1585] Step 4: The system generates a solution using generative AI and the Wolfram Language.

[1586] Step 5: The system generates an explanation for the answer.

[1587] Step 6: The emotion engine recognizes the user's emotion.

[1588] Step 7: Adjust the detail of the answer explanation based on the sentiment recognized by the sentiment engine.

[1589] Example 1

[1590] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1591] In conventional math problem-solving systems, when users submit problems as images, the problem analysis and answer generation may not be performed accurately. Furthermore, there is a lack of emotion recognition functionality to improve user learning efficiency and satisfaction, which means that the difficulty of the problem cannot be adjusted appropriately according to the user's emotions.

[1592] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1593] In this invention, the server includes means for a user to send a question as an image to a communication application, means for analyzing the question from the sent image, means for solving the analyzed question, generative artificial intelligence and mathematical language conversion means for generating an answer, means for generating an explanation for the answer, emotion analysis means for recognizing the user's emotion, and means for adjusting the difficulty of the question based on the recognized emotion. This makes it possible to accurately analyze the question from the image sent by the user and provide appropriate answers and explanations. Furthermore, adjusting the difficulty of the question according to the user's emotion can improve learning efficiency and satisfaction.

[1594] "User" refers to any person or entity that uses the system to solve mathematical problems.

[1595] "Communication application" refers to a software platform used by users to send messages and images.

[1596] "Image" refers to visual data containing a mathematical problem that a user sends through a communication application.

[1597] "Means of analysis" refers to the technology that recognizes mathematical problems from the transmitted images and extracts them as text data.

[1598] "Means of solving" refers to the technique of deriving an appropriate answer to the analyzed mathematical problem.

[1599] "Generative AI" refers to artificial intelligence technology that generates answers and explanations to mathematical problems.

[1600] "Mathematical language conversion means" refers to the technology that converts the generated answers into formats such as mathematical formulas and graphs.

[1601] "Means for generating an explanation for an answer" refers to technology that uses generative artificial intelligence to generate detailed explanations and solutions for the answer.

[1602] "Emotion analysis means" refers to technology for recognizing and analyzing a user's emotions.

[1603] The "means for adjusting the difficulty of a problem" refers to a technology for changing the difficulty of the next mathematical problem provided based on the user's emotions recognized by the emotion analysis means.

[1604] This invention is a system that allows users to send images containing mathematical problems via a communication application, analyzes the problems from the images, and generates answers and explanations. It also has the ability to recognize the user's emotions and adjust the difficulty of the problems based on those emotions.

[1605] Hardware and software used

[1606] Hardware

[1607] Server: A central processing unit for performing processes such as image analysis, answer generation, and emotion recognition.

[1608] Terminal: A device (smartphone, tablet, PC, etc.) through which a user uses a communication application.

[1609] software

[1610] Communication application: a messenger application that allows users to send images containing math problems.

[1611] OCR Engine: Use the Tesseract OCR engine to extract text from images.

[1612] Generative artificial intelligence: Using OpenAI's GPT-4 and other technologies, we generate answers and explanations for mathematical problems.

[1613] Mathematical language converters: Use the Wolfram Language to convert generated answers into mathematical formulas, graphs, and other formats.

[1614] Sentiment analysis engine: Software for recognizing and analyzing user emotions.

[1615] Data processing and calculation

[1616] Image analysis

[1617] The server receives the image submitted by the user and uses the Tesseract OCR engine to extract text from the image. For example, if the image submitted by the user contains the equation "x^2 + 3x + 2 = 0", the OCR engine will accurately extract this text.

[1618] Answer generation

[1619] The server inputs the extracted text into a generative artificial intelligence (GPT-4) to generate an answer to the problem. It then uses the Wolfram Language to convert the answer into a mathematical formula, graph, or other format. For example, the answer "x = -1 or x = -2" is generated.

[1620] Generate explanations for answers

[1621] The server uses generative artificial intelligence to generate explanations for the solution. The explanations include the solution to the problem and related concepts. For example, the server might generate an explanation like, "This equation can be solved by factorization. First, factor the equation as (x + 1)(x + 2) = 0. Then, set each factor equal to 0 to find the solution."

[1622] Emotion recognition and difficulty adjustment

[1623] The server uses an emotion analysis engine to recognize the user's emotions. It analyzes the user's messages and reactions (e.g., emojis and text) to determine whether the user is confused or happy. Based on the recognized emotions, it adjusts the difficulty of the next problem to be presented. For example, if it determines that the user is confused, it lowers the difficulty of the next problem to be presented. Conversely, if it determines that the user is feeling happy or a sense of accomplishment, it increases the difficulty of the next problem to be presented.

[1624] Specific examples

[1625] Prompt Sentence Examples

[1626] "Please answer the math problem contained in this image and provide an explanation for the answer."

[1627] Specific operation flow

[1628] 1. A user sends an image containing "x^2 + 3x + 2 = 0" through a communication application.

[1629] 2. The server receives the image and uses the Tesseract OCR engine to extract the text "x^2 + 3x + 2 = 0".

[1630] 3. The server uses generative artificial intelligence (GPT-4) and the Wolfram Language to generate the answer "x = -1 or x = -2".

[1631] 4. The server uses generative artificial intelligence to generate an explanation of the answer, including the factorization steps.

[1632] 5. The server uses an emotion analysis engine to analyze the user's emotions, and if it determines that the user is confused, it lowers the difficulty of the next question it presents.

[1633] 6. If the user answers the next question and the sentiment analysis engine recognizes the user's joy or sense of accomplishment, the difficulty of the next question provided will be increased.

[1634] This system allows users to study mathematical problems efficiently.

[1635] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1636] Step 1:

[1637] A user uses a communication application to send an image containing a math problem.

[1638] Input: An image containing a math problem taken or saved by the user.

[1639] Output: Image data sent to a server through a communication application.

[1640] Specific operation: The user uses a smartphone or PC to take a photo of a piece of paper with a math problem written on it, then uploads and sends the image to a communication application.

[1641] Step 2:

[1642] The server receives the image sent by the user.

[1643] Input: Image data sent from a communication application.

[1644] Output: Image data stored in the server's storage.

[1645] Specific operation: The server receives image data through the API of the communication application and temporarily stores it in storage.

[1646] Step 3:

[1647] The server analyzes the image to find the math problem.

[1648] Input: Image data stored on the server.

[1649] Output: Text data extracted from the image.

[1650] What it does: The server uses the Tesseract OCR engine to parse the image data and extract text containing the math problem, for example, "x^2 + 3x + 2 = 0."

[1651] Step 4:

[1652] The server answers the math questions based on the extracted text.

[1653] Input: Text data extracted by the OCR engine.

[1654] Output: Answer data generated by generative artificial intelligence.

[1655] How it works: The server inputs the extracted text into a generative artificial intelligence (GPT-4) to generate an answer to the problem. It then uses the Wolfram Language to convert the answer into a mathematical formula, graph, or other format. For example, the answer "x = -1 or x = -2" is generated.

[1656] Step 5:

[1657] The server generates an explanation for the answer.

[1658] Input: Answer data generated by generative artificial intelligence.

[1659] Output: Explanation data of the answer generated by the generative artificial intelligence.

[1660] How it works: The server uses generative artificial intelligence to generate an explanation for the solution. The explanation includes the solution to the problem and related concepts. For example, the server might generate an explanation like, "This equation can be solved by factorization. First, factor the equation as (x + 1)(x + 2) = 0. Then, set each factor equal to 0 to find the solution."

[1661] Step 6:

[1662] The server recognizes the user's emotions.

[1663] Input: User message and response data.

[1664] Output: Sentiment data recognized by the sentiment analysis engine.

[1665] What it does: The server uses a sentiment analysis engine to analyze the user's messages and reactions (e.g., emojis and text) to determine whether the user is confused or happy.

[1666] Step 7:

[1667] The difficulty of the questions is adjusted based on the emotions recognized by the server.

[1668] Input: Emotion data recognized by the sentiment analysis engine.

[1669] Output: Adjusted problem difficulty data.

[1670] Specific operation: The server adjusts the difficulty of the next problem based on the recognized emotion. For example, if it determines that the user is confused, it will lower the difficulty of the next problem. Conversely, if it determines that the user is feeling joy or a sense of accomplishment, it will increase the difficulty of the next problem.

[1671] (Application example 1)

[1672] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1673] Conventional math problem-solving systems can provide answers and explanations for problems submitted by users, but they have the problem of not being able to adjust the difficulty of the problems according to the user's learning situation or emotions. Also, when a user is confused or, conversely, when they get bored with problems that are too easy, it is difficult to provide problems of an appropriate difficulty.

[1674] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1675] In this invention, the server includes means for a user to send a question as an image to a messenger app BOT, means for analyzing the question from the sent image, means for solving the analyzed question, means for converting between a generative AI and a formula processing language to generate an answer, means for generating an explanation of the answer, means for recognizing the user's emotions, and means for adjusting the difficulty of the questions based on the recognized emotions. This makes it possible to provide questions of an appropriate level of difficulty according to the user's learning situation and emotions.

[1676] A "user" is an individual or entity that uses the system to solve mathematical problems.

[1677] A "messenger app BOT" is an automated program that receives mathematical problems sent by users as images, analyzes them, and provides answers.

[1678] "Means to send as an image" is a function that allows users to send math problems in image format to a messenger app BOT.

[1679] The "means for analyzing problems from transmitted images" is a function for extracting mathematical problems as text from received images and analyzing them.

[1680] The "means for solving the analyzed problem" is a function for generating an answer to the analyzed mathematical problem.

[1681] "Generative AI" is a system that uses artificial intelligence technology to generate answers and explanations for mathematical problems.

[1682] A "computer algebra language" is a programming language used to solve mathematical problems.

[1683] "Means for generating explanations for answers" is a function for generating detailed explanations for answers using generative AI.

[1684] The "means for recognizing the user's emotions" is a function for detecting and recognizing emotions from the user's facial expressions and actions.

[1685] The "means for adjusting the difficulty of the question based on the recognized emotion" is a function for changing the difficulty of the next question to be posed in accordance with the emotion of the user.

[1686] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[1687] System Configuration

[1688] The system includes a messenger app BOT that allows users to send math problems as images, an image analysis engine that analyzes the problems from the images, a generative AI and a mathematical processing language that solves the analyzed problems, a generative AI that generates explanations for the answers, an emotion recognition engine that recognizes the user's emotions, and a difficulty adjustment engine that adjusts the difficulty of the problems based on the recognized emotions.

[1689] Hardware and software used

[1690] Hardware: Servers, user devices (smartphones, tablets, PCs, etc.)

[1691] software:

[1692] Image analysis engine: OpenCV, Tesseract OCR

[1693] Generative AI: Hugging Face transformers library

[1694] Mathematical Computation Language: WolframAlpha API

[1695] Emotion Recognition Engine: EmotionRecognizer Library

[1696] Data processing and calculation

[1697] 1. Image analysis: Images containing mathematical problems sent from the user's device are analyzed using an image analysis engine (OpenCV and Tesseract OCR) on the server and extracted as text data.

[1698] 2. Answer generation: The extracted text data is input into a mathematical processing language (WolframAlpha API) to generate an answer.

[1699] 3. Explanation generation: Using generative AI (Hugging Face's transformers library), we generate detailed explanations for the answers.

[1700] 4. Emotion recognition: The user's facial expressions and behavior are analyzed using an emotion recognition engine (EmotionRecognizer library) to recognize the user's emotions.

[1701] 5. Difficulty Adjustment: Based on the recognized emotions, the difficulty of the next question is adjusted by the difficulty adjustment engine.

[1702] Specific examples

[1703] A user sends an image containing a math problem to a messenger app BOT using their smartphone. The server uses an image analysis engine to extract the problem as text from the image and generates an answer using a formula processing language. The generative AI generates an explanation for the answer using a prompt like the following:

[1704] Question: Solve the equation 2x + 3 = 7.

[1705] Answer: x = 2

[1706] Explain the solution.

[1707] The generated description might look something like this:

[1708] To solve the equation 2x + 3 = 7, we first subtract 3 from both sides to get 2x = 4. Then, we divide both sides by 2 to find x = 2.

[1709] Furthermore, the emotion recognition engine analyzes the user's facial expressions and behavior, and if the user is confused, it lowers the difficulty of the next question, and if the user is happy, it raises the difficulty, thus improving the user's learning efficiency and satisfaction.

[1710] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1711] Step 1:

[1712] A user sends an image containing a math problem to a messenger app BOT using a smartphone. The input is the image of the math problem taken by the user, and the output is the image data sent to the server.

[1713] Step 2:

[1714] The server analyzes the received image data using an image analysis engine (OpenCV and Tesseract OCR). The input is the transmitted image data, and the output is the text data extracted from the image. Specifically, the image is read using OpenCV, and the text in the image is extracted using Tesseract OCR.

[1715] Step 3:

[1716] The server inputs the extracted text data into a mathematical processing language (WolframAlpha API) to generate an answer. The input is the extracted text data, and the output is the answer to the mathematical problem. Specifically, the server sends the text data to the WolframAlpha API and obtains the returned answer.

[1717] Step 4:

[1718] The server uses a generative AI (Hugging Face's transformers library) to generate a detailed explanation of the answer. The input is a math problem and its answer, and the output is an explanation of the answer. Specifically, the server inputs the following prompt into the generative AI, which then generates an explanation:

[1719] Question: Solve the equation 2x + 3 = 7.

[1720] Answer: x = 2

[1721] Explain the solution.

[1722] Step 5:

[1723] The server analyzes the user's facial expressions and behavior using an emotion recognition engine (EmotionRecognizer library) to recognize the user's emotions. The input is the user's facial expressions and behavior data, and the output is the recognized emotion. Specifically, the server uses the EmotionRecognizer library to analyze the user's facial expressions and behavior and identify the emotion.

[1724] Step 6:

[1725] The server uses a difficulty adjustment engine to adjust the difficulty of the next question based on the recognized emotion. The input is the recognized emotion data, and the output is the adjusted difficulty of the question. Specifically, the difficulty is set to be lowered if the user is confused, and to be higher if the user is happy.

[1726] Step 7:

[1727] The server selects the next question based on the adjusted difficulty and provides it to the user. The input is the adjusted difficulty of the question, and the output is the next question to be asked. Specifically, the server selects a question of appropriate difficulty from the question database and sends it to the user.

[1728] Example 2

[1729] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1730] Conventional learning support systems lacked the ability to generate appropriate questions based on the user's learning history and level of understanding, which led to problems with reduced learning efficiency. They also lacked the ability to adjust the level of detail in explanations according to the user's emotional state, meaning they were unable to provide appropriate support according to the user's level of understanding.

[1731] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1732] In this invention, the server includes means for a user to send a question as an image to a messenger application, means for analyzing the question from the sent image, means for solving the analyzed question, means for converting between a generative artificial intelligence and a computer algebra language for generating an answer, means for generating an explanation for the answer, means for collecting user learning history data, means for analyzing the collected learning history data, means for generating a prompt sentence based on the analysis result, means for inputting the generated prompt sentence into the generative artificial intelligence to generate a question, means for collecting user emotion data, means for analyzing the collected emotion data, means for adjusting the level of detail of the explanation based on the analysis result, and means for providing the question and explanation to the user. This makes it possible to generate appropriate questions based on the user's learning history and level of understanding, and to adjust the level of detail of the explanation according to the user's emotional state.

[1733] "User" refers to an individual who uses the system to learn.

[1734] "Messenger application" refers to software for sending and receiving messages such as text and images.

[1735] "Image" refers to digital data containing visual information.

[1736] "Question" refers to a task or question for the user to answer.

[1737] "Analysis" refers to the process of examining data in detail to understand its structure and meaning.

[1738] "Answer" refers to the correct answer to a question.

[1739] "Generative artificial intelligence" refers to algorithms and models that use natural language processing and machine learning to generate text and data.

[1740] A "computational mathematics language" refers to a programming language used to perform mathematical calculations and analysis.

[1741] "Explanation" refers to a detailed explanation of the question and answer.

[1742] "Learning history data" refers to data relating to learning activities that a user has performed in the past.

[1743] A "prompt sentence" refers to an instruction sentence to be input into generative artificial intelligence.

[1744] "Emotion data" refers to data that indicates the user's emotional state.

[1745] "Level of detail" refers to the specificity and depth of the explanation.

[1746] MODE FOR CARRYING OUT THE INVENTION

[1747] This invention provides a learning support system that generates appropriate questions based on a user's learning history and level of understanding, and adjusts the level of detail of explanations according to the user's emotional state. Specific embodiments of this system are described below.

[1748] Hardware and software used

[1749] server

[1750] The server uses the following hardware and software:

[1751] Hardware: A server machine with a powerful processor, sufficient memory, and ample storage.

[1752] Software: Python, TensorFlow, OpenCV, Google Cloud Speech-to-Text API, MySQL

[1753] Terminal

[1754] A terminal is a device used by a user that uses the following hardware and software:

[1755] Hardware: A personal computer or smartphone with a camera, microphone, and display

[1756] Software: Messenger application

[1757] Data processing and calculation

[1758] Collecting user learning history data

[1759] The server collects learning history data such as the percentage of correct answers for questions the user has answered in the past, the time it took to answer, and the types of questions they got wrong. A database (MySQL) is used for this purpose.

[1760] Analysis of learning history data

[1761] The server analyzes the collected learning history data and evaluates the user's level of understanding. This is done using Python and TensorFlow. Specifically, it runs an algorithm that quantifies the user's level of understanding based on the user's accuracy rate and answer time.

[1762] Generate prompt statement

[1763] Based on the analysis results, the server generates a prompt sentence to be input to the generative AI model. For example, it generates the following prompt sentence:

[1764] "User is studying 8th grade math and struggled with equation problems on a recent test. Generate equation problems at the appropriate level for the user's understanding."

[1765] Problem generation using generative AI models

[1766] The server inputs the generated prompt sentence into a generative AI model (e.g., GPT-4) to generate an appropriate mathematical problem. The generative AI model generates a problem based on the prompt sentence.

[1767] Collecting user emotion data

[1768] The device collects facial and voice data from the user using a camera and microphone. While the user is solving the problem, the device captures facial expressions with the camera and records voice with the microphone.

[1769] Emotional Data Analysis

[1770] The server analyzes the collected emotion data and assesses the user's emotional state using OpenCV and the Google Cloud Speech-to-Text API. The server analyzes facial expression data to determine whether the user is confused and audio data to assess the user's tone of voice.

[1771] Adjusting the level of detail in the explanation

[1772] The server adjusts the level of detail of the explanation based on the analysis of the emotion data. If it determines that the user is confused, it generates a detailed explanation, and if it determines that the user understands, it generates a concise explanation.

[1773] Providing questions and explanations

[1774] The server provides the generated questions and the adjusted explanations to the user. The server sends the questions and explanations to the terminal, which displays them to the user.

[1775] Specific examples

[1776] For example, if the user is studying eighth-grade math, the server might input the following prompts into the generative AI model:

[1777] "User is studying 8th grade math and struggled with equation problems on a recent test. Generate equation problems at the appropriate level for the user's understanding."

[1778] Based on this prompt, the generative AI model generates the following question:

[1779] "Solve the following equation: 3x + 5 = 20"

[1780] Also, if the emotion engine determines that the user is confused by the issue, the server will provide a detailed explanation, such as:

[1781] "First, subtract 5 from both sides of the equation. Then divide what's left, 3x, by 20 minus 5. This gives you the value of x."

[1782] On the other hand, if the server determines that the user does understand, it provides a brief explanation such as:

[1783] "Solve the equation to find the value of x."

[1784] In this way, the system supports efficient learning by providing appropriate questions and explanations according to the user's level of understanding and emotional state.

[1785] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1786] Step 1:

[1787] Collecting user learning history data

[1788] The server retrieves learning history data from a database (MySQL), such as the accuracy rate of questions the user has answered in the past, the time it took to answer, and the types of questions they got wrong.

[1789] Input: User ID

[1790] Output: Learning history data

[1791] Specific operation: Every time a user logs in, the server queries and obtains the user's learning history data from the database.

[1792] Step 2:

[1793] Analysis of learning history data

[1794] The server analyzes the collected learning history data and evaluates the user's level of understanding, using Python and TensorFlow.

[1795] Input: Learning history data

[1796] Output: User's understanding rating

[1797] Specific operation: The server executes an algorithm that quantifies the user's level of understanding based on the user's accuracy rate and answer time.

[1798] Step 3:

[1799] Generate prompt statement

[1800] Based on the analysis results, the server generates a prompt sentence to be input into the generative AI model.

[1801] Input: User's comprehension rating

[1802] Output: prompt statement

[1803] Specific behavior: The server generates a prompt sentence based on the user's level of understanding, such as: "The user is studying eighth-grade math and has struggled with equation problems on recent tests. Please generate equation problems at an appropriate level based on the user's level of understanding."

[1804] Step 4:

[1805] Problem generation using generative AI models

[1806] The server inputs the generated prompt sentences into a generative AI model (e.g., GPT-4) to generate appropriate math problems.

[1807] Input: prompt statement

[1808] Output: Generated problem

[1809] Specific operation: The server sends a prompt to the generative AI model and receives the generated problem.

[1810] Step 5:

[1811] Collecting user emotion data

[1812] The device collects facial and voice data from the user using a camera and microphone.

[1813] Input: Real-time user data (facial expressions, voice)

[1814] Output: Emotion data

[1815] Specific operation: While the user is solving the problem, the device captures facial expressions with a camera and records audio with a microphone.

[1816] Step 6:

[1817] Emotional Data Analysis

[1818] The server analyzes the collected emotion data and assesses the user's emotional state using OpenCV and the Google Cloud Speech-to-Text API.

[1819] Input: Emotion data

[1820] Output: User's emotional state rating

[1821] Specific operation: The server analyzes facial expression data to determine whether the user is confused, and analyzes voice data to evaluate the user's tone of voice.

[1822] Step 7:

[1823] Adjusting the level of detail in the explanation

[1824] The server adjusts the level of detail of the commentary based on the results of analyzing the emotional data.

[1825] Input: User's emotional state rating

[1826] Output: Adjusted commentary

[1827] Specific behavior: If the server determines that the user is confused, it generates a detailed explanation, and if it determines that the user understands, it generates a concise explanation.

[1828] Step 8:

[1829] Providing questions and explanations

[1830] The server provides the generated questions and tailored explanations to the user.

[1831] Input: Generated questions, adjusted explanations

[1832] Output: The problem and explanation provided to the user

[1833] Specific operation: The server sends the problem and explanation to the terminal, which then displays it to the user.

[1834] (Application example 2)

[1835] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1836] Conventional learning systems do not generate questions based on the user's learning history or level of understanding, nor do they adjust explanations based on the user's emotions, making efficient learning difficult. In particular, they are unable to provide appropriate feedback when the user is confused or has already understood the material, which reduces the effectiveness of learning.

[1837] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1838] In this invention, the server includes means for a user to send a question as an image to a messenger app BOT, means for analyzing the question from the sent image, means for solving the analyzed question, means for converting between a generative AI and a formula processing language for generating an answer, means for generating an explanation of the answer, means for generating questions based on the user's learning history and level of understanding, and means for adjusting the level of detail of the explanation of the answer based on the user's emotions. This makes it possible to generate appropriate questions according to the user's learning history and level of understanding, and to adjust the explanation according to the user's emotions.

[1839] "User" refers to an individual who uses the system to learn.

[1840] "Messenger app bot" refers to a messaging application that has an automated response function that allows users to send images of their problems.

[1841] "Image" refers to a data file containing visual information that a User sends to a Messenger App BOT.

[1842] "Problem" refers to a mathematical or other learning task for which a user seeks an answer.

[1843] "Analysis" refers to the process of recognizing and understanding the problem content from the transmitted image.

[1844] "Answer" refers to the correct response to the analyzed problem.

[1845] "Generative AI" refers to artificial intelligence technology that generates answers and explanations to problems.

[1846] A "computational mathematics language" refers to a programming language for processing mathematical expressions on a computer.

[1847] "Explanation" refers to explanations or supplementary information for the answer.

[1848] "Study history" refers to data that records questions that a user has solved in the past and the answers they gave.

[1849] "Level of understanding" refers to an index that indicates how well the user understands the problem.

[1850] "Emotion" refers to the psychological state that a user feels when solving a problem.

[1851] "Level of detail" refers to the degree to which the content of the commentary is explained in detail.

[1852] A system for implementing this invention includes means for a user to send a question as an image to a messenger app BOT, means for analyzing the question from the sent image, means for solving the analyzed question, means for converting between a generative AI and a mathematical processing language to generate an answer, means for generating an explanation for the answer, means for generating a question based on the user's learning history and level of understanding, and means for adjusting the level of detail of the explanation for the answer based on the user's emotions.

[1853] System configuration

[1854] 1. User device: The device through which the user submits the question, such as a smartphone or tablet.

[1855] 2. Messenger app bot: A messaging application with an automated response function that allows users to send images of their problems.

[1856] 3. Server: A central processing unit for analyzing problems, generating answers, generating explanations, generating questions, and sentiment analysis.

[1857] Software used

[1858] Generative AI: Artificial intelligence technology for generating answers and explanations to problems.

[1859] Computer algebra language: A programming language for processing mathematical expressions on a computer.

[1860] Sentiment analysis engine: Software for analyzing user emotions.

[1861] Learning history management system: A system that records the user's learning history and uses it to generate questions next time.

[1862] Processing flow

[1863] 1. Sending the problem: The user sends the problem image to the messenger app BOT using their smartphone.

[1864] 2. Problem analysis: The server receives the image and analyzes the problem. Image recognition technology is used for the analysis.

[1865] 3. Answer generation: The generative AI generates an answer to the analyzed problem. It uses a mathematical processing language to derive an accurate answer.

[1866] 4. Explanation generation: Generative AI generates an explanation for the answer. The level of detail of the explanation is adjusted by a sentiment analysis engine.

[1867] 5. Question generation: New questions are generated based on the user's learning history and level of understanding.

[1868] 6. Sentiment analysis: Analyze the user's emotions in real time while answering questions and provide appropriate feedback.

[1869] Specific examples

[1870] For example, if the sentiment analysis engine determines that a user is "confused" while solving a math problem, it will provide a detailed explanation, whereas if it determines that the user "understands," it will provide a concise explanation.

[1871] Prompt Sentence Examples

[1872] Generate math problems at the appropriate level based on the user's learning history and understanding, and adjust the level of detail in the explanations depending on the user's emotions: provide detailed explanations if the user is confused, and concise explanations if the user understands.

[1873] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1874] Step 1:

[1875] The user sends the image in question to the messenger app BOT using their smartphone.

[1876] Input: A user-taken image of the problem.

[1877] Output: Image data sent to the messenger app BOT.

[1878] How it works: The user takes a picture of the problem with their smartphone camera and sends it to the messenger app BOT, which then transfers the image data to the server.

[1879] Step 2:

[1880] The server receives the transmitted image and analyzes the problem.

[1881] Input: Image data transferred from the messenger app BOT.

[1882] Output: The parsed text data of the problem.

[1883] Specific operation: The server uses image recognition technology (e.g., OCR) to extract the problem text from the image data. The extracted text data is sent to the problem analysis module.

[1884] Step 3:

[1885] Generative AI generates answers to the analyzed problems.

[1886] Input: The parsed text data of the problem.

[1887] Output: Text data of the generated answer.

[1888] Specific operation: The server uses generative AI to generate answers to the analyzed problems, and derives accurate answers using a mathematical processing language.

[1889] Step 4:

[1890] Generative AI generates explanations for the answers.

[1891] Input: The text data of the generated answer.

[1892] Output: The generated commentary text.

[1893] How it works: The server uses generative AI to generate explanations for the answers. The level of detail of the explanations is adjusted by a sentiment analysis engine.

[1894] Step 5:

[1895] New questions are generated based on the user's learning history and level of understanding.

[1896] Input: User learning history data, comprehension data.

[1897] Output: The text data of the new problem generated.

[1898] Specific operation: The server analyzes the user's learning history and level of understanding, and generates new questions at an appropriate level. The generated questions will be used for the next learning session.

[1899] Step 6:

[1900] Analyzes the user's emotions while answering in real time and provides appropriate feedback.

[1901] Input: Facial expression data, voice data, etc. while the user answers.

[1902] Output: Sentiment analysis results and appropriate feedback text data.

[1903] Specific operation: The server uses an emotion analysis engine to analyze the emotions expressed in the user's answers in real time. Based on the analysis results, it provides detailed or concise explanations.

[1904] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1905] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1906] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1907] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given.

[1908] However, the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1909] [Fourth embodiment]

[1910] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1911] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1912] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1913] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1914] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1915] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1916] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1917] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1918] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1919] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1920] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1921] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1922] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1923] "Example 1"

[1924] In one embodiment of the present invention, a user sends a math problem as an image to a messenger app BOT. The BOT analyzes the problem from the image and solves the analyzed problem. The answer is generated using generative AI and a conversion means in the Wolfram Language. Furthermore, an explanation for the answer is also generated by the generative AI. Specifically, when a user sends an image containing a math problem to the BOT, the BOT reads the problem from the image and solves the problem. The answer is generated using generative AI and a conversion means in the Wolfram Language, and an explanation for the answer is also generated at the same time. "Example 2"

[1925] In another aspect of the present invention, a problem generation function can be added in the future. Specifically, the system can generate mathematics problems at an appropriate level based on the user's learning history and level of understanding. This problem generation function allows users to solve problems tailored to their own level, enabling them to study more efficiently.

[1926] The processing flow of each embodiment will be described below.

[1927] "Example 1"

[1928] Step 1: The user sends an image containing a math problem to the messenger app bot.

[1929] Step 2: The bot analyzes the problem from the image sent. This analysis is done using image recognition technology.

[1930] Step 3: The analyzed problem is solved using generative AI and transformation tools in the Wolfram Language.

[1931] Step 4: An explanation for the answer is also generated by the generative AI and presented to the user.

[1932] "Example 2"

[1933] Step 1: The user sends an image containing a math problem to the messenger app bot.

[1934] Step 2: The bot analyzes the problem from the image sent. This analysis is done using image recognition technology.

[1935] Step 3: The analyzed problem is solved using generative AI and transformation tools in the Wolfram Language.

[1936] Step 4: An explanation for the answer is also generated by the generative AI and presented to the user.

[1937] Step 5: The system generates math problems at an appropriate level based on the user's learning history and level of understanding.

[1938] Example 1

[1939] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1940] Conventional mathematical problem-solving systems require users to input problems in text format, which is time-consuming, and they have the problem of not being able to handle handwritten or image-formatted problems. Furthermore, the generation of the answer and its explanation must be done separately, which makes it difficult for users to understand.

[1941] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1942] In this invention, the server includes means for a user to send a problem as an image to a messenger application, means for transferring the sent image to the server, means for the server to extract text data from the image using image analysis software, means for inputting the extracted text data into a generative AI model to generate an answer to the problem, means for converting the generated answer into a mathematical formula or a calculation result, means for generating an explanation for the answer using the generative AI model, and means for returning the answer and explanation to the user. This allows users to easily solve mathematical problems in handwritten or image format and obtain the answer and its explanation at the same time.

[1943] A "user" is any person or entity that intends to use the system to solve mathematical problems.

[1944] A "messenger application" is communication software that users use to send text and images.

[1945] An "image" is digital data that visually represents a mathematical problem.

[1946] A "server" is a computer system that receives and processes data sent by users.

[1947] "Image analysis software" is a program for extracting text data from images.

[1948] "Text data" is text information extracted by image analysis software.

[1949] A "generative AI model" is an artificial intelligence system that generates answers and explanations to questions based on input text data.

[1950] "Conversion means" refers to techniques or tools for converting generated answers into mathematical formulas or calculation results.

[1951] An "answer" is a solution to a user-submitted math problem.

[1952] An "explanation" is an explanation generated by a generative AI model to help understand the answer.

[1953] "Means for sending back" refers to the method by which the server sends back the generated answers and explanations to the user.

[1954] MODE FOR CARRYING OUT THE INVENTION

[1955] The present invention is a system that allows a user to send an image containing a mathematical problem to a messenger application, analyzes the problem from the image, and generates an answer and explanation. A specific embodiment of this system will be described below.

[1956] Hardware and software used

[1957] Server: A high-performance computer system that analyzes images, runs generative AI models, and transforms and returns data.

[1958] Device: The smartphone or PC used by the user to send images via a messenger application.

[1959] Messenger application: Communication software that allows users to send images.

[1960] Image analysis software: Extract text data from images, using Tesseract OCR as an example.

[1961] Generative AI model: As an example, we use OpenAI's GPT-4 to generate answers and explanations for problems.

[1962] Conversion method: For example, we use the Wolfram Alpha API to convert the generated answers into mathematical formulas and calculation results.

[1963] System Operation

[1964] A user sends an image containing a math problem to a messenger application. The device transfers the image to a server. The server uses image analysis software to extract text data from the image. The extracted text data is input into a generative AI model to generate an answer to the problem. The generated answer is then converted into a mathematical formula or calculation result using a transformation method. The generative AI model also generates an explanation for the answer. Finally, the server returns the answer and explanation to the user.

[1965] Specific examples

[1966] For example, if a user sends an image of the equation "x^2 + 3x + 2 = 0" to a messenger application, the process would proceed as follows:

[1967] 1. The user takes a picture of the equation with their smartphone and sends it to a messenger application.

[1968] 2. The device transfers the image data to the server.

[1969] 3. The server uses Tesseract OCR to extract the text data "x^2 + 3x + 2 = 0" from the image.

[1970] 4. The server sends the prompt "Solve the equation x^2 + 3x + 2 = 0" to the generated AI model, which generates the answer "x = -1, x = -2".

[1971] 5. The server uses the Wolfram Alpha API to materialize the answer in mathematical form.

[1972] 6. The server sends the prompt "Explain the steps to solve the quadratic equation x^2 + 3x + 2 = 0" to the generated AI model and generates an explanation.

[1973] 7. The server returns the answer "x = -1, x = -2" to the user along with the explanation "This equation is a quadratic equation and can be solved using the formula. The formula gives us x = -1 and x = -2."

[1974] In this way, users can receive answers and explanations for math problems simply by submitting an image of the problem.

[1975] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1976] Step 1:

[1977] A user sends an image containing a math problem to a messenger application. Using a smartphone or PC, the user takes a photo of the handwritten equation "x^2 + 3x + 2 = 0" and sends it to the messenger application. The input is the image containing the math problem, and the output is the image sent to the messenger application.

[1978] Step 2:

[1979] The device transfers the image sent to the server. The device transfers the image data sent by the user to the server in an appropriate format (e.g., JPEG, PNG). The input is the image sent to the messenger application, and the output is the image data transferred to the server.

[1980] Step 3:

[1981] The server uses image analysis software to extract text data from the image. The server uses image analysis software (e.g., Tesseract OCR) to extract the text data "x^2 + 3x + 2 = 0" from the image. The input is the image data transferred to the server, and the output is the extracted text data.

[1982] Step 4:

[1983] The server inputs the extracted text data into a generative AI model to generate an answer to the problem. The server inputs the extracted text data into a generative AI model (e.g., OpenAI's GPT-4) and sends the prompt "Solve the equation x^2 + 3x + 2 = 0" to the generative AI model. The generative AI model generates the answer "x = -1, x = -2". The input is the extracted text data and the prompt, and the output is the generated answer.

[1984] Step 5:

[1985] The server materializes the answer using Wolfram Language transformations. The server materializes the generated answer as a formula or result using Wolfram Language transformations (e.g., Wolfram Alpha API). The input is the generated answer, and the output is the materialized formula or result.

[1986] Step 6:

[1987] The server generates an explanation for the answer using a generative AI model. The server sends the generative AI model a prompt, "Explain the steps to solve the quadratic equation x^2 + 3x + 2 = 0," and generates an explanation. The generative AI model generates an explanation, "This equation is a quadratic equation and can be solved using the solution formula. The solution formula gives x = -1 and x = -2." The input is the generated answer and prompt, and the output is the generated explanation.

[1988] Step 7:

[1989] The server returns the answer and explanation to the user. The server returns the generated answer "x = -1, x = -2" and explanation "This equation is a quadratic equation and can be solved using the solution formula. The solution formula gives x = -1 and x = -2." The input is the generated answer and explanation, and the output is the answer and explanation returned to the user.

[1990] (Application example 1)

[1991] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1992] Conventional math problem-solving systems require users to input questions, and it takes time to obtain the answers and explanations. In addition, the accuracy of the answers and the quality of the explanations are often not guaranteed, which can reduce the user's learning efficiency.

[1993] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for a user to send a problem as an image to a messenger app BOT, means for analyzing the problem from the sent image, means for solving the analyzed problem, means for converting between a generative AI and a formula manipulation language to generate an answer, means for generating an explanation of the answer, means including an application to be installed on the smartphone, means for extracting text from an image using an image processing library, means for solving the problem using a formula manipulation language API, means for generating an explanation using a generative AI, and means for displaying the answer and explanation to the user. This allows a user to easily analyze a mathematical problem photographed with their smartphone and quickly obtain an accurate answer and high-quality explanation.

[1994] A "user" is an individual or group that uses the system to solve mathematical problems.

[1995] "Messenger App BOT" is an automatic response program that receives and analyzes mathematical problems sent by users as images.

[1996] "Means of sending as an image" refers to a method in which a user takes a photo of a math problem using a smartphone or other device and sends the image to a messenger app BOT.

[1997] The "means for analyzing problems from transmitted images" is a method for extracting text from images using an image processing library and recognizing the text as a mathematical problem.

[1998] The "means for solving the analyzed problem" is a method for solving the extracted mathematical problem using a computer algebra language API.

[1999] A "generative AI for generating solutions" is an artificial intelligence model used to generate answers to mathematical problems.

[2000] A "computer algebra language transformation means" is a method of using a computer algebra language to solve a mathematical problem.

[2001] "Means for generating explanations for answers" refers to a method for generating detailed explanations for answers using generative AI.

[2002] The "application installed on a smartphone" is software that allows users to take a photo of a math problem using their smartphone and receive the answer and explanation.

[2003] An "image processing library" is a software tool for extracting text from images.

[2004] A "computer algebra language API" is a programmatic interface used to solve mathematical problems.

[2005] "Means for generating explanations using generative AI" refers to a method for generating explanations for answers to mathematical problems using generative AI.

[2006] "Means for displaying answers and explanations to the user" refers to a method for displaying the generated answers and explanations on the user's smartphone.

[2007] The system for implementing this invention starts when a user takes a photo of a math problem using a smartphone and sends the image to a messenger app BOT. A specific embodiment of this system will be described below.

[2008] Hardware and software used

[2009] Hardware: Smartphone (with camera)

[2010] software:

[2011] Messenger app BOT

[2012] Image processing library (e.g. OpenCV)

[2013] OCR (Optical Character Recognition) libraries (e.g., pytesseract)

[2014] Mathematical processing language API (e.g. WolframAlpha API)

[2015] Generative AI models (e.g., GPT-3)

[2016] Data processing and calculation

[2017] 1. Image Acquisition:

[2018] Users take a picture of a math problem using their smartphone camera and send the image to a messenger app BOT.

[2019] 2. Image Analysis:

[2020] The server converts the submitted image to grayscale using an image processing library (OpenCV) and extracts text from the image using an OCR library (pytesseract).

[2021] 3. Problem answer:

[2022] The extracted text is sent to a mathematical processing language API (WolframAlpha API), and the server obtains the answer.

[2023] 4. Description generation:

[2024] The server uses a generative AI model (GPT-3) to generate a detailed explanation for the obtained answer. Specifically, it inputs the following prompt sentence into the generative AI:

[2025] Example prompt sentence:

[2026] Math problem: x^2 + 2x + 1 = 0

[2027] Answer: x = -1

[2028] Description:

[2029] 5. Results display:

[2030] The server sends the generated answers and explanations to the user's smartphone and displays them through the application.

[2031] Specific examples

[2032] If a user photographs and submits the problem "x^2 + 2x + 1 = 0," the system operates as follows:

[2033] 1. Extract the text "x^2 + 2x + 1 = 0" from the image.

[2034] 2. Use the formula processing language API to obtain the answer "x = -1".

[2035] 3. Use generative AI to generate an explanation such as, "This equation is a quadratic equation, and using the formula for the solution, we get x = -1."

[2036] 4. Display the answer and explanation to the user.

[2037] In this way, users can easily analyze math problems photographed with their smartphones and quickly obtain accurate answers and high-quality explanations.

[2038] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2039] Step 1:

[2040] A user takes a picture of a math problem using the camera on their smartphone and sends the image to a messenger app BOT. The input is the image of the math problem that was taken, and the output is the image sent to the messenger app BOT.

[2041] Step 2:

[2042] The server converts the transmitted image to grayscale using an image processing library (OpenCV). The input is the image transmitted by the user, and the output is the image converted to grayscale. Specifically, the server reads the image using OpenCV functions and converts it to grayscale.

[2043] Step 3:

[2044] The server uses an OCR library (pytesseract) to extract text from grayscale images. The input is the image converted to grayscale, and the output is the extracted text. Specifically, it uses pytesseract's functions to read text from the image.

[2045] Step 4:

[2046] The server sends the extracted text to a mathematical expression processing language API (WolframAlpha API) and obtains the answer. The input is the extracted text, and the output is the answer obtained from the mathematical expression processing language API. Specifically, the server sends the text to the WolframAlpha API and receives the answer.

[2047] Step 5:

[2048] The server uses a generative AI model (GPT-3) to generate a detailed explanation for the obtained answer. The input is the answer obtained from the formula processing language API, and the output is the generated explanation. Specifically, the following prompt sentence is input to the generative AI:

[2049] Example prompt sentence:

[2050] Math problem: x^2 + 2x + 1 = 0

[2051] Answer: x = -1

[2052] Description:

[2053] Step 6:

[2054] The server sends the generated answers and explanations to the user's smartphone and displays them through the application. The input is the generated answers and explanations, and the output is the answers and explanations displayed on the user's smartphone. Specifically, the answers and explanations are sent through a messenger app BOT and displayed on the smartphone application.

[2055] Example 2

[2056] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2057] Conventional learning systems have difficulty generating questions based on a user's learning history and level of understanding, making it impossible to provide optimal questions for each individual user. Furthermore, efficient and accurate processing is required for analyzing questions answered by users and generating answers. This reduces the user's learning efficiency, making effective learning difficult.

[2058] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2059] In this invention, the server includes means for a user to send a question to a messenger application as an image, means for analyzing the question from the sent image, and means for answering the analyzed question, thereby making it possible to collect the user's learning history data, evaluate the user's level of understanding based on the analysis results, create prompt sentences to be input to the generative AI based on the level of understanding, generate questions using the generative AI, and provide the generated questions to the user.

[2060] "User" refers to an individual or organization that uses the system to study.

[2061] "Messenger application" refers to communication software that allows users to send and receive text and images.

[2062] "Image" refers to visual data that a user sends through a messenger application.

[2063] "Question" refers to a task or question provided for a user to answer.

[2064] "Analysis" refers to the process of reading and understanding the problem content from the transmitted image.

[2065] "Solution" refers to the answer to the question being analyzed.

[2066] "Generative AI" refers to AI technology that generates questions and answers based on a user's learning history and level of understanding.

[2067] "Programming language" refers to the computer language used by generative artificial intelligence to generate answers.

[2068] "Study history data" refers to data relating to questions that a user has answered in the past and the results of those answers.

[2069] "Analysis results" refers to information obtained after analyzing learning history data.

[2070] "Level of understanding" refers to an index that indicates how well a user can understand a question and answer it.

[2071] A "prompt sentence" refers to an instruction sentence that is input to a generative artificial intelligence when generating a problem.

[2072] "Problem generation" refers to the process by which a generative AI creates new questions based on prompts.

[2073] "Providing" refers to the act of displaying or transmitting the generated question to the user.

[2074] This invention is a system that allows a user to send a question as an image to a messenger application, analyzes the question from the image, generates an answer, and further generates new questions based on the user's learning history data. A specific embodiment of this system will be described below.

[2075] Hardware and software used

[2076] The server uses the following hardware and software:

[2077] Hardware: High-performance server machine

[2078] Software: generative AI models (e.g., GP...

Claims

1. A means for inputting an image including a problem previously solved by a user through a messenger application and analyzing the problem included in the image; means for generating a solution to the analyzed problem using a generative AI model; A means for generating an explanation of an answer by inputting a specific prompt sentence into a generative AI model; an emotion engine that recognizes the emotion of the user; A means for adjusting a level of detail of the commentary based on the emotion; means for providing the adjusted explanation together with the problem to the user; A system including:

2. The system according to claim 1 , further comprising a question generating means for generating questions of a particular level based on the user's learning history and his / her level of understanding of the questions.

3. The system according to claim 2 , wherein the problem generator adjusts the difficulty of the problem at a particular level based on the emotion.

Images