system

Abstract

We provide the system. [Solution] A means for receiving the problem image taken or acquired by the user, A means for extracting text information from a received problem image, A means of analyzing extracted textual information and classifying the learning target, A means of updating learner profiles based on analysis results, A means of generating similar problems and explanations based on the learner's profile, A means for transmitting the generated learning content to the user's device, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There are problems that by leaving the problems that learners have made mistakes in unattended, the same mistakes are repeated and effective learning is hindered. Also, it is difficult to find review problems suitable for individual learners, and general problem sets cannot focus on covering the specific weak areas of learners.

Means for Solving the Problems

[0005] This invention provides a system that receives problem images taken or acquired by learners, extracts textual information from those images, and identifies the learning target. The system can use this information to update the learner's profile and generate similar problems and explanations based on their past learning history. In this way, learning efficiency is improved by providing learning content optimized for each individual learner.

[0006] The term "user" refers to learners or users who utilize the system, and primarily targets those in a position of receiving education.

[0007] "Problem image" refers to data containing visual information about problems that learners answered incorrectly, either by taking a photograph or acquiring the information using a digital device.

[0008] "Textual information" refers to the results of extracting information such as letters, numbers, and symbols contained within the problem image, and is treated as text data.

[0009] "Analysis" refers to the process of analyzing the type and content of a problem using extracted textual information, and is a task aimed at identifying the user's areas of weakness.

[0010] A "learner profile" refers to a dataset containing information about each individual's learning history, areas of difficulty, etc., and is used to provide personalized learning support.

[0011] "Similar problems" refer to problems that have similar content and format to problems that learners have already worked on, and are intended to further deepen the learners' understanding.

[0012] "Explanation" refers to content that provides detailed explanations and introductions to problems that learners have answered incorrectly, and is instructional information aimed at promoting learning.

[0013] "Terminal" refers to a device used by learners to access the system, and examples include smartphones, tablets, and personal computers. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention is a system that supports learners in effectively progressing with their studies, primarily involving data exchange between a server, a terminal, and a user. In implementation, the user first takes a picture of a problem they answered incorrectly with their terminal. This problem image is then sent to the server via a dedicated application.

[0036] The server uses OCR technology to extract text information from the received image. By analyzing this text information, the server identifies the type of problem and the scope of learning, and updates the learner's profile accordingly. Based on the learner's past learning history and current areas of difficulty, the server generates similar learning problems and prepares explanations and lecture videos as needed.

[0037] As a concrete example, if a user makes a mistake in solving a quadratic equation on a math exam, the user takes a picture of the problem. The server receives this and identifies that the problem is related to a quadratic equation. The server then analyzes how the user has failed with similar problems based on past data. As a result, it sends similar quadratic equation problems and detailed explanations of how to solve them to the user's device.

[0038] The device displays the received data to the user, who can then review the provided learning content. Continuous data feedback to the server allows the user's profile to be updated more accurately, improving the precision of subsequent learning support. This system will enable learners to efficiently overcome their weak areas.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The user takes a picture or screenshot of the incorrect answer using their own device. This prepares the question content to be saved in digital format.

[0042] Step 2:

[0043] The device uploads the acquired problem images to the server via a dedicated application. Metadata such as the user ID and problem category is attached to the upload.

[0044] Step 3:

[0045] The server temporarily stores the received image data and uses OCR technology to extract text information from the image. This makes the question text available for processing within the system as digital text.

[0046] Step 4:

[0047] The server analyzes the extracted text data to identify the theme and content of the problem. Natural language processing techniques are used in this analysis to classify the problem.

[0048] Step 5:

[0049] The server references the user's existing profile data and updates the user's learning progress and areas of difficulty based on the analysis results. This update is then used to provide personalized learning support.

[0050] Step 6:

[0051] The server generates similar problems and related explanations based on the user's profile information, and prepares lecture videos as needed. This information is generated to enhance the user's learning.

[0052] Step 7:

[0053] The server sends similar problems and explanatory content it has generated to the device. The device displays the received data, allowing the user to use it for review.

[0054] Step 8:

[0055] Users solve similar problems provided on their devices, deepening their understanding by reading explanations or watching videos. The server continuously collects user responses and behavioral data to improve the accuracy of the profiles.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] Conventional learning support systems have the challenge of not being able to accurately identify individual learners' weak areas and quickly provide them with the most suitable learning problems and explanations. Furthermore, the lack of a mechanism for efficiently updating learning history made it difficult to respond flexibly to learners' progress.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for receiving problem images captured or acquired by the user, means for extracting character information using optical character recognition technology, and means for generating similar problems and related information using a generative AI model. This enables the provision of problems tailored to the learner's weak areas and efficient updating of the learning history.

[0061] A "user" refers to an individual who uses a learning support system to advance their studies.

[0062] A "problem image" refers to image data captured by a camera or obtained from a user who answered a problem incorrectly during their learning process.

[0063] "Optical character recognition technology" is a technology that converts characters in an image into digital text.

[0064] "Character information" refers to the character data within the image in question, extracted using optical character recognition technology.

[0065] A "generative AI model" is a model that uses artificial intelligence technology to analyze a learner's profile and generate optimal learning problems and explanations.

[0066] "Related information" refers to supplementary materials such as explanations and lecture content provided by generative AI models to deepen learners' understanding.

[0067] An "information processing device" is a device used by a user to receive and display content from a learning support system.

[0068] "History information" refers to information that the system retains as a user profile, such as the user's past learning activities and performance data.

[0069] "Feedback" refers to information about one's learning progress and performance that a user sends to the system.

[0070] This invention is a learning support system that allows learners to individually identify areas where they lack understanding and to efficiently advance their learning. The following describes a specific embodiment of this system.

[0071] First, the user takes a picture of the question they answered incorrectly with their device. A dedicated application is installed on the device, and the image of the question is sent to the server via this application. The device compresses the captured image to improve communication efficiency.

[0072] Upon receiving a problem image, the server extracts text information from the image using optical character recognition (OCR) technology. Specifically, software such as Tesseract OCR can be used. Next, the server uses the extracted text information to execute an algorithm to identify the type of problem and the learning area.

[0073] The server updates the learner's history information based on the analysis results. This history information includes past learning data and correct / incorrect information in specific areas. Based on the updated history information, a generative AI model is used to generate similar problems and related information. For example, the generative AI model uses OpenAI's (registered trademark) language model, and optimal learning content is generated by inputting prompt sentences.

[0074] As a concrete example, consider a case where a user makes a mistake on a quadratic equation problem in a math exam. The user takes a picture of the problem and sends it to the server. The server extracts textual information from the image and identifies that the problem is related to a quadratic equation. The server then refers to the learner's history information to generate similar quadratic equation problems and explanations, and sends them to the device.

[0075] Examples of prompt messages include the following:

[0076] "Please create the following learning problem involving quadratic equations. Past data shows that users particularly struggle with the 'completing the square' method. Please provide an explanation and problem proposal based on this."

[0077] The device displays the received learning content to the user, allowing the user to efficiently review the material based on this information. The user enters their answers into the device, and this feedback is sent back to the server and used to update the history information. This cycle further improves the accuracy of learning support for the user.

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] The user takes a picture of the incorrect answer with their device. The input is image data acquired by the camera. The device then imports this image into a dedicated application. Specifically, the image is properly captured and compressed before being prepared for transmission.

[0081] Step 2:

[0082] The terminal sends the prepared problem image data to the server. The input here is the image data captured and compressed in step 1, and the output is the transfer of the image data to the server. The terminal establishes a network connection and performs real-time error checking to ensure that the data is transmitted correctly.

[0083] Step 3:

[0084] The server processes the received image and extracts text information using optical character recognition (OCR) technology. The input is the received image data, and the output is the extracted text information. Specifically, the server uses tools such as Tesseract OCR to efficiently analyze the text portion of the image and convert it into digital text.

[0085] Step 4:

[0086] The server analyzes the extracted text information to identify the type of problem and the learning area. The input is the text information extracted in step 3, and the output is the identified type of problem and learning area. Specifically, the text information is passed through a natural language processing algorithm to structure the content of the problem and compare it with the learner's history information.

[0087] Step 5:

[0088] The server updates the learner's history information based on the identified problem information. The input is the identified problem type and learning area, and the output is the updated history information. Specifically, it refers to the past learning database and adds information to analyze which areas the user makes the most mistakes in.

[0089] Step 6:

[0090] The server uses a generative AI model to generate similar problems and relevant information suitable for the learner. The input is updated historical information and identified problem information, and the output is the generated learning content. Specifically, the server inputs prompt sentences into the AI ​​model and generates content that focuses on what the user particularly needs to understand.

[0091] Step 7:

[0092] The server sends the generated learning content to the user's device. The input is the content generated in step 6, and the output is the delivery of the content to the user's device. Specifically, an efficient data transfer protocol is used to ensure that the content is delivered accurately while minimizing latency.

[0093] Step 8:

[0094] The terminal displays the received learning content to the user. The input is content data received from the server, and the output is a display state that the user can interact with. Specifically, the terminal application formats the content appropriately so that the user can start learning immediately.

[0095] Step 9:

[0096] The user solves problems based on the provided content and inputs their answers into the device. The input is the user's answer data, and the output is the confirmation of the answer data within the device. Specifically, the system is designed to allow users to quickly send data through an interface that makes it easy to input answers.

[0097] Step 10:

[0098] The terminal sends the user's answer results to the server, where they are reflected as feedback. The input is the user's answer data, and the output is the transfer of data to the server. Specifically, the transmitted data is checked to ensure its accuracy and to be used for updating the history information on the server side.

[0099] Step 11:

[0100] The server updates the learner's history information based on the feedback received, and this data is used to generate the next problem. The input is the user's answer data, and the output is the latest history information. Specifically, the server adds the updated information to the database and uses the newly acquired knowledge in the next learning cycle.

[0101] (Application Example 1)

[0102] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0103] In education, it is crucial to support learners in understanding their mistakes and effectively overcoming them. However, existing means of providing individually optimized learning support are limited, and there are challenges in accurately understanding each learner's characteristics and areas of difficulty and adjusting learning content accordingly. Furthermore, current systems cannot efficiently utilize users' learning history, resulting in insufficient learning support.

[0104] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0105] In this invention, the server includes means for receiving information units captured or acquired by the user, means for extracting character information from the received information units, and means for analyzing the extracted character information and classifying the learning target. This makes it possible to provide individually optimized learning support and to provide effective learning content based on the learner's characteristics and past learning history.

[0106] "Users" refers to learners and educators who use this system to receive learning support.

[0107] An "information unit" refers to data, including questions and learning materials, that learners photograph or acquire and send to the system.

[0108] "Textual information" refers to text data extracted from information units using OCR technology.

[0109] "Classification of learning targets" refers to the process of analyzing extracted textual information and classifying its content into specific learning categories or themes.

[0110] "Learner characteristics" refers to profile information including the learner's past learning history, current level of understanding, and areas of difficulty.

[0111] "Similar information" refers to learning questions and content generated based on learners' areas of difficulty and incorrect answers.

[0112] "Explanation" refers to explanatory materials or lecture materials provided for similar information.

[0113] "Portable devices" refer to electronic terminals that users can carry around, such as smartphones and tablets.

[0114] A "learning plan" refers to a learning schedule and content optimized for each individual learner.

[0115] The system implementing this invention provides learning support to help learners effectively understand and overcome their mistakes. The entire system consists of the user's mobile device and a server, and utilizes a variety of technologies.

[0116] First, the user takes a picture of a problem they find difficult or answered incorrectly using their mobile device. The captured information is sent to a server via a dedicated application installed on the mobile device. The server then extracts text information from the received information using OCR technology (for example, AWS® Rekognition or Google® Cloud Vision).

[0117] The server analyzes the extracted text information and classifies the learning targets. Natural language processing techniques are also utilized to interpret the meaning and structure of the text information with greater precision. Based on the analyzed information, the server updates the learner's characteristics and creates a learning plan optimized for each individual learner.

[0118] This system generates similar information and explanations based on the learner's characteristics. The generated learning content is sent from the server to the user's mobile device, where the user can review it and continue learning. This allows for rapid correction of misunderstandings and errors arising from individual learning, resulting in efficient learning.

[0119] As a concrete example of its use, if a high school student takes a picture of a math problem they got wrong on a test, the photo is sent from their mobile device to a server, where OCR technology converts the problem content into text. The server analyzes the content and extracts particularly difficult concepts and problem patterns by comparing them with the learner's past records. Then, using AI, it generates similar problems and detailed explanations of their solutions, which are delivered to the learner's mobile device. A possible prompt in this case would be, "I made a mistake in solving the quadratic equation x² + 3x - 4 = 0. Please provide similar problems and detailed explanations."

[0120] This system allows learners to efficiently deepen their understanding and improve areas where they struggle.

[0121] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0122] Step 1:

[0123] The user takes a picture of the learning content's information unit with a mobile device. The input is image data captured by the camera, and this data is acquired by the device. The image is then sent to the server via a dedicated application.

[0124] Step 2:

[0125] The server extracts text information from received image data using OCR technology. The input is image data sent by the user, and the output is text data present within the image. The specific operation involves using the Google Cloud Vision API to extract the text information.

[0126] Step 3:

[0127] The server analyzes the extracted character information to classify the target of learning. The input is character information obtained by OCR, and the output is the classification result of the character information. This analysis includes using natural language processing techniques to extract meaning from the string and associate it with the learning field.

[0128] Step 4:

[0129] The server updates the learner's characteristics based on the analysis results. The input is the analyzed character information and the existing learner characteristics, and the output is the updated learner characteristics information. It refers to the historical database and performs specific calculations that reflect the learner's existing knowledge level and areas of weakness.

[0130] Step 5:

[0131] The server generates similar information and explanations using a generative AI model based on the learner's characteristics. The input is the updated learner's characteristics, and the output is an individually optimized learning plan and explanatory data. The prompt "Please provide similar problems and solutions for problem XX" is used to process the data in the generative model.

[0132] Step 6:

[0133] The server sends the generated learning content to the user's mobile device. The input is the generated learning plan and explanatory data, and the output is the learning content displayed on the user's mobile device. The device receives this data and operates on an interface that allows the user to review the material immediately.

[0134] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0135] This invention is a system that incorporates an emotion engine to support the learning process when a user engages in learning activities. The process begins with the user taking a picture of a problem they answered incorrectly. The acquired problem image is sent to a server, which uses OCR technology to extract text information from the image. The server then analyzes the text information and classifies the problem.

[0136] The server updates the user's learning profile based on the analysis results. Based on this updated profile, the server generates similar problems, explanations, and even lecture videos. In addition, an emotion engine built into the user's device uses the camera and microphone to capture changes in the user's facial expressions and voice, and analyzes the user's emotional state.

[0137] As a concrete example, when a user posts a problem, the server recognizes that it is related to quadratic equations. In this process, if the emotion engine determines that the user is experiencing stress or lacking understanding while working on the problem, the server prepares an explanatory video tailored to the user's feelings. For example, if the user is experiencing high stress, a concise and easy-to-understand step-by-step explanation is provided.

[0138] The device displays the generated learning content along with a learning approach tailored to the user's emotions. The user uses this information to review effectively, and the server further optimizes the learning strategy based on feedback from the emotion engine.

[0139] By taking user emotions into account, this system can not only provide knowledge but also enhance the user's emotional learning experience. As a result, it can provide more personalized support and improve learning efficiency.

[0140] The following describes the processing flow.

[0141] Step 1:

[0142] The user takes a picture or screenshot of the incorrect answer using their device. This action prepares the problem that was assigned as a learning task to be recorded in digital format.

[0143] Step 2:

[0144] The device uploads the problem image it captures to the server via a dedicated application. Along with the image, metadata such as the user's ID and related problem information is also sent to the server.

[0145] Step 3:

[0146] The server receives image data, saves it to storage, and uses OCR technology to extract text information from the image. This allows the problem statement to be treated as digital text.

[0147] Step 4:

[0148] The server analyzes the character information obtained by OCR to identify the type and content of the problem. Natural language processing technology is used for the analysis, and the problem is classified.

[0149] Step 5:

[0150] The server updates the user's learning profile using the analysis results. This enhances the profile information, ensuring it accurately reflects the user's learning history and areas of difficulty.

[0151] Step 6:

[0152] The device's built-in emotion engine analyzes the user's facial expressions and voice data to evaluate their emotional state. Based on this evaluation, it measures stress levels and interest levels in real time.

[0153] Step 7:

[0154] The server generates similar problems, related explanations, and lecture videos based on the user's profile and sentiment rating. The generated content is optimized to take the user's emotional state into consideration.

[0155] Step 8:

[0156] The server sends generated content to the terminal, which then displays it to the user. The user can then use the provided content to continue their learning.

[0157] Step 9:

[0158] The user deepens their understanding by re-solving the problem and watching the explanation. The device continues to collect sentiment data and sends feedback to the server that is appropriate for the user's learning progress. The server uses this information to further improve the profile.

[0159] (Example 2)

[0160] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0161] Traditional learning support systems focused on providing content to improve users' problem-solving abilities, but they had the challenge of not being able to flexibly respond to changes in users' emotions and understanding. Furthermore, they lacked sufficient methods to provide a learning experience optimized for individual users. As a result, learning efficiency varied greatly from person to person, and there was a problem in that they could not always provide effective learning.

[0162] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0163] In this invention, the server includes means for receiving a problem image taken or acquired by the user, means for extracting textual information from the received problem image, and means for analyzing the extracted textual information and classifying the learning target. This enables the analysis of the problem image and classification of the learning target, the provision of learning content suitable for the user, and further, the provision of an individualized learning experience based on the user's emotional information.

[0164] "User" refers to an individual or group that uses the system to engage in learning activities.

[0165] "Means for receiving problem images" refers to a system for uploading problem images, which are taken or acquired by the user, to a server as digital data.

[0166] "Means for extracting character information" refers to technologies for identifying characters from a received image and extracting them as text data, particularly optical character recognition technology.

[0167] "Methods for analyzing textual information and classifying learning targets" refers to the process of analyzing extracted text and determining and classifying the categories and themes of learning targets based on their content.

[0168] "Means for updating learner profiles" refers to a function within the system that updates information about the user's learning history and characteristics based on analysis results, etc.

[0169] "Means for generating similar problems and explanations" refers to a system that constructs new, similar problems and explanations based on the user's profile information.

[0170] "Means of transmitting to the user's device" refers to the function of transferring learning content and explanations generated on the server to the user's terminal and making them available for display or use.

[0171] "Methods for analyzing emotions" refers to technologies that collect the user's facial expressions and voice data, and then analyze that data to understand the user's emotional state.

[0172] "Means of providing personalized learning content" refers to a function that suggests learning content and support optimized for each user based on the results of an emotional analysis of the user.

[0173] This invention is a system designed to provide more effective support to users when they engage in learning activities. This system primarily consists of three elements: a server, a terminal, and the user.

[0174] The server first uses optical character recognition (OCR) technology to extract text information from the problem image received from the user's terminal. This can be done using a general-purpose server or a cloud-based platform with high data processing capabilities. The extracted text information is analyzed using natural language processing technology and classified into similar problem categories. Furthermore, a generative AI model is used to generate similar problems, explanations, and lecture videos tailored to the user's learning profile. In this process, specific instructions, such as "Generate an explanatory video to deepen the user's understanding," are input to the model as prompts.

[0175] The user's device is equipped with a camera and microphone. This device runs software that analyzes the user's emotions from their facial expressions and voice. The emotion engine analyzes the user's emotions in real time based on the collected data and sends the results to the server. Based on this information, the server provides the user with personalized learning content and explanations.

[0176] As a concrete example, suppose a server analyzes a received image of a problem using OCR and identifies it as a problem involving a quadratic equation. Simultaneously, if sentiment analysis reveals that the user is experiencing high stress related to the problem, the server will present a step-by-step video explanation or a simple, intuitive explanation. In this way, the user can receive learning support that reflects their own emotions.

[0177] This allows the system to suggest the optimal learning approach to the user while reducing their emotional burden. Furthermore, by using generative AI models, it can provide a personalized learning experience for each user.

[0178] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0179] Step 1:

[0180] The terminal uses its camera to photograph the problem the user answered incorrectly. At this time, it obtains image data of the photographed problem as input. The terminal sends this image data to the server. As output, the image data is transferred to the server, and it is ready for character information extraction processing on the server.

[0181] Step 2:

[0182] The server uses optical character recognition (OCR) technology on the received image data. It receives the image data in question, sent from the terminal, as input. This process extracts character information from the image data, obtaining text data as output. The OCR processor identifies character patterns and stores them as digitized text within the server.

[0183] Step 3:

[0184] The server performs natural language processing (NLP) on the extracted text data. The input is text data obtained by optical character recognition (OCR). In this process, the text is analyzed and classified as learning target based on its content. The output is the classification result, and information that should be updated in the user's learning profile is identified. In this process, specific keywords and contexts are analyzed to determine which category the problem belongs to.

[0185] Step 4:

[0186] The server updates the user's learning profile based on the analyzed information. The input is the classification results obtained in step 3. This process reflects the areas where the user made mistakes and their learning tendencies in the profile. As output, the updated learning profile is saved to the database. This process involves calculations to integrate the new data into the existing profile.

[0187] Step 5:

[0188] The server uses a generative AI model to generate similar problems, explanations, and lecture videos based on the updated learning profile. The input is the updated learning profile. The prompt, "Generate an explanatory video to deepen the user's understanding," is input to the model, and the generated content is obtained as output. In this generation process, the AI ​​creates the most suitable learning materials for the user based on its accumulated knowledge.

[0189] Step 6:

[0190] The device receives generated learning content and displays it to the user. The learning content sent from the server serves as input. The device displays this data in a user-friendly format, making it accessible to the user. As output, the user receives visualized learning materials. The device effectively presents content through its user interface (UI).

[0191] Step 7:

[0192] The device analyzes the user's facial expressions and voice data using an emotion engine. It takes real-time data collected through the camera and microphone as input. This process determines the user's emotional state (e.g., stress level and concentration level) and sends the analysis results to the server as output. The engine uses changes in voice tone and facial expressions as indicators during the analysis.

[0193] Step 8:

[0194] The server optimizes the learning strategy based on the sentiment analysis results. It receives the sentiment analysis results obtained from the device as input. These results are reflected in the learning content, providing personalized feedback and adjustments. As output, optimized learning properties corresponding to the user's emotions are managed. The server then adjusts the next learning content based on the sentiment data.

[0195] (Application Example 2)

[0196] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0197] Traditional learning systems provided uniform learning content without adequately considering user emotions or performance data, making it difficult to provide effective learning support tailored to each learner's level of understanding and emotional state. In particular, there was difficulty in reducing the stress and confusion learners experienced during learning, and in improving their motivation.

[0198] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0199] In this invention, the server includes a device for receiving problem images captured or acquired by the user, a device for extracting textual information from the received problem images, and a device for analyzing the extracted textual information and classifying the learning target. This makes it possible to provide optimized learning content based on the individual emotional state and learning profile of the learner.

[0200] "User" refers to an individual who uses the system to engage in learning activities.

[0201] A "device for receiving problem images" is a device that has the function of sending problem image data, which the user has taken or acquired, to a server.

[0202] A "device for extracting text information" refers to software or hardware that has the function of analyzing text from a received image and extracting it as digital information.

[0203] A "device for classifying learning materials" is a system that analyzes extracted textual information and uses that information to determine the category of learning content.

[0204] A "device for updating learner characteristics information" is a system that records, manages, and updates learner profiles and learning progress based on the results of analysis.

[0205] A "device for generating similar problems and explanations" is a system that, based on learner characteristic information, creates problems and explanations tailored to them, providing learning content that enhances learning efficiency.

[0206] A "device for transmitting to an information terminal" is a system that has the function of transmitting generated learning content to the user's digital device using communication means.

[0207] A "device for analyzing emotional states" is a system that uses information such as the user's facial expressions and voice to understand their emotions and utilize that information to improve the effectiveness and progress of their learning.

[0208] A "device for adjusting and optimizing learning content" is a system that has the function of customizing learning methods and content to provide the most suitable learning methods and content to learners based on analyzed emotional state and characteristic information.

[0209] This invention provides an educational support system that enables users to effectively carry out the learning process. The system receives problem images taken by the user and uses optical character recognition (OCR) technology to extract textual information from the images. The extracted information is analyzed using natural language processing technology and categorized as learning material.

[0210] The server updates the user's learning profile based on the information obtained. A process then generates similar problems and related explanations from the updated profile. The generated learning content is sent to the user's information terminal, allowing the user to continue their learning through it.

[0211] The device incorporates an emotion engine that uses a camera and microphone to analyze the user's facial expressions and voice to determine their emotional state. The analyzed emotional state is used to adjust and optimize learning content, and if the user experiences stress or confusion, it provides appropriate learning resources to help improve learning efficiency.

[0212] For example, if the emotion engine determines that a user is stuck on a particular math problem and is feeling stressed, the server will provide relaxing music to the user's device and send a step-by-step explanatory video tailored to the user's understanding. In this way, the user can calm down and more easily tackle the problem.

[0213] An example of a prompt message used to generate specific explanations using a generative AI model is: "If the user is feeling stressed about the problem they are trying to solve, please create a concise and easy-to-understand explanatory video. Also, please recommend music that will help the user relax." Based on this prompt, the generative AI will create content tailored to the user.

[0214] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0215] Step 1:

[0216] The device receives the problem image taken by the user. The input is the problem image, and the output is the image data stored in the device. This image data is then prepared for the next processing step.

[0217] Step 2:

[0218] The terminal uses OCR technology to extract text information from the received problem image. The input is image data, and the output is text data. Through OCR processing, the content of the problem is interpreted as text and sent to the next analysis step.

[0219] Step 3:

[0220] The server analyzes the extracted text information using natural language processing techniques to classify it into categories for learning. The input is text data, and the output is classification information for the problem. This process determines which learning domain the problem belongs to, and this information is used to update the learning profile.

[0221] Step 4:

[0222] The server updates the user's learning profile based on the problem classification information. The input is the problem classification information, and the output is the updated learning profile data. This process generates a profile that reflects the user's progress and strengths and weaknesses.

[0223] Step 5:

[0224] The server references the updated learning profile and generates similar problems and explanations. The input is the learning profile data, and the output is the generated learning content (problems and explanations). This process provides the user with the most suitable learning resources.

[0225] Step 6:

[0226] The device uses a camera and microphone to capture the user's facial expressions and voice, and an emotion engine analyzes their emotional state. The input is the captured video and audio data, and the output is emotional state information. This information is used to adjust the learning content.

[0227] Step 7:

[0228] The server adjusts and optimizes the learning content it provides based on emotional state information. The input is emotional state information and the generated learning content, and the output is the adjusted and optimized learning content. In this step, content that takes into account the user's stress and lack of understanding is completed.

[0229] Step 8:

[0230] The device displays the final learning content to the user, supporting their learning activities. The input is the adjusted learning content, and the output is the content displayed to the user. As a result, learners can effectively progress through their learning using optimized content.

[0231] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0232] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0233] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0234] [Second Embodiment]

[0235] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0236] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0237] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0238] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0239] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0240] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0241] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0242] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0243] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0244] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0245] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0246] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0247] This invention is a system that supports learners in effectively progressing with their studies, primarily involving data exchange between a server, a terminal, and a user. In implementation, the user first takes a picture of a problem they answered incorrectly with their terminal. This problem image is then sent to the server via a dedicated application.

[0248] The server uses OCR technology to extract text information from the received image. By analyzing this text information, the server identifies the type of problem and the scope of learning, and updates the learner's profile accordingly. Based on the learner's past learning history and current areas of difficulty, the server generates similar learning problems and prepares explanations and lecture videos as needed.

[0249] As a concrete example, if a user makes a mistake in solving a quadratic equation on a math exam, the user takes a picture of the problem. The server receives this and identifies that the problem is related to a quadratic equation. The server then analyzes how the user has failed with similar problems based on past data. As a result, it sends similar quadratic equation problems and detailed explanations of how to solve them to the user's device.

[0250] The device displays the received data to the user, who can then review the provided learning content. Continuous data feedback to the server allows the user's profile to be updated more accurately, improving the precision of subsequent learning support. This system will enable learners to efficiently overcome their weak areas.

[0251] The following describes the processing flow.

[0252] Step 1:

[0253] The user takes a picture or screenshot of the incorrect answer using their own device. This prepares the question content to be saved in digital format.

[0254] Step 2:

[0255] The device uploads the acquired problem images to the server via a dedicated application. Metadata such as the user ID and problem category is attached to the upload.

[0256] Step 3:

[0257] The server temporarily stores the received image data and uses OCR technology to extract text information from the image. This makes the question text available for processing within the system as digital text.

[0258] Step 4:

[0259] The server analyzes the extracted text data to identify the theme and content of the problem. Natural language processing techniques are used in this analysis to classify the problem.

[0260] Step 5:

[0261] The server references the user's existing profile data and updates the user's learning progress and areas of difficulty based on the analysis results. This update is then used to provide personalized learning support.

[0262] Step 6:

[0263] The server generates similar problems and related explanations based on the user's profile information, and prepares lecture videos as needed. This information is generated to enhance the user's learning.

[0264] Step 7:

[0265] The server sends similar problems and explanatory content it has generated to the device. The device displays the received data, allowing the user to use it for review.

[0266] Step 8:

[0267] Users solve similar problems provided on their devices, deepening their understanding by reading explanations or watching videos. The server continuously collects user responses and behavioral data to improve the accuracy of the profiles.

[0268] (Example 1)

[0269] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0270] Conventional learning support systems have the challenge of not being able to accurately identify individual learners' weak areas and quickly provide them with the most suitable learning problems and explanations. Furthermore, the lack of a mechanism for efficiently updating learning history made it difficult to respond flexibly to learners' progress.

[0271] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0272] In this invention, the server includes means for receiving problem images captured or acquired by the user, means for extracting character information using optical character recognition technology, and means for generating similar problems and related information using a generative AI model. This enables the provision of problems tailored to the learner's weak areas and efficient updating of the learning history.

[0273] A "user" refers to an individual who uses a learning support system to advance their studies.

[0274] A "problem image" refers to image data captured by a camera or obtained from a user who answered a problem incorrectly during their learning process.

[0275] "Optical character recognition technology" is a technology that converts characters in an image into digital text.

[0276] "Character information" refers to the character data within the image in question, extracted using optical character recognition technology.

[0277] A "generative AI model" is a model that uses artificial intelligence technology to analyze a learner's profile and generate optimal learning problems and explanations.

[0278] "Relevant information" refers to supplementary materials such as explanations and lecture contents provided by the generative AI model to deepen the learner's understanding.

[0279] "Information processing device" refers to a device used by a user to receive and display the content of the learning support system.

[0280] "History information" refers to information that the system retains as the user's profile, such as the user's past learning activities and performance data.

[0281] "Feedback" refers to information about the user's learning status and performance that the user sends to the system.

[0282] This invention is a learning support system for learners to individually identify areas of insufficient understanding and efficiently advance their learning. Hereinafter, specific embodiments of this system will be described.

[0283] First, the user takes a photo of the incorrect problem with a terminal. A dedicated application is installed on the terminal, and the problem image is designed to be sent to the server via this application. The terminal compresses the taken image to improve communication efficiency.

[0284] When the server receives the problem image, it extracts character information from the image using optical character recognition technology (OCR technology). Specifically, software such as Tesseract OCR can be used. Next, the server executes an algorithm for identifying the type of problem and the learning field using the extracted character information.

[0285] Based on the analysis result, the server updates the learner's history information. This history information includes past learning data and correct / incorrect information in specific fields. Based on the updated history information, similar problems and relevant information are generated using a generative AI model. In the generative AI model, for example, the language model of OpenAI is used, and optimal learning content is generated by inputting a prompt sentence.

[0286] As a specific example, consider the case where a user makes a mistake on a quadratic equation problem in a math test. When the user takes a picture of the problem and sends it to the server, the server extracts character information from the image and identifies that the problem is related to a quadratic equation. Then, the server refers to the learner's history information, generates similar quadratic equation problems and explanations, and sends them to the terminal.

[0287] Examples of prompt sentences are as follows.

[0288] "Please create a learning problem for the following quadratic equation. Past data has shown that users have particular difficulty with the 'completing the square' method. Please provide an explanation and problem solution based on this."

[0289] The terminal displays the received learning content to the user, and the user can efficiently proceed with review based on this content. The user inputs the answer result into the terminal, and the feedback is sent back to the server again and utilized for updating the history information. Through this cycle, the accuracy of learning support for the user is further improved.

[0290] The flow of the specific process in Example 1 will be described using FIG. 11.

[0291] Step 1:

[0292] The user takes a picture of the mistaken problem with the terminal. The input is image data acquired by the camera. The terminal captures this image into a dedicated application. As a specific operation, compression processing is performed until the image is appropriately captured and prepared for transmission.

[0293] Step 2:

[0294] The terminal sends the prepared problem image data to the server. The input here is the image data captured and compressed in step 1, and the output is the transfer of the image data to the server. The terminal establishes a network connection and performs real-time error checking to ensure that the data is transmitted correctly.

[0295] Step 3:

[0296] The server processes the received image and extracts text information using optical character recognition (OCR) technology. The input is the received image data, and the output is the extracted text information. Specifically, the server uses tools such as Tesseract OCR to efficiently analyze the text portion of the image and convert it into digital text.

[0297] Step 4:

[0298] The server analyzes the extracted text information to identify the type of problem and the learning area. The input is the text information extracted in step 3, and the output is the identified type of problem and learning area. Specifically, the text information is passed through a natural language processing algorithm to structure the content of the problem and compare it with the learner's history information.

[0299] Step 5:

[0300] The server updates the learner's history information based on the identified problem information. The input is the identified problem type and learning area, and the output is the updated history information. Specifically, it refers to the past learning database and adds information to analyze which areas the user makes the most mistakes in.

[0301] Step 6:

[0302] The server uses a generative AI model to generate similar problems and related information suitable for the learner. The input is the updated historical information and the identified problem information, and the output is the generated learning content. As a specific operation, the server inputs a prompt sentence into the AI model and generates content focusing on what the user should particularly deepen their understanding of.

[0303] Step 7:

[0304] The server sends the generated learning content to the user's terminal. The input is the content generated in Step 6, and the output is the delivery of the content to the user's terminal. As a specific operation, an efficient data transfer protocol is used to ensure that the content is accurately delivered while minimizing latency.

[0305] Step 8:

[0306] The terminal displays the received learning content to the user. The input is the content data received from the server, and the output is a display state that the user can interactively utilize. As a specific operation, the terminal application formats the content appropriately so that the user can start learning immediately.

[0307] Step 9:

[0308] The user solves the problem based on the provided content and inputs the answer result into the terminal. The input is the user's answer data, and the output is the confirmation of the answer data within the terminal. As a specific operation, it is designed to enable the user to quickly send data through an interface that is easy to input answers.

[0309] Step 10:

[0310] The terminal sends the user's answer results to the server, where they are reflected as feedback. The input is the user's answer data, and the output is the transfer of data to the server. Specifically, the transmitted data is checked to ensure its accuracy and to be used for updating the history information on the server side.

[0311] Step 11:

[0312] The server updates the learner's history information based on the feedback received, and this data is used to generate the next problem. The input is the user's answer data, and the output is the latest history information. Specifically, the server adds the updated information to the database and uses the newly acquired knowledge in the next learning cycle.

[0313] (Application Example 1)

[0314] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0315] In education, it is crucial to support learners in understanding their mistakes and effectively overcoming them. However, existing means of providing individually optimized learning support are limited, and there are challenges in accurately understanding each learner's characteristics and areas of difficulty and adjusting learning content accordingly. Furthermore, current systems cannot efficiently utilize users' learning history, resulting in insufficient learning support.

[0316] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0317] In this invention, the server includes means for receiving information units captured or acquired by the user, means for extracting character information from the received information units, and means for analyzing the extracted character information and classifying the learning target. This makes it possible to provide individually optimized learning support and to provide effective learning content based on the learner's characteristics and past learning history.

[0318] "Users" refers to learners and educators who use this system to receive learning support.

[0319] An "information unit" refers to data, including questions and learning materials, that learners photograph or acquire and send to the system.

[0320] "Textual information" refers to text data extracted from information units using OCR technology.

[0321] "Classification of learning targets" refers to the process of analyzing extracted textual information and classifying its content into specific learning categories or themes.

[0322] "Learner characteristics" refers to profile information including the learner's past learning history, current level of understanding, and areas of difficulty.

[0323] "Similar information" refers to learning questions and content generated based on learners' areas of difficulty and incorrect answers.

[0324] "Explanation" refers to explanatory materials or lecture materials provided for similar information.

[0325] "Portable devices" refer to electronic terminals that users can carry around, such as smartphones and tablets.

[0326] A "learning plan" refers to a learning schedule and content optimized for each individual learner.

[0327] The system implementing this invention provides learning support to help learners effectively understand and overcome their mistakes. The entire system consists of the user's mobile device and a server, and utilizes a variety of technologies.

[0328] First, the user takes a picture of a problem they find difficult or answered incorrectly using their mobile device. The captured information is sent to a server via a dedicated application installed on the mobile device. The server then extracts text information from the received information using OCR technology (for example, AWS Rekognition or Google Cloud Vision).

[0329] The server analyzes the extracted text information and classifies the learning targets. Natural language processing techniques are also utilized to interpret the meaning and structure of the text information with greater precision. Based on the analyzed information, the server updates the learner's characteristics and creates a learning plan optimized for each individual learner.

[0330] This system generates similar information and explanations based on the learner's characteristics. The generated learning content is sent from the server to the user's mobile device, where the user can review it and continue learning. This allows for rapid correction of misunderstandings and errors arising from individual learning, resulting in efficient learning.

[0331] As a concrete example of its use, if a high school student takes a picture of a math problem they got wrong on a test, the photo is sent from their mobile device to a server, where OCR technology converts the problem content into text. The server analyzes the content and extracts particularly difficult concepts and problem patterns by comparing them with the learner's past records. Then, using AI, it generates similar problems and detailed explanations of their solutions, which are delivered to the learner's mobile device. A possible prompt in this case would be, "I made a mistake in solving the quadratic equation x² + 3x - 4 = 0. Please provide similar problems and detailed explanations."

[0332] This system allows learners to efficiently deepen their understanding and improve areas where they struggle.

[0333] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0334] Step 1:

[0335] The user takes a picture of the learning content's information unit with a mobile device. The input is image data captured by the camera, and this data is acquired by the device. The image is then sent to the server via a dedicated application.

[0336] Step 2:

[0337] The server extracts text information from received image data using OCR technology. The input is image data sent by the user, and the output is text data present within the image. The specific operation involves using the Google Cloud Vision API to extract the text information.

[0338] Step 3:

[0339] The server analyzes the extracted character information to classify the target of learning. The input is character information obtained by OCR, and the output is the classification result of the character information. This analysis includes using natural language processing techniques to extract meaning from the string and associate it with the learning field.

[0340] Step 4:

[0341] The server updates the learner's characteristics based on the analysis results. The input is the analyzed character information and the existing learner characteristics, and the output is the updated learner characteristics information. It refers to the historical database and performs specific calculations that reflect the learner's existing knowledge level and areas of weakness.

[0342] Step 5:

[0343] The server generates similar information and explanations using a generative AI model based on the learner's characteristics. The input is the updated learner's characteristics, and the output is an individually optimized learning plan and explanatory data. The prompt "Please provide similar problems and solutions for problem XX" is used to process the data in the generative model.

[0344] Step 6:

[0345] The server sends the generated learning content to the user's mobile device. The input is the generated learning plan and explanatory data, and the output is the learning content displayed on the user's mobile device. The device receives this data and operates on an interface that allows the user to review the material immediately.

[0346] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0347] This invention is a system that incorporates an emotion engine to support the learning process when a user engages in learning activities. The process begins with the user taking a picture of a problem they answered incorrectly. The acquired problem image is sent to a server, which uses OCR technology to extract text information from the image. The server then analyzes the text information and classifies the problem.

[0348] The server updates the user's learning profile based on the analysis results. Based on this updated profile, the server generates similar problems, explanations, and even lecture videos. In addition, an emotion engine built into the user's device uses the camera and microphone to capture changes in the user's facial expressions and voice, and analyzes the user's emotional state.

[0349] As a concrete example, when a user posts a problem, the server recognizes that it is related to quadratic equations. In this process, if the emotion engine determines that the user is experiencing stress or lacking understanding while working on the problem, the server prepares an explanatory video tailored to the user's feelings. For example, if the user is experiencing high stress, a concise and easy-to-understand step-by-step explanation is provided.

[0350] The device displays the generated learning content along with a learning approach tailored to the user's emotions. The user uses this information to review effectively, and the server further optimizes the learning strategy based on feedback from the emotion engine.

[0351] By taking user emotions into account, this system can not only provide knowledge but also enhance the user's emotional learning experience. As a result, it can provide more personalized support and improve learning efficiency.

[0352] The following describes the processing flow.

[0353] Step 1:

[0354] The user takes a picture or screenshot of the incorrect answer using their device. This action prepares the problem that was assigned as a learning task to be recorded in digital format.

[0355] Step 2:

[0356] The device uploads the problem image it captures to the server via a dedicated application. Along with the image, metadata such as the user's ID and related problem information is also sent to the server.

[0357] Step 3:

[0358] The server receives image data, saves it to storage, and uses OCR technology to extract text information from the image. This allows the problem statement to be treated as digital text.

[0359] Step 4:

[0360] The server analyzes the character information obtained by OCR to identify the type and content of the problem. Natural language processing technology is used for the analysis, and the problem is classified.

[0361] Step 5:

[0362] The server updates the user's learning profile using the analysis results. This enhances the profile information, ensuring it accurately reflects the user's learning history and areas of difficulty.

[0363] Step 6:

[0364] The device's built-in emotion engine analyzes the user's facial expressions and voice data to evaluate their emotional state. Based on this evaluation, it measures stress levels and interest levels in real time.

[0365] Step 7:

[0366] The server generates similar problems, related explanations, and lecture videos based on the user's profile and sentiment rating. The generated content is optimized to take the user's emotional state into consideration.

[0367] Step 8:

[0368] The server sends generated content to the terminal, which then displays it to the user. The user can then use the provided content to continue their learning.

[0369] Step 9:

[0370] The user deepens their understanding by re-solving the problem and watching the explanation. The device continues to collect sentiment data and sends feedback to the server that is appropriate for the user's learning progress. The server uses this information to further improve the profile.

[0371] (Example 2)

[0372] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0373] Traditional learning support systems focused on providing content to improve users' problem-solving abilities, but they had the challenge of not being able to flexibly respond to changes in users' emotions and understanding. Furthermore, they lacked sufficient methods to provide a learning experience optimized for individual users. As a result, learning efficiency varied greatly from person to person, and there was a problem in that they could not always provide effective learning.

[0374] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0375] In this invention, the server includes means for receiving a problem image taken or acquired by the user, means for extracting textual information from the received problem image, and means for analyzing the extracted textual information and classifying the learning target. This enables the analysis of the problem image and classification of the learning target, the provision of learning content suitable for the user, and further, the provision of an individualized learning experience based on the user's emotional information.

[0376] "User" refers to an individual or group that uses the system to engage in learning activities.

[0377] "Means for receiving problem images" refers to a system for uploading problem images, which are taken or acquired by the user, to a server as digital data.

[0378] "Means for extracting character information" refers to technologies for identifying characters from a received image and extracting them as text data, particularly optical character recognition technology.

[0379] "Methods for analyzing textual information and classifying learning targets" refers to the process of analyzing extracted text and determining and classifying the categories and themes of learning targets based on their content.

[0380] "Means for updating learner profiles" refers to a function within the system that updates information about the user's learning history and characteristics based on analysis results, etc.

[0381] "Means for generating similar problems and explanations" refers to a system that constructs new, similar problems and explanations based on the user's profile information.

[0382] "Means of transmitting to the user's device" refers to the function of transferring learning content and explanations generated on the server to the user's terminal and making them available for display or use.

[0383] "Methods for analyzing emotions" refers to technologies that collect the user's facial expressions and voice data, and then analyze that data to understand the user's emotional state.

[0384] "Means of providing personalized learning content" refers to a function that suggests learning content and support optimized for each user based on the results of an emotional analysis of the user.

[0385] This invention is a system designed to provide more effective support to users when they engage in learning activities. This system primarily consists of three elements: a server, a terminal, and the user.

[0386] The server first uses optical character recognition (OCR) technology to extract text information from the problem image received from the user's terminal. This can be done using a general-purpose server or a cloud-based platform with high data processing capabilities. The extracted text information is analyzed using natural language processing technology and classified into similar problem categories. Furthermore, a generative AI model is used to generate similar problems, explanations, and lecture videos tailored to the user's learning profile. In this process, specific instructions, such as "Generate an explanatory video to deepen the user's understanding," are input to the model as prompts.

[0387] The user's device is equipped with a camera and microphone. This device runs software that analyzes the user's emotions from their facial expressions and voice. The emotion engine analyzes the user's emotions in real time based on the collected data and sends the results to the server. Based on this information, the server provides the user with personalized learning content and explanations.

[0388] As a concrete example, suppose a server analyzes a received image of a problem using OCR and identifies it as a problem involving a quadratic equation. Simultaneously, if sentiment analysis reveals that the user is experiencing high stress related to the problem, the server will present a step-by-step video explanation or a simple, intuitive explanation. In this way, the user can receive learning support that reflects their own emotions.

[0389] This allows the system to suggest the optimal learning approach to the user while reducing their emotional burden. Furthermore, by using generative AI models, it can provide a personalized learning experience for each user.

[0390] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0391] Step 1:

[0392] The terminal uses its camera to photograph the problem the user answered incorrectly. At this time, it obtains image data of the photographed problem as input. The terminal sends this image data to the server. As output, the image data is transferred to the server, and it is ready for character information extraction processing on the server.

[0393] Step 2:

[0394] The server uses optical character recognition (OCR) technology on the received image data. It receives the image data in question, sent from the terminal, as input. This process extracts character information from the image data, obtaining text data as output. The OCR processor identifies character patterns and stores them as digitized text within the server.

[0395] Step 3:

[0396] The server performs natural language processing (NLP) on the extracted text data. The input is text data obtained by optical character recognition (OCR). In this process, the text is analyzed and classified as learning target based on its content. The output is the classification result, and information that should be updated in the user's learning profile is identified. In this process, specific keywords and contexts are analyzed to determine which category the problem belongs to.

[0397] Step 4:

[0398] The server updates the user's learning profile based on the analyzed information. The input is the classification results obtained in step 3. This process reflects the areas where the user made mistakes and their learning tendencies in the profile. As output, the updated learning profile is saved to the database. This process involves calculations to integrate the new data into the existing profile.

[0399] Step 5:

[0400] The server uses a generative AI model to generate similar problems, explanations, and lecture videos based on the updated learning profile. The input is the updated learning profile. The prompt, "Generate an explanatory video to deepen the user's understanding," is input to the model, and the generated content is obtained as output. In this generation process, the AI ​​creates the most suitable learning materials for the user based on its accumulated knowledge.

[0401] Step 6:

[0402] The device receives generated learning content and displays it to the user. The learning content sent from the server serves as input. The device displays this data in a user-friendly format, making it accessible to the user. As output, the user receives visualized learning materials. The device effectively presents content through its user interface (UI).

[0403] Step 7:

[0404] The device analyzes the user's facial expressions and voice data using an emotion engine. It takes real-time data collected through the camera and microphone as input. This process determines the user's emotional state (e.g., stress level and concentration level) and sends the analysis results to the server as output. The engine uses changes in voice tone and facial expressions as indicators during the analysis.

[0405] Step 8:

[0406] The server optimizes the learning strategy based on the sentiment analysis results. It receives the sentiment analysis results obtained from the device as input. These results are reflected in the learning content, providing personalized feedback and adjustments. As output, optimized learning properties corresponding to the user's emotions are managed. The server then adjusts the next learning content based on the sentiment data.

[0407] (Application Example 2)

[0408] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0409] Traditional learning systems provided uniform learning content without adequately considering user emotions or performance data, making it difficult to provide effective learning support tailored to each learner's level of understanding and emotional state. In particular, there was difficulty in reducing the stress and confusion learners experienced during learning, and in improving their motivation.

[0410] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0411] In this invention, the server includes a device for receiving problem images captured or acquired by the user, a device for extracting textual information from the received problem images, and a device for analyzing the extracted textual information and classifying the learning target. This makes it possible to provide optimized learning content based on the individual emotional state and learning profile of the learner.

[0412] "User" refers to an individual who uses the system to engage in learning activities.

[0413] A "device for receiving problem images" is a device that has the function of sending problem image data, which the user has taken or acquired, to a server.

[0414] A "device for extracting text information" refers to software or hardware that has the function of analyzing text from a received image and extracting it as digital information.

[0415] A "device for classifying learning materials" is a system that analyzes extracted textual information and uses that information to determine the category of learning content.

[0416] A "device for updating learner characteristics information" is a system that records, manages, and updates learner profiles and learning progress based on the results of analysis.

[0417] A "device for generating similar problems and explanations" is a system that, based on learner characteristic information, creates problems and explanations tailored to them, providing learning content that enhances learning efficiency.

[0418] A "device for transmitting to an information terminal" is a system that has the function of transmitting generated learning content to the user's digital device using communication means.

[0419] A "device for analyzing emotional states" is a system that uses information such as the user's facial expressions and voice to understand their emotions and utilize that information to improve the effectiveness and progress of their learning.

[0420] A "device for adjusting and optimizing learning content" is a system that has the function of customizing learning methods and content to provide the most suitable learning methods and content to learners based on analyzed emotional state and characteristic information.

[0421] This invention provides an educational support system that enables users to effectively carry out the learning process. The system receives problem images taken by the user and uses optical character recognition (OCR) technology to extract textual information from the images. The extracted information is analyzed using natural language processing technology and categorized as learning material.

[0422] The server updates the user's learning profile based on the information obtained. A process then generates similar problems and related explanations from the updated profile. The generated learning content is sent to the user's information terminal, allowing the user to continue their learning through it.

[0423] The device incorporates an emotion engine that uses a camera and microphone to analyze the user's facial expressions and voice to determine their emotional state. The analyzed emotional state is used to adjust and optimize learning content, and if the user experiences stress or confusion, it provides appropriate learning resources to help improve learning efficiency.

[0424] For example, if the emotion engine determines that a user is stuck on a particular math problem and is feeling stressed, the server will provide relaxing music to the user's device and send a step-by-step explanatory video tailored to the user's understanding. In this way, the user can calm down and more easily tackle the problem.

[0425] An example of a prompt message used to generate specific explanations using a generative AI model is: "If the user is feeling stressed about the problem they are trying to solve, please create a concise and easy-to-understand explanatory video. Also, please recommend music that will help the user relax." Based on this prompt, the generative AI will create content tailored to the user.

[0426] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0427] Step 1:

[0428] The device receives the problem image taken by the user. The input is the problem image, and the output is the image data stored in the device. This image data is then prepared for the next processing step.

[0429] Step 2:

[0430] The terminal uses OCR technology to extract text information from the received problem image. The input is image data, and the output is text data. Through OCR processing, the content of the problem is interpreted as text and sent to the next analysis step.

[0431] Step 3:

[0432] The server analyzes the extracted text information using natural language processing techniques to classify it into categories for learning. The input is text data, and the output is classification information for the problem. This process determines which learning domain the problem belongs to, and this information is used to update the learning profile.

[0433] Step 4:

[0434] The server updates the user's learning profile based on the problem classification information. The input is the problem classification information, and the output is the updated learning profile data. This process generates a profile that reflects the user's progress and strengths and weaknesses.

[0435] Step 5:

[0436] The server references the updated learning profile and generates similar problems and explanations. The input is the learning profile data, and the output is the generated learning content (problems and explanations). This process provides the user with the most suitable learning resources.

[0437] Step 6:

[0438] The device uses a camera and microphone to capture the user's facial expressions and voice, and an emotion engine analyzes their emotional state. The input is the captured video and audio data, and the output is emotional state information. This information is used to adjust the learning content.

[0439] Step 7:

[0440] The server adjusts and optimizes the learning content it provides based on emotional state information. The input is emotional state information and the generated learning content, and the output is the adjusted and optimized learning content. In this step, content that takes into account the user's stress and lack of understanding is completed.

[0441] Step 8:

[0442] The device displays the final learning content to the user, supporting their learning activities. The input is the adjusted learning content, and the output is the content displayed to the user. As a result, learners can effectively progress through their learning using optimized content.

[0443] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0444] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0445] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0446] [Third Embodiment]

[0447] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0448] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0449] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0450] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0451] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0452] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0453] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0454] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0455] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0456] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0457] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0458] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0459] This invention is a system that supports learners in effectively progressing with their studies, primarily involving data exchange between a server, a terminal, and a user. In implementation, the user first takes a picture of a problem they answered incorrectly with their terminal. This problem image is then sent to the server via a dedicated application.

[0460] The server uses OCR technology to extract text information from the received image. By analyzing this text information, the server identifies the type of problem and the scope of learning, and updates the learner's profile accordingly. Based on the learner's past learning history and current areas of difficulty, the server generates similar learning problems and prepares explanations and lecture videos as needed.

[0461] As a concrete example, if a user makes a mistake in solving a quadratic equation on a math exam, the user takes a picture of the problem. The server receives this and identifies that the problem is related to a quadratic equation. The server then analyzes how the user has failed with similar problems based on past data. As a result, it sends similar quadratic equation problems and detailed explanations of how to solve them to the user's device.

[0462] The device displays the received data to the user, who can then review the provided learning content. Continuous data feedback to the server allows the user's profile to be updated more accurately, improving the precision of subsequent learning support. This system will enable learners to efficiently overcome their weak areas.

[0463] The following describes the processing flow.

[0464] Step 1:

[0465] The user takes a picture or screenshot of the incorrect answer using their own device. This prepares the question content to be saved in digital format.

[0466] Step 2:

[0467] The device uploads the acquired problem images to the server via a dedicated application. Metadata such as the user ID and problem category is attached to the upload.

[0468] Step 3:

[0469] The server temporarily stores the received image data and uses OCR technology to extract text information from the image. This makes the question text available for processing within the system as digital text.

[0470] Step 4:

[0471] The server analyzes the extracted text data to identify the theme and content of the problem. Natural language processing techniques are used in this analysis to classify the problem.

[0472] Step 5:

[0473] The server references the user's existing profile data and updates the user's learning progress and areas of difficulty based on the analysis results. This update is then used to provide personalized learning support.

[0474] Step 6:

[0475] The server generates similar problems and related explanations based on the user's profile information, and prepares lecture videos as needed. This information is generated to enhance the user's learning.

[0476] Step 7:

[0477] The server sends similar problems and explanatory content it has generated to the device. The device displays the received data, allowing the user to use it for review.

[0478] Step 8:

[0479] Users solve similar problems provided on their devices, deepening their understanding by reading explanations or watching videos. The server continuously collects user responses and behavioral data to improve the accuracy of the profiles.

[0480] (Example 1)

[0481] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0482] Conventional learning support systems have the challenge of not being able to accurately identify individual learners' weak areas and quickly provide them with the most suitable learning problems and explanations. Furthermore, the lack of a mechanism for efficiently updating learning history made it difficult to respond flexibly to learners' progress.

[0483] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0484] In this invention, the server includes means for receiving problem images captured or acquired by the user, means for extracting character information using optical character recognition technology, and means for generating similar problems and related information using a generative AI model. This enables the provision of problems tailored to the learner's weak areas and efficient updating of the learning history.

[0485] A "user" refers to an individual who uses a learning support system to advance their studies.

[0486] A "problem image" refers to image data captured by a camera or obtained from a user who answered a problem incorrectly during their learning process.

[0487] "Optical character recognition technology" is a technology that converts characters in an image into digital text.

[0488] "Character information" refers to the character data within the image in question, extracted using optical character recognition technology.

[0489] A "generative AI model" is a model that uses artificial intelligence technology to analyze a learner's profile and generate optimal learning problems and explanations.

[0490] "Related information" refers to supplementary materials such as explanations and lecture content provided by generative AI models to deepen learners' understanding.

[0491] An "information processing device" is a device used by a user to receive and display content from a learning support system.

[0492] "History information" refers to information that the system retains as a user profile, such as the user's past learning activities and performance data.

[0493] "Feedback" refers to information about one's learning progress and performance that a user sends to the system.

[0494] This invention is a learning support system that allows learners to individually identify areas where they lack understanding and to efficiently advance their learning. The following describes a specific embodiment of this system.

[0495] First, the user takes a picture of the question they answered incorrectly with their device. A dedicated application is installed on the device, and the image of the question is sent to the server via this application. The device compresses the captured image to improve communication efficiency.

[0496] Upon receiving a problem image, the server extracts text information from the image using optical character recognition (OCR) technology. Specifically, software such as Tesseract OCR can be used. Next, the server uses the extracted text information to execute an algorithm to identify the type of problem and the learning area.

[0497] The server updates the learner's history information based on the analysis results. This history information includes past learning data and correct / incorrect information in specific areas. Based on the updated history information, a generative AI model is used to generate similar problems and related information. For example, the OpenAI language model is used in the generative AI model, and optimal learning content is generated by inputting prompt sentences.

[0498] As a concrete example, consider a case where a user makes a mistake on a quadratic equation problem in a math exam. The user takes a picture of the problem and sends it to the server. The server extracts textual information from the image and identifies that the problem is related to a quadratic equation. The server then refers to the learner's history information to generate similar quadratic equation problems and explanations, and sends them to the device.

[0499] Examples of prompt messages include the following:

[0500] "Please create the following learning problem involving quadratic equations. Past data shows that users particularly struggle with the 'completing the square' method. Please provide an explanation and problem proposal based on this."

[0501] The device displays the received learning content to the user, allowing the user to efficiently review the material based on this information. The user enters their answers into the device, and this feedback is sent back to the server and used to update the history information. This cycle further improves the accuracy of learning support for the user.

[0502] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0503] Step 1:

[0504] The user takes a picture of the incorrect answer with their device. The input is image data acquired by the camera. The device then imports this image into a dedicated application. Specifically, the image is properly captured and compressed before being prepared for transmission.

[0505] Step 2:

[0506] The terminal sends the prepared problem image data to the server. The input here is the image data captured and compressed in step 1, and the output is the transfer of the image data to the server. The terminal establishes a network connection and performs real-time error checking to ensure that the data is transmitted correctly.

[0507] Step 3:

[0508] The server processes the received image and extracts text information using optical character recognition (OCR) technology. The input is the received image data, and the output is the extracted text information. Specifically, the server uses tools such as Tesseract OCR to efficiently analyze the text portion of the image and convert it into digital text.

[0509] Step 4:

[0510] The server analyzes the extracted text information to identify the type of problem and the learning area. The input is the text information extracted in step 3, and the output is the identified type of problem and learning area. Specifically, the text information is passed through a natural language processing algorithm to structure the content of the problem and compare it with the learner's history information.

[0511] Step 5:

[0512] The server updates the learner's history information based on the identified problem information. The input is the identified problem type and learning area, and the output is the updated history information. Specifically, it refers to the past learning database and adds information to analyze which areas the user makes the most mistakes in.

[0513] Step 6:

[0514] The server uses a generative AI model to generate similar problems and relevant information suitable for the learner. The input is updated historical information and identified problem information, and the output is the generated learning content. Specifically, the server inputs prompt sentences into the AI ​​model and generates content that focuses on what the user particularly needs to understand.

[0515] Step 7:

[0516] The server sends the generated learning content to the user's device. The input is the content generated in step 6, and the output is the delivery of the content to the user's device. Specifically, an efficient data transfer protocol is used to ensure that the content is delivered accurately while minimizing latency.

[0517] Step 8:

[0518] The terminal displays the received learning content to the user. The input is content data received from the server, and the output is a display state that the user can interact with. Specifically, the terminal application formats the content appropriately so that the user can start learning immediately.

[0519] Step 9:

[0520] The user solves problems based on the provided content and inputs their answers into the device. The input is the user's answer data, and the output is the confirmation of the answer data within the device. Specifically, the system is designed to allow users to quickly send data through an interface that makes it easy to input answers.

[0521] Step 10:

[0522] The terminal sends the user's answer results to the server, where they are reflected as feedback. The input is the user's answer data, and the output is the transfer of data to the server. Specifically, the transmitted data is checked to ensure its accuracy and to be used for updating the history information on the server side.

[0523] Step 11:

[0524] The server updates the learner's history information based on the feedback received, and this data is used to generate the next problem. The input is the user's answer data, and the output is the latest history information. Specifically, the server adds the updated information to the database and uses the newly acquired knowledge in the next learning cycle.

[0525] (Application Example 1)

[0526] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0527] In education, it is crucial to support learners in understanding their mistakes and effectively overcoming them. However, existing means of providing individually optimized learning support are limited, and there are challenges in accurately understanding each learner's characteristics and areas of difficulty and adjusting learning content accordingly. Furthermore, current systems cannot efficiently utilize users' learning history, resulting in insufficient learning support.

[0528] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0529] In this invention, the server includes means for receiving information units captured or acquired by the user, means for extracting character information from the received information units, and means for analyzing the extracted character information and classifying the learning target. This makes it possible to provide individually optimized learning support and to provide effective learning content based on the learner's characteristics and past learning history.

[0530] "Users" refers to learners and educators who use this system to receive learning support.

[0531] An "information unit" refers to data, including questions and learning materials, that learners photograph or acquire and send to the system.

[0532] "Textual information" refers to text data extracted from information units using OCR technology.

[0533] "Classification of learning targets" refers to the process of analyzing extracted textual information and classifying its content into specific learning categories or themes.

[0534] "Learner characteristics" refers to profile information including the learner's past learning history, current level of understanding, and areas of difficulty.

[0535] "Similar information" refers to learning questions and content generated based on learners' areas of difficulty and incorrect answers.

[0536] "Explanation" refers to explanatory materials or lecture materials provided for similar information.

[0537] "Portable devices" refer to electronic terminals that users can carry around, such as smartphones and tablets.

[0538] A "learning plan" refers to a learning schedule and content optimized for each individual learner.

[0539] The system implementing this invention provides learning support to help learners effectively understand and overcome their mistakes. The entire system consists of the user's mobile device and a server, and utilizes a variety of technologies.

[0540] First, the user takes a picture of a problem they find difficult or answered incorrectly using their mobile device. The captured information is sent to a server via a dedicated application installed on the mobile device. The server then extracts text information from the received information using OCR technology (for example, AWS Rekognition or Google Cloud Vision).

[0541] The server analyzes the extracted text information and classifies the learning targets. Natural language processing techniques are also utilized to interpret the meaning and structure of the text information with greater precision. Based on the analyzed information, the server updates the learner's characteristics and creates a learning plan optimized for each individual learner.

[0542] This system generates similar information and explanations based on the learner's characteristics. The generated learning content is sent from the server to the user's mobile device, where the user can review it and continue learning. This allows for rapid correction of misunderstandings and errors arising from individual learning, resulting in efficient learning.

[0543] As a concrete example of its use, if a high school student takes a picture of a math problem they got wrong on a test, the photo is sent from their mobile device to a server, where OCR technology converts the problem content into text. The server analyzes the content and extracts particularly difficult concepts and problem patterns by comparing them with the learner's past records. Then, using AI, it generates similar problems and detailed explanations of their solutions, which are delivered to the learner's mobile device. A possible prompt in this case would be, "I made a mistake in solving the quadratic equation x² + 3x - 4 = 0. Please provide similar problems and detailed explanations."

[0544] This system allows learners to efficiently deepen their understanding and improve areas where they struggle.

[0545] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0546] Step 1:

[0547] The user takes a picture of the learning content's information unit with a mobile device. The input is image data captured by the camera, and this data is acquired by the device. The image is then sent to the server via a dedicated application.

[0548] Step 2:

[0549] The server extracts text information from received image data using OCR technology. The input is image data sent by the user, and the output is text data present within the image. The specific operation involves using the Google Cloud Vision API to extract the text information.

[0550] Step 3:

[0551] The server analyzes the extracted character information to classify the target of learning. The input is character information obtained by OCR, and the output is the classification result of the character information. This analysis includes using natural language processing techniques to extract meaning from the string and associate it with the learning field.

[0552] Step 4:

[0553] The server updates the learner's characteristics based on the analysis results. The input is the analyzed character information and the existing learner characteristics, and the output is the updated learner characteristics information. It refers to the historical database and performs specific calculations that reflect the learner's existing knowledge level and areas of weakness.

[0554] Step 5:

[0555] The server generates similar information and explanations using a generative AI model based on the learner's characteristics. The input is the updated learner's characteristics, and the output is an individually optimized learning plan and explanatory data. The prompt "Please provide similar problems and solutions for problem XX" is used to process the data in the generative model.

[0556] Step 6:

[0557] The server sends the generated learning content to the user's mobile device. The input is the generated learning plan and explanatory data, and the output is the learning content displayed on the user's mobile device. The device receives this data and operates on an interface that allows the user to review the material immediately.

[0558] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0559] This invention is a system that incorporates an emotion engine to support the learning process when a user engages in learning activities. The process begins with the user taking a picture of a problem they answered incorrectly. The acquired problem image is sent to a server, which uses OCR technology to extract text information from the image. The server then analyzes the text information and classifies the problem.

[0560] The server updates the user's learning profile based on the analysis results. Based on this updated profile, the server generates similar problems, explanations, and even lecture videos. In addition, an emotion engine built into the user's device uses the camera and microphone to capture changes in the user's facial expressions and voice, and analyzes the user's emotional state.

[0561] As a concrete example, when a user posts a problem, the server recognizes that it is related to quadratic equations. In this process, if the emotion engine determines that the user is experiencing stress or lacking understanding while working on the problem, the server prepares an explanatory video tailored to the user's feelings. For example, if the user is experiencing high stress, a concise and easy-to-understand step-by-step explanation is provided.

[0562] The device displays the generated learning content along with a learning approach tailored to the user's emotions. The user uses this information to review effectively, and the server further optimizes the learning strategy based on feedback from the emotion engine.

[0563] By taking user emotions into account, this system can not only provide knowledge but also enhance the user's emotional learning experience. As a result, it can provide more personalized support and improve learning efficiency.

[0564] The following describes the processing flow.

[0565] Step 1:

[0566] The user takes a picture or screenshot of the incorrect answer using their device. This action prepares the problem that was assigned as a learning task to be recorded in digital format.

[0567] Step 2:

[0568] The device uploads the problem image it captures to the server via a dedicated application. Along with the image, metadata such as the user's ID and related problem information is also sent to the server.

[0569] Step 3:

[0570] The server receives image data, saves it to storage, and uses OCR technology to extract text information from the image. This allows the problem statement to be treated as digital text.

[0571] Step 4:

[0572] The server analyzes the character information obtained by OCR to identify the type and content of the problem. Natural language processing technology is used for the analysis, and the problem is classified.

[0573] Step 5:

[0574] The server updates the user's learning profile using the analysis results. This enhances the profile information, ensuring it accurately reflects the user's learning history and areas of difficulty.

[0575] Step 6:

[0576] The device's built-in emotion engine analyzes the user's facial expressions and voice data to evaluate their emotional state. Based on this evaluation, it measures stress levels and interest levels in real time.

[0577] Step 7:

[0578] The server generates similar problems, related explanations, and lecture videos based on the user's profile and sentiment rating. The generated content is optimized to take the user's emotional state into consideration.

[0579] Step 8:

[0580] The server sends generated content to the terminal, which then displays it to the user. The user can then use the provided content to continue their learning.

[0581] Step 9:

[0582] The user deepens their understanding by re-solving the problem and watching the explanation. The device continues to collect sentiment data and sends feedback to the server that is appropriate for the user's learning progress. The server uses this information to further improve the profile.

[0583] (Example 2)

[0584] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0585] Traditional learning support systems focused on providing content to improve users' problem-solving abilities, but they had the challenge of not being able to flexibly respond to changes in users' emotions and understanding. Furthermore, they lacked sufficient methods to provide a learning experience optimized for individual users. As a result, learning efficiency varied greatly from person to person, and there was a problem in that they could not always provide effective learning.

[0586] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0587] In this invention, the server includes means for receiving a problem image taken or acquired by the user, means for extracting textual information from the received problem image, and means for analyzing the extracted textual information and classifying the learning target. This enables the analysis of the problem image and classification of the learning target, the provision of learning content suitable for the user, and further, the provision of an individualized learning experience based on the user's emotional information.

[0588] "User" refers to an individual or group that uses the system to engage in learning activities.

[0589] "Means for receiving problem images" refers to a system for uploading problem images, which are taken or acquired by the user, to a server as digital data.

[0590] "Means for extracting character information" refers to technologies for identifying characters from a received image and extracting them as text data, particularly optical character recognition technology.

[0591] "Methods for analyzing textual information and classifying learning targets" refers to the process of analyzing extracted text and determining and classifying the categories and themes of learning targets based on their content.

[0592] "Means for updating learner profiles" refers to a function within the system that updates information about the user's learning history and characteristics based on analysis results, etc.

[0593] "Means for generating similar problems and explanations" refers to a system that constructs new, similar problems and explanations based on the user's profile information.

[0594] "Means of transmitting to the user's device" refers to the function of transferring learning content and explanations generated on the server to the user's terminal and making them available for display or use.

[0595] "Methods for analyzing emotions" refers to technologies that collect the user's facial expressions and voice data, and then analyze that data to understand the user's emotional state.

[0596] "Means of providing personalized learning content" refers to a function that suggests learning content and support optimized for each user based on the results of an emotional analysis of the user.

[0597] This invention is a system designed to provide more effective support to users when they engage in learning activities. This system primarily consists of three elements: a server, a terminal, and the user.

[0598] The server first uses optical character recognition (OCR) technology to extract text information from the problem image received from the user's terminal. This can be done using a general-purpose server or a cloud-based platform with high data processing capabilities. The extracted text information is analyzed using natural language processing technology and classified into similar problem categories. Furthermore, a generative AI model is used to generate similar problems, explanations, and lecture videos tailored to the user's learning profile. In this process, specific instructions, such as "Generate an explanatory video to deepen the user's understanding," are input to the model as prompts.

[0599] The user's device is equipped with a camera and microphone. This device runs software that analyzes the user's emotions from their facial expressions and voice. The emotion engine analyzes the user's emotions in real time based on the collected data and sends the results to the server. Based on this information, the server provides the user with personalized learning content and explanations.

[0600] As a concrete example, suppose a server analyzes a received image of a problem using OCR and identifies it as a problem involving a quadratic equation. Simultaneously, if sentiment analysis reveals that the user is experiencing high stress related to the problem, the server will present a step-by-step video explanation or a simple, intuitive explanation. In this way, the user can receive learning support that reflects their own emotions.

[0601] This allows the system to suggest the optimal learning approach to the user while reducing their emotional burden. Furthermore, by using generative AI models, it can provide a personalized learning experience for each user.

[0602] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0603] Step 1:

[0604] The terminal uses its camera to photograph the problem the user answered incorrectly. At this time, it obtains image data of the photographed problem as input. The terminal sends this image data to the server. As output, the image data is transferred to the server, and it is ready for character information extraction processing on the server.

[0605] Step 2:

[0606] The server uses optical character recognition (OCR) technology on the received image data. It receives the image data in question, sent from the terminal, as input. This process extracts character information from the image data, obtaining text data as output. The OCR processor identifies character patterns and stores them as digitized text within the server.

[0607] Step 3:

[0608] The server performs natural language processing (NLP) on the extracted text data. The input is text data obtained by optical character recognition (OCR). In this process, the text is analyzed and classified as learning target based on its content. The output is the classification result, and information that should be updated in the user's learning profile is identified. In this process, specific keywords and contexts are analyzed to determine which category the problem belongs to.

[0609] Step 4:

[0610] The server updates the user's learning profile based on the analyzed information. The input is the classification results obtained in step 3. This process reflects the areas where the user made mistakes and their learning tendencies in the profile. As output, the updated learning profile is saved to the database. This process involves calculations to integrate the new data into the existing profile.

[0611] Step 5:

[0612] The server uses a generative AI model to generate similar problems, explanations, and lecture videos based on the updated learning profile. The input is the updated learning profile. The prompt, "Generate an explanatory video to deepen the user's understanding," is input to the model, and the generated content is obtained as output. In this generation process, the AI ​​creates the most suitable learning materials for the user based on its accumulated knowledge.

[0613] Step 6:

[0614] The device receives generated learning content and displays it to the user. The learning content sent from the server serves as input. The device displays this data in a user-friendly format, making it accessible to the user. As output, the user receives visualized learning materials. The device effectively presents content through its user interface (UI).

[0615] Step 7:

[0616] The device analyzes the user's facial expressions and voice data using an emotion engine. It takes real-time data collected through the camera and microphone as input. This process determines the user's emotional state (e.g., stress level and concentration level) and sends the analysis results to the server as output. The engine uses changes in voice tone and facial expressions as indicators during the analysis.

[0617] Step 8:

[0618] The server optimizes the learning strategy based on the sentiment analysis results. It receives the sentiment analysis results obtained from the device as input. These results are reflected in the learning content, providing personalized feedback and adjustments. As output, optimized learning properties corresponding to the user's emotions are managed. The server then adjusts the next learning content based on the sentiment data.

[0619] (Application Example 2)

[0620] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0621] Traditional learning systems provided uniform learning content without adequately considering user emotions or performance data, making it difficult to provide effective learning support tailored to each learner's level of understanding and emotional state. In particular, there was difficulty in reducing the stress and confusion learners experienced during learning, and in improving their motivation.

[0622] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0623] In this invention, the server includes a device for receiving problem images captured or acquired by the user, a device for extracting textual information from the received problem images, and a device for analyzing the extracted textual information and classifying the learning target. This makes it possible to provide optimized learning content based on the individual emotional state and learning profile of the learner.

[0624] "User" refers to an individual who uses the system to engage in learning activities.

[0625] A "device for receiving problem images" is a device that has the function of sending problem image data, which the user has taken or acquired, to a server.

[0626] A "device for extracting text information" refers to software or hardware that has the function of analyzing text from a received image and extracting it as digital information.

[0627] A "device for classifying learning materials" is a system that analyzes extracted textual information and uses that information to determine the category of learning content.

[0628] A "device for updating learner characteristics information" is a system that records, manages, and updates learner profiles and learning progress based on the results of analysis.

[0629] A "device for generating similar problems and explanations" is a system that, based on learner characteristic information, creates problems and explanations tailored to them, providing learning content that enhances learning efficiency.

[0630] A "device for transmitting to an information terminal" is a system that has the function of transmitting generated learning content to the user's digital device using communication means.

[0631] A "device for analyzing emotional states" is a system that uses information such as the user's facial expressions and voice to understand their emotions and utilize that information to improve the effectiveness and progress of their learning.

[0632] A "device for adjusting and optimizing learning content" is a system that has the function of customizing learning methods and content to provide the most suitable learning methods and content to learners based on analyzed emotional state and characteristic information.

[0633] This invention provides an educational support system that enables users to effectively carry out the learning process. The system receives problem images taken by the user and uses optical character recognition (OCR) technology to extract textual information from the images. The extracted information is analyzed using natural language processing technology and categorized as learning material.

[0634] The server updates the user's learning profile based on the information obtained. A process then generates similar problems and related explanations from the updated profile. The generated learning content is sent to the user's information terminal, allowing the user to continue their learning through it.

[0635] The device incorporates an emotion engine that uses a camera and microphone to analyze the user's facial expressions and voice to determine their emotional state. The analyzed emotional state is used to adjust and optimize learning content, and if the user experiences stress or confusion, it provides appropriate learning resources to help improve learning efficiency.

[0636] For example, if the emotion engine determines that a user is stuck on a particular math problem and is feeling stressed, the server will provide relaxing music to the user's device and send a step-by-step explanatory video tailored to the user's understanding. In this way, the user can calm down and more easily tackle the problem.

[0637] An example of a prompt message used to generate specific explanations using a generative AI model is: "If the user is feeling stressed about the problem they are trying to solve, please create a concise and easy-to-understand explanatory video. Also, please recommend music that will help the user relax." Based on this prompt, the generative AI will create content tailored to the user.

[0638] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0639] Step 1:

[0640] The device receives the problem image taken by the user. The input is the problem image, and the output is the image data stored in the device. This image data is then prepared for the next processing step.

[0641] Step 2:

[0642] The terminal uses OCR technology to extract text information from the received problem image. The input is image data, and the output is text data. Through OCR processing, the content of the problem is interpreted as text and sent to the next analysis step.

[0643] Step 3:

[0644] The server analyzes the extracted text information using natural language processing techniques to classify it into categories for learning. The input is text data, and the output is classification information for the problem. This process determines which learning domain the problem belongs to, and this information is used to update the learning profile.

[0645] Step 4:

[0646] The server updates the user's learning profile based on the problem classification information. The input is the problem classification information, and the output is the updated learning profile data. This process generates a profile that reflects the user's progress and strengths and weaknesses.

[0647] Step 5:

[0648] The server references the updated learning profile and generates similar problems and explanations. The input is the learning profile data, and the output is the generated learning content (problems and explanations). This process provides the user with the most suitable learning resources.

[0649] Step 6:

[0650] The device uses a camera and microphone to capture the user's facial expressions and voice, and an emotion engine analyzes their emotional state. The input is the captured video and audio data, and the output is emotional state information. This information is used to adjust the learning content.

[0651] Step 7:

[0652] The server adjusts and optimizes the learning content it provides based on emotional state information. The input is emotional state information and the generated learning content, and the output is the adjusted and optimized learning content. In this step, content that takes into account the user's stress and lack of understanding is completed.

[0653] Step 8:

[0654] The device displays the final learning content to the user, supporting their learning activities. The input is the adjusted learning content, and the output is the content displayed to the user. As a result, learners can effectively progress through their learning using optimized content.

[0655] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0656] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0657] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0658] [Fourth Embodiment]

[0659] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0660] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0661] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0662] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0663] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0664] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0665] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0666] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0667] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0668] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0669] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0670] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0671] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0672] This invention is a system that supports learners in effectively progressing with their studies, primarily involving data exchange between a server, a terminal, and a user. In implementation, the user first takes a picture of a problem they answered incorrectly with their terminal. This problem image is then sent to the server via a dedicated application.

[0673] The server uses OCR technology to extract text information from the received image. By analyzing this text information, the server identifies the type of problem and the scope of learning, and updates the learner's profile accordingly. Based on the learner's past learning history and current areas of difficulty, the server generates similar learning problems and prepares explanations and lecture videos as needed.

[0674] As a concrete example, if a user makes a mistake in solving a quadratic equation on a math exam, the user takes a picture of the problem. The server receives this and identifies that the problem is related to a quadratic equation. The server then analyzes how the user has failed with similar problems based on past data. As a result, it sends similar quadratic equation problems and detailed explanations of how to solve them to the user's device.

[0675] The device displays the received data to the user, who can then review the provided learning content. Continuous data feedback to the server allows the user's profile to be updated more accurately, improving the precision of subsequent learning support. This system will enable learners to efficiently overcome their weak areas.

[0676] The following describes the processing flow.

[0677] Step 1:

[0678] The user takes a picture or screenshot of the incorrect answer using their own device. This prepares the question content to be saved in digital format.

[0679] Step 2:

[0680] The device uploads the acquired problem images to the server via a dedicated application. Metadata such as the user ID and problem category is attached to the upload.

[0681] Step 3:

[0682] The server temporarily stores the received image data and uses OCR technology to extract text information from the image. This makes the question text available for processing within the system as digital text.

[0683] Step 4:

[0684] The server analyzes the extracted text data to identify the theme and content of the problem. Natural language processing techniques are used in this analysis to classify the problem.

[0685] Step 5:

[0686] The server references the user's existing profile data and updates the user's learning progress and areas of difficulty based on the analysis results. This update is then used to provide personalized learning support.

[0687] Step 6:

[0688] The server generates similar problems and related explanations based on the user's profile information, and prepares lecture videos as needed. This information is generated to enhance the user's learning.

[0689] Step 7:

[0690] The server sends similar problems and explanatory content it has generated to the device. The device displays the received data, allowing the user to use it for review.

[0691] Step 8:

[0692] Users solve similar problems provided on their devices, deepening their understanding by reading explanations or watching videos. The server continuously collects user responses and behavioral data to improve the accuracy of the profiles.

[0693] (Example 1)

[0694] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0695] Conventional learning support systems have the challenge of not being able to accurately identify individual learners' weak areas and quickly provide them with the most suitable learning problems and explanations. Furthermore, the lack of a mechanism for efficiently updating learning history made it difficult to respond flexibly to learners' progress.

[0696] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0697] In this invention, the server includes means for receiving problem images captured or acquired by the user, means for extracting character information using optical character recognition technology, and means for generating similar problems and related information using a generative AI model. This enables the provision of problems tailored to the learner's weak areas and efficient updating of the learning history.

[0698] A "user" refers to an individual who uses a learning support system to advance their studies.

[0699] A "problem image" refers to image data captured by a camera or obtained from a user who answered a problem incorrectly during their learning process.

[0700] "Optical character recognition technology" is a technology that converts characters in an image into digital text.

[0701] "Character information" refers to the character data within the image in question, extracted using optical character recognition technology.

[0702] A "generative AI model" is a model that uses artificial intelligence technology to analyze a learner's profile and generate optimal learning problems and explanations.

[0703] "Related information" refers to supplementary materials such as explanations and lecture content provided by generative AI models to deepen learners' understanding.

[0704] An "information processing device" is a device used by a user to receive and display content from a learning support system.

[0705] "History information" refers to information that the system retains as a user profile, such as the user's past learning activities and performance data.

[0706] "Feedback" refers to information about one's learning progress and performance that a user sends to the system.

[0707] This invention is a learning support system that allows learners to individually identify areas where they lack understanding and to efficiently advance their learning. The following describes a specific embodiment of this system.

[0708] First, the user takes a picture of the question they answered incorrectly with their device. A dedicated application is installed on the device, and the image of the question is sent to the server via this application. The device compresses the captured image to improve communication efficiency.

[0709] Upon receiving a problem image, the server extracts text information from the image using optical character recognition (OCR) technology. Specifically, software such as Tesseract OCR can be used. Next, the server uses the extracted text information to execute an algorithm to identify the type of problem and the learning area.

[0710] The server updates the learner's history information based on the analysis results. This history information includes past learning data and correct / incorrect information in specific areas. Based on the updated history information, a generative AI model is used to generate similar problems and related information. For example, the OpenAI language model is used in the generative AI model, and optimal learning content is generated by inputting prompt sentences.

[0711] As a concrete example, consider a case where a user makes a mistake on a quadratic equation problem in a math exam. The user takes a picture of the problem and sends it to the server. The server extracts textual information from the image and identifies that the problem is related to a quadratic equation. The server then refers to the learner's history information to generate similar quadratic equation problems and explanations, and sends them to the device.

[0712] Examples of prompt messages include the following:

[0713] "Please create the following learning problem involving quadratic equations. Past data shows that users particularly struggle with the 'completing the square' method. Please provide an explanation and problem proposal based on this."

[0714] The device displays the received learning content to the user, allowing the user to efficiently review the material based on this information. The user enters their answers into the device, and this feedback is sent back to the server and used to update the history information. This cycle further improves the accuracy of learning support for the user.

[0715] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0716] Step 1:

[0717] The user takes a picture of the incorrect answer with their device. The input is image data acquired by the camera. The device then imports this image into a dedicated application. Specifically, the image is properly captured and compressed before being prepared for transmission.

[0718] Step 2:

[0719] The terminal sends the prepared problem image data to the server. The input here is the image data captured and compressed in step 1, and the output is the transfer of the image data to the server. The terminal establishes a network connection and performs real-time error checking to ensure that the data is transmitted correctly.

[0720] Step 3:

[0721] The server processes the received image and extracts text information using optical character recognition (OCR) technology. The input is the received image data, and the output is the extracted text information. Specifically, the server uses tools such as Tesseract OCR to efficiently analyze the text portion of the image and convert it into digital text.

[0722] Step 4:

[0723] The server analyzes the extracted text information to identify the type of problem and the learning area. The input is the text information extracted in step 3, and the output is the identified type of problem and learning area. Specifically, the text information is passed through a natural language processing algorithm to structure the content of the problem and compare it with the learner's history information.

[0724] Step 5:

[0725] The server updates the learner's history information based on the identified problem information. The input is the identified problem type and learning area, and the output is the updated history information. Specifically, it refers to the past learning database and adds information to analyze which areas the user makes the most mistakes in.

[0726] Step 6:

[0727] The server uses a generative AI model to generate similar problems and relevant information suitable for the learner. The input is updated historical information and identified problem information, and the output is the generated learning content. Specifically, the server inputs prompt sentences into the AI ​​model and generates content that focuses on what the user particularly needs to understand.

[0728] Step 7:

[0729] The server sends the generated learning content to the user's device. The input is the content generated in step 6, and the output is the delivery of the content to the user's device. Specifically, an efficient data transfer protocol is used to ensure that the content is delivered accurately while minimizing latency.

[0730] Step 8:

[0731] The terminal displays the received learning content to the user. The input is content data received from the server, and the output is a display state that the user can interact with. Specifically, the terminal application formats the content appropriately so that the user can start learning immediately.

[0732] Step 9:

[0733] The user solves problems based on the provided content and inputs their answers into the device. The input is the user's answer data, and the output is the confirmation of the answer data within the device. Specifically, the system is designed to allow users to quickly send data through an interface that makes it easy to input answers.

[0734] Step 10:

[0735] The terminal sends the user's answer results to the server, where they are reflected as feedback. The input is the user's answer data, and the output is the transfer of data to the server. Specifically, the transmitted data is checked to ensure its accuracy and to be used for updating the history information on the server side.

[0736] Step 11:

[0737] The server updates the learner's history information based on the feedback received, and this data is used to generate the next problem. The input is the user's answer data, and the output is the latest history information. Specifically, the server adds the updated information to the database and uses the newly acquired knowledge in the next learning cycle.

[0738] (Application Example 1)

[0739] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0740] In education, it is crucial to support learners in understanding their mistakes and effectively overcoming them. However, existing means of providing individually optimized learning support are limited, and there are challenges in accurately understanding each learner's characteristics and areas of difficulty and adjusting learning content accordingly. Furthermore, current systems cannot efficiently utilize users' learning history, resulting in insufficient learning support.

[0741] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0742] In this invention, the server includes means for receiving information units captured or acquired by the user, means for extracting character information from the received information units, and means for analyzing the extracted character information and classifying the learning target. This makes it possible to provide individually optimized learning support and to provide effective learning content based on the learner's characteristics and past learning history.

[0743] "Users" refers to learners and educators who use this system to receive learning support.

[0744] An "information unit" refers to data, including questions and learning materials, that learners photograph or acquire and send to the system.

[0745] "Textual information" refers to text data extracted from information units using OCR technology.

[0746] "Classification of learning targets" refers to the process of analyzing extracted textual information and classifying its content into specific learning categories or themes.

[0747] "Learner characteristics" refers to profile information including the learner's past learning history, current level of understanding, and areas of difficulty.

[0748] "Similar information" refers to learning questions and content generated based on learners' areas of difficulty and incorrect answers.

[0749] "Explanation" refers to explanatory materials or lecture materials provided for similar information.

[0750] "Portable devices" refer to electronic terminals that users can carry around, such as smartphones and tablets.

[0751] A "learning plan" refers to a learning schedule and content optimized for each individual learner.

[0752] The system implementing this invention provides learning support to help learners effectively understand and overcome their mistakes. The entire system consists of the user's mobile device and a server, and utilizes a variety of technologies.

[0753] First, the user takes a picture of a problem they find difficult or answered incorrectly using their mobile device. The captured information is sent to a server via a dedicated application installed on the mobile device. The server then extracts text information from the received information using OCR technology (for example, AWS Rekognition or Google Cloud Vision).

[0754] The server analyzes the extracted text information and classifies the learning targets. Natural language processing techniques are also utilized to interpret the meaning and structure of the text information with greater precision. Based on the analyzed information, the server updates the learner's characteristics and creates a learning plan optimized for each individual learner.

[0755] This system generates similar information and explanations based on the learner's characteristics. The generated learning content is sent from the server to the user's mobile device, where the user can review it and continue learning. This allows for rapid correction of misunderstandings and errors arising from individual learning, resulting in efficient learning.

[0756] As a concrete example of its use, if a high school student takes a picture of a math problem they got wrong on a test, the photo is sent from their mobile device to a server, where OCR technology converts the problem content into text. The server analyzes the content and extracts particularly difficult concepts and problem patterns by comparing them with the learner's past records. Then, using AI, it generates similar problems and detailed explanations of their solutions, which are delivered to the learner's mobile device. A possible prompt in this case would be, "I made a mistake in solving the quadratic equation x² + 3x - 4 = 0. Please provide similar problems and detailed explanations."

[0757] This system allows learners to efficiently deepen their understanding and improve areas where they struggle.

[0758] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0759] Step 1:

[0760] The user takes a picture of the learning content's information unit with a mobile device. The input is image data captured by the camera, and this data is acquired by the device. The image is then sent to the server via a dedicated application.

[0761] Step 2:

[0762] The server extracts text information from received image data using OCR technology. The input is image data sent by the user, and the output is text data present within the image. The specific operation involves using the Google Cloud Vision API to extract the text information.

[0763] Step 3:

[0764] The server analyzes the extracted character information to classify the target of learning. The input is character information obtained by OCR, and the output is the classification result of the character information. This analysis includes using natural language processing techniques to extract meaning from the string and associate it with the learning field.

[0765] Step 4:

[0766] The server updates the learner's characteristics based on the analysis results. The input is the analyzed character information and the existing learner characteristics, and the output is the updated learner characteristics information. It refers to the historical database and performs specific calculations that reflect the learner's existing knowledge level and areas of weakness.

[0767] Step 5:

[0768] The server generates similar information and explanations using a generative AI model based on the learner's characteristics. The input is the updated learner's characteristics, and the output is an individually optimized learning plan and explanatory data. The prompt "Please provide similar problems and solutions for problem XX" is used to process the data in the generative model.

[0769] Step 6:

[0770] The server sends the generated learning content to the user's mobile device. The input is the generated learning plan and explanatory data, and the output is the learning content displayed on the user's mobile device. The device receives this data and operates on an interface that allows the user to review the material immediately.

[0771] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0772] This invention is a system that incorporates an emotion engine to support the learning process when a user engages in learning activities. The process begins with the user taking a picture of a problem they answered incorrectly. The acquired problem image is sent to a server, which uses OCR technology to extract text information from the image. The server then analyzes the text information and classifies the problem.

[0773] The server updates the user's learning profile based on the analysis results. Based on this updated profile, the server generates similar problems, explanations, and even lecture videos. In addition, an emotion engine built into the user's device uses the camera and microphone to capture changes in the user's facial expressions and voice, and analyzes the user's emotional state.

[0774] As a concrete example, when a user posts a problem, the server recognizes that it is related to quadratic equations. In this process, if the emotion engine determines that the user is experiencing stress or lacking understanding while working on the problem, the server prepares an explanatory video tailored to the user's feelings. For example, if the user is experiencing high stress, a concise and easy-to-understand step-by-step explanation is provided.

[0775] The device displays the generated learning content along with a learning approach tailored to the user's emotions. The user uses this information to review effectively, and the server further optimizes the learning strategy based on feedback from the emotion engine.

[0776] By taking user emotions into account, this system can not only provide knowledge but also enhance the user's emotional learning experience. As a result, it can provide more personalized support and improve learning efficiency.

[0777] The following describes the processing flow.

[0778] Step 1:

[0779] The user takes a picture or screenshot of the incorrect answer using their device. This action prepares the problem that was assigned as a learning task to be recorded in digital format.

[0780] Step 2:

[0781] The device uploads the problem image it captures to the server via a dedicated application. Along with the image, metadata such as the user's ID and related problem information is also sent to the server.

[0782] Step 3:

[0783] The server receives image data, saves it to storage, and uses OCR technology to extract text information from the image. This allows the problem statement to be treated as digital text.

[0784] Step 4:

[0785] The server analyzes the character information obtained by OCR to identify the type and content of the problem. Natural language processing technology is used for the analysis, and the problem is classified.

[0786] Step 5:

[0787] The server updates the user's learning profile using the analysis results. This enhances the profile information, ensuring it accurately reflects the user's learning history and areas of difficulty.

[0788] Step 6:

[0789] The device's built-in emotion engine analyzes the user's facial expressions and voice data to evaluate their emotional state. Based on this evaluation, it measures stress levels and interest levels in real time.

[0790] Step 7:

[0791] The server generates similar problems, related explanations, and lecture videos based on the user's profile and sentiment rating. The generated content is optimized to take the user's emotional state into consideration.

[0792] Step 8:

[0793] The server sends generated content to the terminal, which then displays it to the user. The user can then use the provided content to continue their learning.

[0794] Step 9:

[0795] The user deepens their understanding by re-solving the problem and watching the explanation. The device continues to collect sentiment data and sends feedback to the server that is appropriate for the user's learning progress. The server uses this information to further improve the profile.

[0796] (Example 2)

[0797] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0798] Traditional learning support systems focused on providing content to improve users' problem-solving abilities, but they had the challenge of not being able to flexibly respond to changes in users' emotions and understanding. Furthermore, they lacked sufficient methods to provide a learning experience optimized for individual users. As a result, learning efficiency varied greatly from person to person, and there was a problem in that they could not always provide effective learning.

[0799] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0800] In this invention, the server includes means for receiving a problem image taken or acquired by the user, means for extracting textual information from the received problem image, and means for analyzing the extracted textual information and classifying the learning target. This enables the analysis of the problem image and classification of the learning target, the provision of learning content suitable for the user, and further, the provision of an individualized learning experience based on the user's emotional information.

[0801] "User" refers to an individual or group that uses the system to engage in learning activities.

[0802] "Means for receiving problem images" refers to a system for uploading problem images, which are taken or acquired by the user, to a server as digital data.

[0803] "Means for extracting character information" refers to technologies for identifying characters from a received image and extracting them as text data, particularly optical character recognition technology.

[0804] "Methods for analyzing textual information and classifying learning targets" refers to the process of analyzing extracted text and determining and classifying the categories and themes of learning targets based on their content.

[0805] "Means for updating learner profiles" refers to a function within the system that updates information about the user's learning history and characteristics based on analysis results, etc.

[0806] "Means for generating similar problems and explanations" refers to a system that constructs new, similar problems and explanations based on the user's profile information.

[0807] "Means of transmitting to the user's device" refers to the function of transferring learning content and explanations generated on the server to the user's terminal and making them available for display or use.

[0808] "Methods for analyzing emotions" refers to technologies that collect the user's facial expressions and voice data, and then analyze that data to understand the user's emotional state.

[0809] "Means of providing personalized learning content" refers to a function that suggests learning content and support optimized for each user based on the results of an emotional analysis of the user.

[0810] This invention is a system designed to provide more effective support to users when they engage in learning activities. This system primarily consists of three elements: a server, a terminal, and the user.

[0811] The server first uses optical character recognition (OCR) technology to extract text information from the problem image received from the user's terminal. This can be done using a general-purpose server or a cloud-based platform with high data processing capabilities. The extracted text information is analyzed using natural language processing technology and classified into similar problem categories. Furthermore, a generative AI model is used to generate similar problems, explanations, and lecture videos tailored to the user's learning profile. In this process, specific instructions, such as "Generate an explanatory video to deepen the user's understanding," are input to the model as prompts.

[0812] The user's device is equipped with a camera and microphone. This device runs software that analyzes the user's emotions from their facial expressions and voice. The emotion engine analyzes the user's emotions in real time based on the collected data and sends the results to the server. Based on this information, the server provides the user with personalized learning content and explanations.

[0813] As a concrete example, suppose a server analyzes a received image of a problem using OCR and identifies it as a problem involving a quadratic equation. Simultaneously, if sentiment analysis reveals that the user is experiencing high stress related to the problem, the server will present a step-by-step video explanation or a simple, intuitive explanation. In this way, the user can receive learning support that reflects their own emotions.

[0814] This allows the system to suggest the optimal learning approach to the user while reducing their emotional burden. Furthermore, by using generative AI models, it can provide a personalized learning experience for each user.

[0815] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0816] Step 1:

[0817] The terminal uses its camera to photograph the problem the user answered incorrectly. At this time, it obtains image data of the photographed problem as input. The terminal sends this image data to the server. As output, the image data is transferred to the server, and it is ready for character information extraction processing on the server.

[0818] Step 2:

[0819] The server uses optical character recognition (OCR) technology on the received image data. It receives the image data in question, sent from the terminal, as input. This process extracts character information from the image data, obtaining text data as output. The OCR processor identifies character patterns and stores them as digitized text within the server.

[0820] Step 3:

[0821] The server performs natural language processing (NLP) on the extracted text data. The input is text data obtained by optical character recognition (OCR). In this process, the text is analyzed and classified as learning target based on its content. The output is the classification result, and information that should be updated in the user's learning profile is identified. In this process, specific keywords and contexts are analyzed to determine which category the problem belongs to.

[0822] Step 4:

[0823] The server updates the user's learning profile based on the analyzed information. The input is the classification results obtained in step 3. This process reflects the areas where the user made mistakes and their learning tendencies in the profile. As output, the updated learning profile is saved to the database. This process involves calculations to integrate the new data into the existing profile.

[0824] Step 5:

[0825] The server uses a generative AI model to generate similar problems, explanations, and lecture videos based on the updated learning profile. The input is the updated learning profile. The prompt, "Generate an explanatory video to deepen the user's understanding," is input to the model, and the generated content is obtained as output. In this generation process, the AI ​​creates the most suitable learning materials for the user based on its accumulated knowledge.

[0826] Step 6:

[0827] The device receives generated learning content and displays it to the user. The learning content sent from the server serves as input. The device displays this data in a user-friendly format, making it accessible to the user. As output, the user receives visualized learning materials. The device effectively presents content through its user interface (UI).

[0828] Step 7:

[0829] The device analyzes the user's facial expressions and voice data using an emotion engine. It takes real-time data collected through the camera and microphone as input. This process determines the user's emotional state (e.g., stress level and concentration level) and sends the analysis results to the server as output. The engine uses changes in voice tone and facial expressions as indicators during the analysis.

[0830] Step 8:

[0831] The server optimizes the learning strategy based on the sentiment analysis results. It receives the sentiment analysis results obtained from the device as input. These results are reflected in the learning content, providing personalized feedback and adjustments. As output, optimized learning properties corresponding to the user's emotions are managed. The server then adjusts the next learning content based on the sentiment data.

[0832] (Application Example 2)

[0833] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0834] Traditional learning systems provided uniform learning content without adequately considering user emotions or performance data, making it difficult to provide effective learning support tailored to each learner's level of understanding and emotional state. In particular, there was difficulty in reducing the stress and confusion learners experienced during learning, and in improving their motivation.

[0835] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0836] In this invention, the server includes a device for receiving problem images captured or acquired by the user, a device for extracting textual information from the received problem images, and a device for analyzing the extracted textual information and classifying the learning target. This makes it possible to provide optimized learning content based on the individual emotional state and learning profile of the learner.

[0837] "User" refers to an individual who uses the system to engage in learning activities.

[0838] A "device for receiving problem images" is a device that has the function of sending problem image data, which the user has taken or acquired, to a server.

[0839] A "device for extracting text information" refers to software or hardware that has the function of analyzing text from a received image and extracting it as digital information.

[0840] A "device for classifying learning materials" is a system that analyzes extracted textual information and uses that information to determine the category of learning content.

[0841] A "device for updating learner characteristics information" is a system that records, manages, and updates learner profiles and learning progress based on the results of analysis.

[0842] A "device for generating similar problems and explanations" is a system that, based on learner characteristic information, creates problems and explanations tailored to them, providing learning content that enhances learning efficiency.

[0843] A "device for transmitting to an information terminal" is a system that has the function of transmitting generated learning content to the user's digital device using communication means.

[0844] A "device for analyzing emotional states" is a system that uses information such as the user's facial expressions and voice to understand their emotions and utilize that information to improve the effectiveness and progress of their learning.

[0845] A "device for adjusting and optimizing learning content" is a system that has the function of customizing learning methods and content to provide the most suitable learning methods and content to learners based on analyzed emotional state and characteristic information.

[0846] This invention provides an educational support system that enables users to effectively carry out the learning process. The system receives problem images taken by the user and uses optical character recognition (OCR) technology to extract textual information from the images. The extracted information is analyzed using natural language processing technology and categorized as learning material.

[0847] The server updates the user's learning profile based on the information obtained. A process then generates similar problems and related explanations from the updated profile. The generated learning content is sent to the user's information terminal, allowing the user to continue their learning through it.

[0848] The device incorporates an emotion engine that uses a camera and microphone to analyze the user's facial expressions and voice to determine their emotional state. The analyzed emotional state is used to adjust and optimize learning content, and if the user experiences stress or confusion, it provides appropriate learning resources to help improve learning efficiency.

[0849] For example, if the emotion engine determines that a user is stuck on a particular math problem and is feeling stressed, the server will provide relaxing music to the user's device and send a step-by-step explanatory video tailored to the user's understanding. In this way, the user can calm down and more easily tackle the problem.

[0850] An example of a prompt message used to generate specific explanations using a generative AI model is: "If the user is feeling stressed about the problem they are trying to solve, please create a concise and easy-to-understand explanatory video. Also, please recommend music that will help the user relax." Based on this prompt, the generative AI will create content tailored to the user.

[0851] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0852] Step 1:

[0853] The device receives the problem image taken by the user. The input is the problem image, and the output is the image data stored in the device. This image data is then prepared for the next processing step.

[0854] Step 2:

[0855] The terminal uses OCR technology to extract text information from the received problem image. The input is image data, and the output is text data. Through OCR processing, the content of the problem is interpreted as text and sent to the next analysis step.

[0856] Step 3:

[0857] The server analyzes the extracted text information using natural language processing techniques to classify it into categories for learning. The input is text data, and the output is classification information for the problem. This process determines which learning domain the problem belongs to, and this information is used to update the learning profile.

[0858] Step 4:

[0859] The server updates the user's learning profile based on the problem classification information. The input is the problem classification information, and the output is the updated learning profile data. This process generates a profile that reflects the user's progress and strengths and weaknesses.

[0860] Step 5:

[0861] The server references the updated learning profile and generates similar problems and explanations. The input is the learning profile data, and the output is the generated learning content (problems and explanations). This process provides the user with the most suitable learning resources.

[0862] Step 6:

[0863] The device uses a camera and microphone to capture the user's facial expressions and voice, and an emotion engine analyzes their emotional state. The input is the captured video and audio data, and the output is emotional state information. This information is used to adjust the learning content.

[0864] Step 7:

[0865] The server adjusts and optimizes the learning content it provides based on emotional state information. The input is emotional state information and the generated learning content, and the output is the adjusted and optimized learning content. In this step, content that takes into account the user's stress and lack of understanding is completed.

[0866] Step 8:

[0867] The device displays the final learning content to the user, supporting their learning activities. The input is the adjusted learning content, and the output is the content displayed to the user. As a result, learners can effectively progress through their learning using optimized content.

[0868] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0869] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0870] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0871] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0872] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0873] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0874] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0875] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0876] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0877] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0878] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0879] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0880] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0881] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0882] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0883] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0884] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0885] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0886] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0887] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0888] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0889] The following is further disclosed regarding the embodiments described above.

[0890] (Claim 1)

[0891] A means for receiving the problem image taken or acquired by the user,

[0892] A means for extracting text information from a received problem image,

[0893] A means of analyzing extracted textual information and classifying the learning target,

[0894] A means of updating learner profiles based on analysis results,

[0895] A means of generating similar problems and explanations based on the learner's profile,

[0896] A means for transmitting the generated learning content to the user's device,

[0897] A system that includes this.

[0898] (Claim 2)

[0899] The system according to claim 1, which uses natural language processing technology in the analysis of extracted character information.

[0900] (Claim 3)

[0901] The system according to claim 1, comprising a means for generating a profile that accumulates learning data for each user and identifies areas of weakness for the learner.

[0902] "Example 1"

[0903] (Claim 1)

[0904] A means for receiving the problem image taken or acquired by the user,

[0905] A means for extracting character information from a received problem image using optical character recognition technology,

[0906] A means for analyzing extracted character information and identifying the type and scope of the learning target,

[0907] A means of updating learners' history information based on the analysis results,

[0908] A means of generating similar problems and related information using a generative AI model based on learner history information,

[0909] A means for transmitting the generated learning content to the user's information processing device,

[0910] A means of continuously updating learner history information based on user feedback,

[0911] A system that includes this.

[0912] (Claim 2)

[0913] The system according to claim 1, which uses natural language processing technology in the analysis of extracted character information.

[0914] (Claim 3)

[0915] The system according to claim 1, comprising means for generating historical information that accumulates learning data for each user and identifies areas of weakness for the learner.

[0916] "Application Example 1"

[0917] (Claim 1)

[0918] A means for receiving information units captured or acquired by the user,

[0919] A means for extracting character information from received information units,

[0920] A means of analyzing extracted textual information and classifying the learning target,

[0921] A means of updating learner characteristics based on analysis results,

[0922] A means of generating similar information and explanations based on the characteristics of learners,

[0923] A means of transmitting the generated educational content to the user's mobile device,

[0924] Methods for individually optimizing learning plans,

[0925] A system that includes this.

[0926] (Claim 2)

[0927] The system according to claim 1, which uses natural language processing technology in the analysis of extracted character information.

[0928] (Claim 3)

[0929] The system according to claim 1, comprising means for optimizing information distribution based on the characteristics of educators using natural language processing technology.

[0930] "Example 2 of combining an emotion engine"

[0931] (Claim 1)

[0932] A means for receiving the problem image taken or acquired by the user,

[0933] A means for extracting text information from a received problem image,

[0934] A means of analyzing extracted textual information and classifying the learning target,

[0935] A means of updating learner profiles based on analysis results,

[0936] A means of generating similar problems and explanations based on the learner's profile,

[0937] A means for transmitting the generated learning content to the user's device,

[0938] A means of analyzing emotions from the user's video and audio,

[0939] A means of providing personalized learning content based on analyzed emotional information,

[0940] A system that includes this.

[0941] (Claim 2)

[0942] The system according to claim 1, which uses natural language processing technology in the analysis of extracted character information.

[0943] (Claim 3)

[0944] The system according to claim 1, comprising a means for generating a profile that accumulates learning data for each user and identifies areas of weakness for the learner.

[0945] "Application example 2 when combining with an emotional engine"

[0946] (Claim 1)

[0947] A device that receives the problem image captured or acquired by the user,

[0948] A device for extracting text information from a received problem image,

[0949] A device that analyzes extracted character information and classifies the learning target,

[0950] A device that updates learner characteristic information based on analysis results,

[0951] A device that generates similar problems and explanations based on learner characteristics information,

[0952] A device that transmits the generated learning content to the user's information terminal,

[0953] A device that analyzes the emotional state of the user from their facial expressions and voice,

[0954] A device that adjusts and optimizes learning content based on emotional state,

[0955] A system that includes this.

[0956] (Claim 2)

[0957] The system according to claim 1, which uses natural language processing technology in the analysis of extracted character information.

[0958] (Claim 3)

[0959] The system according to claim 1, comprising a characteristic generation device that accumulates learning data for each user and identifies areas of weakness for the learner. [Explanation of symbols]

[0960] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving the problem image taken or acquired by the user, A means for extracting text information from a received problem image, A means of analyzing extracted textual information and classifying the learning target, A means of updating learner profiles based on analysis results, A means of generating similar problems and explanations based on the learner's profile, A means for transmitting the generated learning content to the user's device, A system that includes this.

2. The system according to claim 1, which uses natural language processing technology in the analysis of extracted character information.

3. The system according to claim 1, comprising a profile generation means for accumulating learning data for each user and identifying areas of weakness for the learner.

Images