system

Abstract

We provide the system. [Solution] A means of receiving problems that learners were unable to solve, A means for analyzing the received problem and identifying its characteristics and category, A means for generating or searching for similar problems and their explanations based on the characteristics and categories of the aforementioned problems, A system including means for providing learners with similar problems that have been generated or retrieved, along with explanations thereof.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional learning support systems, there is a problem that personalized guidance according to the individual weaknesses and learning progress of learners cannot be sufficiently provided. In particular, it is difficult to provide appropriate feedback or similar problems for problems that learners could not solve, making efficient learning difficult. Also, it is difficult to continuously manage individual explanations and learning records for these problems. As a result, it becomes difficult to improve the understanding of learners and maintain their learning motivation.

Means for Solving the Problems

[0005] The present invention solves the above problems by providing a system that includes means for receiving problems that learners were unable to solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, and means for providing the generated or searched similar problems and their explanations to the learners.

[0006] Specifically, by using natural language processing or image recognition technology to identify the characteristics and categories of the aforementioned problems, highly accurate problem analysis is achieved. Furthermore, when providing similar problems and their explanations, the system includes means for recording the learner's answers and optimizing the next learning plan, thereby supporting personalized learning tailored to the learner's level of understanding. This allows learners to efficiently overcome their weak areas and continue learning sustainably.

[0007] A "learner" is a person who seeks to acquire knowledge or skills for educational or personal development purposes.

[0008] A "problem" is a task or question that learners must solve, and requires a specific answer.

[0009] The "means of receiving" refer to the mechanism for incorporating questions entered by learners into the system.

[0010] "Means of analysis" refer to the techniques and processes used to understand the content of a given problem and to identify its characteristics and categories.

[0011] "Characteristics" refer to the characteristics related to the difficulty level, format, and content of the problem.

[0012] A "category" is a classification that indicates the specific field or topic to which a problem belongs.

[0013] "Means of generation" refer to the techniques and processes used to create new, similar problems or explanations.

[0014] "The means for searching" refers to the technology or process for finding related similar problems and explanations from an existing database.

[0015] "Similar problems" refer to other issues or questions that are similar in content or form to the received problem.

[0016] "Explanation" refers to information or guidelines for explaining the answer to a problem in detail to assist understanding.

[0017] "The means for providing" refers to the technology or process for conveying the generated or searched similar problems and their explanations to the learner.

[0018] "Natural language processing technology" refers to algorithms or technologies for a computer to understand, analyze, and generate human language.

[0019] "Image recognition technology" refers to algorithms or technologies for a computer to read and analyze information from images.

[0020] "Answer result" refers to answer information such as correct or incorrect answers to the problems solved by the learner.

[0021] "The means for recording" refers to the technology or process for saving the learner's answer results and learning progress and reflecting them in future learning plans.

[0022] "The means for optimizing" refers to the technology or process for adjusting the content of the next learning based on the learner's progress and understanding level to support effective learning.

Brief Explanation of Drawings

[0023] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3]It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0024] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0025] First, the language used in the following description will be explained.

[0026] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0027] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0028] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0029] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0031] [First Embodiment]

[0032] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0033] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0036] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0039] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0043] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0044] System Overview

[0045] This system aims to allow learners to study at their own pace and provide effective supplementary lessons, especially for problems they find difficult. This includes a process of inputting problems that learners were unable to solve, analyzing those problems, generating or searching for similar problems and explanations, and then providing them to the learners.

[0046] User actions

[0047] Users utilize an interface to input problems they were unable to solve. They can enter and submit problems in text or image format. Users are not required to input problems in a specific format; the system is designed to appropriately analyze problems using natural language processing (NLP) and image recognition technologies.

[0048] Server operation

[0049] 1. Receiving and saving the problem:

[0050] The issues submitted by the user are received by the server and stored in the database. In this step, the user ID and the issue are associated and stored together.

[0051] 2. Problem Analysis:

[0052] The server analyzes the stored problems to identify their characteristics (e.g., difficulty level, problem format) and category (e.g., mathematics, factorization). This analysis utilizes NLP and image recognition technologies.

[0053] 3. Generate or search for similar problems and explanations:

[0054] Based on the characteristics and category of the problem, the server searches the database for similar problems and their explanations, or generates new ones. In doing so, the server utilizes the existing problem database to select the most suitable similar problems and explanations.

[0055] 4. Provision of problems and explanations:

[0056] The server sends generated or searched similar problems and their explanations to the user's terminal. This allows the user to work on new problems and deepen their understanding by referring to the explanations.

[0057] Specific example

[0058] For example, if a user is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", the user inputs this problem into the system. The server then receives the problem and analyzes its characteristics and category. In this case, the problem is classified as a "quadratic equation" and "factorization". Next, the server searches the database for similar quadratic equations such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0", and prepares the solutions and explanations for each.

[0059] This information is sent to the user's device, allowing them to work on similar problems. Furthermore, each time a user submits an answer, the result is recorded on the server, optimizing the next learning plan. This entire process allows users to focus on problems they find particularly difficult, resulting in more effective learning.

[0060] Tracking learning progress

[0061] The user's answers to problems are sent from the device to the server. The server receives this data and records it in a database. This record is managed as the user's learning history and used to optimize the learning plan for the next learning session. This allows users to visualize their progress and receive individually customized learning content.

[0062] Through the above process, this system can provide learners with an individually optimized learning experience and support efficient learning.

[0063] The following describes the processing flow.

[0064] Step 1:

[0065] Users input and submit problems they were unable to solve using their devices. Users input problems in text or image format through a dedicated interface. By pressing the submit button, the device sends the input data to the server.

[0066] Step 2:

[0067] The server receives the problem sent from the terminal and saves it to the database. When saving, the user ID is associated with the problem.

[0068] Step 3:

[0069] The server analyzes the stored problems. This analysis uses natural language processing (NLP) and image recognition technologies. As a result of the analysis, the characteristics of the problem (e.g., difficulty level, format) and category (e.g., mathematics, factorization) are identified.

[0070] Step 4:

[0071] The server generates or searches for similar problems and their explanations based on the analysis results. First, it searches the database for existing problems that match the characteristics and categories. If no suitable problem is found, it may generate a new similar problem. Similarly, explanations are either retrieved from the existing database or newly created.

[0072] Step 5:

[0073] The server sends similar problems, generated or found, along with their explanations, to the user's terminal. This allows the user to receive materials to work on similar problems.

[0074] Step 6:

[0075] Users work on similar problems sent via their devices. They answer the problems and input their answers into their devices.

[0076] Step 7:

[0077] The device sends the user's answer to the server. This transmission includes not only the answer but also which question the answer corresponds to.

[0078] Step 8:

[0079] The server receives the user's answer and records it in the database. The user ID, question ID, and answer are stored together in an associated manner.

[0080] Step 9:

[0081] The server analyzes the user's learning progress based on the saved answer results. This analysis evaluates the user's level of understanding and learning progress.

[0082] Step 10:

[0083] The server optimizes the next learning plan. Based on the user's learning progress and answer results, it selects the next problems and explanations to tackle and prepares individually optimized learning content.

[0084] Through these steps, the system provides learners with an individually optimized learning experience, supporting efficient learning.

[0085] (Example 1)

[0086] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0087] Traditional learning systems struggled to efficiently provide supplementary lessons tailored to individual learners' weak areas, hindering learners from effectively progressing at their own pace. Furthermore, they often lacked adequate explanations for unsolved problems and insufficient provision of similar exercises, leading to decreased learning efficiency. Additionally, they lacked features to track learners' progress and optimize subsequent learning plans.

[0088] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0089] In this invention, the server includes means for receiving problems that the learner was unable to solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, means for recording the learner's answer results and optimizing the next learning plan, and means for generating similar problems and their explanations using a generative AI model. This allows learners to learn efficiently at their own pace and receive effective supplementary lessons, especially for problems they find difficult.

[0090] A "learner" is an individual who seeks to acquire knowledge and skills through an educational program.

[0091] A "problem" is a question or task presented to learners with the expectation that they will answer it.

[0092] "Analysis" is the process of analyzing the content and characteristics of a problem and classifying it into specific attributes or categories.

[0093] "Characteristics" refer to specific features or attributes of a problem, such as its difficulty level or format.

[0094] A "category" is a classification framework used to categorize the academic field or type of problem to which it belongs.

[0095] A "similar problem" is another problem that has similar characteristics or categories to the problem that has been analyzed.

[0096] "Explanation" refers to explanations and supplementary information that help solve or understand a problem.

[0097] "Generation" refers to the artificial creation of new problems or explanations.

[0098] "Searching" is the process of finding information that matches specific criteria from an existing database.

[0099] A "learning plan" refers to a learning schedule and content optimized based on the learner's progress and level of understanding.

[0100] "Natural language processing technology" refers to the technology that enables computers to understand, process, and generate human language.

[0101] "Image recognition technology" is a technology that analyzes image data and enables computers to understand its content.

[0102] A "generative AI model" is a model that uses artificial intelligence technology to generate problems and explanations.

[0103] "Answer results" refer to information such as the answers that learners gave to the questions and whether or not they were correct.

[0104] Modes for carrying out the invention

[0105] System Overview

[0106] This invention relates to a system that allows learners to study at their own pace and provides effective supplementary lessons for specific areas of difficulty. Specifically, it includes a series of processes in which learners input problems they were unable to solve, the system analyzes them, and provides similar problems and explanations. The system can also record the learner's answers and optimize their next learning plan.

[0107] Server operation

[0108] The core of this system resides in the server. The server has the following main functions:

[0109] 1. Receiving and saving the problem:

[0110] The server receives the questions submitted by learners and saves them to a database. By associating the questions with the user ID, the server tracks the progress of individual learners.

[0111] 2. Problem Analysis:

[0112] The server analyzes received and stored problems using natural language processing (NLP) and image recognition technologies. This analysis identifies the problem's characteristics (difficulty level, problem format) and category (mathematics, factorization, etc.). NLP technologies such as Spacy and BERT are used, while image recognition technologies such as OpenCV and Tensorflow® are employed.

[0113] 3. Generate or search for similar problems and explanations:

[0114] Based on the analysis results, the server searches the database for similar problems and their explanations. Alternatively, it uses a generative AI model to generate new problems and explanations. Advanced models such as 'GPT-3' and 'GPT-4' are used for the generative AI model.

[0115] 4. Provision of problems and explanations:

[0116] The server sends generated or retrieved similar problems and their explanations to the learner's terminal. This allows the learner to work on new problems and deepen their understanding by referring to the explanations.

[0117] 5. Tracking learning progress:

[0118] The server receives the results of the questions answered by the learner and records them in a database. This data is used to optimize the next learning plan.

[0119] Terminal operation

[0120] Learners will perform the following operations using their own devices.

[0121] 1. Enter the question:

[0122] Learners use the interface to input problems they were unable to solve. These problems can be entered in text or image format.

[0123] 2. Submit your answer:

[0124] Each time a learner submits an answer to a problem, the result is automatically sent to the server.

[0125] Specific example

[0126] For example, if a learner is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", they input the problem into the system. The server receives the problem and analyzes its characteristics and category. In this case, the problem is classified as a "quadratic equation" and "factorization". Next, the server searches the database for similar quadratic equations such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0", and prepares the solutions and explanations for each. This information is sent to the learner's terminal, allowing them to work on these similar problems.

[0127] Example of a prompt

[0128] When using a generative AI model to perform problem analysis or similar problem searches, the following prompt statements can be input to the generative AI model.

[0129] Please analyze the following quadratic equation problem that the user was unable to solve: "x^2 - 4x + 4 = 0". Identify the characteristics of this problem (difficulty level, category, etc.) and generate similar problems and their solutions.

[0130] This process allows the system to provide learners with a learning experience that is individually optimized, supporting effective learning.

[0131] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0132] Step 1:

[0133] User Action: The user inputs the problem they were unable to solve through the interface. Input can be in text or image format. The input data is sent to the system.

[0134] Input: The text or image data of the problem.

[0135] Output: Problem data sent to the server

[0136] Step 2:

[0137] Server operation: The server receives problem data sent by the user and stores it in the database. It associates the user ID with the problem data and registers it.

[0138] Input: Problem data submitted by the user

[0139] Output: Problem data and user ID stored in the database

[0140] Step 3:

[0141] Server Operation: The server analyzes stored problems using NLP and image recognition technologies. It identifies the characteristics of the problem (difficulty level, problem format) and its category (e.g., mathematics, factorization). For this analysis, NLP technologies such as 'Spacy' and 'BERT' are used, and image recognition technologies such as 'OpenCV' and 'TensorFlow' are used.

[0142] Input: Saved problem data

[0143] Output: Problem characteristics and categories

[0144] Step 4:

[0145] Server operation: Based on the analysis results, the server searches the database for similar problems and their explanations, or generates new ones using a generation AI model (e.g., 'GPT-3' or 'GPT-4'). By inputting generation prompts into the AI ​​model, similar problems and explanations are generated.

[0146] Input: Problem characteristics and categories

[0147] Output: Similar problems and explanations

[0148] Example prompt:

[0149] "Analyze the following quadratic equation problem that the user was unable to solve: 'x^2 - 4x + 4 = 0'. Identify the characteristics of this problem (difficulty level, category, etc.) and generate similar problems and their solutions."

[0150] Step 5:

[0151] Server operation: The server sends similar problems generated or found, along with their explanations, to the user's terminal. This allows the user to refer to them and work on new problems.

[0152] Input: Similar problems and explanations

[0153] Output: Similar problems and explanations sent to the user's terminal.

[0154] Step 6:

[0155] User actions: The user works on similar problems submitted to them and submits their solutions through the interface.

[0156] Input: Data provided by the user

[0157] Output: Answer data sent to the server

[0158] Step 7:

[0159] Server operation: The server receives answer data submitted by users and records it in a database. This record is used to optimize the next learning plan.

[0160] Input: User-submitted answer data

[0161] Output: Answer results and learning progress recorded in the database

[0162] Through the above processing steps, the system can provide learners with appropriate supplementary lessons and support effective learning.

[0163] (Application Example 1)

[0164] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0165] Conventional learning support systems and operational management systems have struggled to effectively and quickly resolve problems faced by users. Furthermore, these systems lacked the functionality to adequately support user skill development. As a result, users repeatedly encountered the same problems, leading to decreased learning and operational efficiency. This invention aims to solve these problems.

[0166] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0167] In this invention, the server includes means for receiving problems that learners were unable to solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, means for providing the generated or searched similar problems and their explanations to the learners, means for analyzing mechanical problems and error codes entered by an operator and providing similar solutions and explanations, an interface for users to input problems in text or image format, means for analyzing the input problems using natural language processing or image recognition technology, means for providing the user with the optimal solution based on the analysis results, and means for recording the user's operation history and optimizing the next solution. This enables users to obtain quick and accurate solutions to problems they face, effectively supporting skill improvement.

[0168] A "learner" refers to a user who uses the system to improve their knowledge and skills.

[0169] A "problem" refers to a question that learners cannot solve, or one that illustrates a mechanical error or situation that an operator might encounter.

[0170] "Analysis" refers to the process of identifying the characteristics and categories of a problem received and understanding its content.

[0171] A "similar problem" refers to another problem that has the same or very similar characteristics or categories as the problem being analyzed.

[0172] "Explanation" refers to information that provides a detailed explanation or solution to a problem.

[0173] An "operator" refers to a person responsible for operating factory robots, operational equipment, and other similar devices.

[0174] "Natural language processing technology" refers to technologies for analyzing and understanding human language.

[0175] "Image recognition technology" refers to the technology of extracting and analyzing specific information from images.

[0176] "Interface" refers to the screen or means by which a user inputs information about a problem.

[0177] A "server" refers to a computer system that stores, analyzes, and responds to users' data.

[0178] "Operation history" refers to a record of operations and inputs performed by a user using the system.

[0179] This invention is a system for analyzing problems faced by learners and factory operators and providing appropriate solutions. This system is implemented using the following hardware and software.

[0180] Hardware to use

[0181] 1. Server: A system capable of high-speed data processing and large-scale data storage. Specific examples include using cloud-based servers such as AWS® or Google® Cloud.

[0182] 2. User terminals: Devices such as smartphones, tablets, and PCs. In factory operations, dedicated operating terminals, such as industrial tablets, are also used.

[0183] Software to use

[0184] 1. Natural Language Processing (NLP): A technique for analyzing human language and understanding its meaning. As a concrete example, we will use the Transformers library (provided by Hugging Face).

[0185] 2. Image Recognition Technology: A technology for analyzing image data. As a specific example, we will use OpenCV (an open-source computer vision library).

[0186] 3. Database: A system for storing data on problems and solutions. Specific examples include MySQL® and PostgreSQL.

[0187] Overall system operation

[0188] The system receives problems entered by users, identifies their characteristics and categories, and then provides similar problems and solutions based on that. Users can enter problems in text or image format, and the system analyzes the input to provide the best solution.

[0189] Specific example

[0190] For example, if a factory operator enters a problem in text format such as "The robot suddenly stopped," the system analyzes the problem and searches its database for similar problems and their solutions. This provides specific solutions such as "Check the robot's main switch."

[0191] Furthermore, if a learner is unable to solve a math problem such as "x^2 - 4x + 4 = 0", they can input the problem in text or image format. The system will then identify the problem's category (e.g., quadratic equations) and search its database for similar problems and their solutions.

[0192] Examples of prompt statements

[0193] "The robot suddenly stopped. What should I do?"

[0194] "I can't solve the problem x^2 - 4x + 4 = 0. Please help me."

[0195] This invention allows users to obtain problem-solving solutions quickly and accurately, and is expected to improve learning and operational efficiency. Furthermore, it can record the user's operation history and optimize future problem-solving processes for greater efficiency.

[0196] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0197] Step 1:

[0198] The user inputs the problem using a terminal. The user inputs the problem in text or image format and presses the submit button. There is no specific format required for the problem the user inputs. Examples of input include sentences such as "The robot suddenly stopped" or images of mathematical formulas such as "x^2 - 4x + 4 = 0". The input data is sent to the server.

[0199] Step 2:

[0200] The server saves the received problems. The server records the problem data submitted by the user in a database. The problem data is associated with the problem content and the user ID. This saving step allows for future analysis of the problems and tracking of learning history.

[0201] Step 3:

[0202] The server analyzes the problem. It analyzes the received problem and uses natural language processing (NLP) or image recognition techniques to identify the problem's characteristics (e.g., difficulty, format) and category (e.g., mathematics, mechanical failure). For example, the text "The robot stopped" would be classified into the categories "robot" and "operation stopped" using NLP. The analysis results are temporarily stored in memory.

[0203] Step 4:

[0204] The server searches for or generates similar problems and explanations. Based on the analysis results, it searches the database for similar problems and their explanations. In this process, it can also generate new explanations using a generation AI model. From the search results or generated results, the most suitable similar problem and explanation are selected. For example, an explanation such as "the possibility of the robot stopping at the main switch" might be obtained.

[0205] Step 5:

[0206] The server sends similar problems and explanations to the user's terminal. Similar problems and their explanations, whether found or generated, are sent to the user's terminal. The user can then view this information on their terminal. For example, an explanation such as "Check the robot's main switch" might be displayed.

[0207] Step 6:

[0208] The user implements the suggested solution. The user follows the instructions displayed on the terminal and takes specific action. For example, the operator might instruct the user to "check the robot's main switch" and then check the main switch.

[0209] Step 7:

[0210] The user submits their solution or problem-solving result. After implementing the suggested solution, the user enters the result on their device and sends it to the server. For example, they might enter feedback such as, "I checked the main switch, but the robot is not working."

[0211] Step 8:

[0212] The server records the user's progress and optimizes the next approach. Received solutions are stored in a database and managed as learning or operation history. Based on this history, the server generates information to optimize future problem-solving and learning plans. This improves the user's skills and problem-solving efficiency.

[0213] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0214] System Overview

[0215] This system aims to allow learners to learn at their own pace and provide effective supplementary lessons, especially for areas where they struggle. A new emotion engine has been added, which recognizes and analyzes learners' emotions and adjusts the learning content accordingly. This enables efficient learning while maintaining learner motivation.

[0216] User actions

[0217] Users utilize an interface to input problems they were unable to solve. They input and submit problems in text or image format. At the same time, facial expressions and voice are recorded via camera and microphone to capture user emotion data.

[0218] Server operation

[0219] 1. Receiving and saving the problem:

[0220] Issues submitted by users are received by the server and stored in the database. In this step, the user ID, issue, and sentiment data are stored together.

[0221] 2. Problem Analysis:

[0222] The server analyzes the stored problems to identify their characteristics (e.g., difficulty, format) and category (e.g., mathematics, factorization). This analysis uses natural language processing (NLP) and image recognition technologies.

[0223] 3. Analysis of emotional data:

[0224] The emotion engine analyzes recorded emotion data to identify the learner's emotional state (e.g., stress level, motivation). This data is used in conjunction with the problem analysis results.

[0225] 4. Generate or search for similar problems and explanations:

[0226] Based on the characteristics and category of the problem, as well as sentiment data, the server searches the database for similar problems and their explanations, or generates new ones. Depending on the sentiment data, the difficulty level may be adjusted and the depth of the explanations may be changed.

[0227] 5. Provision of problems and explanations:

[0228] The server generates or searches for similar problems and their explanations, adjusts them appropriately based on sentiment data, and sends them to the user's terminal. This allows the user to learn with content that is appropriate to their current emotional state.

[0229] Specific example

[0230] For example, if a user is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", they input this problem into the system. The server then receives the problem and analyzes its characteristics and categories. In this case, the problem is classified as a "quadratic equation" and "factorization". Simultaneously, the emotion engine detects "anxiety" from the user's facial expressions and voice.

[0231] The server then searches for similar problems in the "quadratic equations" and "factorization" categories and prepares appropriate explanations. Based on the detected sentiment, the server provides explanations in a clearer, step-by-step manner. For example, problems such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0" are selected, and their solutions and explanations are sent to the user.

[0232] Users work on these similar problems and input their answers into their device. The device then retrieves sentiment data along with these answers and sends it to the server. The server records and analyzes the answers in a database. The next learning plan is further optimized based on the recorded answers and sentiment data.

[0233] Tracking learning progress

[0234] The user's answers to problems are sent from their device to the server. The server receives this data and records it in a database. This record is managed as the user's learning history and used to optimize the learning plan for the next learning session. Since emotional data is also recorded, a learning plan is created that takes into account the learner's motivation and emotional changes. As a result, users can not only visualize their own progress but also receive individually customized learning content.

[0235] This entire process allows learners to have an optimal learning experience and to progress through their studies efficiently. The introduction of an emotion engine improves the quality of learning by providing personalized support that takes into account the learner's emotional state.

[0236] The following describes the processing flow.

[0237] Step 1:

[0238] The user inputs and submits a problem they were unable to solve using their device. The user inputs the problem in text or image format through a dedicated interface and presses the submit button. During this process, the user's facial expressions and voice data are simultaneously recorded using the camera and microphone.

[0239] Step 2:

[0240] The terminal sends the entered problem data, along with the user's facial expressions and voice data, to the server. The problem data is transmitted with the user ID associated with it.

[0241] Step 3:

[0242] The server receives problem data sent from the terminal and saves it to the database. During saving, the user ID is associated with the problem data.

[0243] Step 4:

[0244] The server analyzes the problem data it receives. Natural language processing (NLP) and image recognition technologies are used to identify the characteristics and categories of the problems. These characteristics include the difficulty level and format of the problem.

[0245] Step 5:

[0246] The server analyzes facial expressions and voice data to identify the user's emotional state. The emotion engine evaluates the user's stress level and motivation. This data is also stored in a database.

[0247] Step 6:

[0248] The server searches the database for similar problems and their explanations based on the problem's characteristics and category, as well as sentiment data. It may also generate new problems and explanations as needed.

[0249] Step 7:

[0250] The server adjusts the searched or generated similar problems and their explanations to match the user's emotional state. For example, if the user is showing a high stress level, it will start with easier problems and adjust the explanations to be more gradual.

[0251] Step 8:

[0252] The server sends similar problems and their explanations, tailored to the user's device. The user receives these problems and explanations on their device and continues learning.

[0253] Step 9:

[0254] Users work on similar problems submitted via their devices and input their answers. After solving the problems, sentiment data is recorded again.

[0255] Step 10:

[0256] The device sends the user's answers, facial expressions, and voice data to the server. This data includes the user ID.

[0257] Step 11:

[0258] The server receives the answer results and saves them to the database. The user ID, question ID, answer result, and sentiment data are stored together.

[0259] Step 12:

[0260] The server analyzes the user's learning progress based on stored answer results and sentiment data. This analysis evaluates the user's level of understanding and learning progress.

[0261] Step 13:

[0262] The server optimizes the next learning plan. Based on the user's learning progress and sentiment data, it selects the next problems and explanations to tackle and prepares individually optimized learning content.

[0263] Through the steps outlined above, this system provides learners with an individually optimized learning experience, supporting efficient learning. The introduction of an emotion engine enables personalized learning support that takes into account the learner's emotional state.

[0264] (Example 2)

[0265] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0266] Traditional learning support systems often provide problems and explanations without considering the learner's emotional state, making it difficult to provide an optimal learning experience for each individual learner. This can lead to decreased learner motivation and hinder efficient learning.

[0267] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving problems that the learner could not solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for simultaneously recording the learner's emotional data when receiving the problems, means for analyzing the emotional data and identifying the learner's emotional state, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems and the emotional data, means for providing the generated or searched similar problems and their explanations to the learner, and means for recording the learner's answer results and emotional data and optimizing the next learning plan. This makes it possible to provide effective learning support and improve motivation in accordance with the learner's emotional state.

[0268] "Learner" refers to an individual who uses the system to engage in learning activities.

[0269] A "problem" refers to an academic or educational task that learners attempt to solve.

[0270] "Emotional data" refers to data that indicates an emotional state, extracted from the learner's facial expressions and voice.

[0271] A "server" refers to a computer system that is responsible for analyzing, processing, and storing the data it receives.

[0272] A "database" refers to a collection of information that a server uses to efficiently store, manage, and retrieve data.

[0273] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human language.

[0274] "Image recognition technology" refers to the technology used by computers to extract and analyze information from images.

[0275] The "Emotion Engine" refers to software or hardware for analyzing the recorded emotion data of learners to identify their emotional states.

[0276] "Analysis" refers to the process of examining data or information in detail and finding characteristics or specific patterns based on it.

[0277] "Similar problems" refer to other problems having the same characteristics or categories as the problems that learners could not solve.

[0278] "Explanation" refers to the content that explains the solutions to problems and related knowledge to learners.

[0279] "Learning plan" refers to a plan that proposes optimal learning content and schedule based on the progress and abilities of learners.

[0280] "Optimization" refers to adjusting a system or plan to its most effective state to achieve a specific goal.

[0281] Modes for Implementing the Invention

[0282] This invention relates to a system that collects problems that learners could not solve and learners' emotion data, analyzes these data, and provides appropriate learning support. Specifically, it is a process in which a server receives the problems and emotion data sent by learners, analyzes them, and optimizes the learning plan.

[0283] Hardware and Software to be Used

[0284] This system uses the following hardware and software.

[0285] 1. Terminal:

[0286] Devices used by learners, such as PCs, smartphones, tablets, etc.

[0287] It is desirable to have a camera and microphone, as well as sensors for recording emotional data.

[0288] 2. Server:

[0289] It is a high-performance computer system that stores, analyzes, and processes data.

[0290] The system efficiently manages learner data by using an internal or external database.

[0291] 3. Natural Language Processing Techniques:

[0292] We will use the Python NLP library "NLTK" or similar tools to analyze text data.

[0293] 4. Image recognition technology:

[0294] We use libraries such as "OpenCV" to perform image analysis.

[0295] 5. Emotional Engine:

[0296] Software such as "Emotion API" is used to analyze the learner's facial expressions and voice to identify their emotional state.

[0297] 6. Generative AI Models:

[0298] The "GPT-3" model and its successor models are used to generate similar problems and explanations.

[0299] System Description

[0300] User actions

[0301] Users use their devices to input problems they couldn't solve into the interface. Problems can be entered in text or image format, and simultaneously, facial expressions and voice are recorded via the camera and microphone. This also allows for the collection of emotional data.

[0302] As a specific example, when a user fails to solve the quadratic equation "x^2 - 4x + 4 = 0", the user enters this problem into a text box and clicks the "Send" button. At this time, the camera records the user's facial expression and the microphone captures the voice.

[0303] Server Operations

[0304] The server receives the problem data and emotion data sent by the user. The problem data is saved in a database and analyzed using natural language processing technology (e.g., NLTK) and image recognition technology (e.g., OpenCV). Through the analysis, the characteristics (e.g., difficulty level, format) and category (e.g., quadratic equation, factorization) of the problem are identified.

[0305] At the same time, the emotion engine analyzes the recorded facial expressions and voices to identify the learner's emotional state (e.g., anxiety, stress). The information on the identified emotional state is used together with the analysis results of the problem.

[0306] Based on these analysis results, the server searches the database for similar problems and their explanations or generates them newly using a generated AI model (e.g., GPT-3). The difficulty level and depth of the explanation are adjusted according to the emotion data.

[0307] Specific Example

[0308] For example, when a user enters the problem "x^2 - 4x + 4 = 0" and "anxiety" is identified as the emotional state, the server prepares similar problems and their explanations from the categories of "quadratic equation" and "factorization". At this time, based on the emotion data, the explanations are provided in an increasingly understandable manner step by step. Specifically, problems such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0" are selected, and the solution methods for each are explained in detail.

[0309] Examples of Prompt Sentences

[0310] Examples of prompts to input into a generative AI model include the following:

[0311] To help a student who is struggling with the quadratic equation "x^2 - 4x + 4 = 0" and is feeling frustrated, please generate and provide the following problem and explanation. Please include several similar problems and explain the solution to each step-by-step.

[0312] This allows the server to provide a learning experience optimized for the learner's emotional state, thereby maintaining motivation and improving learning efficiency.

[0313] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0314] Step 1:

[0315] The user inputs the problem they were unable to solve into the device's interface. Specifically, the user enters the problem into a text box or uploads an image file. Simultaneously, the camera and microphone automatically activate to record the user's facial expressions and voice. The input data includes problem data (text or image) and emotion data (facial expressions, voice).

[0316] Step 2:

[0317] The terminal sends user-entered problem data and recorded sentiment data to the server. This data includes user ID, problem data, and sentiment data. This provides the server with the necessary input for subsequent analysis.

[0318] Step 3:

[0319] The server stores the received problem data and analyzes the text data using natural language processing techniques (such as NLTK). It also analyzes image data using image recognition techniques (such as OpenCV). As a result of the analysis, the characteristics of the problem (difficulty level, format) and its category (e.g., mathematics, factorization) are identified. The input is the problem data, and the output is the problem characteristics and category information.

[0320] Step 4:

[0321] The server uses an emotion engine (such as the Emotion API) to analyze the recorded emotion data. This analysis identifies the learner's emotional state (e.g., stress level, motivation). The input is the emotion data, and the output is the identified emotional state.

[0322] Step 5:

[0323] The server searches the database for similar problems and their explanations, or generates new ones, based on the problem's characteristics, category, and sentiment data. A generative AI model (such as GPT-3) is used for generation. Specifically, prompts are used to instruct the generative AI model to obtain similar problems and their explanations. The input consists of the problem's characteristics, category, and sentiment state, while the output is similar problems and their explanations.

[0324] Step 6:

[0325] The server sends generated or retrieved similar problems and their explanations to the user's terminal. Based on sentiment data, the difficulty level and depth of the explanations are adjusted, so learners receive learning content that is best suited to their state. The input is similar problems and explanations, and the user's terminal information; the output is the transmission of the adjusted learning content.

[0326] Step 7:

[0327] The user works on a similar problem provided and inputs their answer into the terminal. The terminal then sends the answer and recorded sentiment data back to the server. The input is the user's answer and sentiment data, and the output is the data sent to the server.

[0328] Step 8:

[0329] The server records the received answers and sentiment data in a database. This optimizes the next learning plan. Specifically, it uses an adaptive learning algorithm to analyze the learner's progress and sentiment state and propose the optimal content for the next learning session. The input is the answers and sentiment data, and the output is the optimized next learning plan.

[0330] (Application Example 2)

[0331] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0332] Traditional learning systems provided learning content without considering learners' emotions, resulting in insufficient maintenance of learner motivation and optimization of learning effectiveness. Furthermore, in in-store customer service support, it was difficult to grasp customers' emotional states in real time and respond appropriately. Consequently, improvements in customer satisfaction and effective product recommendations were not fully achieved. This system aims to solve these problems.

[0333] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving problems that the learner could not solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, means for providing the generated or searched similar problems and their explanations to the learner, means for recognizing and analyzing the learner's emotional state, and means for appropriately adjusting the learning content based on the emotional state. This makes it possible to provide personalized learning content that takes into account the learner's emotional state, thereby maintaining motivation and optimizing learning effectiveness. Furthermore, even in physical stores, customer satisfaction can be improved by analyzing the customer's emotional state in real time and providing appropriate customer service methods.

[0334] "A means of receiving problems that learners were unable to solve" refers to an interface for learners to input or upload problems they were unable to solve.

[0335] "Means for identifying the characteristics and categories of a problem" refers to a function that analyzes a received problem and identifies its characteristics and classification, such as difficulty level, format, subject matter, etc.

[0336] "Means for generating or searching for similar problems and their explanations" means a function that generates new similar problems and their explanations or searches for them in an existing database based on the aforementioned characteristics and categories.

[0337] "Means to provide to learners" means the function of displaying or transmitting generated or retrieved similar problems and their explanations to the learner's device.

[0338] "Means for recognizing and analyzing emotional states" refers to a function that analyzes emotions from a learner's facial expressions and voice data and identifies their emotional state (e.g., stress, motivation).

[0339] "Means for appropriately adjusting learning content based on emotional state" refers to a function that dynamically adjusts learning content and problem difficulty levels according to analyzed emotional data, providing an optimal learning plan tailored to the learner.

[0340] "Natural language processing technology" is a general term for algorithms and methods used to analyze text data and identify its characteristics and categories.

[0341] "Image recognition technology" is a general term for algorithms and methods used to analyze image data and identify its characteristics and categories.

[0342] "Facial expression analysis technology" is a general term for technologies that analyze facial expressions from image data acquired by a camera and estimate emotions from the results.

[0343] "Voice analysis technology" is a general term for technologies that analyze voice data acquired by a microphone and estimate emotions from the results.

[0344] System Configuration

[0345] This invention is a system consisting of the following main elements:

[0346] 1. Device: A smartphone used by learners, store clerks, etc.

[0347] 2. Server: A server that performs data analysis, generates learning content, sentiment analysis, etc.

[0348] 3. Camera and microphone: Devices for acquiring facial expressions and voice data.

[0349] Program processing

[0350] For learners

[0351] 1. Inputting the problem: Learners input the problems they were unable to solve into their device in text or image format and submit them. At this time, the camera and microphone are used to collect learner sentiment data.

[0352] 2. Problem Analysis: The server analyzes the received problem using natural language processing (NLP) or image recognition techniques to identify its characteristics and categories.

[0353] 3. Analysis of emotional data: The server uses data acquired from the camera and microphone to analyze the learner's emotional state using facial expression analysis and voice analysis technologies.

[0354] 4. Generation and search of similar problems and explanations: Based on the characteristics of the problem and the sentiment analysis results, the server searches the database for similar problems and their explanations or generates new ones.

[0355] 5. Provision of tailored learning content: The server sends optimized problems and explanations based on the analysis data to the learner's terminal, which the learner then works on.

[0356] In the case of customer service support at physical stores

[0357] 1. Acquisition of customer emotion data: While serving customers, store employees use the camera and microphone on their devices to capture the customer's facial expressions and voice, thereby acquiring emotion data.

[0358] 2. Analysis of emotional data: Based on the acquired data, the server analyzes the customer's emotional state in real time using facial expression analysis technology and voice analysis technology.

[0359] 3. Adjusting responses: Based on the analyzed sentiment data, the server provides appropriate responses and product suggestions that the store staff should take, and these are displayed on the terminal.

[0360] Hardware and software to be used

[0361] Hardware: Smartphone, camera, microphone.

[0362] Software: OpenCV (video acquisition and display), DeepFace (facial expression analysis), natural language processing technology (NLP, text analysis), image recognition technology (image analysis).

[0363] Examples of specific cases and prompt statements

[0364] Specific example

[0365] For example, if a customer is having trouble choosing a product during a consultation at a physical store, the system uses the customer's smartphone camera and microphone to analyze their emotional state. If the analysis reveals "anxiety" or "unease," the system instructs the salesperson to take actions such as "provide a detailed product description" or "suggest alternative options."

[0366] Example of a prompt

[0367] Develop an application that analyzes user emotions and suggests appropriate customer service methods. Implement the emotion analysis function shown in the following code:

[0368] 1. The camera captures facial images, and DeepFace is used to analyze facial expressions.

[0369] 2. Based on the emotional data obtained, we adjust the customer service methods and recommended products to best suit each customer.

[0370] As described above, in the embodiment of this invention, it is possible to analyze user input information and emotional data and dynamically provide appropriate learning content and customer service responses. As a result, improved learning effectiveness and increased customer satisfaction can be expected.

[0371] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0372] Step 1:

[0373] The user inputs a problem they were unable to solve. The input problem is in text or image format and is sent from the learner's device to the server. At the same time, the learner's facial expressions and voice data are captured using the device's camera and microphone and sent to the server as emotion data.

[0374] Input: Problems the learner was unable to solve, learner's facial expression data, learner's voice data

[0375] Output: Problem data and sentiment data sent to the server

[0376] Specific steps: The learner opens a dedicated application on their smartphone, enters or photographs the question, and clicks the submit button. Simultaneously, video and audio data is captured.

[0377] Step 2:

[0378] The server analyzes the problem data it receives. Natural language processing (NLP) and image recognition techniques are used to identify the characteristics of the problem (difficulty level, format, etc.) and its category (e.g., mathematics, factorization).

[0379] Input: Problem data

[0380] Output: Problem characteristics and categories

[0381] Specific operation: The server analyzes the received text or image data using NLP and image recognition algorithms to identify the nature of the problem.

[0382] Step 3:

[0383] The server analyzes emotional data. Using facial expression analysis and voice analysis technologies, it identifies the learner's emotional state (e.g., stress level, motivation) from the received video and audio data.

[0384] Input: Facial expression data, voice data

[0385] Output: Emotional state

[0386] Specific operation: The server uses DeepFace and other facial expression analysis software to analyze the learner's video data and identify the dominant emotion. Voice analysis software is also used in conjunction to estimate the emotional state with high accuracy.

[0387] Step 4:

[0388] Based on the characteristics and category of the problem, as well as the sentiment analysis results, the server generates or searches for similar problems and their explanations in the database. The level of detail and difficulty of the explanation are adjusted according to the emotional state.

[0389] Input: Problem characteristics and categories, emotional state

[0390] Output: Similar problems and their explanations

[0391] Specific operation: The server searches the database using problem characteristic data and category data to find similar problems and their explanations. It may also generate new similar problems and explanations using a generative AI model. Based on the results of sentiment analysis, it automatically adjusts the difficulty and detail of the explanations.

[0392] Step 5:

[0393] The server sends similar problems and their explanations, generated or retrieved, to the learner's device. This allows the learner to progress through the learning process with content that best suits their current emotional state.

[0394] Input: Similar problems and their explanations

[0395] Output: Learning content sent to the learner's device.

[0396] Specific operation: The server sends the formatted data to the learner's device and displays it on the device's application. During this process, notification functions and interface optimization are performed.

[0397] Step 6:

[0398] The learner works on additional learning material and enters their answers into the device. The device sends the answers and newly acquired sentiment data to the server.

[0399] Input: Answer results, additional sentiment data

[0400] Output: Answer results and additional sentiment data sent to the server

[0401] Specific operation: The learner works on similar problems provided and enters their answers. Simultaneously, the device's camera and microphone capture the learner's facial expressions and voice again.

[0402] Step 7:

[0403] The server receives the answer results and additional sentiment data, and records them in a database as the learner's learning history. This allows for further optimization of the next learning plan.

[0404] Input: Answer results, additional sentiment data

[0405] Output: Update of the learning history database

[0406] Specific actions: The server analyzes the answer results and sentiment data, and updates parameters to adjust the next learning plan. It adds new entries to the database and optimizes the learning history.

[0407] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0408] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0409] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0410] [Second Embodiment]

[0411] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0412] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0413] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0414] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0415] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0416] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0417] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0418] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0419] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0420] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0421] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0422] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0423] System Overview

[0424] This system aims to allow learners to study at their own pace and provide effective supplementary lessons, especially for problems they find difficult. This includes a process of inputting problems that learners were unable to solve, analyzing those problems, generating or searching for similar problems and explanations, and then providing them to the learners.

[0425] User actions

[0426] Users utilize an interface to input problems they were unable to solve. They can enter and submit problems in text or image format. Users are not required to input problems in a specific format; the system is designed to appropriately analyze problems using natural language processing (NLP) and image recognition technologies.

[0427] Server operation

[0428] 1. Receiving and saving the problem:

[0429] The issues submitted by the user are received by the server and stored in the database. In this step, the user ID and the issue are associated and stored together.

[0430] 2. Problem Analysis:

[0431] The server analyzes the stored problems to identify their characteristics (e.g., difficulty level, problem format) and category (e.g., mathematics, factorization). This analysis utilizes NLP and image recognition technologies.

[0432] 3. Generate or search for similar problems and explanations:

[0433] Based on the characteristics and category of the problem, the server searches the database for similar problems and their explanations, or generates new ones. In doing so, the server utilizes the existing problem database to select the most suitable similar problems and explanations.

[0434] 4. Provision of problems and explanations:

[0435] The server sends generated or searched similar problems and their explanations to the user's terminal. This allows the user to work on new problems and deepen their understanding by referring to the explanations.

[0436] Specific example

[0437] For example, if a user is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", the user inputs this problem into the system. The server then receives the problem and analyzes its characteristics and category. In this case, the problem is classified as a "quadratic equation" and "factorization". Next, the server searches the database for similar quadratic equations such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0", and prepares the solutions and explanations for each.

[0438] This information is sent to the user's device, allowing them to work on similar problems. Furthermore, each time a user submits an answer, the result is recorded on the server, optimizing the next learning plan. This entire process allows users to focus on problems they find particularly difficult, resulting in more effective learning.

[0439] Tracking learning progress

[0440] The user's answers to problems are sent from the device to the server. The server receives this data and records it in a database. This record is managed as the user's learning history and used to optimize the learning plan for the next learning session. This allows users to visualize their progress and receive individually customized learning content.

[0441] Through the above process, this system can provide learners with an individually optimized learning experience and support efficient learning.

[0442] The following describes the processing flow.

[0443] Step 1:

[0444] Users input and submit problems they were unable to solve using their devices. Users input problems in text or image format through a dedicated interface. By pressing the submit button, the device sends the input data to the server.

[0445] Step 2:

[0446] The server receives the problem sent from the terminal and saves it to the database. When saving, the user ID is associated with the problem.

[0447] Step 3:

[0448] The server analyzes the stored problems. This analysis uses natural language processing (NLP) and image recognition technologies. As a result of the analysis, the characteristics of the problem (e.g., difficulty level, format) and category (e.g., mathematics, factorization) are identified.

[0449] Step 4:

[0450] The server generates or searches for similar problems and their explanations based on the analysis results. First, it searches the database for existing problems that match the characteristics and categories. If no suitable problem is found, it may generate a new similar problem. Similarly, explanations are either retrieved from the existing database or newly created.

[0451] Step 5:

[0452] The server sends similar problems, generated or found, along with their explanations, to the user's terminal. This allows the user to receive materials to work on similar problems.

[0453] Step 6:

[0454] Users work on similar problems sent via their devices. They answer the problems and input their answers into their devices.

[0455] Step 7:

[0456] The device sends the user's answer to the server. This transmission includes not only the answer but also which question the answer corresponds to.

[0457] Step 8:

[0458] The server receives the user's answer and records it in the database. The user ID, question ID, and answer are stored together in an associated manner.

[0459] Step 9:

[0460] The server analyzes the user's learning progress based on the saved answer results. This analysis evaluates the user's level of understanding and learning progress.

[0461] Step 10:

[0462] The server optimizes the next learning plan. Based on the user's learning progress and answer results, it selects the next problems and explanations to tackle and prepares individually optimized learning content.

[0463] Through these steps, the system provides learners with an individually optimized learning experience, supporting efficient learning.

[0464] (Example 1)

[0465] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0466] Traditional learning systems struggled to efficiently provide supplementary lessons tailored to individual learners' weak areas, hindering learners from effectively progressing at their own pace. Furthermore, they often lacked adequate explanations for unsolved problems and insufficient provision of similar exercises, leading to decreased learning efficiency. Additionally, they lacked features to track learners' progress and optimize subsequent learning plans.

[0467] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0468] In this invention, the server includes means for receiving problems that the learner was unable to solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, means for recording the learner's answer results and optimizing the next learning plan, and means for generating similar problems and their explanations using a generative AI model. This allows learners to learn efficiently at their own pace and receive effective supplementary lessons, especially for problems they find difficult.

[0469] A "learner" is an individual who seeks to acquire knowledge and skills through an educational program.

[0470] A "problem" is a question or task presented to learners with the expectation that they will answer it.

[0471] "Analysis" is the process of analyzing the content and characteristics of a problem and classifying it into specific attributes or categories.

[0472] "Characteristics" refer to specific features or attributes of a problem, such as its difficulty level or format.

[0473] A "category" is a classification framework used to categorize the academic field or type of problem to which it belongs.

[0474] A "similar problem" is another problem that has similar characteristics or categories to the problem that has been analyzed.

[0475] "Explanation" refers to explanations and supplementary information that help solve or understand a problem.

[0476] "Generation" refers to the artificial creation of new problems or explanations.

[0477] "Searching" is the process of finding information that matches specific criteria from an existing database.

[0478] A "learning plan" refers to a learning schedule and content optimized based on the learner's progress and level of understanding.

[0479] "Natural language processing technology" refers to the technology that enables computers to understand, process, and generate human language.

[0480] "Image recognition technology" is a technology that analyzes image data and enables computers to understand its content.

[0481] A "generative AI model" is a model that uses artificial intelligence technology to generate problems and explanations.

[0482] "Answer results" refer to information such as the answers that learners gave to the questions and whether or not they were correct.

[0483] Modes for carrying out the invention

[0484] System Overview

[0485] This invention relates to a system that allows learners to study at their own pace and provides effective supplementary lessons for specific areas of difficulty. Specifically, it includes a series of processes in which learners input problems they were unable to solve, the system analyzes them, and provides similar problems and explanations. The system can also record the learner's answers and optimize their next learning plan.

[0486] Server operation

[0487] The core of this system resides in the server. The server has the following main functions:

[0488] 1. Receiving and saving the problem:

[0489] The server receives the questions submitted by learners and saves them to a database. By associating the questions with the user ID, the server tracks the progress of individual learners.

[0490] 2. Problem Analysis:

[0491] The server analyzes received and stored problems using natural language processing (NLP) and image recognition technologies. This analysis identifies the problem's characteristics (difficulty level, problem format) and category (mathematics, factorization, etc.). NLP technologies such as Spacy and BERT are used, while image recognition technologies such as OpenCV and TensorFlow are employed.

[0492] 3. Generate or search for similar problems and explanations:

[0493] Based on the analysis results, the server searches the database for similar problems and their explanations. Alternatively, it uses a generative AI model to generate new problems and explanations. Advanced models such as 'GPT-3' and 'GPT-4' are used for generative AI modeling.

[0494] 4. Provision of problems and explanations:

[0495] The server sends generated or retrieved similar problems and their explanations to the learner's terminal. This allows the learner to work on new problems and deepen their understanding by referring to the explanations.

[0496] 5. Tracking learning progress:

[0497] The server receives the results of the questions answered by the learner and records them in a database. This data is used to optimize the next learning plan.

[0498] Terminal operation

[0499] Learners will perform the following operations using their own devices.

[0500] 1. Enter the question:

[0501] Learners use the interface to input problems they were unable to solve. These problems can be entered in text or image format.

[0502] 2. Submit your answer:

[0503] Each time a learner submits an answer to a problem, the result is automatically sent to the server.

[0504] Specific example

[0505] For example, if a learner is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", they input the problem into the system. The server receives the problem and analyzes its characteristics and category. In this case, the problem is classified as a "quadratic equation" and "factorization". Next, the server searches the database for similar quadratic equations such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0", and prepares the solutions and explanations for each. This information is sent to the learner's terminal, allowing them to work on these similar problems.

[0506] Example of a prompt

[0507] When using a generative AI model to perform problem analysis or similar problem searches, the following prompt statements can be input to the generative AI model.

[0508] Please analyze the following quadratic equation problem that the user was unable to solve: "x^2 - 4x + 4 = 0". Identify the characteristics of this problem (difficulty level, category, etc.) and generate similar problems and their solutions.

[0509] This process allows the system to provide learners with a learning experience that is individually optimized, supporting effective learning.

[0510] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0511] Step 1:

[0512] User Action: The user inputs the problem they were unable to solve through the interface. Input can be in text or image format. The input data is sent to the system.

[0513] Input: The text or image data of the problem.

[0514] Output: Problem data sent to the server

[0515] Step 2:

[0516] Server operation: The server receives problem data sent by the user and stores it in the database. It associates the user ID with the problem data and registers it.

[0517] Input: Problem data submitted by the user

[0518] Output: Problem data and user ID stored in the database

[0519] Step 3:

[0520] Server Operation: The server analyzes stored problems using NLP and image recognition technologies. It identifies the characteristics of the problem (difficulty level, problem format) and its category (e.g., mathematics, factorization). For this analysis, NLP technologies such as 'Spacy' and 'BERT' are used, and image recognition technologies such as 'OpenCV' and 'TensorFlow' are used.

[0521] Input: Saved problem data

[0522] Output: Problem characteristics and categories

[0523] Step 4:

[0524] Server operation: Based on the analysis results, the server searches the database for similar problems and their explanations, or generates new ones using a generation AI model (e.g., 'GPT-3' or 'GPT-4'). By inputting generation prompts into the AI ​​model, similar problems and explanations are generated.

[0525] Input: Problem characteristics and categories

[0526] Output: Similar problems and explanations

[0527] Example prompt:

[0528] "Analyze the following quadratic equation problem that the user was unable to solve: 'x^2 - 4x + 4 = 0'. Identify the characteristics of this problem (difficulty level, category, etc.) and generate similar problems and their solutions."

[0529] Step 5:

[0530] Server operation: The server sends similar problems generated or found, along with their explanations, to the user's terminal. This allows the user to refer to them and work on new problems.

[0531] Input: Similar problems and explanations

[0532] Output: Similar problems and explanations sent to the user's terminal.

[0533] Step 6:

[0534] User actions: The user works on similar problems submitted to them and submits their solutions through the interface.

[0535] Input: Data provided by the user

[0536] Output: Answer data sent to the server

[0537] Step 7:

[0538] Server operation: The server receives answer data submitted by users and records it in a database. This record is used to optimize the next learning plan.

[0539] Input: User-submitted answer data

[0540] Output: Answer results and learning progress recorded in the database

[0541] Through the above processing steps, the system can provide learners with appropriate supplementary lessons and support effective learning.

[0542] (Application Example 1)

[0543] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0544] Conventional learning support systems and operational management systems have struggled to effectively and quickly resolve problems faced by users. Furthermore, these systems lacked the functionality to adequately support user skill development. As a result, users repeatedly encountered the same problems, leading to decreased learning and operational efficiency. This invention aims to solve these problems.

[0545] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0546] In this invention, the server includes means for receiving problems that learners were unable to solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, means for providing the generated or searched similar problems and their explanations to the learners, means for analyzing mechanical problems and error codes entered by an operator and providing similar solutions and explanations, an interface for users to input problems in text or image format, means for analyzing the input problems using natural language processing or image recognition technology, means for providing the user with the optimal solution based on the analysis results, and means for recording the user's operation history and optimizing the next solution. This enables users to obtain quick and accurate solutions to problems they face, effectively supporting skill improvement.

[0547] A "learner" refers to a user who uses the system to improve their knowledge and skills.

[0548] A "problem" refers to a question that learners cannot solve, or one that illustrates a mechanical error or situation that an operator might encounter.

[0549] "Analysis" refers to the process of identifying the characteristics and categories of a problem received and understanding its content.

[0550] A "similar problem" refers to another problem that has the same or very similar characteristics or categories as the problem being analyzed.

[0551] "Explanation" refers to information that provides a detailed explanation or solution to a problem.

[0552] An "operator" refers to a person responsible for operating factory robots, operational equipment, and other similar devices.

[0553] "Natural language processing technology" refers to technologies for analyzing and understanding human language.

[0554] "Image recognition technology" refers to the technology of extracting and analyzing specific information from images.

[0555] "Interface" refers to the screen or means by which a user inputs information about a problem.

[0556] A "server" refers to a computer system that stores, analyzes, and responds to users' data.

[0557] "Operation history" refers to a record of operations and inputs performed by a user using the system.

[0558] This invention is a system for analyzing problems faced by learners and factory operators and providing appropriate solutions. This system is implemented using the following hardware and software.

[0559] Hardware to use

[0560] 1. Server: A system capable of high-speed data processing and large-scale data storage. Specific examples include using cloud-based servers such as AWS or Google Cloud.

[0561] 2. User terminals: Devices such as smartphones, tablets, and PCs. In factory operations, dedicated operating terminals, such as industrial tablets, are also used.

[0562] Software to use

[0563] 1. Natural Language Processing (NLP): A technique for analyzing human language and understanding its meaning. As a concrete example, we will use the Transformers library (provided by Hugging Face).

[0564] 2. Image Recognition Technology: A technology for analyzing image data. As a specific example, we will use OpenCV (an open-source computer vision library).

[0565] 3. Database: A system for storing data on problems and solutions. Specific examples include MySQL and PostgreSQL.

[0566] Overall system operation

[0567] The system receives problems entered by users, identifies their characteristics and categories, and then provides similar problems and solutions based on that. Users can enter problems in text or image format, and the system analyzes the input to provide the best solution.

[0568] Specific example

[0569] For example, if a factory operator enters a problem in text format such as "The robot suddenly stopped," the system analyzes the problem and searches its database for similar problems and their solutions. This provides specific solutions such as "Check the robot's main switch."

[0570] Furthermore, if a learner is unable to solve a math problem such as "x^2 - 4x + 4 = 0", they can input the problem in text or image format. The system will then identify the problem's category (e.g., quadratic equations) and search its database for similar problems and their solutions.

[0571] Examples of prompt statements

[0572] "The robot suddenly stopped. What should I do?"

[0573] "I can't solve the problem x^2 - 4x + 4 = 0. Please help me."

[0574] This invention allows users to obtain problem-solving solutions quickly and accurately, and is expected to improve learning and operational efficiency. Furthermore, it can record the user's operation history and optimize future problem-solving processes for greater efficiency.

[0575] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0576] Step 1:

[0577] The user inputs the problem using a terminal. The user inputs the problem in text or image format and presses the submit button. There is no specific format required for the problem the user inputs. Examples of input include sentences such as "The robot suddenly stopped" or images of mathematical formulas such as "x^2 - 4x + 4 = 0". The input data is sent to the server.

[0578] Step 2:

[0579] The server saves the received problems. The server records the problem data submitted by the user in a database. The problem data is associated with the problem content and the user ID. This saving step allows for future analysis of the problems and tracking of learning history.

[0580] Step 3:

[0581] The server analyzes the problem. It analyzes the received problem and uses natural language processing (NLP) or image recognition techniques to identify the problem's characteristics (e.g., difficulty, format) and category (e.g., mathematics, mechanical failure). For example, the text "The robot stopped" would be classified into the categories "robot" and "operation stopped" using NLP. The analysis results are temporarily stored in memory.

[0582] Step 4:

[0583] The server searches for or generates similar problems and explanations. Based on the analysis results, it searches the database for similar problems and their explanations. In this process, it can also generate new explanations using a generation AI model. From the search results or generated results, the most suitable similar problem and explanation are selected. For example, an explanation such as "the possibility of the robot stopping at the main switch" might be obtained.

[0584] Step 5:

[0585] The server sends similar problems and explanations to the user's terminal. Similar problems and their explanations, whether found or generated, are sent to the user's terminal. The user can then view this information on their terminal. For example, an explanation such as "Check the robot's main switch" might be displayed.

[0586] Step 6:

[0587] The user implements the suggested solution. The user follows the instructions displayed on the terminal and takes specific action. For example, the operator checks the main switch of the robot, following the instruction "Check the robot's main switch."

[0588] Step 7:

[0589] The user submits their solution or problem-solving result. After implementing the suggested solution, the user enters the result on their device and sends it to the server. For example, they might enter feedback such as, "I checked the main switch, but the robot is not working."

[0590] Step 8:

[0591] The server records the user's progress and optimizes the next approach. Received solutions are stored in a database and managed as learning or operation history. Based on this history, the server generates information to optimize future problem-solving and learning plans. This improves the user's skills and problem-solving efficiency.

[0592] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0593] System Overview

[0594] This system aims to allow learners to learn at their own pace and provide effective supplementary lessons, especially for areas where they struggle. A new emotion engine has been added, which recognizes and analyzes learners' emotions and adjusts the learning content accordingly. This enables efficient learning while maintaining learner motivation.

[0595] User actions

[0596] Users utilize an interface to input problems they were unable to solve. They input and submit problems in text or image format. At the same time, facial expressions and voice are recorded via camera and microphone to capture user emotion data.

[0597] Server operation

[0598] 1. Receiving and saving the problem:

[0599] Issues submitted by users are received by the server and stored in the database. In this step, the user ID, issue, and sentiment data are stored together.

[0600] 2. Problem Analysis:

[0601] The server analyzes the stored problems to identify their characteristics (e.g., difficulty, format) and category (e.g., mathematics, factorization). This analysis uses natural language processing (NLP) and image recognition technologies.

[0602] 3. Analysis of emotional data:

[0603] The emotion engine analyzes recorded emotion data to identify the learner's emotional state (e.g., stress level, motivation). This data is used in conjunction with the problem analysis results.

[0604] 4. Generate or search for similar problems and explanations:

[0605] Based on the characteristics and category of the problem, as well as sentiment data, the server searches the database for similar problems and their explanations, or generates new ones. Depending on the sentiment data, the difficulty level may be adjusted and the depth of the explanations may be changed.

[0606] 5. Provision of problems and explanations:

[0607] The server generates or searches for similar problems and their explanations, adjusts them appropriately based on sentiment data, and sends them to the user's terminal. This allows the user to learn with content that is appropriate to their current emotional state.

[0608] Specific example

[0609] For example, if a user is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", they input this problem into the system. The server then receives the problem and analyzes its characteristics and categories. In this case, the problem is classified as a "quadratic equation" and "factorization". Simultaneously, the emotion engine detects "anxiety" from the user's facial expressions and voice.

[0610] The server then searches for similar problems in the "quadratic equations" and "factorization" categories and prepares appropriate explanations. Based on the detected sentiment, the server provides explanations in a clearer, step-by-step manner. For example, problems such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0" are selected, and their solutions and explanations are sent to the user.

[0611] Users work on these similar problems and input their answers into their device. The device then retrieves sentiment data along with these answers and sends it to the server. The server records and analyzes the answers in a database. The next learning plan is further optimized based on the recorded answers and sentiment data.

[0612] Tracking learning progress

[0613] The user's answers to problems are sent from their device to the server. The server receives this data and records it in a database. This record is managed as the user's learning history and used to optimize the learning plan for the next learning session. Since emotional data is also recorded, a learning plan is created that takes into account the learner's motivation and emotional changes. As a result, users can not only visualize their own progress but also receive individually customized learning content.

[0614] This entire process allows learners to have an optimal learning experience and to progress through their studies efficiently. The introduction of an emotion engine improves the quality of learning by providing personalized support that takes into account the learner's emotional state.

[0615] The following describes the processing flow.

[0616] Step 1:

[0617] The user inputs and submits a problem they were unable to solve using their device. The user inputs the problem in text or image format through a dedicated interface and presses the submit button. During this process, the user's facial expressions and voice data are simultaneously recorded using the camera and microphone.

[0618] Step 2:

[0619] The terminal sends the entered problem data, along with the user's facial expressions and voice data, to the server. The problem data is transmitted with the user ID associated with it.

[0620] Step 3:

[0621] The server receives problem data sent from the terminal and saves it to the database. During saving, the user ID is associated with the problem data.

[0622] Step 4:

[0623] The server analyzes the problem data it receives. Natural language processing (NLP) and image recognition technologies are used to identify the characteristics and categories of the problems. These characteristics include the difficulty level and format of the problem.

[0624] Step 5:

[0625] The server analyzes facial expressions and voice data to identify the user's emotional state. The emotion engine evaluates the user's stress level and motivation. This data is also stored in a database.

[0626] Step 6:

[0627] The server searches the database for similar problems and their explanations based on the problem's characteristics and category, as well as sentiment data. It may also generate new problems and explanations as needed.

[0628] Step 7:

[0629] The server adjusts the searched or generated similar problems and their explanations to match the user's emotional state. For example, if the user is showing a high stress level, it will start with easier problems and adjust the explanations to be more gradual.

[0630] Step 8:

[0631] The server sends similar problems and their explanations, tailored to the user's device. The user receives these problems and explanations on their device and continues learning.

[0632] Step 9:

[0633] Users work on similar problems submitted via their devices and input their answers. After solving the problems, sentiment data is recorded again.

[0634] Step 10:

[0635] The device sends the user's answers, facial expressions, and voice data to the server. This data includes the user ID.

[0636] Step 11:

[0637] The server receives the answer results and saves them to the database. The user ID, question ID, answer result, and sentiment data are stored together.

[0638] Step 12:

[0639] The server analyzes the user's learning progress based on stored answer results and sentiment data. This analysis evaluates the user's level of understanding and learning progress.

[0640] Step 13:

[0641] The server optimizes the next learning plan. Based on the user's learning progress and sentiment data, it selects the next problems and explanations to tackle and prepares individually optimized learning content.

[0642] Through the steps outlined above, this system provides learners with an individually optimized learning experience, supporting efficient learning. The introduction of an emotion engine enables personalized learning support that takes into account the learner's emotional state.

[0643] (Example 2)

[0644] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0645] Traditional learning support systems often provide problems and explanations without considering the learner's emotional state, making it difficult to provide an optimal learning experience for each individual learner. This can lead to decreased learner motivation and hinder efficient learning.

[0646] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving problems that the learner could not solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for simultaneously recording the learner's emotional data when receiving the problems, means for analyzing the emotional data and identifying the learner's emotional state, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems and the emotional data, means for providing the generated or searched similar problems and their explanations to the learner, and means for recording the learner's answer results and emotional data and optimizing the next learning plan. This makes it possible to provide effective learning support and improve motivation in accordance with the learner's emotional state.

[0647] "Learner" refers to an individual who uses the system to engage in learning activities.

[0648] A "problem" refers to an academic or educational task that learners attempt to solve.

[0649] "Emotional data" refers to data that indicates an emotional state, extracted from the learner's facial expressions and voice.

[0650] A "server" refers to a computer system that is responsible for analyzing, processing, and storing the data it receives.

[0651] A "database" refers to a collection of information that a server uses to efficiently store, manage, and retrieve data.

[0652] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human language.

[0653] "Image recognition technology" refers to the technology used by computers to extract and analyze information from images.

[0654] An "emotion engine" refers to software or hardware that analyzes a learner's recorded emotional data to identify their emotional state.

[0655] "Analysis" refers to the process of examining data and information in detail and using that information to identify characteristics or specific patterns.

[0656] A "similar problem" refers to another problem that has similar characteristics or categories to the problem the learner was unable to solve.

[0657] "Explanation" refers to content that explains to learners how to solve problems and related knowledge.

[0658] A "learning plan" refers to a plan that proposes the optimal learning content and schedule based on the learner's progress and abilities.

[0659] "Optimization" refers to adjusting a system or plan to its most effective state in order to achieve a specific objective.

[0660] Modes for carrying out the invention

[0661] This invention relates to a system that collects problems that learners were unable to solve and data on the learners' emotions, analyzes this data, and provides appropriate learning support. Specifically, the system involves a server receiving problems and emotion data sent by learners, analyzing them, and optimizing the learning plan.

[0662] Hardware and software to be used

[0663] This system uses the following hardware and software:

[0664] 1. Terminal:

[0665] These are devices used by learners, such as PCs, smartphones, and tablets.

[0666] It is desirable to have a camera and microphone, as well as sensors for recording emotional data.

[0667] 2. Server:

[0668] It is a high-performance computer system that stores, analyzes, and processes data.

[0669] The system efficiently manages learner data by using an internal or external database.

[0670] 3. Natural Language Processing Techniques:

[0671] We will use the Python NLP library "NLTK" or similar tools to analyze text data.

[0672] 4. Image recognition technology:

[0673] We use libraries such as "OpenCV" to perform image analysis.

[0674] 5. Emotional Engine:

[0675] Software such as "Emotion API" is used to analyze the learner's facial expressions and voice to identify their emotional state.

[0676] 6. Generative AI Models:

[0677] The "GPT-3" model and its successor models are used to generate similar problems and explanations.

[0678] System Description

[0679] User actions

[0680] Users use their devices to input problems they couldn't solve into the interface. Problems can be entered in text or image format, and simultaneously, facial expressions and voice are recorded via the camera and microphone. This also collects emotional data.

[0681] As a concrete example, if a user is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", they enter the problem into a text box and click the "Submit" button. At this time, the camera records the user's facial expressions and the microphone captures their voice.

[0682] Server operation

[0683] The server receives problem data and sentiment data submitted by users. The problem data is stored in a database and analyzed using natural language processing techniques (e.g., NLTK) and image recognition techniques (e.g., OpenCV). The analysis identifies the characteristics of the problem (e.g., difficulty level, format) and its category (e.g., quadratic equation, factorization).

[0684] Simultaneously, the emotion engine analyzes recorded facial expressions and voice to identify the learner's emotional state (e.g., anxiety, stress). This identified emotional state information is used in conjunction with the problem analysis results.

[0685] Based on these analysis results, the server searches the database for similar problems and their explanations, or generates new problems using a newly generated AI model (e.g., GPT-3). The difficulty level and depth of the explanations are adjusted according to the sentiment data.

[0686] Specific example

[0687] For example, if a user inputs the problem "x^2 - 4x + 4 = 0" and their emotional state is identified as "anxiety," the server will prepare similar problems and their explanations from the "quadratic equations" and "factorization" categories. In this case, the explanations will be provided in a step-by-step manner, based on the emotional data. Specifically, problems such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0" will be selected, and the solutions to each will be explained in detail.

[0688] Example of a prompt

[0689] Examples of prompts to input into a generative AI model include the following:

[0690] To help a student who is struggling with the quadratic equation "x^2 - 4x + 4 = 0" and is feeling frustrated, please generate and provide the following problem and explanation. Please include several similar problems and explain the solution to each step-by-step.

[0691] This allows the server to provide a learning experience optimized for the learner's emotional state, thereby maintaining motivation and improving learning efficiency.

[0692] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0693] Step 1:

[0694] The user inputs the problem they were unable to solve into the device's interface. Specifically, the user enters the problem into a text box or uploads an image file. Simultaneously, the camera and microphone automatically activate to record the user's facial expressions and voice. The input data includes problem data (text or image) and emotion data (facial expressions, voice).

[0695] Step 2:

[0696] The terminal sends user-entered problem data and recorded sentiment data to the server. This data includes user ID, problem data, and sentiment data. This provides the server with the necessary input for subsequent analysis.

[0697] Step 3:

[0698] The server stores the received problem data and analyzes the text data using natural language processing techniques (such as NLTK). It also analyzes image data using image recognition techniques (such as OpenCV). As a result of the analysis, the characteristics of the problem (difficulty level, format) and its category (e.g., mathematics, factorization) are identified. The input is the problem data, and the output is the problem characteristics and category information.

[0699] Step 4:

[0700] The server uses an emotion engine (such as the Emotion API) to analyze the recorded emotion data. This analysis identifies the learner's emotional state (e.g., stress level, motivation). The input is the emotion data, and the output is the identified emotional state.

[0701] Step 5:

[0702] The server searches the database for similar problems and their explanations, or generates new ones, based on the problem's characteristics, category, and sentiment data. A generative AI model (such as GPT-3) is used for generation. Specifically, prompts are used to instruct the generative AI model to obtain similar problems and their explanations. The input consists of the problem's characteristics, category, and sentiment state, while the output is similar problems and their explanations.

[0703] Step 6:

[0704] The server sends generated or retrieved similar problems and their explanations to the user's terminal. Based on sentiment data, the difficulty level and depth of the explanations are adjusted, so learners receive learning content that is best suited to their state. The input is similar problems and explanations, and the user's terminal information; the output is the transmission of the adjusted learning content.

[0705] Step 7:

[0706] The user works on a similar problem provided and inputs their answer into the terminal. The terminal then sends the answer and recorded sentiment data back to the server. The input is the user's answer and sentiment data, and the output is the data sent to the server.

[0707] Step 8:

[0708] The server records the received answers and sentiment data in a database. This optimizes the next learning plan. Specifically, it uses an adaptive learning algorithm to analyze the learner's progress and sentiment state and propose the optimal content for the next learning session. The input is the answers and sentiment data, and the output is the optimized next learning plan.

[0709] (Application Example 2)

[0710] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0711] Traditional learning systems provided learning content without considering learners' emotions, resulting in insufficient maintenance of learner motivation and optimization of learning effectiveness. Furthermore, in in-store customer service support, it was difficult to grasp customers' emotional states in real time and respond appropriately. Consequently, improvements in customer satisfaction and effective product recommendations were not fully achieved. This system aims to solve these problems.

[0712] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving problems that the learner could not solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, means for providing the generated or searched similar problems and their explanations to the learner, means for recognizing and analyzing the learner's emotional state, and means for appropriately adjusting the learning content based on the emotional state. This makes it possible to provide personalized learning content that takes into account the learner's emotional state, thereby maintaining motivation and optimizing learning effectiveness. Furthermore, even in physical stores, customer satisfaction can be improved by analyzing the customer's emotional state in real time and providing appropriate customer service methods.

[0713] "A means of receiving problems that learners were unable to solve" refers to an interface for learners to input or upload problems they were unable to solve.

[0714] "Means for identifying the characteristics and categories of a problem" refers to a function that analyzes a received problem and identifies its characteristics and classification, such as difficulty level, format, and subject matter.

[0715] "Means for generating or searching for similar problems and their explanations" means a function that generates new similar problems and their explanations, or searches for them in an existing database, based on the aforementioned characteristics and categories.

[0716] "Means to provide to learners" means the function of displaying or transmitting generated or retrieved similar problems and their explanations to the learner's device.

[0717] "Means for recognizing and analyzing emotional states" refers to a function that analyzes emotions from a learner's facial expressions and voice data and identifies their emotional state (e.g., stress, motivation).

[0718] "Means for appropriately adjusting learning content based on emotional state" refers to a function that dynamically adjusts learning content and problem difficulty levels according to analyzed emotional data, providing an optimal learning plan tailored to the learner.

[0719] "Natural language processing technology" is a general term for algorithms and methods used to analyze text data and identify its characteristics and categories.

[0720] "Image recognition technology" is a general term for algorithms and methods used to analyze image data and identify its characteristics and categories.

[0721] "Facial expression analysis technology" is a general term for technologies that analyze facial expressions from image data acquired by a camera and estimate emotions from the results.

[0722] "Voice analysis technology" is a general term for technologies that analyze voice data acquired by a microphone and estimate emotions from the results.

[0723] System Configuration

[0724] This invention is a system consisting of the following main elements:

[0725] 1. Device: A smartphone used by learners, store clerks, etc.

[0726] 2. Server: A server that performs data analysis, generates learning content, sentiment analysis, etc.

[0727] 3. Camera and microphone: Devices for acquiring facial expressions and voice data.

[0728] Program processing

[0729] For learners

[0730] 1. Inputting the problem: Learners input the problems they were unable to solve into their device in text or image format and submit them. At this time, the camera and microphone are used to collect learner sentiment data.

[0731] 2. Problem Analysis: The server analyzes the received problem using natural language processing (NLP) or image recognition techniques to identify its characteristics and categories.

[0732] 3. Analysis of emotional data: The server uses data acquired from the camera and microphone to analyze the learner's emotional state using facial expression analysis and voice analysis technologies.

[0733] 4. Generation and search of similar problems and explanations: Based on the characteristics of the problem and the sentiment analysis results, the server searches the database for similar problems and their explanations or generates new ones.

[0734] 5. Provision of tailored learning content: The server sends optimized problems and explanations based on the analysis data to the learner's terminal, which the learner then works on.

[0735] In the case of customer service support at physical stores

[0736] 1. Acquisition of customer emotion data: While serving customers, store employees use the camera and microphone on their devices to capture the customer's facial expressions and voice, thereby acquiring emotion data.

[0737] 2. Analysis of emotional data: Based on the acquired data, the server analyzes the customer's emotional state in real time using facial expression analysis technology and voice analysis technology.

[0738] 3. Adjusting responses: Based on the analyzed sentiment data, the server provides appropriate responses and product suggestions that the store staff should take, and these are displayed on the terminal.

[0739] Hardware and software to be used

[0740] Hardware: Smartphone, camera, microphone.

[0741] Software: OpenCV (video acquisition and display), DeepFace (facial expression analysis), natural language processing technology (NLP, text analysis), image recognition technology (image analysis).

[0742] Examples of specific cases and prompt statements

[0743] Specific example

[0744] For example, if a customer is having trouble choosing a product during a consultation at a physical store, the system uses the customer's smartphone camera and microphone to analyze their emotional state. If the analysis reveals "anxiety" or "unease," the system instructs the salesperson to take actions such as "provide a detailed product description" or "suggest alternative options."

[0745] Example of a prompt

[0746] Develop an application that analyzes user emotions and suggests appropriate customer service methods. Implement the emotion analysis function shown in the following code:

[0747] 1. The camera captures facial images, and DeepFace is used to analyze facial expressions.

[0748] 2. Based on the emotional data obtained, we adjust the customer service methods and recommended products to best suit each customer.

[0749] As described above, in the embodiment of this invention, it is possible to analyze user input information and emotional data and dynamically provide appropriate learning content and customer service responses. As a result, improved learning effectiveness and increased customer satisfaction can be expected.

[0750] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0751] Step 1:

[0752] The user inputs a problem they were unable to solve. The input problem is in text or image format and is sent from the learner's device to the server. At the same time, the learner's facial expressions and voice data are captured using the device's camera and microphone and sent to the server as emotion data.

[0753] Input: Problems the learner was unable to solve, learner's facial expression data, learner's voice data

[0754] Output: Problem data and sentiment data sent to the server

[0755] Specific steps: The learner opens a dedicated application on their smartphone, enters or photographs the question, and clicks the submit button. Simultaneously, video and audio data is captured.

[0756] Step 2:

[0757] The server analyzes the problem data it receives. Natural language processing (NLP) and image recognition techniques are used to identify the characteristics of the problem (difficulty level, format, etc.) and its category (e.g., mathematics, factorization).

[0758] Input: Problem data

[0759] Output: Problem characteristics and categories

[0760] Specific operation: The server analyzes the received text or image data using NLP and image recognition algorithms to identify the nature of the problem.

[0761] Step 3:

[0762] The server analyzes emotional data. Using facial expression analysis and voice analysis technologies, it identifies the learner's emotional state (e.g., stress level, motivation) from the received video and audio data.

[0763] Input: Facial expression data, voice data

[0764] Output: Emotional state

[0765] Specific operation: The server uses DeepFace and other facial expression analysis software to analyze the learner's video data and identify the dominant emotion. Voice analysis software is also used in conjunction to estimate the emotional state with high accuracy.

[0766] Step 4:

[0767] Based on the characteristics and category of the problem, as well as the sentiment analysis results, the server generates or searches for similar problems and their explanations in the database. The level of detail and difficulty of the explanation are adjusted according to the emotional state.

[0768] Input: Problem characteristics and categories, emotional state

[0769] Output: Similar problems and their explanations

[0770] Specific operation: The server searches the database using problem characteristic data and category data to find similar problems and their explanations. It may also generate new similar problems and explanations using a generative AI model. Based on the results of sentiment analysis, it automatically adjusts the difficulty and detail of the explanations.

[0771] Step 5:

[0772] The server sends similar problems and their explanations, generated or retrieved, to the learner's device. This allows the learner to progress through the learning process with content that best suits their current emotional state.

[0773] Input: Similar problems and their explanations

[0774] Output: Learning content sent to the learner's device.

[0775] Specific operation: The server sends the formatted data to the learner's device and displays it on the device's application. During this process, notification functions and interface optimization are performed.

[0776] Step 6:

[0777] The learner works on additional learning material and enters their answers into the device. The device sends the answers and newly acquired sentiment data to the server.

[0778] Input: Answer results, additional sentiment data

[0779] Output: Answer results and additional sentiment data sent to the server

[0780] Specific operation: The learner works on similar problems provided and enters their answers. Simultaneously, the device's camera and microphone capture the learner's facial expressions and voice again.

[0781] Step 7:

[0782] The server receives the answer results and additional sentiment data, and records them in a database as the learner's learning history. This allows for further optimization of the next learning plan.

[0783] Input: Answer results, additional sentiment data

[0784] Output: Update of the learning history database

[0785] Specific actions: The server analyzes the answer results and sentiment data, and updates parameters to adjust the next learning plan. It adds new entries to the database and optimizes the learning history.

[0786] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0787] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0788] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0789] [Third Embodiment]

[0790] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0791] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0792] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0793] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0794] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0795] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0796] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0797] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0798] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0799] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0800] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0801] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0802] System Overview

[0803] This system aims to allow learners to study at their own pace and provide effective supplementary lessons, especially for problems they find difficult. This includes a process of inputting problems that learners were unable to solve, analyzing those problems, generating or searching for similar problems and explanations, and then providing them to the learners.

[0804] User actions

[0805] Users utilize an interface to input problems they were unable to solve. They can enter and submit problems in text or image format. Users are not required to input problems in a specific format; the system is designed to appropriately analyze problems using natural language processing (NLP) and image recognition technologies.

[0806] Server operation

[0807] 1. Receiving and saving the problem:

[0808] The issues submitted by the user are received by the server and stored in the database. In this step, the user ID and the issue are associated and stored together.

[0809] 2. Problem Analysis:

[0810] The server analyzes the stored problems to identify their characteristics (e.g., difficulty level, problem format) and category (e.g., mathematics, factorization). This analysis utilizes NLP and image recognition technologies.

[0811] 3. Generate or search for similar problems and explanations:

[0812] Based on the characteristics and category of the problem, the server searches the database for similar problems and their explanations, or generates new ones. In doing so, the server utilizes the existing problem database to select the most suitable similar problems and explanations.

[0813] 4. Provision of problems and explanations:

[0814] The server sends generated or searched similar problems and their explanations to the user's terminal. This allows the user to work on new problems and deepen their understanding by referring to the explanations.

[0815] Specific example

[0816] For example, if a user is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", the user inputs this problem into the system. The server then receives the problem and analyzes its characteristics and category. In this case, the problem is classified as a "quadratic equation" and "factorization". Next, the server searches the database for similar quadratic equations such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0", and prepares the solutions and explanations for each.

[0817] This information is sent to the user's device, allowing them to work on similar problems. Furthermore, each time a user submits an answer, the result is recorded on the server, optimizing the next learning plan. This entire process allows users to focus on problems they find particularly difficult, resulting in more effective learning.

[0818] Tracking learning progress

[0819] The user's answers to problems are sent from the device to the server. The server receives this data and records it in a database. This record is managed as the user's learning history and used to optimize the learning plan for the next learning session. This allows users to visualize their progress and receive individually customized learning content.

[0820] Through the above process, this system can provide learners with an individually optimized learning experience and support efficient learning.

[0821] The following describes the processing flow.

[0822] Step 1:

[0823] Users input and submit problems they were unable to solve using their devices. Users input problems in text or image format through a dedicated interface. By pressing the submit button, the device sends the input data to the server.

[0824] Step 2:

[0825] The server receives the problem sent from the terminal and saves it to the database. When saving, the user ID is associated with the problem.

[0826] Step 3:

[0827] The server analyzes the stored problems. This analysis uses natural language processing (NLP) and image recognition technologies. As a result of the analysis, the characteristics of the problem (e.g., difficulty level, format) and category (e.g., mathematics, factorization) are identified.

[0828] Step 4:

[0829] The server generates or searches for similar problems and their explanations based on the analysis results. First, it searches the database for existing problems that match the characteristics and categories. If no suitable problem is found, it may generate a new similar problem. Similarly, explanations are either retrieved from the existing database or newly created.

[0830] Step 5:

[0831] The server sends similar problems, generated or found, along with their explanations, to the user's terminal. This allows the user to receive materials to work on similar problems.

[0832] Step 6:

[0833] Users work on similar problems sent via their devices. They answer the problems and input their answers into their devices.

[0834] Step 7:

[0835] The device sends the user's answer to the server. This transmission includes not only the answer but also which question the answer corresponds to.

[0836] Step 8:

[0837] The server receives the user's answer and records it in the database. The user ID, question ID, and answer are stored together in an associated manner.

[0838] Step 9:

[0839] The server analyzes the user's learning progress based on the saved answer results. This analysis evaluates the user's level of understanding and learning progress.

[0840] Step 10:

[0841] The server optimizes the next learning plan. Based on the user's learning progress and answer results, it selects the next problems and explanations to tackle and prepares individually optimized learning content.

[0842] Through these steps, the system provides learners with an individually optimized learning experience, supporting efficient learning.

[0843] (Example 1)

[0844] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0845] Traditional learning systems struggled to efficiently provide supplementary lessons tailored to individual learners' weak areas, hindering learners from effectively progressing at their own pace. Furthermore, they often lacked adequate explanations for unsolved problems and insufficient provision of similar exercises, leading to decreased learning efficiency. Additionally, they lacked features to track learners' progress and optimize subsequent learning plans.

[0846] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0847] In this invention, the server includes means for receiving problems that the learner was unable to solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, means for recording the learner's answer results and optimizing the next learning plan, and means for generating similar problems and their explanations using a generative AI model. This allows learners to learn efficiently at their own pace and receive effective supplementary lessons, especially for problems they find difficult.

[0848] A "learner" is an individual who seeks to acquire knowledge and skills through an educational program.

[0849] A "problem" is a question or task presented to learners with the expectation that they will answer it.

[0850] "Analysis" is the process of analyzing the content and characteristics of a problem and classifying it into specific attributes or categories.

[0851] "Characteristics" refer to specific features or attributes of a problem, such as its difficulty level or format.

[0852] A "category" is a classification framework used to categorize the academic field or type of problem to which it belongs.

[0853] A "similar problem" is another problem that has similar characteristics or categories to the problem that has been analyzed.

[0854] "Explanation" refers to explanations and supplementary information that help solve or understand a problem.

[0855] "Generation" refers to the artificial creation of new problems or explanations.

[0856] "Searching" is the process of finding information that matches specific criteria from an existing database.

[0857] A "learning plan" refers to a learning schedule and content optimized based on the learner's progress and level of understanding.

[0858] "Natural language processing technology" refers to the technology that enables computers to understand, process, and generate human language.

[0859] "Image recognition technology" is a technology that analyzes image data and enables computers to understand its content.

[0860] A "generative AI model" is a model that uses artificial intelligence technology to generate problems and explanations.

[0861] "Answer results" refer to information such as the answers that learners gave to the questions and whether or not they were correct.

[0862] Modes for carrying out the invention

[0863] System Overview

[0864] This invention relates to a system that allows learners to study at their own pace and provides effective supplementary lessons for specific areas of difficulty. Specifically, it includes a series of processes in which learners input problems they were unable to solve, the system analyzes them, and provides similar problems and explanations. The system can also record the learner's answers and optimize their next learning plan.

[0865] Server operation

[0866] The core of this system resides in the server. The server has the following main functions:

[0867] 1. Receiving and saving the problem:

[0868] The server receives the questions submitted by learners and saves them to a database. By associating the questions with the user ID, the server tracks the progress of individual learners.

[0869] 2. Problem Analysis:

[0870] The server analyzes received and stored problems using natural language processing (NLP) and image recognition technologies. This analysis identifies the problem's characteristics (difficulty level, problem format) and category (mathematics, factorization, etc.). NLP technologies such as Spacy and BERT are used, while image recognition technologies such as OpenCV and TensorFlow are employed.

[0871] 3. Generate or search for similar problems and explanations:

[0872] Based on the analysis results, the server searches the database for similar problems and their explanations. Alternatively, it uses a generative AI model to generate new problems and explanations. Advanced models such as 'GPT-3' and 'GPT-4' are used for generative AI modeling.

[0873] 4. Provision of problems and explanations:

[0874] The server sends generated or retrieved similar problems and their explanations to the learner's terminal. This allows the learner to work on new problems and deepen their understanding by referring to the explanations.

[0875] 5. Tracking learning progress:

[0876] The server receives the results of the questions answered by the learner and records them in a database. This data is used to optimize the next learning plan.

[0877] Terminal operation

[0878] Learners will perform the following operations using their own devices.

[0879] 1. Enter the question:

[0880] Learners use the interface to input problems they were unable to solve. These problems can be entered in text or image format.

[0881] 2. Submit your answer:

[0882] Each time a learner submits an answer to a problem, the result is automatically sent to the server.

[0883] Specific example

[0884] For example, if a learner is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", they input the problem into the system. The server receives the problem and analyzes its characteristics and category. In this case, the problem is classified as a "quadratic equation" and "factorization". Next, the server searches the database for similar quadratic equations such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0", and prepares the solutions and explanations for each. This information is sent to the learner's terminal, allowing them to work on these similar problems.

[0885] Example of a prompt

[0886] When using a generative AI model to perform problem analysis or similar problem searches, the following prompt statements can be input to the generative AI model.

[0887] Please analyze the following quadratic equation problem that the user was unable to solve: "x^2 - 4x + 4 = 0". Identify the characteristics of this problem (difficulty level, category, etc.) and generate similar problems and their solutions.

[0888] This process allows the system to provide learners with a learning experience that is individually optimized, supporting effective learning.

[0889] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0890] Step 1:

[0891] User Action: The user inputs the problem they were unable to solve through the interface. Input can be in text or image format. The input data is sent to the system.

[0892] Input: The text or image data of the problem.

[0893] Output: Problem data sent to the server

[0894] Step 2:

[0895] Server operation: The server receives problem data sent by the user and stores it in the database. It associates the user ID with the problem data and registers it.

[0896] Input: Problem data submitted by the user

[0897] Output: Problem data and user ID stored in the database

[0898] Step 3:

[0899] Server Operation: The server analyzes stored problems using NLP and image recognition technologies. It identifies the characteristics of the problem (difficulty level, problem format) and its category (e.g., mathematics, factorization). For this analysis, NLP technologies such as 'Spacy' and 'BERT' are used, and image recognition technologies such as 'OpenCV' and 'TensorFlow' are used.

[0900] Input: Saved problem data

[0901] Output: Problem characteristics and categories

[0902] Step 4:

[0903] Server operation: Based on the analysis results, the server searches the database for similar problems and their explanations, or generates new ones using a generation AI model (e.g., 'GPT-3' or 'GPT-4'). By inputting generation prompts into the AI ​​model, similar problems and explanations are generated.

[0904] Input: Problem characteristics and categories

[0905] Output: Similar problems and explanations

[0906] Example prompt:

[0907] "Analyze the following quadratic equation problem that the user was unable to solve: 'x^2 - 4x + 4 = 0'. Identify the characteristics of this problem (difficulty level, category, etc.) and generate similar problems and their solutions."

[0908] Step 5:

[0909] Server operation: The server sends similar problems generated or found, along with their explanations, to the user's terminal. This allows the user to refer to them and work on new problems.

[0910] Input: Similar problems and explanations

[0911] Output: Similar problems and explanations sent to the user's terminal.

[0912] Step 6:

[0913] User actions: The user works on similar problems submitted to them and submits their solutions through the interface.

[0914] Input: Data provided by the user

[0915] Output: Answer data sent to the server

[0916] Step 7:

[0917] Server operation: The server receives answer data submitted by users and records it in a database. This record is used to optimize the next learning plan.

[0918] Input: User-submitted answer data

[0919] Output: Answer results and learning progress recorded in the database

[0920] Through the above processing steps, the system can provide learners with appropriate supplementary lessons and support effective learning.

[0921] (Application Example 1)

[0922] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0923] Conventional learning support systems and operational management systems have struggled to effectively and quickly resolve problems faced by users. Furthermore, these systems lacked the functionality to adequately support user skill development. As a result, users repeatedly encountered the same problems, leading to decreased learning and operational efficiency. This invention aims to solve these problems.

[0924] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0925] In this invention, the server includes means for receiving problems that learners were unable to solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, means for providing the generated or searched similar problems and their explanations to the learners, means for analyzing mechanical problems and error codes entered by an operator and providing similar solutions and explanations, an interface for users to input problems in text or image format, means for analyzing the input problems using natural language processing or image recognition technology, means for providing the user with the optimal solution based on the analysis results, and means for recording the user's operation history and optimizing the next solution. This enables users to obtain quick and accurate solutions to problems they face, effectively supporting skill improvement.

[0926] A "learner" refers to a user who uses the system to improve their knowledge and skills.

[0927] A "problem" refers to a question that learners cannot solve, or one that illustrates a mechanical error or situation that an operator might encounter.

[0928] "Analysis" refers to the process of identifying the characteristics and categories of a problem received and understanding its content.

[0929] A "similar problem" refers to another problem that has the same or very similar characteristics or categories as the problem being analyzed.

[0930] "Explanation" refers to information that provides a detailed explanation or solution to a problem.

[0931] An "operator" refers to a person responsible for operating factory robots, operational equipment, and other similar devices.

[0932] "Natural language processing technology" refers to technologies for analyzing and understanding human language.

[0933] "Image recognition technology" refers to the technology of extracting and analyzing specific information from images.

[0934] "Interface" refers to the screen or means by which a user inputs information about a problem.

[0935] A "server" refers to a computer system that stores, analyzes, and responds to users' data.

[0936] "Operation history" refers to a record of operations and inputs performed by a user using the system.

[0937] This invention is a system for analyzing problems faced by learners and factory operators and providing appropriate solutions. This system is implemented using the following hardware and software.

[0938] Hardware to use

[0939] 1. Server: A system capable of high-speed data processing and large-scale data storage. Specific examples include using cloud-based servers such as AWS or Google Cloud.

[0940] 2. User terminals: Devices such as smartphones, tablets, and PCs. In factory operations, dedicated operating terminals, such as industrial tablets, are also used.

[0941] Software to use

[0942] 1. Natural Language Processing (NLP): A technique for analyzing human language and understanding its meaning. As a concrete example, we will use the Transformers library (provided by Hugging Face).

[0943] 2. Image Recognition Technology: A technology for analyzing image data. As a specific example, we will use OpenCV (an open-source computer vision library).

[0944] 3. Database: A system for storing data on problems and solutions. Specific examples include MySQL and PostgreSQL.

[0945] Overall system operation

[0946] The system receives problems entered by users, identifies their characteristics and categories, and then provides similar problems and solutions based on that. Users can enter problems in text or image format, and the system analyzes the input to provide the best solution.

[0947] Specific example

[0948] For example, if a factory operator enters a problem in text format such as "The robot suddenly stopped," the system analyzes the problem and searches its database for similar problems and their solutions. This provides specific solutions such as "Check the robot's main switch."

[0949] Furthermore, if a learner is unable to solve a math problem such as "x^2 - 4x + 4 = 0", they can input the problem in text or image format. The system will then identify the problem's category (e.g., quadratic equations) and search its database for similar problems and their solutions.

[0950] Examples of prompt statements

[0951] "The robot suddenly stopped. What should I do?"

[0952] "I can't solve the problem x^2 - 4x + 4 = 0. Please help me."

[0953] This invention allows users to obtain problem-solving solutions quickly and accurately, and is expected to improve learning and operational efficiency. Furthermore, it can record the user's operation history and optimize future problem-solving processes for greater efficiency.

[0954] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0955] Step 1:

[0956] The user inputs the problem using a terminal. The user inputs the problem in text or image format and presses the submit button. There is no specific format required for the problem the user inputs. Examples of input include sentences such as "The robot suddenly stopped" or images of mathematical formulas such as "x^2 - 4x + 4 = 0". The input data is sent to the server.

[0957] Step 2:

[0958] The server saves the received problems. The server records the problem data submitted by the user in a database. The problem data is associated with the problem content and the user ID. This saving step allows for future analysis of the problems and tracking of learning history.

[0959] Step 3:

[0960] The server analyzes the problem. It analyzes the received problem and uses natural language processing (NLP) or image recognition techniques to identify the problem's characteristics (e.g., difficulty, format) and category (e.g., mathematics, mechanical failure). For example, the text "The robot stopped" would be classified into the categories "robot" and "operation stopped" using NLP. The analysis results are temporarily stored in memory.

[0961] Step 4:

[0962] The server searches for or generates similar problems and explanations. Based on the analysis results, it searches the database for similar problems and their explanations. In this process, it can also generate new explanations using a generation AI model. From the search results or generated results, the most suitable similar problem and explanation are selected. For example, an explanation such as "the possibility of the robot stopping at the main switch" might be obtained.

[0963] Step 5:

[0964] The server sends similar problems and explanations to the user's terminal. Similar problems and their explanations, whether found or generated, are sent to the user's terminal. The user can then view this information on their terminal. For example, an explanation such as "Check the robot's main switch" might be displayed.

[0965] Step 6:

[0966] The user implements the suggested solution. The user follows the instructions displayed on the terminal and takes specific action. For example, the operator checks the main switch of the robot, following the instruction "Check the robot's main switch."

[0967] Step 7:

[0968] The user submits their solution or problem-solving result. After implementing the suggested solution, the user enters the result on their device and sends it to the server. For example, they might enter feedback such as, "I checked the main switch, but the robot is not working."

[0969] Step 8:

[0970] The server records the user's progress and optimizes the next approach. Received solutions are stored in a database and managed as learning or operation history. Based on this history, the server generates information to optimize future problem-solving and learning plans. This improves the user's skills and problem-solving efficiency.

[0971] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0972] System Overview

[0973] This system aims to allow learners to learn at their own pace and provide effective supplementary lessons, especially for areas where they struggle. A new emotion engine has been added, which recognizes and analyzes learners' emotions and adjusts the learning content accordingly. This enables efficient learning while maintaining learner motivation.

[0974] User actions

[0975] Users utilize an interface to input problems they were unable to solve. They input and submit problems in text or image format. At the same time, facial expressions and voice are recorded via camera and microphone to capture user emotion data.

[0976] Server operation

[0977] 1. Receiving and saving the problem:

[0978] Issues submitted by users are received by the server and stored in the database. In this step, the user ID, issue, and sentiment data are stored together.

[0979] 2. Problem Analysis:

[0980] The server analyzes the stored problems to identify their characteristics (e.g., difficulty, format) and category (e.g., mathematics, factorization). This analysis uses natural language processing (NLP) and image recognition technologies.

[0981] 3. Analysis of emotional data:

[0982] The emotion engine analyzes recorded emotion data to identify the learner's emotional state (e.g., stress level, motivation). This data is used in conjunction with the problem analysis results.

[0983] 4. Generate or search for similar problems and explanations:

[0984] Based on the characteristics and category of the problem, as well as sentiment data, the server searches the database for similar problems and their explanations, or generates new ones. Depending on the sentiment data, the difficulty level may be adjusted and the depth of the explanations may be changed.

[0985] 5. Provision of problems and explanations:

[0986] The server generates or searches for similar problems and their explanations, adjusts them appropriately based on sentiment data, and sends them to the user's terminal. This allows the user to learn with content that is appropriate to their current emotional state.

[0987] Specific example

[0988] For example, if a user is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", they input this problem into the system. The server then receives the problem and analyzes its characteristics and categories. In this case, the problem is classified as a "quadratic equation" and "factorization". Simultaneously, the emotion engine detects "anxiety" from the user's facial expressions and voice.

[0989] The server then searches for similar problems in the "quadratic equations" and "factorization" categories and prepares appropriate explanations. Based on the detected sentiment, the server provides explanations in a clearer, step-by-step manner. For example, problems such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0" are selected, and their solutions and explanations are sent to the user.

[0990] Users work on these similar problems and input their answers into their device. The device then retrieves sentiment data along with these answers and sends it to the server. The server records and analyzes the answers in a database. The next learning plan is further optimized based on the recorded answers and sentiment data.

[0991] Tracking learning progress

[0992] The user's answers to problems are sent from their device to the server. The server receives this data and records it in a database. This record is managed as the user's learning history and used to optimize the learning plan for the next learning session. Since emotional data is also recorded, a learning plan is created that takes into account the learner's motivation and emotional changes. As a result, users can not only visualize their own progress but also receive individually customized learning content.

[0993] This entire process allows learners to have an optimal learning experience and to progress through their studies efficiently. The introduction of an emotion engine improves the quality of learning by providing personalized support that takes into account the learner's emotional state.

[0994] The following describes the processing flow.

[0995] Step 1:

[0996] The user inputs and submits a problem they were unable to solve using their device. The user inputs the problem in text or image format through a dedicated interface and presses the submit button. During this process, the user's facial expressions and voice data are simultaneously recorded using the camera and microphone.

[0997] Step 2:

[0998] The terminal sends the entered problem data, along with the user's facial expressions and voice data, to the server. The problem data is transmitted with the user ID associated with it.

[0999] Step 3:

[1000] The server receives problem data sent from the terminal and saves it to the database. During saving, the user ID is associated with the problem data.

[1001] Step 4:

[1002] The server analyzes the problem data it receives. Natural language processing (NLP) and image recognition technologies are used to identify the characteristics and categories of the problems. These characteristics include the difficulty level and format of the problem.

[1003] Step 5:

[1004] The server analyzes facial expressions and voice data to identify the user's emotional state. The emotion engine evaluates the user's stress level and motivation. This data is also stored in a database.

[1005] Step 6:

[1006] The server searches the database for similar problems and their explanations based on the problem's characteristics and category, as well as sentiment data. It may also generate new problems and explanations as needed.

[1007] Step 7:

[1008] The server adjusts the searched or generated similar problems and their explanations to match the user's emotional state. For example, if the user is showing a high stress level, it will start with easier problems and adjust the explanations to be more gradual.

[1009] Step 8:

[1010] The server sends similar problems and their explanations, tailored to the user's device. The user receives these problems and explanations on their device and continues learning.

[1011] Step 9:

[1012] Users work on similar problems submitted via their devices and input their answers. After solving the problems, sentiment data is recorded again.

[1013] Step 10:

[1014] The device sends the user's answers, facial expressions, and voice data to the server. This data includes the user ID.

[1015] Step 11:

[1016] The server receives the answer results and saves them to the database. The user ID, question ID, answer result, and sentiment data are stored together.

[1017] Step 12:

[1018] The server analyzes the user's learning progress based on stored answer results and sentiment data. This analysis evaluates the user's level of understanding and learning progress.

[1019] Step 13:

[1020] The server optimizes the next learning plan. Based on the user's learning progress and sentiment data, it selects the next problems and explanations to tackle and prepares individually optimized learning content.

[1021] Through the steps outlined above, this system provides learners with an individually optimized learning experience, supporting efficient learning. The introduction of an emotion engine enables personalized learning support that takes into account the learner's emotional state.

[1022] (Example 2)

[1023] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1024] Traditional learning support systems often provide problems and explanations without considering the learner's emotional state, making it difficult to provide an optimal learning experience for each individual learner. This can lead to decreased learner motivation and hinder efficient learning.

[1025] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving problems that the learner could not solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for simultaneously recording the learner's emotional data when receiving the problems, means for analyzing the emotional data and identifying the learner's emotional state, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems and the emotional data, means for providing the generated or searched similar problems and their explanations to the learner, and means for recording the learner's answer results and emotional data and optimizing the next learning plan. This makes it possible to provide effective learning support and improve motivation in accordance with the learner's emotional state.

[1026] "Learner" refers to an individual who uses the system to engage in learning activities.

[1027] A "problem" refers to an academic or educational task that learners attempt to solve.

[1028] "Emotional data" refers to data that indicates an emotional state, extracted from the learner's facial expressions and voice.

[1029] A "server" refers to a computer system that is responsible for analyzing, processing, and storing the data it receives.

[1030] A "database" refers to a collection of information that a server uses to efficiently store, manage, and retrieve data.

[1031] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human language.

[1032] "Image recognition technology" refers to the technology used by computers to extract and analyze information from images.

[1033] An "emotion engine" refers to software or hardware that analyzes a learner's recorded emotional data to identify their emotional state.

[1034] "Analysis" refers to the process of examining data and information in detail and using that information to identify characteristics or specific patterns.

[1035] A "similar problem" refers to another problem that has similar characteristics or categories to the problem the learner was unable to solve.

[1036] "Explanation" refers to content that explains to learners how to solve problems and related knowledge.

[1037] A "learning plan" refers to a plan that proposes the optimal learning content and schedule based on the learner's progress and abilities.

[1038] "Optimization" refers to adjusting a system or plan to its most effective state in order to achieve a specific objective.

[1039] Modes for carrying out the invention

[1040] This invention relates to a system that collects problems that learners were unable to solve and data on the learners' emotions, analyzes this data, and provides appropriate learning support. Specifically, the system involves a server receiving problems and emotion data sent by learners, analyzing them, and optimizing the learning plan.

[1041] Hardware and software to be used

[1042] This system uses the following hardware and software:

[1043] 1. Terminal:

[1044] These are devices used by learners, such as PCs, smartphones, and tablets.

[1045] It is desirable to have a camera and microphone, as well as sensors for recording emotional data.

[1046] 2. Server:

[1047] It is a high-performance computer system that stores, analyzes, and processes data.

[1048] The system efficiently manages learner data by using an internal or external database.

[1049] 3. Natural Language Processing Techniques:

[1050] We will use the Python NLP library "NLTK" or similar tools to analyze text data.

[1051] 4. Image recognition technology:

[1052] We use libraries such as "OpenCV" to perform image analysis.

[1053] 5. Emotional Engine:

[1054] Software such as "Emotion API" is used to analyze the learner's facial expressions and voice to identify their emotional state.

[1055] 6. Generative AI Models:

[1056] The "GPT-3" model and its successor models are used to generate similar problems and explanations.

[1057] System Description

[1058] User actions

[1059] Users use their devices to input problems they couldn't solve into the interface. Problems can be entered in text or image format, and simultaneously, facial expressions and voice are recorded via the camera and microphone. This also collects emotional data.

[1060] As a concrete example, if a user is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", they enter the problem into a text box and click the "Submit" button. At this time, the camera records the user's facial expressions and the microphone captures their voice.

[1061] Server operation

[1062] The server receives problem data and sentiment data submitted by users. The problem data is stored in a database and analyzed using natural language processing techniques (e.g., NLTK) and image recognition techniques (e.g., OpenCV). The analysis identifies the characteristics of the problem (e.g., difficulty level, format) and its category (e.g., quadratic equation, factorization).

[1063] Simultaneously, the emotion engine analyzes recorded facial expressions and voice to identify the learner's emotional state (e.g., anxiety, stress). This identified emotional state information is used in conjunction with the problem analysis results.

[1064] Based on these analysis results, the server searches the database for similar problems and their explanations, or generates new problems using a newly generated AI model (e.g., GPT-3). The difficulty level and depth of the explanations are adjusted according to the sentiment data.

[1065] Specific example

[1066] For example, if a user inputs the problem "x^2 - 4x + 4 = 0" and their emotional state is identified as "anxiety," the server will prepare similar problems and their explanations from the "quadratic equations" and "factorization" categories. In this case, the explanations will be provided in a step-by-step manner, based on the emotional data. Specifically, problems such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0" will be selected, and the solutions to each will be explained in detail.

[1067] Example of a prompt

[1068] Examples of prompts to input into a generative AI model include the following:

[1069] To help a student who is struggling with the quadratic equation "x^2 - 4x + 4 = 0" and is feeling frustrated, please generate and provide the following problem and explanation. Please include several similar problems and explain the solution to each step-by-step.

[1070] This allows the server to provide a learning experience optimized for the learner's emotional state, thereby maintaining motivation and improving learning efficiency.

[1071] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1072] Step 1:

[1073] The user inputs the problem they were unable to solve into the device's interface. Specifically, the user enters the problem into a text box or uploads an image file. Simultaneously, the camera and microphone automatically activate to record the user's facial expressions and voice. The input data includes problem data (text or image) and emotion data (facial expressions, voice).

[1074] Step 2:

[1075] The terminal sends user-entered problem data and recorded sentiment data to the server. This data includes user ID, problem data, and sentiment data. This provides the server with the necessary input for subsequent analysis.

[1076] Step 3:

[1077] The server stores the received problem data and analyzes the text data using natural language processing techniques (such as NLTK). It also analyzes image data using image recognition techniques (such as OpenCV). As a result of the analysis, the characteristics of the problem (difficulty level, format) and its category (e.g., mathematics, factorization) are identified. The input is the problem data, and the output is the problem characteristics and category information.

[1078] Step 4:

[1079] The server uses an emotion engine (such as the Emotion API) to analyze the recorded emotion data. This analysis identifies the learner's emotional state (e.g., stress level, motivation). The input is the emotion data, and the output is the identified emotional state.

[1080] Step 5:

[1081] The server searches the database for similar problems and their explanations, or generates new ones, based on the problem's characteristics, category, and sentiment data. A generative AI model (such as GPT-3) is used for generation. Specifically, prompts are used to instruct the generative AI model to obtain similar problems and their explanations. The input consists of the problem's characteristics, category, and sentiment state, while the output is similar problems and their explanations.

[1082] Step 6:

[1083] The server sends generated or retrieved similar problems and their explanations to the user's terminal. Based on sentiment data, the difficulty level and depth of the explanations are adjusted, so learners receive learning content that is best suited to their state. The input is similar problems and explanations, and the user's terminal information; the output is the transmission of the adjusted learning content.

[1084] Step 7:

[1085] The user works on a similar problem provided and inputs their answer into the terminal. The terminal then sends the answer and recorded sentiment data back to the server. The input is the user's answer and sentiment data, and the output is the data sent to the server.

[1086] Step 8:

[1087] The server records the received answers and sentiment data in a database. This optimizes the next learning plan. Specifically, it uses an adaptive learning algorithm to analyze the learner's progress and sentiment state and propose the optimal content for the next learning session. The input is the answers and sentiment data, and the output is the optimized next learning plan.

[1088] (Application Example 2)

[1089] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1090] Traditional learning systems provided learning content without considering learners' emotions, resulting in insufficient maintenance of learner motivation and optimization of learning effectiveness. Furthermore, in in-store customer service support, it was difficult to grasp customers' emotional states in real time and respond appropriately. Consequently, improvements in customer satisfaction and effective product recommendations were not fully achieved. This system aims to solve these problems.

[1091] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving problems that the learner could not solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, means for providing the generated or searched similar problems and their explanations to the learner, means for recognizing and analyzing the learner's emotional state, and means for appropriately adjusting the learning content based on the emotional state. This makes it possible to provide personalized learning content that takes into account the learner's emotional state, thereby maintaining motivation and optimizing learning effectiveness. Furthermore, even in physical stores, customer satisfaction can be improved by analyzing the customer's emotional state in real time and providing appropriate customer service methods.

[1092] "A means of receiving problems that learners were unable to solve" refers to an interface for learners to input or upload problems they were unable to solve.

[1093] "Means for identifying the characteristics and categories of a problem" refers to a function that analyzes a received problem and identifies its characteristics and classification, such as difficulty level, format, and subject matter.

[1094] "Means for generating or searching for similar problems and their explanations" means a function that generates new similar problems and their explanations, or searches for them in an existing database, based on the aforementioned characteristics and categories.

[1095] "Means to provide to learners" means the function of displaying or transmitting generated or retrieved similar problems and their explanations to the learner's device.

[1096] "Means for recognizing and analyzing emotional states" refers to a function that analyzes emotions from a learner's facial expressions and voice data and identifies their emotional state (e.g., stress, motivation).

[1097] "Means for appropriately adjusting learning content based on emotional state" refers to a function that dynamically adjusts learning content and problem difficulty levels according to analyzed emotional data, providing an optimal learning plan tailored to the learner.

[1098] "Natural language processing technology" is a general term for algorithms and methods used to analyze text data and identify its characteristics and categories.

[1099] "Image recognition technology" is a general term for algorithms and methods used to analyze image data and identify its characteristics and categories.

[1100] "Facial expression analysis technology" is a general term for technologies that analyze facial expressions from image data acquired by a camera and estimate emotions from the results.

[1101] "Voice analysis technology" is a general term for technologies that analyze voice data acquired by a microphone and estimate emotions from the results.

[1102] System Configuration

[1103] This invention is a system consisting of the following main elements:

[1104] 1. Device: A smartphone used by learners, store clerks, etc.

[1105] 2. Server: A server that performs data analysis, generates learning content, sentiment analysis, etc.

[1106] 3. Camera and microphone: Devices for acquiring facial expressions and voice data.

[1107] Program processing

[1108] For learners

[1109] 1. Inputting the problem: Learners input the problems they were unable to solve into their device in text or image format and submit them. At this time, the camera and microphone are used to collect learner sentiment data.

[1110] 2. Problem Analysis: The server analyzes the received problem using natural language processing (NLP) or image recognition techniques to identify its characteristics and categories.

[1111] 3. Analysis of emotional data: The server uses data acquired from the camera and microphone to analyze the learner's emotional state using facial expression analysis and voice analysis technologies.

[1112] 4. Generation and search of similar problems and explanations: Based on the characteristics of the problem and the sentiment analysis results, the server searches the database for similar problems and their explanations or generates new ones.

[1113] 5. Provision of tailored learning content: The server sends optimized problems and explanations based on the analysis data to the learner's terminal, which the learner then works on.

[1114] In the case of customer service support at physical stores

[1115] 1. Acquisition of customer emotion data: While serving customers, store employees use the camera and microphone on their devices to capture the customer's facial expressions and voice, thereby acquiring emotion data.

[1116] 2. Analysis of emotional data: Based on the acquired data, the server analyzes the customer's emotional state in real time using facial expression analysis technology and voice analysis technology.

[1117] 3. Adjusting responses: Based on the analyzed sentiment data, the server provides appropriate responses and product suggestions that the store staff should take, and these are displayed on the terminal.

[1118] Hardware and software to be used

[1119] Hardware: Smartphone, camera, microphone.

[1120] Software: OpenCV (video acquisition and display), DeepFace (facial expression analysis), natural language processing technology (NLP, text analysis), image recognition technology (image analysis).

[1121] Examples of specific cases and prompt statements

[1122] Specific example

[1123] For example, if a customer is having trouble choosing a product during a consultation at a physical store, the system uses the customer's smartphone camera and microphone to analyze their emotional state. If the analysis reveals "anxiety" or "unease," the system instructs the salesperson to take actions such as "provide a detailed product description" or "suggest alternative options."

[1124] Example of a prompt

[1125] Develop an application that analyzes user emotions and suggests appropriate customer service methods. Implement the emotion analysis function shown in the following code:

[1126] 1. The camera captures facial images, and DeepFace is used to analyze facial expressions.

[1127] 2. Based on the emotional data obtained, we adjust the customer service methods and recommended products to best suit each customer.

[1128] As described above, in the embodiment of this invention, it is possible to analyze user input information and emotional data and dynamically provide appropriate learning content and customer service responses. As a result, improved learning effectiveness and increased customer satisfaction can be expected.

[1129] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1130] Step 1:

[1131] The user inputs a problem they were unable to solve. The input problem is in text or image format and is sent from the learner's device to the server. At the same time, the learner's facial expressions and voice data are captured using the device's camera and microphone and sent to the server as emotion data.

[1132] Input: Problems the learner was unable to solve, learner's facial expression data, learner's voice data

[1133] Output: Problem data and sentiment data sent to the server

[1134] Specific steps: The learner opens a dedicated application on their smartphone, enters or photographs the question, and clicks the submit button. Simultaneously, video and audio data is captured.

[1135] Step 2:

[1136] The server analyzes the problem data it receives. Natural language processing (NLP) and image recognition techniques are used to identify the characteristics of the problem (difficulty level, format, etc.) and its category (e.g., mathematics, factorization).

[1137] Input: Problem data

[1138] Output: Problem characteristics and categories

[1139] Specific operation: The server analyzes the received text or image data using NLP and image recognition algorithms to identify the nature of the problem.

[1140] Step 3:

[1141] The server analyzes emotional data. Using facial expression analysis and voice analysis technologies, it identifies the learner's emotional state (e.g., stress level, motivation) from the received video and audio data.

[1142] Input: Facial expression data, voice data

[1143] Output: Emotional state

[1144] Specific operation: The server uses DeepFace and other facial expression analysis software to analyze the learner's video data and identify the dominant emotion. Voice analysis software is also used in conjunction to estimate the emotional state with high accuracy.

[1145] Step 4:

[1146] Based on the characteristics and category of the problem, as well as the sentiment analysis results, the server generates or searches for similar problems and their explanations in the database. The level of detail and difficulty of the explanation are adjusted according to the emotional state.

[1147] Input: Problem characteristics and categories, emotional state

[1148] Output: Similar problems and their explanations

[1149] Specific operation: The server searches the database using problem characteristic data and category data to find similar problems and their explanations. It may also generate new similar problems and explanations using a generative AI model. Based on the results of sentiment analysis, it automatically adjusts the difficulty and detail of the explanations.

[1150] Step 5:

[1151] The server sends similar problems and their explanations, generated or retrieved, to the learner's device. This allows the learner to progress through the learning process with content that best suits their current emotional state.

[1152] Input: Similar problems and their explanations

[1153] Output: Learning content sent to the learner's device.

[1154] Specific operation: The server sends the formatted data to the learner's device and displays it on the device's application. During this process, notification functions and interface optimization are performed.

[1155] Step 6:

[1156] The learner works on additional learning material and enters their answers into the device. The device sends the answers and newly acquired sentiment data to the server.

[1157] Input: Answer results, additional sentiment data

[1158] Output: Answer results and additional sentiment data sent to the server

[1159] Specific operation: The learner works on similar problems provided and enters their answers. Simultaneously, the device's camera and microphone capture the learner's facial expressions and voice again.

[1160] Step 7:

[1161] The server receives the answer results and additional sentiment data, and records them in a database as the learner's learning history. This allows for further optimization of the next learning plan.

[1162] Input: Answer results, additional sentiment data

[1163] Output: Update of the learning history database

[1164] Specific actions: The server analyzes the answer results and sentiment data, and updates parameters to adjust the next learning plan. It adds new entries to the database and optimizes the learning history.

[1165] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1166] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1167] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1168] [Fourth Embodiment]

[1169] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1170] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1171] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1172] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1173] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1174] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1175] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1176] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1177] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1178] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1179] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1180] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1181] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1182] System Overview

[1183] This system aims to allow learners to study at their own pace and provide effective supplementary lessons, especially for problems they find difficult. This includes a process of inputting problems that learners were unable to solve, analyzing those problems, generating or searching for similar problems and explanations, and then providing them to the learners.

[1184] User actions

[1185] Users utilize an interface to input problems they were unable to solve. They can enter and submit problems in text or image format. Users are not required to input problems in a specific format; the system is designed to appropriately analyze problems using natural language processing (NLP) and image recognition technologies.

[1186] Server operation

[1187] 1. Receiving and saving the problem:

[1188] The issues submitted by the user are received by the server and stored in the database. In this step, the user ID and the issue are associated and stored together.

[1189] 2. Problem Analysis:

[1190] The server analyzes the stored problems to identify their characteristics (e.g., difficulty level, problem format) and category (e.g., mathematics, factorization). This analysis utilizes NLP and image recognition technologies.

[1191] 3. Generate or search for similar problems and explanations:

[1192] Based on the characteristics and category of the problem, the server searches the database for similar problems and their explanations, or generates new ones. In doing so, the server utilizes the existing problem database to select the most suitable similar problems and explanations.

[1193] 4. Provision of problems and explanations:

[1194] The server sends generated or searched similar problems and their explanations to the user's terminal. This allows the user to work on new problems and deepen their understanding by referring to the explanations.

[1195] Specific example

[1196] For example, if a user is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", the user inputs this problem into the system. The server then receives the problem and analyzes its characteristics and category. In this case, the problem is classified as a "quadratic equation" and "factorization". Next, the server searches the database for similar quadratic equations such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0", and prepares the solutions and explanations for each.

[1197] This information is sent to the user's device, allowing them to work on similar problems. Furthermore, each time a user submits an answer, the result is recorded on the server, optimizing the next learning plan. This entire process allows users to focus on problems they find particularly difficult, resulting in more effective learning.

[1198] Tracking learning progress

[1199] The user's answers to problems are sent from the device to the server. The server receives this data and records it in a database. This record is managed as the user's learning history and used to optimize the learning plan for the next learning session. This allows users to visualize their progress and receive individually customized learning content.

[1200] Through the above process, this system can provide learners with an individually optimized learning experience and support efficient learning.

[1201] The following describes the processing flow.

[1202] Step 1:

[1203] Users input and submit problems they were unable to solve using their devices. Users input problems in text or image format through a dedicated interface. By pressing the submit button, the device sends the input data to the server.

[1204] Step 2:

[1205] The server receives the problem sent from the terminal and saves it to the database. When saving, the user ID is associated with the problem.

[1206] Step 3:

[1207] The server analyzes the stored problems. This analysis uses natural language processing (NLP) and image recognition technologies. As a result of the analysis, the characteristics of the problem (e.g., difficulty level, format) and category (e.g., mathematics, factorization) are identified.

[1208] Step 4:

[1209] The server generates or searches for similar problems and their explanations based on the analysis results. First, it searches the database for existing problems that match the characteristics and categories. If no suitable problem is found, it may generate a new similar problem. Similarly, explanations are either retrieved from the existing database or newly created.

[1210] Step 5:

[1211] The server sends similar problems, generated or found, along with their explanations, to the user's terminal. This allows the user to receive materials to work on similar problems.

[1212] Step 6:

[1213] Users work on similar problems sent via their devices. They answer the problems and input their answers into their devices.

[1214] Step 7:

[1215] The device sends the user's answer to the server. This transmission includes not only the answer but also which question the answer corresponds to.

[1216] Step 8:

[1217] The server receives the user's answer and records it in the database. The user ID, question ID, and answer are stored together in an associated manner.

[1218] Step 9:

[1219] The server analyzes the user's learning progress based on the saved answer results. This analysis evaluates the user's level of understanding and learning progress.

[1220] Step 10:

[1221] The server optimizes the next learning plan. Based on the user's learning progress and answer results, it selects the next problems and explanations to tackle and prepares individually optimized learning content.

[1222] Through these steps, the system provides learners with an individually optimized learning experience, supporting efficient learning.

[1223] (Example 1)

[1224] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1225] Traditional learning systems struggled to efficiently provide supplementary lessons tailored to individual learners' weak areas, hindering learners from effectively progressing at their own pace. Furthermore, they often lacked adequate explanations for unsolved problems and insufficient provision of similar exercises, leading to decreased learning efficiency. Additionally, they lacked features to track learners' progress and optimize subsequent learning plans.

[1226] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1227] In this invention, the server includes means for receiving problems that the learner was unable to solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, means for recording the learner's answer results and optimizing the next learning plan, and means for generating similar problems and their explanations using a generative AI model. This allows learners to learn efficiently at their own pace and receive effective supplementary lessons, especially for problems they find difficult.

[1228] A "learner" is an individual who seeks to acquire knowledge and skills through an educational program.

[1229] A "problem" is a question or task presented to learners with the expectation that they will answer it.

[1230] "Analysis" is the process of analyzing the content and characteristics of a problem and classifying it into specific attributes or categories.

[1231] "Characteristics" refer to specific features or attributes of a problem, such as its difficulty level or format.

[1232] A "category" is a classification framework used to categorize the academic field or type of problem to which it belongs.

[1233] A "similar problem" is another problem that has similar characteristics or categories to the problem that has been analyzed.

[1234] "Explanation" refers to explanations and supplementary information that help solve or understand a problem.

[1235] "Generation" refers to the artificial creation of new problems or explanations.

[1236] "Searching" is the process of finding information that matches specific criteria from an existing database.

[1237] A "learning plan" refers to a learning schedule and content optimized based on the learner's progress and level of understanding.

[1238] "Natural language processing technology" refers to the technology that enables computers to understand, process, and generate human language.

[1239] "Image recognition technology" is a technology that analyzes image data and enables computers to understand its content.

[1240] A "generative AI model" is a model that uses artificial intelligence technology to generate problems and explanations.

[1241] "Answer results" refer to information such as the answers that learners gave to the questions and whether or not they were correct.

[1242] Modes for carrying out the invention

[1243] System Overview

[1244] This invention relates to a system that allows learners to study at their own pace and provides effective supplementary lessons for specific areas of difficulty. Specifically, it includes a series of processes in which learners input problems they were unable to solve, the system analyzes them, and provides similar problems and explanations. The system can also record the learner's answers and optimize their next learning plan.

[1245] Server operation

[1246] The core of this system resides in the server. The server has the following main functions:

[1247] 1. Receiving and saving the problem:

[1248] The server receives the questions submitted by learners and saves them to a database. By associating the questions with the user ID, the server tracks the progress of individual learners.

[1249] 2. Problem Analysis:

[1250] The server analyzes received and stored problems using natural language processing (NLP) and image recognition technologies. This analysis identifies the problem's characteristics (difficulty level, problem format) and category (mathematics, factorization, etc.). NLP technologies such as Spacy and BERT are used, while image recognition technologies such as OpenCV and TensorFlow are employed.

[1251] 3. Generate or search for similar problems and explanations:

[1252] Based on the analysis results, the server searches the database for similar problems and their explanations. Alternatively, it uses a generative AI model to generate new problems and explanations. Advanced models such as 'GPT-3' and 'GPT-4' are used for generative AI modeling.

[1253] 4. Provision of problems and explanations:

[1254] The server sends generated or retrieved similar problems and their explanations to the learner's terminal. This allows the learner to work on new problems and deepen their understanding by referring to the explanations.

[1255] 5. Tracking learning progress:

[1256] The server receives the results of the questions answered by the learner and records them in a database. This data is used to optimize the next learning plan.

[1257] Terminal operation

[1258] Learners will perform the following operations using their own devices.

[1259] 1. Enter the question:

[1260] Learners use the interface to input problems they were unable to solve. These problems can be entered in text or image format.

[1261] 2. Submit your answer:

[1262] Each time a learner submits an answer to a problem, the result is automatically sent to the server.

[1263] Specific example

[1264] For example, if a learner is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", they input the problem into the system. The server receives the problem and analyzes its characteristics and category. In this case, the problem is classified as a "quadratic equation" and "factorization". Next, the server searches the database for similar quadratic equations such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0", and prepares the solutions and explanations for each. This information is sent to the learner's terminal, allowing them to work on these similar problems.

[1265] Example of a prompt

[1266] When using a generative AI model to perform problem analysis or similar problem searches, the following prompt statements can be input to the generative AI model.

[1267] Please analyze the following quadratic equation problem that the user was unable to solve: "x^2 - 4x + 4 = 0". Identify the characteristics of this problem (difficulty level, category, etc.) and generate similar problems and their solutions.

[1268] This process allows the system to provide learners with a learning experience that is individually optimized, supporting effective learning.

[1269] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1270] Step 1:

[1271] User Action: The user inputs the problem they were unable to solve through the interface. Input can be in text or image format. The input data is sent to the system.

[1272] Input: The text or image data of the problem.

[1273] Output: Problem data sent to the server

[1274] Step 2:

[1275] Server operation: The server receives problem data sent by the user and stores it in the database. It associates the user ID with the problem data and registers it.

[1276] Input: Problem data submitted by the user

[1277] Output: Problem data and user ID stored in the database

[1278] Step 3:

[1279] Server Operation: The server analyzes stored problems using NLP and image recognition technologies. It identifies the characteristics of the problem (difficulty level, problem format) and its category (e.g., mathematics, factorization). For this analysis, NLP technologies such as 'Spacy' and 'BERT' are used, and image recognition technologies such as 'OpenCV' and 'TensorFlow' are used.

[1280] Input: Saved problem data

[1281] Output: Problem characteristics and categories

[1282] Step 4:

[1283] Server operation: Based on the analysis results, the server searches the database for similar problems and their explanations, or generates new ones using a generation AI model (e.g., 'GPT-3' or 'GPT-4'). By inputting generation prompts into the AI ​​model, similar problems and explanations are generated.

[1284] Input: Problem characteristics and categories

[1285] Output: Similar problems and explanations

[1286] Example prompt:

[1287] "Analyze the following quadratic equation problem that the user was unable to solve: 'x^2 - 4x + 4 = 0'. Identify the characteristics of this problem (difficulty level, category, etc.) and generate similar problems and their solutions."

[1288] Step 5:

[1289] Server operation: The server sends similar problems generated or found, along with their explanations, to the user's terminal. This allows the user to refer to them and work on new problems.

[1290] Input: Similar problems and explanations

[1291] Output: Similar problems and explanations sent to the user's terminal.

[1292] Step 6:

[1293] User actions: The user works on similar problems submitted to them and submits their solutions through the interface.

[1294] Input: Data provided by the user

[1295] Output: Answer data sent to the server

[1296] Step 7:

[1297] Server operation: The server receives answer data submitted by users and records it in a database. This record is used to optimize the next learning plan.

[1298] Input: User-submitted answer data

[1299] Output: Answer results and learning progress recorded in the database

[1300] Through the above processing steps, the system can provide learners with appropriate supplementary lessons and support effective learning.

[1301] (Application Example 1)

[1302] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1303] Conventional learning support systems and operational management systems have struggled to effectively and quickly resolve problems faced by users. Furthermore, these systems lacked the functionality to adequately support user skill development. As a result, users repeatedly encountered the same problems, leading to decreased learning and operational efficiency. This invention aims to solve these problems.

[1304] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1305] In this invention, the server includes means for receiving problems that learners were unable to solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, means for providing the generated or searched similar problems and their explanations to the learners, means for analyzing mechanical problems and error codes entered by an operator and providing similar solutions and explanations, an interface for users to input problems in text or image format, means for analyzing the input problems using natural language processing or image recognition technology, means for providing the user with the optimal solution based on the analysis results, and means for recording the user's operation history and optimizing the next solution. This enables users to obtain quick and accurate solutions to problems they face, effectively supporting skill improvement.

[1306] A "learner" refers to a user who uses the system to improve their knowledge and skills.

[1307] A "problem" refers to a question that learners cannot solve, or one that illustrates a mechanical error or situation that an operator might encounter.

[1308] "Analysis" refers to the process of identifying the characteristics and categories of a problem received and understanding its content.

[1309] A "similar problem" refers to another problem that has the same or very similar characteristics or categories as the problem being analyzed.

[1310] "Explanation" refers to information that provides a detailed explanation or solution to a problem.

[1311] An "operator" refers to a person responsible for operating factory robots, operational equipment, and other similar devices.

[1312] "Natural language processing technology" refers to technologies for analyzing and understanding human language.

[1313] "Image recognition technology" refers to the technology of extracting and analyzing specific information from images.

[1314] "Interface" refers to the screen or means by which a user inputs information about a problem.

[1315] A "server" refers to a computer system that stores, analyzes, and responds to users' data.

[1316] "Operation history" refers to a record of operations and inputs performed by a user using the system.

[1317] This invention is a system for analyzing problems faced by learners and factory operators and providing appropriate solutions. This system is implemented using the following hardware and software.

[1318] Hardware to use

[1319] 1. Server: A system capable of high-speed data processing and large-scale data storage. Specific examples include using cloud-based servers such as AWS or Google Cloud.

[1320] 2. User terminals: Devices such as smartphones, tablets, and PCs. In factory operations, dedicated operating terminals, such as industrial tablets, are also used.

[1321] Software to use

[1322] 1. Natural Language Processing (NLP): A technique for analyzing human language and understanding its meaning. As a concrete example, we will use the Transformers library (provided by Hugging Face).

[1323] 2. Image Recognition Technology: A technology for analyzing image data. As a specific example, we will use OpenCV (an open-source computer vision library).

[1324] 3. Database: A system for storing data on problems and solutions. Specific examples include MySQL and PostgreSQL.

[1325] Overall system operation

[1326] The system receives problems entered by users, identifies their characteristics and categories, and then provides similar problems and solutions based on that. Users can enter problems in text or image format, and the system analyzes the input to provide the best solution.

[1327] Specific example

[1328] For example, if a factory operator enters a problem in text format such as "The robot suddenly stopped," the system analyzes the problem and searches its database for similar problems and their solutions. This provides specific solutions such as "Check the robot's main switch."

[1329] Furthermore, if a learner is unable to solve a math problem such as "x^2 - 4x + 4 = 0", they can input the problem in text or image format. The system will then identify the problem's category (e.g., quadratic equations) and search its database for similar problems and their solutions.

[1330] Examples of prompt statements

[1331] "The robot suddenly stopped. What should I do?"

[1332] "I can't solve the problem x^2 - 4x + 4 = 0. Please help me."

[1333] This invention allows users to obtain problem-solving solutions quickly and accurately, and is expected to improve learning and operational efficiency. Furthermore, it can record the user's operation history and optimize future problem-solving processes for greater efficiency.

[1334] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1335] Step 1:

[1336] The user inputs the problem using a terminal. The user inputs the problem in text or image format and presses the submit button. There is no specific format required for the problem the user inputs. Examples of input include sentences such as "The robot suddenly stopped" or images of mathematical formulas such as "x^2 - 4x + 4 = 0". The input data is sent to the server.

[1337] Step 2:

[1338] The server saves the received problems. The server records the problem data submitted by the user in a database. The problem data is associated with the problem content and the user ID. This saving step allows for future analysis of the problems and tracking of learning history.

[1339] Step 3:

[1340] The server analyzes the problem. It analyzes the received problem and uses natural language processing (NLP) or image recognition techniques to identify the problem's characteristics (e.g., difficulty, format) and category (e.g., mathematics, mechanical failure). For example, the text "The robot stopped" would be classified into the categories "robot" and "operation stopped" using NLP. The analysis results are temporarily stored in memory.

[1341] Step 4:

[1342] The server searches for or generates similar problems and explanations. Based on the analysis results, it searches the database for similar problems and their explanations. In this process, it can also generate new explanations using a generation AI model. From the search results or generated results, the most suitable similar problem and explanation are selected. For example, an explanation such as "the possibility of the robot stopping at the main switch" might be obtained.

[1343] Step 5:

[1344] The server sends similar problems and explanations to the user's terminal. Similar problems and their explanations, whether found or generated, are sent to the user's terminal. The user can then view this information on their terminal. For example, an explanation such as "Check the robot's main switch" might be displayed.

[1345] Step 6:

[1346] The user implements the suggested solution. The user follows the instructions displayed on the terminal and takes specific action. For example, the operator checks the main switch of the robot, following the instruction "Check the robot's main switch."

[1347] Step 7:

[1348] The user submits their solution or problem-solving result. After implementing the suggested solution, the user enters the result on their device and sends it to the server. For example, they might enter feedback such as, "I checked the main switch, but the robot is not working."

[1349] Step 8:

[1350] The server records the user's progress and optimizes the next approach. Received solutions are stored in a database and managed as learning or operation history. Based on this history, the server generates information to optimize future problem-solving and learning plans. This improves the user's skills and problem-solving efficiency.

[1351] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1352] System Overview

[1353] This system aims to allow learners to learn at their own pace and provide effective supplementary lessons, especially for areas where they struggle. A new emotion engine has been added, which recognizes and analyzes learners' emotions and adjusts the learning content accordingly. This enables efficient learning while maintaining learner motivation.

[1354] User actions

[1355] Users utilize an interface to input problems they were unable to solve. They input and submit problems in text or image format. At the same time, facial expressions and voice are recorded via camera and microphone to capture user emotion data.

[1356] Server operation

[1357] 1. Receiving and saving the problem:

[1358] Issues submitted by users are received by the server and stored in the database. In this step, the user ID, issue, and sentiment data are stored together.

[1359] 2. Problem Analysis:

[1360] The server analyzes the stored problems to identify their characteristics (e.g., difficulty, format) and category (e.g., mathematics, factorization). This analysis uses natural language processing (NLP) and image recognition technologies.

[1361] 3. Analysis of emotional data:

[1362] The emotion engine analyzes recorded emotion data to identify the learner's emotional state (e.g., stress level, motivation). This data is used in conjunction with the problem analysis results.

[1363] 4. Generate or search for similar problems and explanations:

[1364] Based on the characteristics and category of the problem, as well as sentiment data, the server searches the database for similar problems and their explanations, or generates new ones. Depending on the sentiment data, the difficulty level may be adjusted and the depth of the explanations may be changed.

[1365] 5. Provision of problems and explanations:

[1366] The server generates or searches for similar problems and their explanations, adjusts them appropriately based on sentiment data, and sends them to the user's terminal. This allows the user to learn with content that is appropriate to their current emotional state.

[1367] Specific example

[1368] For example, if a user is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", they input this problem into the system. The server then receives the problem and analyzes its characteristics and categories. In this case, the problem is classified as a "quadratic equation" and "factorization". Simultaneously, the emotion engine detects "anxiety" from the user's facial expressions and voice.

[1369] The server then searches for similar problems in the "quadratic equations" and "factorization" categories and prepares appropriate explanations. Based on the detected sentiment, the server provides explanations in a clearer, step-by-step manner. For example, problems such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0" are selected, and their solutions and explanations are sent to the user.

[1370] Users work on these similar problems and input their answers into their device. The device then retrieves sentiment data along with these answers and sends it to the server. The server records and analyzes the answers in a database. The next learning plan is further optimized based on the recorded answers and sentiment data.

[1371] Tracking learning progress

[1372] The user's answers to problems are sent from their device to the server. The server receives this data and records it in a database. This record is managed as the user's learning history and used to optimize the learning plan for the next learning session. Since emotional data is also recorded, a learning plan is created that takes into account the learner's motivation and emotional changes. As a result, users can not only visualize their own progress but also receive individually customized learning content.

[1373] This entire process allows learners to have an optimal learning experience and to progress through their studies efficiently. The introduction of an emotion engine improves the quality of learning by providing personalized support that takes into account the learner's emotional state.

[1374] The following describes the processing flow.

[1375] Step 1:

[1376] The user inputs and submits a problem they were unable to solve using their device. The user inputs the problem in text or image format through a dedicated interface and presses the submit button. During this process, the user's facial expressions and voice data are simultaneously recorded using the camera and microphone.

[1377] Step 2:

[1378] The terminal sends the entered problem data, along with the user's facial expressions and voice data, to the server. The problem data is transmitted with the user ID associated with it.

[1379] Step 3:

[1380] The server receives problem data sent from the terminal and saves it to the database. During saving, the user ID is associated with the problem data.

[1381] Step 4:

[1382] The server analyzes the problem data it receives. Natural language processing (NLP) and image recognition technologies are used to identify the characteristics and categories of the problems. These characteristics include the difficulty level and format of the problem.

[1383] Step 5:

[1384] The server analyzes facial expressions and voice data to identify the user's emotional state. The emotion engine evaluates the user's stress level and motivation. This data is also stored in a database.

[1385] Step 6:

[1386] The server searches the database for similar problems and their explanations based on the problem's characteristics and category, as well as sentiment data. It may also generate new problems and explanations as needed.

[1387] Step 7:

[1388] The server adjusts the searched or generated similar problems and their explanations to match the user's emotional state. For example, if the user is showing a high stress level, it will start with easier problems and adjust the explanations to be more gradual.

[1389] Step 8:

[1390] The server sends similar problems and their explanations, tailored to the user's device. The user receives these problems and explanations on their device and continues learning.

[1391] Step 9:

[1392] Users work on similar problems submitted via their devices and input their answers. After solving the problems, sentiment data is recorded again.

[1393] Step 10:

[1394] The device sends the user's answers, facial expressions, and voice data to the server. This data includes the user ID.

[1395] Step 11:

[1396] The server receives the answer results and saves them to the database. The user ID, question ID, answer result, and sentiment data are stored together.

[1397] Step 12:

[1398] The server analyzes the user's learning progress based on stored answer results and sentiment data. This analysis evaluates the user's level of understanding and learning progress.

[1399] Step 13:

[1400] The server optimizes the next learning plan. Based on the user's learning progress and sentiment data, it selects the next problems and explanations to tackle and prepares individually optimized learning content.

[1401] Through the steps outlined above, this system provides learners with an individually optimized learning experience, supporting efficient learning. The introduction of an emotion engine enables personalized learning support that takes into account the learner's emotional state.

[1402] (Example 2)

[1403] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1404] Traditional learning support systems often provide problems and explanations without considering the learner's emotional state, making it difficult to provide an optimal learning experience for each individual learner. This can lead to decreased learner motivation and hinder efficient learning.

[1405] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving problems that the learner could not solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for simultaneously recording the learner's emotional data when receiving the problems, means for analyzing the emotional data and identifying the learner's emotional state, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems and the emotional data, means for providing the generated or searched similar problems and their explanations to the learner, and means for recording the learner's answer results and emotional data and optimizing the next learning plan. This makes it possible to provide effective learning support and improve motivation in accordance with the learner's emotional state.

[1406] "Learner" refers to an individual who uses the system to engage in learning activities.

[1407] A "problem" refers to an academic or educational task that learners attempt to solve.

[1408] "Emotional data" refers to data that indicates an emotional state, extracted from the learner's facial expressions and voice.

[1409] A "server" refers to a computer system that is responsible for analyzing, processing, and storing the data it receives.

[1410] A "database" refers to a collection of information that a server uses to efficiently store, manage, and retrieve data.

[1411] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human language.

[1412] "Image recognition technology" refers to the technology used by computers to extract and analyze information from images.

[1413] An "emotion engine" refers to software or hardware that analyzes a learner's recorded emotional data to identify their emotional state.

[1414] "Analysis" refers to the process of examining data and information in detail and using that information to identify characteristics or specific patterns.

[1415] A "similar problem" refers to another problem that has similar characteristics or categories to the problem the learner was unable to solve.

[1416] "Explanation" refers to content that explains to learners how to solve problems and related knowledge.

[1417] A "learning plan" refers to a plan that proposes the optimal learning content and schedule based on the learner's progress and abilities.

[1418] "Optimization" refers to adjusting a system or plan to its most effective state in order to achieve a specific objective.

[1419] Modes for carrying out the invention

[1420] This invention relates to a system that collects problems that learners were unable to solve and data on the learners' emotions, analyzes this data, and provides appropriate learning support. Specifically, the system involves a server receiving problems and emotion data sent by learners, analyzing them, and optimizing the learning plan.

[1421] Hardware and software to be used

[1422] This system uses the following hardware and software:

[1423] 1. Terminal:

[1424] These are devices used by learners, such as PCs, smartphones, and tablets.

[1425] It is desirable to have a camera and microphone, as well as sensors for recording emotional data.

[1426] 2. Server:

[1427] It is a high-performance computer system that stores, analyzes, and processes data.

[1428] The system efficiently manages learner data by using an internal or external database.

[1429] 3. Natural Language Processing Techniques:

[1430] We will use the Python NLP library "NLTK" or similar tools to analyze text data.

[1431] 4. Image recognition technology:

[1432] We use libraries such as "OpenCV" to perform image analysis.

[1433] 5. Emotional Engine:

[1434] Software such as "Emotion API" is used to analyze the learner's facial expressions and voice to identify their emotional state.

[1435] 6. Generative AI Models:

[1436] The "GPT-3" model and its successor models are used to generate similar problems and explanations.

[1437] System Description

[1438] User actions

[1439] Users use their devices to input problems they couldn't solve into the interface. Problems can be entered in text or image format, and simultaneously, facial expressions and voice are recorded via the camera and microphone. This also collects emotional data.

[1440] As a concrete example, if a user is unable to solve the quadratic equation "x^2 - 4x + 4 = 0", they enter the problem into a text box and click the "Submit" button. At this time, the camera records the user's facial expressions and the microphone captures their voice.

[1441] Server operation

[1442] The server receives problem data and sentiment data submitted by users. The problem data is stored in a database and analyzed using natural language processing techniques (e.g., NLTK) and image recognition techniques (e.g., OpenCV). The analysis identifies the characteristics of the problem (e.g., difficulty level, format) and its category (e.g., quadratic equation, factorization).

[1443] Simultaneously, the emotion engine analyzes recorded facial expressions and voice to identify the learner's emotional state (e.g., anxiety, stress). This identified emotional state information is used in conjunction with the problem analysis results.

[1444] Based on these analysis results, the server searches the database for similar problems and their explanations, or generates new problems using a newly generated AI model (e.g., GPT-3). The difficulty level and depth of the explanations are adjusted according to the sentiment data.

[1445] Specific example

[1446] For example, if a user inputs the problem "x^2 - 4x + 4 = 0" and their emotional state is identified as "anxiety," the server will prepare similar problems and their explanations from the "quadratic equations" and "factorization" categories. In this case, the explanations will be provided in a step-by-step manner, based on the emotional data. Specifically, problems such as "2x^2 - 8x + 8 = 0" and "x^2 - 2x + 1 = 0" will be selected, and the solutions to each will be explained in detail.

[1447] Example of a prompt

[1448] Examples of prompts to input into a generative AI model include the following:

[1449] To help a student who is struggling with the quadratic equation "x^2 - 4x + 4 = 0" and is feeling frustrated, please generate and provide the following problem and explanation. Please include several similar problems and explain the solution to each step-by-step.

[1450] This allows the server to provide a learning experience optimized for the learner's emotional state, thereby maintaining motivation and improving learning efficiency.

[1451] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1452] Step 1:

[1453] The user inputs the problem they were unable to solve into the device's interface. Specifically, the user enters the problem into a text box or uploads an image file. Simultaneously, the camera and microphone automatically activate to record the user's facial expressions and voice. The input data includes problem data (text or image) and emotion data (facial expressions, voice).

[1454] Step 2:

[1455] The terminal sends user-entered problem data and recorded sentiment data to the server. This data includes user ID, problem data, and sentiment data. This provides the server with the necessary input for subsequent analysis.

[1456] Step 3:

[1457] The server stores the received problem data and analyzes the text data using natural language processing techniques (such as NLTK). It also analyzes image data using image recognition techniques (such as OpenCV). As a result of the analysis, the characteristics of the problem (difficulty level, format) and its category (e.g., mathematics, factorization) are identified. The input is the problem data, and the output is the problem characteristics and category information.

[1458] Step 4:

[1459] The server uses an emotion engine (such as the Emotion API) to analyze the recorded emotion data. This analysis identifies the learner's emotional state (e.g., stress level, motivation). The input is the emotion data, and the output is the identified emotional state.

[1460] Step 5:

[1461] The server searches the database for similar problems and their explanations, or generates new ones, based on the problem's characteristics, category, and sentiment data. A generative AI model (such as GPT-3) is used for generation. Specifically, prompts are used to instruct the generative AI model to obtain similar problems and their explanations. The input consists of the problem's characteristics, category, and sentiment state, while the output is similar problems and their explanations.

[1462] Step 6:

[1463] The server sends generated or retrieved similar problems and their explanations to the user's terminal. Based on sentiment data, the difficulty level and depth of the explanations are adjusted, so learners receive learning content that is best suited to their state. The input is similar problems and explanations, and the user's terminal information; the output is the transmission of the adjusted learning content.

[1464] Step 7:

[1465] The user works on a similar problem provided and inputs their answer into the terminal. The terminal then sends the answer and recorded sentiment data back to the server. The input is the user's answer and sentiment data, and the output is the data sent to the server.

[1466] Step 8:

[1467] The server records the received answers and sentiment data in a database. This optimizes the next learning plan. Specifically, it uses an adaptive learning algorithm to analyze the learner's progress and sentiment state and propose the optimal content for the next learning session. The input is the answers and sentiment data, and the output is the optimized next learning plan.

[1468] (Application Example 2)

[1469] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1470] Traditional learning systems provided learning content without considering learners' emotions, resulting in insufficient maintenance of learner motivation and optimization of learning effectiveness. Furthermore, in in-store customer service support, it was difficult to grasp customers' emotional states in real time and respond appropriately. Consequently, improvements in customer satisfaction and effective product recommendations were not fully achieved. This system aims to solve these problems.

[1471] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving problems that the learner could not solve, means for analyzing the received problems and identifying the characteristics and categories of the problems, means for generating or searching for similar problems and their explanations based on the characteristics and categories of the problems, means for providing the generated or searched similar problems and their explanations to the learner, means for recognizing and analyzing the learner's emotional state, and means for appropriately adjusting the learning content based on the emotional state. This makes it possible to provide personalized learning content that takes into account the learner's emotional state, thereby maintaining motivation and optimizing learning effectiveness. Furthermore, even in physical stores, customer satisfaction can be improved by analyzing the customer's emotional state in real time and providing appropriate customer service methods.

[1472] "A means of receiving problems that learners were unable to solve" refers to an interface for learners to input or upload problems they were unable to solve.

[1473] "Means for identifying the characteristics and categories of a problem" refers to a function that analyzes a received problem and identifies its characteristics and classification, such as difficulty level, format, and subject matter.

[1474] "Means for generating or searching for similar problems and their explanations" means a function that generates new similar problems and their explanations, or searches for them in an existing database, based on the aforementioned characteristics and categories.

[1475] "Means to provide to learners" means the function of displaying or transmitting generated or retrieved similar problems and their explanations to the learner's device.

[1476] "Means for recognizing and analyzing emotional states" refers to a function that analyzes emotions from a learner's facial expressions and voice data and identifies their emotional state (e.g., stress, motivation).

[1477] "Means for appropriately adjusting learning content based on emotional state" refers to a function that dynamically adjusts learning content and problem difficulty levels according to analyzed emotional data, providing an optimal learning plan tailored to the learner.

[1478] "Natural language processing technology" is a general term for algorithms and methods used to analyze text data and identify its characteristics and categories.

[1479] "Image recognition technology" is a general term for algorithms and methods used to analyze image data and identify its characteristics and categories.

[1480] "Facial expression analysis technology" is a general term for technologies that analyze facial expressions from image data acquired by a camera and estimate emotions from the results.

[1481] "Voice analysis technology" is a general term for technologies that analyze voice data acquired by a microphone and estimate emotions from the results.

[1482] System Configuration

[1483] This invention is a system consisting of the following main elements:

[1484] 1. Device: A smartphone used by learners, store clerks, etc.

[1485] 2. Server: A server that performs data analysis, generates learning content, sentiment analysis, etc.

[1486] 3. Camera and microphone: Devices for acquiring facial expressions and voice data.

[1487] Program processing

[1488] For learners

[1489] 1. Inputting the problem: Learners input the problems they were unable to solve into their device in text or image format and submit them. At this time, the camera and microphone are used to collect learner sentiment data.

[1490] 2. Problem Analysis: The server analyzes the received problem using natural language processing (NLP) or image recognition techniques to identify its characteristics and categories.

[1491] 3. Analysis of emotional data: The server uses data acquired from the camera and microphone to analyze the learner's emotional state using facial expression analysis and voice analysis technologies.

[1492] 4. Generation and search of similar problems and explanations: Based on the characteristics of the problem and the sentiment analysis results, the server searches the database for similar problems and their explanations or generates new ones.

[1493] 5. Provision of tailored learning content: The server sends optimized problems and explanations based on the analysis data to the learner's terminal, which the learner then works on.

[1494] In the case of customer service support at physical stores

[1495] 1. Acquisition of customer emotion data: While serving customers, store employees use the camera and microphone on their devices to capture the customer's facial expressions and voice, thereby acquiring emotion data.

[1496] 2. Analysis of emotional data: Based on the acquired data, the server analyzes the customer's emotional state in real time using facial expression analysis technology and voice analysis technology.

[1497] 3. Adjusting responses: Based on the analyzed sentiment data, the server provides appropriate responses and product suggestions that the store staff should take, and these are displayed on the terminal.

[1498] Hardware and software to be used

[1499] Hardware: Smartphone, camera, microphone.

[1500] Software: OpenCV (video acquisition and display), DeepFace (facial expression analysis), natural language processing technology (NLP, text analysis), image recognition technology (image analysis).

[1501] Examples of specific cases and prompt statements

[1502] Specific example

[1503] For example, if a customer is having trouble choosing a product during a consultation at a physical store, the system uses the customer's smartphone camera and microphone to analyze their emotional state. If the analysis reveals "anxiety" or "unease," the system instructs the salesperson to take actions such as "provide a detailed product description" or "suggest alternative options."

[1504] Example of a prompt

[1505] Develop an application that analyzes user emotions and suggests appropriate customer service methods. Implement the emotion analysis function shown in the following code:

[1506] 1. The camera captures facial images, and DeepFace is used to analyze facial expressions.

[1507] 2. Based on the emotional data obtained, we adjust the customer service methods and recommended products to best suit each customer.

[1508] As described above, in the embodiment of this invention, it is possible to analyze user input information and emotional data and dynamically provide appropriate learning content and customer service responses. As a result, improved learning effectiveness and increased customer satisfaction can be expected.

[1509] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1510] Step 1:

[1511] The user inputs a problem they were unable to solve. The input problem is in text or image format and is sent from the learner's device to the server. At the same time, the learner's facial expressions and voice data are captured using the device's camera and microphone and sent to the server as emotion data.

[1512] Input: Problems the learner was unable to solve, learner's facial expression data, learner's voice data

[1513] Output: Problem data and sentiment data sent to the server

[1514] Specific steps: The learner opens a dedicated application on their smartphone, enters or photographs the question, and clicks the submit button. Simultaneously, video and audio data is captured.

[1515] Step 2:

[1516] The server analyzes the problem data it receives. Natural language processing (NLP) and image recognition techniques are used to identify the characteristics of the problem (difficulty level, format, etc.) and its category (e.g., mathematics, factorization).

[1517] Input: Problem data

[1518] Output: Problem characteristics and categories

[1519] Specific operation: The server analyzes the received text or image data using NLP and image recognition algorithms to identify the nature of the problem.

[1520] Step 3:

[1521] The server analyzes emotional data. Using facial expression analysis and voice analysis technologies, it identifies the learner's emotional state (e.g., stress level, motivation) from the received video and audio data.

[1522] Input: Facial expression data, voice data

[1523] Output: Emotional state

[1524] Specific operation: The server uses DeepFace and other facial expression analysis software to analyze the learner's video data and identify the dominant emotion. Voice analysis software is also used in conjunction to estimate the emotional state with high accuracy.

[1525] Step 4:

[1526] Based on the characteristics and category of the problem, as well as the sentiment analysis results, the server generates or searches for similar problems and their explanations in the database. The level of detail and difficulty of the explanation are adjusted according to the emotional state.

[1527] Input: Problem characteristics and categories, emotional state

[1528] Output: Similar problems and their explanations

[1529] Specific operation: The server searches the database using problem characteristic data and category data to find similar problems and their explanations. It may also generate new similar problems and explanations using a generative AI model. Based on the results of sentiment analysis, it automatically adjusts the difficulty and detail of the explanations.

[1530] Step 5:

[1531] The server sends similar problems and their explanations, generated or retrieved, to the learner's device. This allows the learner to progress through the learning process with content that best suits their current emotional state.

[1532] Input: Similar problems and their explanations

[1533] Output: Learning content sent to the learner's device.

[1534] Specific operation: The server sends the formatted data to the learner's device and displays it on the device's application. During this process, notification functions and interface optimization are performed.

[1535] Step 6:

[1536] The learner works on additional learning material and enters their answers into the device. The device sends the answers and newly acquired sentiment data to the server.

[1537] Input: Answer results, additional sentiment data

[1538] Output: Answer results and additional sentiment data sent to the server

[1539] Specific operation: The learner works on similar problems provided and enters their answers. Simultaneously, the device's camera and microphone capture the learner's facial expressions and voice again.

[1540] Step 7:

[1541] The server receives the answer results and additional sentiment data, and records them in a database as the learner's learning history. This allows for further optimization of the next learning plan.

[1542] Input: Answer results, additional sentiment data

[1543] Output: Update of the learning history database

[1544] Specific actions: The server analyzes the answer results and sentiment data, and updates parameters to adjust the next learning plan. It adds new entries to the database and optimizes the learning history.

[1545] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1546] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1547] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1548] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1549] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1550] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1551] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1552] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1553] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1554] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1555] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1556] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1557] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1558] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1559] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1560] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1561] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1562] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1563] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1564] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1565] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1566] The following is further disclosed regarding the embodiments described above.

[1567] (Claim 1)

[1568] A means of receiving problems that learners were unable to solve,

[1569] A means for analyzing the received problem and identifying its characteristics and category,

[1570] A means for generating or searching for similar problems and their explanations based on the characteristics and categories of the aforementioned problems,

[1571] A system including means for providing learners with similar problems that have been generated or retrieved, along with explanations thereof.

[1572] (Claim 2)

[1573] The system according to claim 1, comprising means for using natural language processing techniques or image recognition techniques in identifying the characteristics and categories of the aforementioned problem.

[1574] (Claim 3)

[1575] The system according to claim 1, which includes means for recording the learner's answer results and optimizing the next learning plan when providing the aforementioned similar problems and their explanations.

[1576] "Example 1"

[1577] (Claim 1)

[1578] A means of receiving problems that learners were unable to solve,

[1579] A means for analyzing the received problem and identifying its characteristics and category,

[1580] A means for generating or searching for similar problems and their explanations based on the characteristics and categories of the aforementioned problems,

[1581] Means for providing learners with the generated or retrieved similar problems and their explanations,

[1582] A system that includes means for recording learners' answer results and optimizing their next learning plan.

[1583] (Claim 2)

[1584] The system according to claim 1, comprising means for using natural language processing techniques or image recognition techniques in identifying the characteristics and categories of the aforementioned problem.

[1585] (Claim 3)

[1586] The system according to claim 1, comprising means for generating similar problems and their explanations using a generative AI model.

[1587] "Application Example 1"

[1588] (Claim 1)

[1589] A means of receiving problems that learners were unable to solve,

[1590] A means for analyzing the received problem and identifying its characteristics and category,

[1591] A means for generating or searching for similar problems and their explanations based on the characteristics and categories of the aforementioned problems,

[1592] Means for providing learners with the generated or retrieved similar problems and their explanations,

[1593] A means of analyzing mechanical problems and error codes entered by operators and providing similar solutions and explanations,

[1594] An interface in which the user inputs questions in text or image format,

[1595] Means for analyzing the input problem using natural language processing technology or image recognition technology,

[1596] A means of providing users with the optimal solution based on the analysis results,

[1597] A system that includes means for recording user operation history and optimizing the next course of action.

[1598] (Claim 2)

[1599] The system according to claim 1, comprising means for using natural language processing techniques or image recognition techniques in identifying the characteristics and categories of the aforementioned problem.

[1600] (Claim 3)

[1601] The system according to claim 1, which includes means for recording the learner's answer results and optimizing the next learning plan when providing the aforementioned similar problems and their explanations.

[1602] "Example 2 of combining an emotion engine"

[1603] (Claim 1)

[1604] A means of receiving problems that learners were unable to solve,

[1605] A means for analyzing the received problem and identifying its characteristics and category,

[1606] When receiving the aforementioned problem, a means for simultaneously recording learner's emotional data,

[1607] A means for analyzing the aforementioned emotional data and identifying the learner's emotional state,

[1608] Means for generating or searching for similar problems and their explanations based on the characteristics and categories of the aforementioned problems and the sentiment data,

[1609] Means for providing learners with the generated or retrieved similar problems and their explanations,

[1610] A means to record learners' answer results and sentiment data, and to optimize the next learning plan,

[1611] A system that includes this.

[1612] (Claim 2)

[1613] The system according to claim 1, comprising means for using natural language processing techniques or image recognition techniques in identifying the characteristics and categories of the aforementioned problem.

[1614] (Claim 3)

[1615] The system according to claim 1, which includes means for adjusting the content based on the learner's emotional state when providing the aforementioned similar problems and their explanations.

[1616] "Application example 2 when combining with an emotional engine"

[1617] (Claim 1)

[1618] A means of receiving problems that learners were unable to solve,

[1619] A means for analyzing the received problem and identifying its characteristics and category,

[1620] A means for generating or searching for similar problems and their explanations based on the characteristics and categories of the aforementioned problems,

[1621] Means for providing learners with the generated or retrieved similar problems and their explanations,

[1622] Means for recognizing and analyzing the emotional state of the learner,

[1623] A system including means for appropriately adjusting learning content based on the aforementioned emotional state.

[1624] (Claim 2)

[1625] In identifying the characteristics and categories of the aforementioned problem, means of using natural language processing techniques or image recognition techniques,

[1626] The system according to claim 1, further comprising means for using facial expression analysis technology or voice analysis technology for the recognition and analysis of the aforementioned emotional state.

[1627] (Claim 3)

[1628] When providing the aforementioned similar problems and their explanations, a means is provided to record the learner's answer results and optimize the next learning plan.

[1629] The system according to claim 1, further comprising means for adjusting the learning content based on the learner's emotional state. [Explanation of symbols]

[1630] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of receiving problems that learners were unable to solve, A means for analyzing the received problem and identifying its characteristics and category, A means for generating or searching for similar problems and their explanations based on the characteristics and categories of the aforementioned problems, A system including means for providing learners with similar problems that have been generated or retrieved, along with explanations thereof.

2. The system according to claim 1, comprising means for using natural language processing techniques or image recognition techniques when identifying the characteristics and categories of the aforementioned problem.

3. The system according to claim 1, which includes means for recording the learner's answer results and optimizing the next learning plan when providing the aforementioned similar problems and their explanations.

Images