System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving question content input by a user; means for analyzing the received question content and identifying a corresponding question; means for generating an explanation for the identified question using a generative AI model; means for transmitting the generated explanation to the user; and means for displaying the explanation transmitted to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, many elementary schools face the problem of not being able to properly follow up on points that students feel "unclear" during class, and the lesson simply continues as is. Furthermore, in today's society, where the number of dual-income households is increasing, it is becoming more difficult for parents to effectively support their children's learning. Therefore, there is a need for a way to properly resolve any "uncertainty" that children may feel while studying at home, rather than leaving it unaddressed. Furthermore, there is a need for a system that allows students to study efficiently at home, without relying on cram schools or teaching materials. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes a means for receiving a question entered by a user, a means for analyzing the received question to identify a corresponding problem, a means for generating an explanation for the identified problem using a generative AI model, a means for sending the generated explanation to the user, and a means for displaying the explanation sent to the user. This system can resolve any confusion a child may have during homework or self-study in real time. Furthermore, by including a means for converting the received question into a data format and sending it to a server, and for analyzing and converting the explanation data received from the server into a displayable format, the system achieves efficient data processing and a user-friendly interface. Furthermore, by supporting audio output of the explanations displayed to the user, the system can accommodate a variety of learning styles. This creates an environment in which parents can support their children's learning even when they are busy, and is expected to improve the quality of home learning.

[0006] A "user" is an individual or representative who utilizes the system to enter questions and receive explanations.

[0007] "Question content" is information related to the problem the user wants to solve.

[0008] The "receiving means" is a means for acquiring and saving the content of the question entered by the user.

[0009] The "means for analyzing" refers to a means for carrying out a process of examining the received question content and identifying the relevant problem and explanation content.

[0010] "Means of identification" refers to the means for finding specific learning problems or issues from the analyzed question content.

[0011] A "generative AI model" is a mathematical model or algorithm that uses artificial intelligence to generate explanations or explanations of a problem.

[0012] A "means for generating an explanation" is a means for creating an explanation for an identified problem using a generative AI model.

[0013] The "transmitting means" is a means for sending the generated commentary to the user's terminal.

[0014] The "means for displaying" is a means for providing the explanation visually or audibly in a format that is easy for the user to understand.

[0015] A "data format" is a method or standard for expressing information in a standardized form.

[0016] A "server" is hardware or software that provides data processing and storage over a network.

[0017] The "audio output means" is a means for conveying the generated commentary to the user as audio. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention is a system for solving "unclear" problems that a user encounters in the course of his / her own learning. Specific embodiments will now be described.

[0040] User input

[0041] When a user has a question about a problem, they enter the question into the terminal. For example, they might say, "I don't understand page 30(2) of the calculation drill." The terminal receives the user's input and processes it as properly formatted data.

[0042] Sending data from the device to the server

[0043] The terminal converts the question entered by the user into a data format (for example, JSON format) and sends it to the specified endpoint of the server. This allows the server to receive the user's question and prepare for the next analysis process.

[0044] Server analysis and identification of the problem

[0045] The server analyzes the received question data and identifies the relevant question from the question content. Specifically, it extracts information such as the name of the learning material, page number, and question number from the received data and searches the database on the server for the relevant question.

[0046] Server-generated description

[0047] After identifying the problem, the server uses a generative AI model to generate a solution for the problem. The server creates a detailed solution that includes step-by-step instructions, important points to note, and background information on the problem. For example, the solution might include, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator. The least common multiple of 4 and 3 is 12."

[0048] Sending and displaying commentary

[0049] The server packages the generated commentary in an appropriate data format (for example, JSON or HTML format) and sends it to the terminal. The terminal analyzes the commentary data received from the server and displays it on the screen in a format that is easy for the user to understand. If necessary, the terminal can also provide audio commentary using audio output means.

[0050] Specific examples

[0051] User input

[0052] User: "I don't understand page 30(2) of the math drill."

[0053] Terminal: (Display the input contents in the form and confirm)

[0054] Sending data from the device to the server

[0055] Terminal: (Convert input to JSON and send to server)

[0056] Server analysis and identification of the problem

[0057] Server: (Analysis of "I don't understand page 30(2) of the calculation drill")

[0058] Server: (Identifies the problem in an internal database)

[0059] Server-generated description

[0060] Server: (Generates explanations using AI models)

[0061] Server: (Configure generated commentary)

[0062] Sending and displaying commentary

[0063] Server: (Package the explanation in JSON format)

[0064] Server: (Sent to the device)

[0065] Display explanation

[0066] Terminal: (Analyzes received data and converts it to HTML)

[0067] Terminal: (Explanation displayed on screen)

[0068] Device: (Audio description available if needed)

[0069] In this way, this system can provide real-time explanations to users' (especially learners') questions and effectively support home learning. It can be used as a tool to properly support children's learning even when parents are busy, and is expected to improve the quality of home learning.

[0070] The processing flow will be explained below.

[0071] Step 1: User enters question

[0072] If a user has a question about a homework assignment or problem, they can use the terminal to enter their question, for example, "I don't understand page 30(2) of the calculation drill."

[0073] Step 2: Check input on the terminal and convert to data format

[0074] The device checks the user's input and converts it into an appropriate format (e.g., JSON), checking that the input has been converted correctly into the data format.

[0075] Step 3: Send to the server

[0076] The terminal sends formatted data to the server, which then sends the data to the specified endpoint while checking the stability of the communication and the integrity of the data.

[0077] Step 4: Data reception and analysis by the server

[0078] The server receives the data sent from the terminal, analyzes the question content based on the received data, and extracts information such as the teaching material name, page number, and question number.

[0079] Step 5: Identify the problem

[0080] Based on the information analyzed by the server, the server searches and identifies the relevant problem in the internal teaching material database. For example, it retrieves the problem on page 30(2) of the arithmetic drill from the database.

[0081] Step 6: Server-generated description

[0082] The server uses a generative AI model to generate detailed explanations for the identified problems, including the background to the problem, steps to solve it, and common mistakes.

[0083] Step 7: Converting the description to a data format

[0084] The server converts the generated description into an appropriate format (e.g., JSON or HTML), and the data is organized in a format that can be displayed on the user's device.

[0085] Step 8: Send data to your device

[0086] The server sends the formatted explanatory data to the terminal, and checks the integrity of the data while ensuring the stability and security of the communication.

[0087] Step 9: Data reception and analysis by the device

[0088] The terminal receives the commentary data sent from the server, analyzes the received data, and converts it into a format that can be displayed to the user (for example, HTML or text).

[0089] Step 10: Display the description

[0090] The device displays instructions on the user's screen, and optionally provides audio instructions using a voice output means. For example, detailed instructions such as "First, find the common denominator, then add the fractions."

[0091] This process allows users to receive explanations that resolve their questions in real time, enabling them to study effectively.

[0092] Example 1

[0093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] Conventional learning support systems lack the means to instantly resolve questions users have while studying. This can lead to a decline in learning effectiveness and a loss of motivation to study. Furthermore, when parents or teachers are busy, there is also the problem that learners are unable to receive appropriate support. The present invention aims to solve these problems and provide a system that allows learners to instantly resolve their questions.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0096] In this invention, the server includes means for receiving a question entered by a user, means for converting the received question into a data format, means for transmitting the converted data to the server, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for transmitting the generated explanation to the user, and means for displaying the explanation transmitted to the user. This makes it possible to provide quick and accurate explanations to questions that users may have while studying.

[0097] "User" refers to an individual who uses the system to receive learning support.

[0098] "Terminal" refers to an electronic device that allows a user to access and input data into the system.

[0099] "Question content" refers to information that expresses doubts that arise during the user's studies or problems that the user wants to solve.

[0100] "Data format" refers to the method by which data is organized and converted into an appropriate form for transmission from a device to a server.

[0101] "Server" refers to a central computer that receives, analyzes, and processes data sent from the terminals.

[0102] "Analysis" refers to the process by which the server breaks down and understands the data it receives and extracts specific information.

[0103] "Applicable questions" refer to specific study tasks related to the question submitted by the user.

[0104] A "generative AI model" refers to artificial intelligence that uses natural language processing technology to generate explanations for questions.

[0105] "Explanation" refers to the explanation and specific solution steps provided by the generative AI model for an identified problem.

[0106] "Data transmission" refers to the process of sending information from a terminal to a server or from a server to a terminal.

[0107] "Display" refers to the terminal providing the received commentary data to the user visually or audibly.

[0108] This invention relates to a learning support system that allows users to instantly resolve questions they may have while studying. This system receives questions entered by users and provides appropriate explanations via a server. Specific embodiments of this system are described below.

[0109] The user inputs the question using their own device (e.g., PC, smartphone, tablet). For example, they might input, "I don't understand page 30(2) of the math drill." The device then displays this input in a form and asks the user to confirm it.

[0110] The terminal converts the questions entered by the user into a data format such as JSON. For example, it converts the questions into JSON data like this:

[0111] json

[0112] {

[0113] "Material name": "Calculation drill",

[0114] "Page Number": 30,

[0115] "Problem Number": 2,

[0116] "Question": "I don't understand page 30(2)"

[0117] }

[0118] The converted data is sent over a network connection to a specific endpoint on the server, which receives the data.

[0119] The server parses the received JSON data. During parsing, it extracts information such as the teaching material name, page number, and question number. For example, it searches the database on the server using the condition "teaching material name = arithmetic drill AND page number = 30 AND question number = 2" to identify the corresponding question.

[0120] After the problem is identified, the server uses a generative AI model (e.g., a model using natural language processing technology) to generate an explanation for the problem. An example of a prompt that the server inputs to the generative AI model is, "Please provide an explanation for page 30(2) of the calculation drill."

[0121] The generated explanation includes specific calculation steps and important points to note. For example, the explanation generated might say, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator. The least common multiple of 4 and 3 is 12."

[0122] After the description is generated, the server repackages it into JSON format and sends it to the device using an HTTP POST request. The device then parses the received description data and converts it into a user-readable HTML format. It can also provide audio description using speech synthesis software if needed.

[0123] This system allows users to instantly resolve any questions that arise during their studies, improving the quality of their learning and effectively supporting home study. A specific example of a prompt sentence is "I don't understand page 30(2) of the math drill," and real-time explanations are provided.

[0124] In this way, the present invention is expected to increase learners' motivation to learn and improve their learning effectiveness by quickly and accurately resolving the learning problems that users have. The overall configuration and operation of the system is realized based on the cooperation between terminals and a server.

[0125] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0126] Step 1:

[0127] If a user has a question while studying, they can input it using a device (e.g., PC, smartphone, tablet). An example of the input content is "I don't understand page 30(2) of the calculation drill." The device displays this input in a form for the user to confirm. Input: User's question. Output: Confirmed question.

[0128] Step 2:

[0129] The terminal converts the questions the user has confirmed into JSON format. For example, it converts it into JSON data like this:

[0130] json

[0131] {

[0132] "Material name": "Calculation drill",

[0133] "Page Number": 30,

[0134] "Problem Number": 2,

[0135] "Question": "I don't understand page 30(2)"

[0136] }

[0137] In this conversion process, each element of the question (teaching material name, page number, question number, question content) is extracted and put into a JSON structure. Input: Confirmed question content. Output: Data in JSON format.

[0138] Step 3:

[0139] The device sends the converted JSON data to the server using an HTTP POST request, using a network connection to securely send the data to a specific endpoint on the server. Input: JSON format data. Output: Data sent to the server.

[0140] Step 4:

[0141] The server parses the received JSON data. During parsing, it extracts information such as the learning material name, page number, and question number. For example, it parses the data as follows:

[0142] Material name: Calculation drill

[0143] Page number: 30

[0144] Issue Number: 2

[0145] Input: JSON formatted data. Output: Extracted question elements.

[0146] Step 5:

[0147] The server uses the extracted information to search its internal database and identify the relevant questions. Specifically, it searches using the query "Material name = Calculation drills AND page number = 30 AND question number = 2." Input: Extracted question elements. Output: Identification of the relevant questions.

[0148] Step 6:

[0149] The server generates an explanation for the identified problem using a generative AI model (e.g., a model using natural language processing technology). The prompt sentence that the server inputs to the generative AI model is, "Please provide an explanation for page 30(2) of the calculation drill." The generative AI model returns a specific explanation. Input: Prompt sentence. Output: Generated explanation.

[0150] Step 7:

[0151] The server then packages the generated description back into JSON, for example into the following JSON data:

[0152] json

[0153] {

[0154] "Explanation": "To solve the equation 3 / 4 + 2 / 3, we first need to find a common denominator. The least common multiple of 4 and 3 is 12..."

[0155] }

[0156] Input: Generated description. Output: Description data in JSON format.

[0157] Step 8:

[0158] The server sends the packaged description data to the device using an HTTP POST request. In this process, data is transferred to the device via the network. Input: Description data in JSON format. Output: Data sent to the device.

[0159] Step 9:

[0160] The device analyzes the JSON-formatted explanatory data received from the server. It then converts the analyzed data into HTML format and displays it on the screen in a format that is easy for the user to read. Input: JSON-formatted explanatory data. Output: Explanation displayed in HTML format.

[0161] Step 10:

[0162] If necessary, the device will use speech synthesis software to provide audio descriptions, allowing users to hear as well as see the information. Input: Description data in HTML format. Output: Audio description.

[0163] In this way, the questions entered by the user are processed through a number of steps and an explanation is quickly generated and displayed. By providing immediate answers to questions during learning, learners can deepen their understanding and increase their motivation to learn.

[0164] (Application example 1)

[0165] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0166] In factories, when operational problems occur with machines or robots, it is difficult for engineers to respond quickly and resolve them. In particular, if specific explanations about operation are lacking, it takes time to resolve the problem, resulting in a decrease in production efficiency. The present invention aims to provide technology for quickly and appropriately resolving operational problems and machine issues encountered by workers and engineers in factories.

[0167] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0168] In this invention, the server includes means for receiving a question entered by a user, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for sending the generated explanation to the user, means for displaying the explanation sent to the user, means for receiving and analyzing a question about an operation problem of a factory robot and generating an explanation using a generative AI model, and means for sending and displaying the generated explanation to the display and audio output means of the factory robot, thereby enabling workers and engineers in a factory to quickly and appropriately solve operational problems and issues with machines and robots.

[0169] "Users" refer to factory workers and engineers who use the system.

[0170] "Questions" refer to inquiries about operational problems and machine issues entered by factory workers and engineers through the system.

[0171] "Means for receiving" refers to a method or device that allows the system to acquire and store the content of the question entered by the user.

[0172] "Means for analyzing" refers to a method or device that analyzes the content of the received query and identifies the problem based on the information contained therein.

[0173] "Means of identification" refers to the methods and devices used to identify specific operational problems or machine issues from the analyzed questions.

[0174] A "generative AI model" refers to an algorithm or program that uses machine learning and natural language processing techniques to generate explanations for identified problems.

[0175] "Means for generating explanations" refers to a method or device that utilizes a generative AI model to create detailed explanations and solutions to an identified problem.

[0176] "Means for transmitting to a user" refers to a method or device for transmitting the generated commentary from the system to a user.

[0177] "Means for displaying" refers to a method or device for displaying the transmitted commentary in a form that can be understood by the user.

[0178] "Factory robot" refers to an industrial robot on which the present system is installed.

[0179] "Operational trouble" refers to problems or malfunctions that occur during the operation of factory robots and other related machinery.

[0180] "Display" refers to a screen or monitor for visually displaying the generated commentary.

[0181] The "audio output means" refers to a speaker or a voice synthesizer for transmitting the generated commentary to the user by voice.

[0182] "Data format" refers to the standard data structure and format used when communicating question content and explanation data between servers and systems.

[0183] The system program that realizes this application example is designed to solve operational problems of factory robots. Below is a description of how this system works.

[0184] Hardware and software used

[0185] The system mainly uses the following hardware and software:

[0186] Hardware: Factory robot body, touch panel, speaker, central server

[0187] Software: Question receiving program, data analysis program, generative AI model (e.g., OpenAI's GPT series), data transmission and reception program

[0188] User operations and question input

[0189] Workers and engineers (users) use the factory robot's touch panel or voice input function to input questions about operational problems into the system. For example, they can input a question like, "I don't know why the robot arm on line 3 isn't moving."

[0190] Sending data about questions

[0191] The question is converted to JSON format and sent to a central server. The server uses an analysis program to analyze the received question and identify the problem. For example, the JSON-formatted question data may contain information such as "Question: 'The robot arm on line 3 is not moving.'"

[0192] Server-based problem analysis and explanation generation

[0193] The server uses an analysis program to analyze the question and generates an explanation for the identified problem using a generative AI model. For example, after analyzing the question, the generative AI model generates an explanation such as, "It is highly likely that a foreign object is stuck in the sensor. Please check the sensor first."

[0194] Data transmission and display of commentary

[0195] The generated explanation is then packaged again in JSON format and sent to the factory robot, which then displays the explanation on its display and, if necessary, outputs it aloud, allowing workers and technicians to quickly understand how to solve the problem.

[0196] Examples and prompts

[0197] For example, a factory worker may input a question through a display, such as, "A sensor error is displayed. What should I do?" This question data is converted into JSON format and sent to a central server for analysis. The generative AI model generates a specific explanation, such as, "The possible cause of the sensor error is that a foreign object is stuck in the sensor. First, check the sensor and see if there is a foreign object." The explanation is sent to a factory robot, which then provides the explanation on the display and in voice.

[0198] Example prompt sentence:

[0199] "Please tell me why the machine isn't working. For example, the robot arm on line 3 has stopped working."

[0200] In this way, the system helps factory workers and engineers quickly and efficiently resolve operational problems they encounter.

[0201] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0202] Step 1:

[0203] A user uses the factory robot's touch panel or voice input function to input questions about operational problems. This input is in the form of a string, such as "I don't know why the robot arm on line 3 isn't moving."

[0204] Step 2:

[0205] The terminal converts the question entered by the user into JSON format. The entered string is analyzed and converted into JSON data in the format of, for example, "Question: 'The robot arm on line 3 is not moving.'" This data format conversion results in structured data.

[0206] Step 3:

[0207] The device sends the converted JSON-formatted question data to the central server, which receives the data sent from the device via an API endpoint, thereby obtaining JSON data containing the question content.

[0208] Step 4:

[0209] The server analyzes the received JSON data and identifies the relevant problem from the question. Specifically, it analyzes the "question" field of the received data and extracts information about the relevant machine and the problem. For example, it performs an analysis based on the information, "The robot arm on line 3 is not moving."

[0210] Step 5:

[0211] The server uses a generative AI model to generate an explanation based on the analyzed question. Using a generative AI model (e.g., OpenAI's GPT series), the server generates an explanation based on the analysis results. For example, it creates an explanation such as, "There is a high possibility that a foreign object is stuck in the sensor. Please check the sensor first."

[0212] Step 6:

[0213] The server then packages the generated explanation into JSON format again and sends it to the device. This operation sends explanation data from the server to the device. Specifically, JSON data in the format "Explanation: 'There is a high possibility that a foreign object is stuck in the sensor. Please check the sensor first.'" is sent.

[0214] Step 7:

[0215] The device analyzes the received JSON-formatted explanatory data and converts it into a displayable format. Based on the analysis results, the explanation is converted into text or HTML format. This data conversion makes it possible to provide information to the user visually and audibly.

[0216] Step 8:

[0217] The device displays the converted commentary on the display and outputs it as audio if necessary. The user can check the commentary on the display and, in some cases, listen to the commentary via audio guidance. For example, the display may say, "There is a high possibility that a foreign object is stuck in the sensor. Please check the sensor first."

[0218] By following the steps explained in this manual, users can quickly and accurately identify the cause of operational problems and find solutions, thereby improving production efficiency and reducing the time required to resolve problems.

[0219] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0220] The present invention combines a system that solves the "I don't understand" problem that users experience during the learning process with an emotion engine that recognizes the user's emotions and adjusts the response content. The purpose of this invention is to improve the learning experience by taking into account the user's emotional state while providing appropriate explanations.

[0221] User input of question

[0222] When a user has a question about a problem, they enter the question into the terminal. For example, they might say, "I don't understand page 30(2) of the calculation drill." The terminal receives the user's input and processes it as properly formatted data.

[0223] Sending data from the device to the server

[0224] The terminal converts the question entered by the user into a data format (for example, JSON format) and sends it to the specified endpoint of the server. This allows the server to receive the user's question and prepare for the next analysis process.

[0225] Server analysis and identification of the problem

[0226] The server analyzes the received question data and identifies the relevant question from the question content. Specifically, it extracts information such as the name of the learning material, page number, and question number from the received data and searches the database on the server for the relevant question.

[0227] Server-generated description

[0228] After identifying the problem, the server uses a generative AI model to generate a solution for the problem. The server creates a detailed solution that includes step-by-step instructions, important points to note, and background information on the problem. For example, it might say, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator."

[0229] Emotion recognition by emotion engine

[0230] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's facial expressions, tone of voice, typing speed, etc. to identify the emotion the user is feeling. For example, it detects whether the user is feeling anxious or irritated.

[0231] Adjustment of commentary content

[0232] The server adjusts the content and tone of the generated commentary based on the user's emotions recognized by the emotion engine. For example, if the user is feeling anxious, the server changes the content of the commentary to be more friendly and encouraging.

[0233] Sending and displaying commentary

[0234] The server packages the generated commentary in an appropriate data format (for example, JSON or HTML format) and sends it to the terminal. The terminal analyzes the commentary data received from the server and displays it on the screen in a format that is easy for the user to understand. If necessary, the terminal can also provide audio commentary using audio output means.

[0235] Specific examples

[0236] User input

[0237] User: "I don't understand page 30(2) of the math drill."

[0238] Terminal: (Display the input contents in the form and confirm)

[0239] Sending data from the device to the server

[0240] Terminal: (Convert input to JSON and send to server)

[0241] Server analysis and identification of the problem

[0242] Server: (Analysis of "I don't understand page 30(2) of the calculation drill")

[0243] Server: (Identifies the problem in an internal database)

[0244] Server-generated description

[0245] Server: (Generates explanations using AI models)

[0246] Server: (Configure generated commentary)

[0247] Emotion recognition by emotion engine

[0248] Server: (Analyzes user emotions and detects anxiety)

[0249] Adjustment of commentary content

[0250] Server: (adjusts commentary based on perceived emotion)

[0251] Sending and displaying commentary

[0252] Server: (Package the explanation in JSON format)

[0253] Server: (Sent to the device)

[0254] Display explanation

[0255] Terminal: (Analyzes received data and converts it to HTML)

[0256] Terminal: (Explanation displayed on screen)

[0257] Device: (Audio description available if needed)

[0258] In this way, the system can improve the learning experience by providing accurate explanations in real time to any questions or anxieties the user may have while studying, and by providing optimal responses based on the user's emotional state.

[0259] The processing flow will be explained below.

[0260] Step 1: User enters question

[0261] If a user has a question about a problem, they can enter the question using the terminal. For example, they can enter "I don't understand page 30(2) of the calculation drill."

[0262] Step 2: Check input on the terminal and convert to data format

[0263] The device checks the user's input and converts it into an appropriate format (e.g., JSON), checking that the input has been converted correctly into the data format.

[0264] Step 3: Send to the server

[0265] The terminal sends formatted data to the server, which then sends the data to the specified endpoint while checking the stability of the communication and the integrity of the data.

[0266] Step 4: Data reception and analysis by the server

[0267] The server receives the data sent from the terminal, analyzes the question content based on the received data, and extracts information such as the teaching material name, page number, and question number.

[0268] Step 5: Identify the problem

[0269] Based on the information analyzed by the server, the server searches and identifies the relevant problem in the internal teaching material database. For example, it retrieves the problem on page 30(2) of the arithmetic drill from the database.

[0270] Step 6: Server-generated description

[0271] The server uses a generative AI model to generate detailed explanations for the identified problems, including the background to the problem, steps to solve it, and common mistakes.

[0272] Step 7: Emotion Recognition with the Emotion Engine

[0273] The server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial expressions, tone of voice, typing speed, etc. to determine whether the user is feeling anxious or annoyed.

[0274] Step 8: Adjust the commentary

[0275] The server adjusts the content and tone of the generated commentary based on the user's emotions recognized by the emotion engine. For example, if the user is feeling anxious, the server changes the commentary to be more friendly and encouraging.

[0276] Step 9: Converting the description to a data format

[0277] The server packages the adjusted description into an appropriate format (e.g., JSON or HTML), and the data is organized in a format that can be displayed on the user's device.

[0278] Step 10: Send data to your device

[0279] The server sends the formatted explanatory data to the terminal, and checks the integrity of the data while ensuring the stability and security of the communication.

[0280] Step 11: Receiving and analyzing data on the device

[0281] The terminal receives the commentary data sent from the server, analyzes the received data, and converts it into a format that can be displayed to the user (for example, HTML or text).

[0282] Step 12: Display the description

[0283] The device displays instructions on the user's screen, and optionally provides audio instructions using a voice output means. For example, detailed instructions such as "First, find the common denominator, then add the fractions."

[0284] Specific examples

[0285] 1. User: "I don't understand page 30(2) of the math drill."

[0286] 2. Terminal: (Converts input into JSON and sends it to the server)

[0287] 3. Server: (Analysis of "I don't understand page 30(2) of the math drill")

[0288] 4. Server: (Identify the problem in the internal database)

[0289] 5. Server: (Generates explanations using AI models)

[0290] 6. Server: (Analyze user emotions and detect anxiety)

[0291] 7. Server: (adjusts commentary based on perceived emotion)

[0292] 8. Server: (Package the description in JSON format)

[0293] 9. Server: (Send to device)

[0294] 10. Terminal: (Analyzes received data and converts it to HTML)

[0295] 11. Terminal: (Displays explanation on the screen)

[0296] 12. Terminal: (Audio description provided if needed)

[0297] This allows users to receive real-time explanations that resolve their questions and improve their learning experience by responding to their emotions.

[0298] Example 2

[0299] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0300] Conventional learning support systems have limited functionality for providing solutions to users' "unclear" problems in real time, and have difficulty providing appropriate explanations that take the user's emotional state into account. Furthermore, in order to provide accurate explanations to users' questions, it is necessary to analyze the user's input in detail and accurately recognize the user's emotional state. In this context, the present invention aims to realize a system that not only responds quickly and accurately to users' questions, but also provides optimal explanations based on the user's emotional state.

[0301] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0302] In this invention, the server includes means for receiving a question entered by a user, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for recognizing the user's emotion, means for adjusting the content of the generated explanation based on the recognized user emotion, means for sending the adjusted explanation to the user, and means for displaying the explanation sent to the user. This makes it possible to provide accurate and prompt answers to questions that the user has during the learning process, and further to provide optimal explanations according to the user's emotional state.

[0303] A "user" is a subject who uses the system to learn, enter questions, and receive explanations.

[0304] A "terminal" is a device that a user uses to input questions and that sends and receives information.

[0305] The "server" is a central system that analyzes the received question, generates an explanation, and transmits the explanation to the user.

[0306] "Question content" refers to a specific problem or question that a user wants to solve while studying.

[0307] The "receiving means" is a system for taking into the server the content of the question entered by the user through the terminal.

[0308] "Analysis means" refers to procedures and techniques for analyzing the content of the received question and identifying the relevant problem.

[0309] A "generative AI model" is an artificial intelligence model used to generate appropriate explanations for identified problems.

[0310] "Explanation" is information that includes answers and explanations to the user's questions.

[0311] "Emotion recognition means" refers to technology or systems for analyzing and identifying a user's emotional state.

[0312] "Adjustment means" refers to techniques or methods for adjusting the content and tone of generated commentary based on the perceived user sentiment.

[0313] "Transmission means" refers to a system or technology for transmitting the adjusted commentary to the user.

[0314] The "display means" refers to a system or function for displaying the commentary sent to the user on the screen.

[0315] This invention is a system that solves the "I don't understand" problem that users experience during the learning process, and combines it with an emotion engine that recognizes the user's emotions and adjusts the response content. The purpose of this invention is to improve the learning experience by providing appropriate explanations and taking into account the user's emotional state.

[0316] Hardware and software used

[0317] The main components of the system are:

[0318] 1. Terminal: The device through which the user enters their question, such as a PC, tablet, or smartphone.

[0319] 2. Server: A central system that analyzes questions, generates explanations, and recognizes user emotions. This can be a cloud server or an on-premise server.

[0320] 3. Generative AI model: An artificial intelligence model used to generate an appropriate explanation for an identified problem.

[0321] 4. Emotion engine: Software for recognizing the user's emotional state. This includes facial expression recognition software and voice analysis software.

[0322] Data processing and calculation

[0323] 1. Entering and receiving questions

[0324] When a user has a question while studying, they can enter it into the device. For example, they might write, "I don't understand page 30(2) of the math drill." The device will then check the input and convert it into the appropriate data format.

[0325] 2. Data Transmission

[0326] The device converts the input question into JSON format and sends it to the server, to an endpoint pre-specified within the system.

[0327] 3. Analysis and identification of questions

[0328] The server extracts the learning material name, page number, and question number from the received JSON data and searches the database for the corresponding question. Once the question is identified, it uses that information to generate a prompt for the generative AI model.

[0329] 4. Generating Explanations

[0330] The server uses a generative AI model to generate a commentary. Enter the following prompt:

[0331] "A user says he doesn't understand page 30(2) of the math drill. Please provide him with a step-by-step explanation starting with finding the common denominator. Also include an encouraging message to ease any anxiety he may be feeling."

[0332] The generative AI model generates step-by-step instructions and provides detailed explanations.

[0333] 5. Emotion recognition

[0334] The server uses an emotion engine to analyze the user's facial expression and voice data received from the device and identify the user's emotional state, for example, determining whether the user is feeling anxious or irritated.

[0335] 6. Adjustment of explanation content

[0336] The server adjusts the generated commentary based on information obtained from the emotion engine: for example, if the user is feeling anxious, it makes the commentary more friendly and adds encouraging elements.

[0337] 7. Sending and Displaying Commentary

[0338] The server converts the adjusted commentary into JSON or HTML format and sends it to the device. The device analyzes the received data and displays it in a format that is easy for the user to read. It also outputs audio if necessary.

[0339] Specific examples

[0340] The user enters "I don't understand page 30(2) of the calculation drill."

[0341] Terminal: Display input and ask user for confirmation

[0342] Terminal: Convert to JSON format and send to server

[0343] The server receives and analyzes the data

[0344] Server: Extract the teaching material name, page number, and question number

[0345] Server: Search database for matching problem

[0346] Generate explanations

[0347] Server: Enter prompt text using generative AI model: "A user says he doesn't understand page 30(2) of the math drill. Please provide him with a step-by-step explanation, starting with finding the common denominator. Include an encouraging message to ease any anxiety he may be feeling."

[0348] Generative AI model: Generate detailed explanations

[0349] emotion recognition

[0350] Server: Analyzes the user's facial expression and voice data to determine their emotional state

[0351] Adjustment of commentary content

[0352] Server: Adjust the tone and content of commentary depending on the emotional state

[0353] Sending and displaying commentary

[0354] Server: Converts the adjusted commentary into JSON or HTML format and sends it to the device.

[0355] Terminal: Analyzes received data and displays it on the screen. Also outputs audio if necessary.

[0356] In this way, the system can provide accurate explanations in real time to address any questions or concerns users may have while studying, providing an optimal learning experience.

[0357] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0358] Program processing flow

[0359] Step 1: Enter your question

[0360] If a user has a question while studying, they can enter it into the text field using the terminal. For example, they can enter "I don't understand page 30(2) of the calculation drill."

[0361] Input: Questions entered by the user

[0362] Output: The question displayed on the terminal

[0363] Specific behavior:

[0364] A user opens a learning app and writes a question in a text input field.

[0365] The terminal displays the entered text as is for the user to review.

[0366] Step 2: Data format conversion

[0367] The device converts the questions entered by the user into JSON format, which prepares the data in a suitable format for transmission.

[0368] Input: Text data entered by the user

[0369] Output: Data converted to JSON format

[0370] Specific behavior:

[0371] The terminal converts the input information into JSON format, dividing it into fields such as "teaching material name," "page number," and "question number."

[0372] The converted JSON data is passed to the next processing step.

[0373] Step 3: Send data

[0374] The device sends the question converted into JSON format to the server, to an endpoint pre-specified within the system.

[0375] Input: Data converted to JSON format

[0376] Output: Data sent to the server

[0377] Specific behavior:

[0378] The device sends the JSON data as an HTTPS request to the server's URL.

[0379] Step 4: Analyze and identify questions

[0380] The server parses the received JSON data, extracts the learning material name, page number, and question number, and uses this information to search and identify the corresponding question in its internal database.

[0381] Input: JSON data sent to the server

[0382] Output: Identification of the problem

[0383] Specific behavior:

[0384] The server extracts the fields "teaching material name," "page number," and "question number" from the JSON data.

[0385] Based on the extracted information, a query is run against the database to search for the relevant problem.

[0386] Step 5: Generate a description

[0387] The server uses a generative AI model to create a prompt based on the identified problem, which is then input into the AI ​​model to generate an explanation.

[0388] Input: Identified problem information

[0389] Output: Commentary by the generative AI model

[0390] Specific behavior:

[0391] The server creates a prompt sentence and inputs it into the generative AI model.

[0392] The generative AI model generates step-by-step instructions based on the prompt.

[0393] Example prompt statement:

[0394] "A user says he doesn't understand page 30(2) of the math drill. Please provide him with a step-by-step explanation starting with finding the common denominator. Also include an encouraging message to ease any anxiety he may be feeling."

[0395] Step 6: Recognize emotions

[0396] The server uses an emotion engine to analyze the user's emotions, analyzing the user's facial expressions, tone of voice, typing speed, etc. to identify the emotion the user is feeling.

[0397] Input: User's facial expression data, voice data

[0398] Output: User's emotional state

[0399] Specific behavior:

[0400] The server passes the webcam and microphone data received from the device to the emotion engine.

[0401] The emotion engine analyzes this data to determine the user's emotional state.

[0402] Step 7: Adjust the commentary

[0403] The server then adjusts the generated commentary based on the information obtained through emotion recognition, for example adding encouraging elements to the commentary if the user is feeling anxious.

[0404] Input: Emotion recognition data, generated commentary

[0405] Output: Adjusted description

[0406] Specific behavior:

[0407] The server reconstructs the generated commentary based on the emotional state.

[0408] The adjusted description is passed to the next processing step.

[0409] Step 8: Send and view your commentary

[0410] The post-adjustment explanation is converted into JSON or HTML format and sent to the device. The device analyzes the received data and displays it in a format that is easy for the user to read. It also outputs audio if necessary.

[0411] Input: Adjusted description

[0412] Output: The description displayed to the user

[0413] Specific behavior:

[0414] The server converts the adjusted description into JSON or HTML format.

[0415] The server sends this content to the terminal as an HTTPS response.

[0416] The device analyzes the received data and displays commentary on the screen, and also provides audio commentary if necessary.

[0417] In this way, the system can provide accurate explanations in real time to any questions or concerns users may have while studying, providing an optimal learning experience.

[0418] (Application example 2)

[0419] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0420] The problem that this invention aims to solve is to realize an effective learning experience by resolving the doubts and anxieties that users feel during learning in real time while providing emotionally sensitive explanations. Conventional learning support systems provide explanations without taking the user's emotional state into consideration, which has the problem of not being able to alleviate the anxiety and frustration that users feel during the learning process. This poses a risk of reducing the user's motivation to learn and reducing their level of understanding.

[0421] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0422] In this invention, the server includes means for receiving a question entered by a user, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for sending the generated explanation to the user, means for displaying the explanation sent to the user, means for analyzing the user's emotions, and means for adjusting the content of the explanation based on the analyzed emotions. This makes it possible to provide appropriate and friendly explanations while taking the user's emotional state into consideration.

[0423] The "means for receiving question content" refers to a hardware or software function for receiving a question entered by a user via a device.

[0424] "Means for analyzing the content of the question and identifying the relevant problem" refers to technology that analyzes the content of the received question using natural language processing and database search, and identifies the specific problem to be solved within the teaching materials or problem collection.

[0425] "Means for generating explanations using generative AI models" refers to technology that uses artificial intelligence to generate appropriate explanations and answers for identified problems.

[0426] The "means for transmitting the generated commentary to the user" refers to a communication means or protocol for transmitting the commentary generated on the server side to the user's device.

[0427] The "means for displaying the commentary sent to the user" refers to a screen display function or interface for visually displaying the commentary on the user's device.

[0428] "Means for analyzing user emotions" refers to technology that analyzes the user's facial expressions, tone of voice, input speed, etc., to identify the emotions the user is feeling.

[0429] The "means for adjusting the content of the commentary based on the analyzed emotions" is a technology for changing the tone and content of the commentary according to the user's emotional state, and adding friendly explanations and encouraging words.

[0430] This invention is a system that solves the problem of "I don't understand" that users experience during the learning process, and in particular, it combines an "emotion engine" that recognizes the user's emotions and adjusts the response accordingly. This system aims to improve the learning experience by taking into account the user's emotional state while providing appropriate explanations.

[0431] User input of question

[0432] If a user has a question while studying, they can input it into a device such as a smartphone or head-mounted display. For example, they can input a question into the device, such as, "I don't understand page 30(2) of the math drill." The device receives the user's input and processes it as appropriately formatted data.

[0433] Sending data from the device to the server

[0434] The terminal converts the question entered by the user into a data format (for example, JSON format) and sends it to the specified endpoint of the server. The server receives the user's question and is ready to move on to the next analysis process.

[0435] Server analysis and identification of the problem

[0436] The server analyzes the received question data and identifies the relevant question from the question content. Specifically, it extracts information such as the name of the learning material, page number, and question number from the received data and searches the database on the server for the relevant question.

[0437] Server-generated description

[0438] After identifying the problem, the server uses a generative AI model to generate a solution for the problem. The server creates a detailed solution that includes step-by-step instructions, important points to note, and background information on the problem. For example, it might say, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator."

[0439] Emotion recognition by emotion engine

[0440] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's facial expressions, tone of voice, typing speed, etc. to identify the emotion the user is feeling. For example, it detects whether the user is feeling anxious or irritated.

[0441] Adjustment of commentary content

[0442] The server adjusts the content and tone of the generated commentary based on the user's emotions recognized by the emotion engine. For example, if the user is feeling anxious, the server changes the content of the commentary to be more friendly and encouraging.

[0443] Sending and displaying commentary

[0444] The server packages the generated commentary in an appropriate data format (for example, JSON or HTML format) and sends it to the terminal. The terminal analyzes the commentary data received from the server and displays it on the screen in a format that is easy for the user to understand. If necessary, the terminal can also provide audio commentary using audio output means.

[0445] Specific examples of hardware and software used

[0446] Hardware used: smartphone, head-mounted display (HMD)

[0447] Software used:

[0448] Emotion Recognizer: Identifies emotions by analyzing the user's facial expressions and tone of voice.

[0449] Generative AI model (Explanation Generator): Generates appropriate explanations for the user's questions.

[0450] Text to Speech Engine (Text to Speech): Provides the generated commentary in audio format.

[0451] Virtual Display Engine (VirtualDisplay): Visually displays the generated commentary to the user.

[0452] Specific examples

[0453] User input

[0454] User: "I don't know how to add fractions."

[0455] Terminal: (Display the input contents in the form and confirm)

[0456] Sending data from the device to the server

[0457] Terminal: (Convert input to JSON and send to server)

[0458] Server analysis and identification of the problem

[0459] Server: ("I don't know how to add fractions" analysis)

[0460] Server: (Identifies the problem in an internal database)

[0461] Server-generated description

[0462] Server: (Generate explanations using generative AI models)

[0463] Server: (Configure generated commentary)

[0464] Emotion recognition by emotion engine

[0465] Server: (Analyzes user emotions and detects anxiety)

[0466] Adjustment of commentary content

[0467] Server: (adjusts commentary based on perceived emotion)

[0468] Sending and displaying commentary

[0469] Server: (Package the explanation in JSON format)

[0470] Server: (Sent to the device)

[0471] Display explanation

[0472] Terminal: (Analyzes received data and converts it to HTML)

[0473] Terminal: (Explanation displayed on screen)

[0474] Device: (Audio description available if needed)

[0475] Prompt Sentence Examples

[0476] If the user types "I don't know how to add fractions":

[0477] (Prompt for generative AI model)

[0478] "The user feels that they don't understand how to add fractions. Please explain carefully, especially emphasizing the step of finding the 'common denominator.' The user seems confused, so please explain slowly and clearly."

[0479] In this way, the system can improve the learning experience by providing accurate explanations in real time to any questions or anxieties the user may have while studying, and by providing optimal responses based on the user's emotional state.

[0480] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0481] Step 1:

[0482] User enters question

[0483] Input: The user uses a device (smartphone or head-mounted display) to input a question, such as "I don't know how to add fractions," using text or voice.

[0484] Specific operation: The terminal receives the user's input and displays it in a form for confirmation.

[0485] Step 2:

[0486] Sending data from the device to the server

[0487] Input: The question typed by the user

[0488] Data processing: The terminal converts the input content into JSON format.

[0489] Output: JSON data sent to the server

[0490] Specific operation: The terminal sends the converted JSON data to the specified endpoint on the server.

[0491] Step 3:

[0492] Server analysis and identification of the problem

[0493] Input: JSON data of the received question

[0494] Data calculation: The server analyzes the received question data, extracts information such as the name of the teaching material, page number, and question number from the question content, and performs a database search.

[0495] Output: Identifying the problem in question

[0496] Specific operation: The server searches the database for the corresponding problem and identifies detailed information about the problem.

[0497] Step 4:

[0498] Server-generated description

[0499] Input: Details of the problem in question

[0500] Data Computation: Uses a generative AI model to generate a step-by-step explanation for the problem.

[0501] Output: Generated commentary

[0502] How it works: The server uses a generative AI model to create an explanation, such as "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator."

[0503] Step 5:

[0504] Emotion recognition by emotion engine

[0505] Input: Data such as the user's facial expression, tone of voice, and typing speed

[0506] Data Calculation: An emotion engine is used to analyze this data and identify the emotions the user is feeling.

[0507] Output: Recognized user emotion information

[0508] Specific operation: The server analyzes the user's emotions and detects feelings such as anxiety or irritation.

[0509] Step 6:

[0510] Adjustment of commentary content

[0511] Input: Generated commentary and user's emotional information

[0512] Data arithmetic: Adjust the tone and content of your commentary based on emotion, for example, using friendly language if the user is feeling anxious.

[0513] Output: Adjusted commentary

[0514] Specific operation: The server generates explanatory content that takes the user's feelings into consideration, such as, "Don't worry. Let's take it step by step. To find the common denominator..."

[0515] Step 7:

[0516] Sending and displaying commentary

[0517] Input: Adjusted commentary

[0518] Data processing: The generated commentary is packaged into an appropriate data format (JSON or HTML format) and sent to the terminal.

[0519] Output: Descriptive data converted into a displayable format

[0520] Specific operation: The device analyzes the commentary data received from the server and displays it in a format that is easy for the user to understand. Audio commentary is also provided if necessary.

[0521] Through the above steps, the system provides accurate explanations in real time to any questions or anxieties the user may have while studying, and responds optimally according to the user's emotional state.

[0522] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0523] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0524] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0525] [Second embodiment]

[0526] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0527] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0528] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0529] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0530] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0531] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0532] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0533] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0534] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0535] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0536] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0537] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0538] The present invention is a system for solving "unclear" problems that a user encounters in the course of his / her own learning. Specific embodiments will now be described.

[0539] User input

[0540] When a user has a question about a problem, they enter the question into the terminal. For example, they might say, "I don't understand page 30(2) of the calculation drill." The terminal receives the user's input and processes it as properly formatted data.

[0541] Sending data from the device to the server

[0542] The terminal converts the question entered by the user into a data format (for example, JSON format) and sends it to the specified endpoint of the server. This allows the server to receive the user's question and prepare for the next analysis process.

[0543] Server analysis and identification of the problem

[0544] The server analyzes the received question data and identifies the relevant question from the question content. Specifically, it extracts information such as the name of the learning material, page number, and question number from the received data and searches the database on the server for the relevant question.

[0545] Server-generated description

[0546] After identifying the problem, the server uses a generative AI model to generate a solution for the problem. The server creates a detailed solution that includes step-by-step instructions, important points to note, and background information on the problem. For example, the solution might include, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator. The least common multiple of 4 and 3 is 12."

[0547] Sending and displaying commentary

[0548] The server packages the generated commentary in an appropriate data format (for example, JSON or HTML format) and sends it to the terminal. The terminal analyzes the commentary data received from the server and displays it on the screen in a format that is easy for the user to understand. If necessary, the terminal can also provide audio commentary using audio output means.

[0549] Specific examples

[0550] User input

[0551] User: "I don't understand page 30(2) of the math drill."

[0552] Terminal: (Display the input contents in the form and confirm)

[0553] Sending data from the device to the server

[0554] Terminal: (Convert input to JSON and send to server)

[0555] Server analysis and identification of the problem

[0556] Server: (Analysis of "I don't understand page 30(2) of the calculation drill")

[0557] Server: (Identifies the problem in an internal database)

[0558] Server-generated description

[0559] Server: (Generates explanations using AI models)

[0560] Server: (Configure generated commentary)

[0561] Sending and displaying commentary

[0562] Server: (Package the explanation in JSON format)

[0563] Server: (Sent to the device)

[0564] Display explanation

[0565] Terminal: (Analyzes received data and converts it to HTML)

[0566] Terminal: (Explanation displayed on screen)

[0567] Device: (Audio description available if needed)

[0568] In this way, this system can provide real-time explanations to users' (especially learners') questions and effectively support home learning. It can be used as a tool to properly support children's learning even when parents are busy, and is expected to improve the quality of home learning.

[0569] The processing flow will be explained below.

[0570] Step 1: User enters question

[0571] If a user has a question about a homework assignment or problem, they can use the terminal to enter their question, for example, "I don't understand page 30(2) of the calculation drill."

[0572] Step 2: Check input on the terminal and convert to data format

[0573] The device checks the user's input and converts it into an appropriate format (e.g., JSON), checking that the input has been converted correctly into the data format.

[0574] Step 3: Send to the server

[0575] The terminal sends formatted data to the server, which then sends the data to the specified endpoint while checking the stability of the communication and the integrity of the data.

[0576] Step 4: Data reception and analysis by the server

[0577] The server receives the data sent from the terminal, analyzes the question content based on the received data, and extracts information such as the name of the learning material, page number, and question number.

[0578] Step 5: Identify the problem

[0579] Based on the information analyzed by the server, the server searches and identifies the relevant problem in the internal teaching material database. For example, it retrieves the problem on page 30(2) of the arithmetic drill from the database.

[0580] Step 6: Server-generated description

[0581] The server uses a generative AI model to generate a detailed explanation for the identified problem, including the background to the problem, the steps to solving it, and common mistakes.

[0582] Step 7: Converting the description to a data format

[0583] The server converts the generated description into an appropriate format (e.g., JSON or HTML), and the data is organized in a format that can be displayed on the user's device.

[0584] Step 8: Send data to your device

[0585] The server sends the formatted explanatory data to the terminal, and checks the integrity of the data while ensuring the stability and security of the communication.

[0586] Step 9: Data reception and analysis by the device

[0587] The terminal receives the commentary data sent from the server, analyzes the received data, and converts it into a format that can be displayed to the user (for example, HTML or text).

[0588] Step 10: Display the description

[0589] The device displays instructions on the user's screen, and optionally provides audio instructions using a voice output means. For example, detailed instructions such as "First, find the common denominator, then add the fractions."

[0590] This process allows users to receive explanations that resolve their questions in real time, enabling them to study effectively.

[0591] Example 1

[0592] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0593] Conventional learning support systems lack the means to instantly resolve questions users have while studying. This can lead to a decline in learning effectiveness and a loss of motivation to study. Furthermore, when parents or teachers are busy, there is also the problem that learners are unable to receive appropriate support. The present invention aims to solve these problems and provide a system that allows learners to instantly resolve their questions.

[0594] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0595] In this invention, the server includes means for receiving a question entered by a user, means for converting the received question into a data format, means for transmitting the converted data to the server, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for transmitting the generated explanation to the user, and means for displaying the explanation transmitted to the user. This makes it possible to provide quick and accurate explanations to questions that users may have while studying.

[0596] "User" refers to an individual who uses the system to receive learning support.

[0597] "Terminal" refers to an electronic device that allows a user to access and input data into the system.

[0598] "Question content" refers to information that expresses doubts that arise during the user's studies or problems that the user wants to solve.

[0599] "Data format" refers to the method by which data is organized and converted into an appropriate form for transmission from a device to a server.

[0600] "Server" refers to a central computer that receives, analyzes, and processes data sent from the terminals.

[0601] "Analysis" refers to the process by which the server breaks down and understands the data it receives and extracts specific information.

[0602] "Applicable questions" refer to specific study tasks related to the question submitted by the user.

[0603] A "generative AI model" refers to artificial intelligence that uses natural language processing technology to generate explanations for questions.

[0604] "Explanation" refers to the explanation and specific solution steps provided by the generative AI model for an identified problem.

[0605] "Data transmission" refers to the process of sending information from a terminal to a server or from a server to a terminal.

[0606] "Display" refers to the terminal providing the received commentary data to the user visually or audibly.

[0607] This invention relates to a learning support system that allows users to instantly resolve questions they may have while studying. This system receives questions entered by users and provides appropriate explanations via a server. Specific embodiments of this system are described below.

[0608] The user inputs the question using their own device (e.g., PC, smartphone, tablet). For example, they might input, "I don't understand page 30(2) of the math drill." The device then displays this input in a form and asks the user to confirm it.

[0609] The terminal converts the questions entered by the user into a data format such as JSON. For example, it converts the questions into JSON data like this:

[0610] json

[0611] {

[0612] "Material name": "Calculation drill",

[0613] "Page Number": 30,

[0614] "Problem Number": 2,

[0615] "Question": "I don't understand page 30(2)"

[0616] }

[0617] The converted data is sent over a network connection to a specific endpoint on the server, which receives the data.

[0618] The server parses the received JSON data. During parsing, it extracts information such as the teaching material name, page number, and question number. For example, it searches the database on the server using the condition "teaching material name = arithmetic drill AND page number = 30 AND question number = 2" to identify the corresponding question.

[0619] After the problem is identified, the server uses a generative AI model (e.g., a model using natural language processing technology) to generate an explanation for the problem. An example of a prompt that the server inputs to the generative AI model is, "Please provide an explanation for page 30(2) of the calculation drill."

[0620] The generated explanation includes specific calculation steps and important points to note. For example, the explanation generated might say, "To solve the calculation 3 / 4 + 2 / 3, you first need to find a common denominator. The least common multiple of 4 and 3 is 12."

[0621] After the description is generated, the server repackages it into JSON format and sends it to the device using an HTTP POST request. The device then parses the received description data and converts it into a user-readable HTML format. It can also provide audio description using speech synthesis software if needed.

[0622] This system allows users to instantly resolve any questions that arise during their studies, improving the quality of their learning and effectively supporting home study. A specific example of a prompt sentence is "I don't understand page 30(2) of the math drill," and real-time explanations are provided.

[0623] In this way, the present invention is expected to increase learners' motivation to learn and improve their learning effectiveness by quickly and accurately resolving the learning problems that users have. The overall configuration and operation of the system is realized based on the cooperation between terminals and a server.

[0624] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0625] Step 1:

[0626] If a user has a question while studying, they can input it using a device (e.g., PC, smartphone, tablet). An example of the input content is "I don't understand page 30(2) of the calculation drill." The device displays this input in a form for the user to confirm. Input: User's question. Output: Confirmed question.

[0627] Step 2:

[0628] The terminal converts the questions the user has confirmed into JSON format. For example, it converts it into JSON data like this:

[0629] json

[0630] {

[0631] "Material name": "Calculation drill",

[0632] "Page Number": 30,

[0633] "Problem Number": 2,

[0634] "Question": "I don't understand page 30(2)"

[0635] }

[0636] In this conversion process, each element of the question (teaching material name, page number, question number, question content) is extracted and put into a JSON structure. Input: Confirmed question content. Output: Data in JSON format.

[0637] Step 3:

[0638] The device sends the converted JSON data to the server using an HTTP POST request, using a network connection to securely send the data to a specific endpoint on the server. Input: JSON format data. Output: Data sent to the server.

[0639] Step 4:

[0640] The server parses the received JSON data. During parsing, it extracts information such as the learning material name, page number, and question number. For example, it parses the data as follows:

[0641] Material name: Calculation drill

[0642] Page number: 30

[0643] Issue Number: 2

[0644] Input: JSON formatted data. Output: Extracted question elements.

[0645] Step 5:

[0646] The server uses the extracted information to search its internal database and identify the relevant questions. Specifically, it searches using the query "Material name = Calculation drill AND Page number = 30 AND Question number = 2". Input: Extracted question elements. Output: Identification of the relevant questions.

[0647] Step 6:

[0648] The server generates an explanation for the identified problem using a generative AI model (e.g., a model using natural language processing technology). The prompt sentence that the server inputs to the generative AI model is, "Please provide an explanation for page 30(2) of the calculation drill." The generative AI model returns a specific explanation. Input: Prompt sentence. Output: Generated explanation.

[0649] Step 7:

[0650] The server then packages the generated description back into JSON, for example into the following JSON data:

[0651] json

[0652] {

[0653] "Explanation": "To solve the equation 3 / 4 + 2 / 3, we first need to find a common denominator. The least common multiple of 4 and 3 is 12..."

[0654] }

[0655] Input: Generated description. Output: Description data in JSON format.

[0656] Step 8:

[0657] The server sends the packaged description data to the device using an HTTP POST request. In this process, data is transferred to the device via the network. Input: Description data in JSON format. Output: Data sent to the device.

[0658] Step 9:

[0659] The device analyzes the explanatory data in JSON format received from the server. The analyzed data is converted into HTML format and displayed on the screen in a format that is easy for the user to read. Input: Explanation data in JSON format. Output: Explanation displayed in HTML format.

[0660] Step 10:

[0661] If necessary, the device will use speech synthesis software to provide audio descriptions, allowing users to hear as well as see the information. Input: Description data in HTML format. Output: Audio description.

[0662] In this way, the questions entered by the user are processed through a number of steps and an explanation is quickly generated and displayed. By providing immediate answers to questions during learning, learners can deepen their understanding and increase their motivation to learn.

[0663] (Application example 1)

[0664] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0665] In factories, when operational problems occur with machines or robots, it is difficult for engineers to respond quickly and resolve them. In particular, if specific explanations about operation are lacking, it takes time to resolve the problem, resulting in a decrease in production efficiency. The present invention aims to provide technology for quickly and appropriately resolving operational problems and machine issues encountered by workers and engineers in factories.

[0666] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0667] In this invention, the server includes means for receiving a question entered by a user, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for sending the generated explanation to the user, means for displaying the explanation sent to the user, means for receiving and analyzing a question about an operation problem of a factory robot and generating an explanation using a generative AI model, and means for sending and displaying the generated explanation to the display and audio output means of the factory robot, thereby enabling workers and engineers in a factory to quickly and appropriately solve operational problems and issues with machines and robots.

[0668] "Users" refer to factory workers and engineers who use the system.

[0669] "Questions" refer to inquiries about operational problems and machine issues entered by factory workers and engineers through the system.

[0670] "Means for receiving" refers to a method or device that allows the system to acquire and store the content of the question entered by the user.

[0671] "Means for analyzing" refers to a method or device that analyzes the content of the received query and identifies the problem based on the information contained therein.

[0672] "Means of identification" refers to the methods and devices used to identify specific operational problems or machine issues from the analyzed questions.

[0673] A "generative AI model" refers to an algorithm or program that uses machine learning and natural language processing techniques to generate explanations for identified problems.

[0674] "Means for generating explanations" refers to a method or device that utilizes a generative AI model to create detailed explanations and solutions to an identified problem.

[0675] "Means for transmitting to a user" refers to a method or device for transmitting the generated commentary from the system to a user.

[0676] "Means for displaying" refers to a method or device for displaying the transmitted commentary in a form that can be understood by the user.

[0677] "Factory robot" refers to an industrial robot on which the present system is installed.

[0678] "Operational trouble" refers to problems or malfunctions that occur during the operation of factory robots and other related machinery.

[0679] "Display" refers to a screen or monitor for visually displaying the generated commentary.

[0680] The "audio output means" refers to a speaker or a voice synthesizer for transmitting the generated commentary to the user by voice.

[0681] "Data format" refers to the standard data structure and format used when communicating question content and explanation data between servers and systems.

[0682] The system program that realizes this application example is designed to solve operational problems of factory robots. Below we explain how this system works.

[0683] Hardware and software used

[0684] The system mainly uses the following hardware and software:

[0685] Hardware: Factory robot body, touch panel, speaker, central server

[0686] Software: Question receiving program, data analysis program, generative AI model (e.g., OpenAI's GPT series), data transmission and reception program

[0687] User operations and question input

[0688] Workers and engineers (users) use the factory robot's touch panel or voice input function to input questions about operational problems into the system. For example, they can input a question like, "I don't know why the robot arm on line 3 isn't moving."

[0689] Sending data of question content

[0690] The question is converted to JSON format and sent to a central server. The server uses an analysis program to analyze the received question and identify the problem. For example, the JSON-formatted question data may contain information such as "Question: 'The robot arm on line 3 is not moving.'"

[0691] Server-based problem analysis and explanation generation

[0692] The server uses an analysis program to analyze the question and generates an explanation for the identified problem using a generative AI model. For example, after analyzing the question, the generative AI model generates an explanation such as, "It is highly likely that a foreign object is stuck in the sensor. Please check the sensor first."

[0693] Data transmission and display of commentary

[0694] The generated explanation is then packaged again in JSON format and sent to the factory robot, which then displays the explanation on its display and, if necessary, outputs it aloud, allowing workers and technicians to quickly understand how to solve the problem.

[0695] Examples and prompts

[0696] For example, a factory worker may input a question through a display, such as, "A sensor error is displayed. What should I do?" This question data is converted into JSON format and sent to a central server for analysis. The generative AI model generates a specific explanation, such as, "The possible cause of the sensor error is that a foreign object is stuck in the sensor. First, check the sensor and see if there is a foreign object." The explanation is sent to a factory robot, which then provides the explanation on the display and in voice.

[0697] Example prompt sentence:

[0698] "Please tell me why the machine isn't working. For example, the robot arm on line 3 has stopped working."

[0699] In this way, the system helps factory workers and engineers quickly and efficiently resolve operational problems they encounter.

[0700] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0701] Step 1:

[0702] A user uses the factory robot's touch panel or voice input function to input questions about operational problems. This input is in the form of a string, such as "I don't know why the robot arm on line 3 isn't moving."

[0703] Step 2:

[0704] The terminal converts the question entered by the user into JSON format. It parses the entered string and converts it into JSON data in the format of, for example, "Question: 'The robot arm on line 3 is not moving.'" This data format conversion results in structured data.

[0705] Step 3:

[0706] The device sends the converted JSON-formatted question data to the central server, which receives the data sent from the device via an API endpoint, thereby obtaining JSON data containing the question content.

[0707] Step 4:

[0708] The server analyzes the received JSON data and identifies the relevant problem from the question. Specifically, it analyzes the "question" field of the received data and extracts information about the relevant machine and the problem. For example, it performs an analysis based on the information, "The robot arm on line 3 is not moving."

[0709] Step 5:

[0710] The server uses a generative AI model to generate an explanation based on the analyzed question. Using a generative AI model (e.g., OpenAI's GPT series), the server generates an explanation based on the analysis results. For example, it creates an explanation such as, "There is a high possibility that a foreign object is stuck in the sensor. Please check the sensor first."

[0711] Step 6:

[0712] The server then packages the generated explanation into JSON format again and sends it to the device. This operation sends explanation data from the server to the device. Specifically, JSON data in the format "Explanation: 'There is a high possibility that a foreign object is stuck in the sensor. Please check the sensor first.'" is sent.

[0713] Step 7:

[0714] The device analyzes the received JSON-formatted explanatory data and converts it into a displayable format. Based on the analysis results, the explanation is converted into text or HTML format. This data conversion makes it possible to provide information to the user visually and audibly.

[0715] Step 8:

[0716] The device displays the converted commentary on the display and outputs it as audio if necessary. The user can check the commentary on the display and, in some cases, listen to the commentary via audio guidance. For example, the display may say, "There is a high possibility that a foreign object is stuck in the sensor. Please check the sensor first."

[0717] By following the steps explained in this manual, users can quickly and accurately identify the cause and solution of operational problems, thereby improving production efficiency and reducing the time required to resolve problems.

[0718] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0719] The present invention combines a system that solves the "I don't understand" problem that users experience during the learning process with an emotion engine that recognizes the user's emotions and adjusts the response content. The purpose of this invention is to improve the learning experience by taking into account the user's emotional state as well as providing appropriate explanations.

[0720] User input of question

[0721] When a user has a question about a problem, they enter the question into the terminal. For example, they might say, "I don't understand page 30(2) of the calculation drill." The terminal receives the user's input and processes it as properly formatted data.

[0722] Sending data from the device to the server

[0723] The terminal converts the question entered by the user into a data format (for example, JSON format) and sends it to the specified endpoint of the server. This allows the server to receive the user's question and prepare to move on to the next analysis process.

[0724] Server analysis and identification of the problem

[0725] The server analyzes the received question data and identifies the relevant question from the question content. Specifically, it extracts information such as the name of the learning material, page number, and question number from the received data and searches the database on the server for the relevant question.

[0726] Server-generated description

[0727] After identifying the problem, the server uses a generative AI model to generate a solution for the problem. The server creates a detailed solution that includes step-by-step instructions, important points to note, and background information on the problem. For example, it might say, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator."

[0728] Emotion recognition by emotion engine

[0729] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's facial expressions, tone of voice, typing speed, etc. to identify the emotion the user is feeling. For example, it detects whether the user is feeling anxious or irritated.

[0730] Adjustment of commentary content

[0731] The server adjusts the content and tone of the generated commentary based on the user's emotions recognized by the emotion engine. For example, if the user is feeling anxious, the server changes the content of the commentary to be more friendly and encouraging.

[0732] Sending and displaying commentary

[0733] The server packages the generated commentary in an appropriate data format (for example, JSON or HTML format) and sends it to the terminal. The terminal analyzes the commentary data received from the server and displays it on the screen in a format that is easy for the user to understand. If necessary, the terminal can also provide audio commentary using audio output means.

[0734] Specific examples

[0735] User input

[0736] User: "I don't understand page 30(2) of the math drill."

[0737] Terminal: (Display the input contents in the form and confirm)

[0738] Sending data from the device to the server

[0739] Terminal: (Convert input to JSON and send to server)

[0740] Server analysis and identification of the problem

[0741] Server: (Analysis of "I don't understand page 30(2) of the calculation drill")

[0742] Server: (Identifies the problem in an internal database)

[0743] Server-generated description

[0744] Server: (Generates explanations using AI models)

[0745] Server: (Configure generated commentary)

[0746] Emotion recognition by emotion engine

[0747] Server: (Analyzes user emotions and detects anxiety)

[0748] Adjustment of commentary content

[0749] Server: (adjusts commentary based on perceived emotion)

[0750] Sending and displaying commentary

[0751] Server: (Package the explanation in JSON format)

[0752] Server: (Sent to the device)

[0753] Display explanation

[0754] Terminal: (Analyzes received data and converts it to HTML)

[0755] Terminal: (Explanation displayed on screen)

[0756] Device: (Audio description available if needed)

[0757] In this way, the system can improve the learning experience by providing accurate explanations in real time to any questions or anxieties the user may have while studying, and by providing optimal responses based on the user's emotional state.

[0758] The processing flow will be explained below.

[0759] Step 1: User enters question

[0760] If a user has a question about a problem, they can enter the question using the terminal. For example, they can enter "I don't understand page 30(2) of the calculation drill."

[0761] Step 2: Check input on the terminal and convert to data format

[0762] The device checks the user's input and converts it into an appropriate format (e.g., JSON), checking that the input has been converted correctly into the data format.

[0763] Step 3: Send to the server

[0764] The terminal sends formatted data to the server, which then sends the data to the specified endpoint while checking the stability of the communication and the integrity of the data.

[0765] Step 4: Data reception and analysis by the server

[0766] The server receives the data sent from the terminal, analyzes the question content based on the received data, and extracts information such as the name of the learning material, page number, and question number.

[0767] Step 5: Identify the problem

[0768] Based on the information analyzed by the server, the server searches and identifies the relevant problem in the internal teaching material database. For example, it retrieves the problem on page 30(2) of the arithmetic drill from the database.

[0769] Step 6: Server-generated description

[0770] The server uses a generative AI model to generate a detailed explanation for the identified problem, including the background to the problem, the steps to solving it, and common mistakes.

[0771] Step 7: Emotion Recognition with the Emotion Engine

[0772] The server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial expressions, tone of voice, typing speed, etc. to determine whether the user is feeling anxious or annoyed.

[0773] Step 8: Adjust the commentary

[0774] The server adjusts the content and tone of the generated commentary based on the user's emotions recognized by the emotion engine. For example, if the user is feeling anxious, the server changes the commentary to be more friendly and encouraging.

[0775] Step 9: Converting the description to a data format

[0776] The server packages the adjusted description into an appropriate format (e.g., JSON or HTML), and the data is organized in a format that can be displayed on the user's device.

[0777] Step 10: Send data to your device

[0778] The server sends the formatted explanatory data to the terminal, and checks the integrity of the data while ensuring the stability and security of the communication.

[0779] Step 11: Receiving and analyzing data on the device

[0780] The terminal receives the commentary data sent from the server, analyzes the received data, and converts it into a format that can be displayed to the user (for example, HTML or text).

[0781] Step 12: Display the description

[0782] The device displays instructions on the user's screen, and optionally provides audio instructions using a voice output means. For example, detailed instructions such as "First, find the common denominator, then add the fractions."

[0783] Specific examples

[0784] 1. User: "I don't understand page 30(2) of the math drill."

[0785] 2. Terminal: (Converts input into JSON and sends it to the server)

[0786] 3. Server: (Analysis of "I don't understand page 30(2) of the math drill")

[0787] 4. Server: (Identify the problem in the internal database)

[0788] 5. Server: (Generates explanations using AI models)

[0789] 6. Server: (Analyze user emotions and detect anxiety)

[0790] 7. Server: (adjusts commentary based on perceived emotion)

[0791] 8. Server: (Package the description in JSON format)

[0792] 9. Server: (Send to device)

[0793] 10. Terminal: (Analyzes received data and converts it to HTML)

[0794] 11. Terminal: (Displays explanation on the screen)

[0795] 12. Terminal: (Audio description provided if needed)

[0796] This allows users to receive real-time explanations that resolve their questions and improve their learning experience by responding to their emotions.

[0797] Example 2

[0798] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0799] Conventional learning support systems have limited functionality for providing solutions to users' "unclear" problems in real time, and have difficulty providing appropriate explanations that take the user's emotional state into account. Furthermore, in order to provide accurate explanations to users' questions, it is necessary to analyze the user's input in detail and accurately recognize the user's emotional state. In this context, the present invention aims to realize a system that not only responds quickly and accurately to users' questions, but also provides optimal explanations based on the user's emotional state.

[0800] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0801] In this invention, the server includes means for receiving a question entered by a user, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for recognizing the user's emotion, means for adjusting the content of the generated explanation based on the recognized user emotion, means for sending the adjusted explanation to the user, and means for displaying the explanation sent to the user. This makes it possible to provide accurate and prompt answers to questions that the user has during the learning process, and further to provide optimal explanations according to the user's emotional state.

[0802] A "user" is a subject who uses the system to learn, enter questions, and receive explanations.

[0803] A "terminal" is a device that a user uses to input questions and that sends and receives information.

[0804] The "server" is a central system that analyzes the received question, generates an explanation, and transmits the explanation to the user.

[0805] "Question content" refers to a specific problem or question that a user wants to solve while studying.

[0806] The "receiving means" is a system for taking into the server the content of the question entered by the user through the terminal.

[0807] "Analysis means" refers to procedures and techniques for analyzing the content of the received question and identifying the relevant problem.

[0808] A "generative AI model" is an artificial intelligence model used to generate appropriate explanations for identified problems.

[0809] "Explanation" is information that includes answers and explanations to the user's questions.

[0810] "Emotion recognition means" refers to technology or systems for analyzing and identifying a user's emotional state.

[0811] "Adjustment means" refers to techniques or methods for adjusting the content and tone of generated commentary based on the perceived user sentiment.

[0812] "Transmission means" refers to a system or technology for transmitting the adjusted commentary to the user.

[0813] The "display means" refers to a system or function for displaying the commentary sent to the user on the screen.

[0814] This invention is a system that solves the "I don't understand" problem that users experience during the learning process, and combines it with an emotion engine that recognizes the user's emotions and adjusts the response content. The purpose of this invention is to improve the learning experience by providing appropriate explanations and taking into account the user's emotional state.

[0815] Hardware and software used

[0816] The main components of the system are:

[0817] 1. Terminal: The device through which the user enters their question, such as a PC, tablet, or smartphone.

[0818] 2. Server: A central system that analyzes questions, generates explanations, and recognizes user emotions. This can be a cloud server or an on-premise server.

[0819] 3. Generative AI model: An artificial intelligence model used to generate an appropriate explanation for an identified problem.

[0820] 4. Emotion engine: Software for recognizing the user's emotional state. This includes facial expression recognition software and voice analysis software.

[0821] Data processing and calculation

[0822] 1. Entering and receiving questions

[0823] When a user has a question while studying, they can enter it into the device. For example, they might write, "I don't understand page 30(2) of the math drill." The device will then check the input and convert it into the appropriate data format.

[0824] 2. Data Transmission

[0825] The device converts the input question into JSON format and sends it to the server, to an endpoint pre-specified within the system.

[0826] 3. Analysis and identification of questions

[0827] The server extracts the learning material name, page number, and question number from the received JSON data and searches the database for the corresponding question. Once the question is identified, it uses that information to generate a prompt for the generative AI model.

[0828] 4. Generating Explanations

[0829] The server uses a generative AI model to generate a commentary. Enter the following prompt:

[0830] "A user says he doesn't understand page 30(2) of the math drill. Please provide him with a step-by-step explanation starting with finding the common denominator. Also include an encouraging message to ease any anxiety he may be feeling."

[0831] The generative AI model generates step-by-step instructions and provides detailed explanations.

[0832] 5. Emotion recognition

[0833] The server uses an emotion engine to analyze the user's facial expression and voice data received from the device and identify the user's emotional state, for example, determining whether the user is feeling anxious or irritated.

[0834] 6. Adjustment of explanation content

[0835] The server adjusts the generated commentary based on information obtained from the emotion engine: for example, if the user is feeling anxious, it makes the commentary more friendly and adds encouraging elements.

[0836] 7. Sending and Displaying Commentary

[0837] The server converts the adjusted commentary into JSON or HTML format and sends it to the device. The device analyzes the received data and displays it in a format that is easy for the user to read. It also outputs audio if necessary.

[0838] Specific examples

[0839] The user enters "I don't understand page 30(2) of the calculation drill."

[0840] Terminal: Display input and ask user for confirmation

[0841] Terminal: Convert to JSON format and send to server

[0842] The server receives and analyzes the data

[0843] Server: Extract the teaching material name, page number, and question number

[0844] Server: Search database for matching problem

[0845] Generate explanations

[0846] Server: Enter prompt text using generative AI model: "A user says he doesn't understand page 30(2) of the math drill. Please provide him with a step-by-step explanation, starting with finding the common denominator. Include an encouraging message to ease any anxiety he may be feeling."

[0847] Generative AI model: Generate detailed explanations

[0848] emotion recognition

[0849] Server: Analyzes the user's facial expression and voice data to determine their emotional state

[0850] Adjustment of commentary content

[0851] Server: Adjust the tone and content of commentary depending on the emotional state

[0852] Sending and displaying commentary

[0853] Server: Converts the adjusted commentary into JSON or HTML format and sends it to the device.

[0854] Terminal: Analyzes received data and displays it on the screen. Also outputs audio if necessary.

[0855] In this way, the system can provide accurate explanations in real time to address any questions or concerns users may have while studying, providing an optimal learning experience.

[0856] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0857] Program processing flow

[0858] Step 1: Enter your question

[0859] If a user has a question while studying, they can enter it into the text field using the terminal. For example, they can enter "I don't understand page 30(2) of the calculation drill."

[0860] Input: Questions entered by the user

[0861] Output: The question displayed on the terminal

[0862] Specific behavior:

[0863] A user opens a learning app and writes a question in a text input field.

[0864] The terminal displays the entered text as is for the user to review.

[0865] Step 2: Data format conversion

[0866] The device converts the questions entered by the user into JSON format, which prepares the data in a suitable format for transmission.

[0867] Input: Text data entered by the user

[0868] Output: Data converted to JSON format

[0869] Specific behavior:

[0870] The terminal converts the input information into JSON format, dividing it into fields such as "teaching material name," "page number," and "question number."

[0871] The converted JSON data is passed to the next processing step.

[0872] Step 3: Send data

[0873] The device sends the question converted into JSON format to the server, to an endpoint pre-specified within the system.

[0874] Input: Data converted to JSON format

[0875] Output: Data sent to the server

[0876] Specific behavior:

[0877] The device sends the JSON data as an HTTPS request to the server's URL.

[0878] Step 4: Analyze and identify questions

[0879] The server parses the received JSON data, extracts the learning material name, page number, and question number, and uses this information to search and identify the corresponding question in its internal database.

[0880] Input: JSON data sent to the server

[0881] Output: Identification of the problem

[0882] Specific behavior:

[0883] The server extracts the fields "teaching material name," "page number," and "question number" from the JSON data.

[0884] Based on the extracted information, a query is run against the database to search for the relevant problem.

[0885] Step 5: Generate a description

[0886] The server uses a generative AI model to create a prompt based on the identified problem, which is then input into the AI ​​model to generate an explanation.

[0887] Input: Identified problem information

[0888] Output: Commentary by the generative AI model

[0889] Specific behavior:

[0890] The server creates a prompt sentence and inputs it into the generative AI model.

[0891] The generative AI model generates step-by-step instructions based on the prompt.

[0892] Example prompt statement:

[0893] "A user says he doesn't understand page 30(2) of the math drill. Please provide him with a step-by-step explanation starting with finding the common denominator. Also include an encouraging message to ease any anxiety he may be feeling."

[0894] Step 6: Recognize emotions

[0895] The server uses an emotion engine to analyze the user's emotions, analyzing the user's facial expressions, tone of voice, typing speed, etc. to identify the emotion the user is feeling.

[0896] Input: User's facial expression data, voice data

[0897] Output: User's emotional state

[0898] Specific behavior:

[0899] The server passes the webcam and microphone data received from the device to the emotion engine.

[0900] The emotion engine analyzes this data to determine the user's emotional state.

[0901] Step 7: Adjust the commentary

[0902] The server then adjusts the generated commentary based on the information obtained through emotion recognition, for example adding encouraging elements to the commentary if the user is feeling anxious.

[0903] Input: Emotion recognition data, generated commentary

[0904] Output: Adjusted description

[0905] Specific behavior:

[0906] The server reconstructs the generated commentary based on the emotional state.

[0907] The adjusted description is passed to the next processing step.

[0908] Step 8: Send and view your commentary

[0909] The post-adjustment explanation is converted into JSON or HTML format and sent to the device. The device analyzes the received data and displays it in a format that is easy for the user to read. It also outputs audio if necessary.

[0910] Input: Adjusted description

[0911] Output: The description displayed to the user

[0912] Specific behavior:

[0913] The server converts the adjusted description into JSON or HTML format.

[0914] The server sends this content to the terminal as an HTTPS response.

[0915] The device analyzes the received data and displays commentary on the screen, and also provides audio commentary if necessary.

[0916] In this way, the system can provide accurate explanations in real time to any questions or concerns users may have while studying, providing an optimal learning experience.

[0917] (Application example 2)

[0918] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0919] The problem that this invention aims to solve is to realize an effective learning experience by resolving the doubts and anxieties that users feel during learning in real time while providing emotionally sensitive explanations. Conventional learning support systems provide explanations without taking the user's emotional state into consideration, which has the problem of not being able to alleviate the anxiety and frustration that users feel during the learning process. This poses a risk of reducing the user's motivation to learn and reducing their level of understanding.

[0920] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0921] In this invention, the server includes means for receiving a question entered by a user, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for sending the generated explanation to the user, means for displaying the explanation sent to the user, means for analyzing the user's emotions, and means for adjusting the content of the explanation based on the analyzed emotions. This makes it possible to provide appropriate and friendly explanations while taking the user's emotional state into consideration.

[0922] The "means for receiving question content" refers to a hardware or software function for receiving a question entered by a user via a device.

[0923] "Means for analyzing the content of the question and identifying the relevant problem" refers to technology that analyzes the content of the received question using natural language processing and database search, and identifies the specific problem to be solved within the teaching materials or problem collection.

[0924] "Means for generating explanations using generative AI models" refers to technology that uses artificial intelligence to generate appropriate explanations and answers for identified problems.

[0925] The "means for transmitting the generated commentary to the user" refers to a communication means or protocol for transmitting the commentary generated on the server side to the user's device.

[0926] The "means for displaying the commentary sent to the user" refers to a screen display function or interface for visually displaying the commentary on the user's device.

[0927] "Means for analyzing user emotions" refers to technology that analyzes the user's facial expressions, tone of voice, input speed, etc., to identify the emotions the user is feeling.

[0928] The "means for adjusting the content of the commentary based on the analyzed emotions" is a technology for changing the tone and content of the commentary according to the user's emotional state, and adding friendly explanations and encouraging words.

[0929] This invention is a system that solves the problem of "I don't understand" that users experience during the learning process, and in particular, it combines an "emotion engine" that recognizes the user's emotions and adjusts the response accordingly. This system aims to improve the learning experience by taking into account the user's emotional state while providing appropriate explanations.

[0930] User input of question

[0931] If a user has a question while studying, they can input it into a device such as a smartphone or head-mounted display. For example, they can input a question into the device, such as, "I don't understand page 30(2) of the math drill." The device receives the user's input and processes it as appropriately formatted data.

[0932] Sending data from the device to the server

[0933] The terminal converts the question entered by the user into a data format (for example, JSON format) and sends it to the specified endpoint of the server. The server receives the user's question and is ready to move on to the next analysis process.

[0934] Server analysis and identification of the problem

[0935] The server analyzes the received question data and identifies the relevant question from the question content. Specifically, it extracts information such as the name of the learning material, page number, and question number from the received data and searches the database on the server for the relevant question.

[0936] Server-generated description

[0937] After identifying the problem, the server uses a generative AI model to generate a solution for the problem. The server creates a detailed solution that includes step-by-step instructions, important points to note, and background information on the problem. For example, it might say, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator."

[0938] Emotion recognition by emotion engine

[0939] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's facial expressions, tone of voice, typing speed, etc. to identify the emotion the user is feeling. For example, it detects whether the user is feeling anxious or irritated.

[0940] Adjustment of commentary content

[0941] The server adjusts the content and tone of the generated commentary based on the user's emotions recognized by the emotion engine. For example, if the user is feeling anxious, the server changes the content of the commentary to be more friendly and encouraging.

[0942] Sending and displaying commentary

[0943] The server packages the generated commentary in an appropriate data format (for example, JSON or HTML format) and sends it to the terminal. The terminal analyzes the commentary data received from the server and displays it on the screen in a format that is easy for the user to understand. If necessary, the terminal can also provide audio commentary using audio output means.

[0944] Specific examples of hardware and software used

[0945] Hardware used: smartphone, head-mounted display (HMD)

[0946] Software used:

[0947] Emotion Recognizer: Identifies emotions by analyzing the user's facial expressions and tone of voice.

[0948] Generative AI model (Explanation Generator): Generates appropriate explanations for the user's questions.

[0949] Text to Speech Engine (Text to Speech): Provides the generated commentary in audio format.

[0950] Virtual Display Engine (VirtualDisplay): Visually displays the generated commentary to the user.

[0951] Specific examples

[0952] User input

[0953] User: "I don't know how to add fractions."

[0954] Terminal: (Display the input contents in the form and confirm)

[0955] Sending data from the device to the server

[0956] Terminal: (Convert input to JSON and send to server)

[0957] Server analysis and identification of the problem

[0958] Server: ("I don't know how to add fractions" analysis)

[0959] Server: (Identifies the problem in an internal database)

[0960] Server-generated description

[0961] Server: (Generate explanations using generative AI models)

[0962] Server: (Configure generated commentary)

[0963] Emotion recognition by emotion engine

[0964] Server: (Analyzes user emotions and detects anxiety)

[0965] Adjustment of commentary content

[0966] Server: (adjusts commentary based on perceived emotion)

[0967] Sending and displaying commentary

[0968] Server: (Package the explanation in JSON format)

[0969] Server: (Sent to the device)

[0970] Display explanation

[0971] Terminal: (Analyzes received data and converts it to HTML)

[0972] Terminal: (Explanation displayed on screen)

[0973] Device: (Audio description available if needed)

[0974] Prompt Sentence Examples

[0975] If the user types "I don't know how to add fractions":

[0976] (Prompt for generative AI model)

[0977] "The user feels that they don't understand how to add fractions. Please explain carefully, especially emphasizing the step of finding the 'common denominator.' The user seems confused, so please explain slowly and clearly."

[0978] In this way, the system can improve the learning experience by providing accurate explanations in real time to any questions or anxieties the user may have while studying, and by providing optimal responses based on the user's emotional state.

[0979] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0980] Step 1:

[0981] User enters question

[0982] Input: The user uses a device (smartphone or head-mounted display) to input a question, such as "I don't know how to add fractions," using text or voice.

[0983] Specific operation: The terminal receives the user's input and displays it in a form for confirmation.

[0984] Step 2:

[0985] Sending data from the device to the server

[0986] Input: The question typed by the user

[0987] Data processing: The terminal converts the input content into JSON format.

[0988] Output: JSON data sent to the server

[0989] Specific operation: The terminal sends the converted JSON data to the specified endpoint on the server.

[0990] Step 3:

[0991] Server analysis and identification of the problem

[0992] Input: JSON data of the received question

[0993] Data calculation: The server analyzes the received question data, extracts information such as the name of the teaching material, page number, and question number from the question content, and performs a database search.

[0994] Output: Identifying the problem in question

[0995] Specific operation: The server searches the database for the corresponding problem and identifies detailed information about the problem.

[0996] Step 4:

[0997] Server-generated description

[0998] Input: Details of the problem in question

[0999] Data Computation: Uses a generative AI model to generate a step-by-step explanation for the problem.

[1000] Output: Generated commentary

[1001] How it works: The server uses a generative AI model to create an explanation, such as "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator."

[1002] Step 5:

[1003] Emotion recognition by emotion engine

[1004] Input: Data such as the user's facial expression, tone of voice, and typing speed

[1005] Data Calculation: An emotion engine is used to analyze this data and identify the emotions the user is feeling.

[1006] Output: Recognized user emotion information

[1007] Specific operation: The server analyzes the user's emotions and detects feelings such as anxiety or irritation.

[1008] Step 6:

[1009] Adjustment of commentary content

[1010] Input: Generated commentary and user's emotional information

[1011] Data arithmetic: Adjust the tone and content of your commentary based on emotion, for example, using friendly language if the user is feeling anxious.

[1012] Output: Adjusted commentary

[1013] Specific operation: The server generates explanatory content that takes the user's feelings into consideration, such as, "Don't worry. Let's take it step by step. To find the common denominator..."

[1014] Step 7:

[1015] Sending and displaying commentary

[1016] Input: Adjusted commentary

[1017] Data processing: The generated commentary is packaged into an appropriate data format (JSON or HTML format) and sent to the terminal.

[1018] Output: Descriptive data converted into a displayable format

[1019] Specific operation: The device analyzes the commentary data received from the server and displays it in a format that is easy for the user to understand. Audio commentary is also provided if necessary.

[1020] Through the above steps, the system provides accurate explanations in real time to any questions or anxieties the user may have while studying, and responds optimally according to the user's emotional state.

[1021] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1022] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1023] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1024] [Third embodiment]

[1025] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1026] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1028] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1029] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1030] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1032] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1033] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1035] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1036] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1037] The present invention is a system for solving "unclear" problems that a user encounters in the course of his / her own learning. Specific embodiments will now be described.

[1038] User input

[1039] When a user has a question about a problem, they enter the question into the terminal. For example, they might say, "I don't understand page 30(2) of the calculation drill." The terminal receives the user's input and processes it as properly formatted data.

[1040] Sending data from the device to the server

[1041] The terminal converts the question entered by the user into a data format (for example, JSON format) and sends it to the specified endpoint of the server. This allows the server to receive the user's question and prepare for the next analysis process.

[1042] Server analysis and identification of the problem

[1043] The server analyzes the received question data and identifies the relevant question from the question content. Specifically, it extracts information such as the name of the learning material, page number, and question number from the received data and searches the database on the server for the relevant question.

[1044] Server-generated description

[1045] After identifying the problem, the server uses a generative AI model to generate a solution for the problem. The server creates a detailed solution that includes step-by-step instructions, important points to note, and background information on the problem. For example, the solution might include, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator. The least common multiple of 4 and 3 is 12."

[1046] Sending and displaying commentary

[1047] The server packages the generated commentary in an appropriate data format (for example, JSON or HTML format) and sends it to the terminal. The terminal analyzes the commentary data received from the server and displays it on the screen in a format that is easy for the user to understand. If necessary, the terminal can also provide audio commentary using audio output means.

[1048] Specific examples

[1049] User input

[1050] User: "I don't understand page 30(2) of the math drill."

[1051] Terminal: (Display the input contents in the form and confirm)

[1052] Sending data from the device to the server

[1053] Terminal: (Convert input to JSON and send to server)

[1054] Server analysis and identification of the problem

[1055] Server: (Analysis of "I don't understand page 30(2) of the calculation drill")

[1056] Server: (Identifies the problem in an internal database)

[1057] Server-generated description

[1058] Server: (Generates explanations using AI models)

[1059] Server: (Configure generated commentary)

[1060] Sending and displaying commentary

[1061] Server: (Package the explanation in JSON format)

[1062] Server: (Sent to the device)

[1063] Display explanation

[1064] Terminal: (Analyzes received data and converts it to HTML)

[1065] Terminal: (Explanation displayed on screen)

[1066] Device: (Audio description available if needed)

[1067] In this way, this system can provide real-time explanations to users' (especially learners') questions and effectively support home learning. It can be used as a tool to properly support children's learning even when parents are busy, and is expected to improve the quality of home learning.

[1068] The processing flow will be explained below.

[1069] Step 1: User enters question

[1070] If a user has a question about a homework assignment or problem, they can use the terminal to enter their question, for example, "I don't understand page 30(2) of the calculation drill."

[1071] Step 2: Check input on the terminal and convert to data format

[1072] The device checks the user's input and converts it into an appropriate format (e.g., JSON), checking that the input has been converted correctly into the data format.

[1073] Step 3: Send to the server

[1074] The terminal sends formatted data to the server, which then sends the data to the specified endpoint while checking the stability of the communication and the integrity of the data.

[1075] Step 4: Data reception and analysis by the server

[1076] The server receives the data sent from the terminal, analyzes the question content based on the received data, and extracts information such as the name of the learning material, page number, and question number.

[1077] Step 5: Identify the problem

[1078] Based on the information analyzed by the server, the server searches and identifies the relevant problem in the internal teaching material database. For example, it retrieves the problem on page 30(2) of the arithmetic drill from the database.

[1079] Step 6: Server-generated description

[1080] The server uses a generative AI model to generate a detailed explanation for the identified problem, including the background to the problem, the steps to solving it, and common mistakes.

[1081] Step 7: Converting the description to a data format

[1082] The server converts the generated description into an appropriate format (e.g., JSON or HTML), and the data is organized in a format that can be displayed on the user's device.

[1083] Step 8: Send data to your device

[1084] The server sends the formatted explanatory data to the terminal, and checks the integrity of the data while ensuring the stability and security of the communication.

[1085] Step 9: Data reception and analysis by the device

[1086] The terminal receives the commentary data sent from the server, analyzes the received data, and converts it into a format that can be displayed to the user (for example, HTML or text).

[1087] Step 10: Display the description

[1088] The device displays instructions on the user's screen, and optionally provides audio instructions using a voice output means. For example, detailed instructions such as "First, find the common denominator, then add the fractions."

[1089] This process allows users to receive explanations that resolve their questions in real time, enabling them to study effectively.

[1090] Example 1

[1091] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1092] Conventional learning support systems lack the means to instantly resolve questions users have while studying. This can lead to a decline in learning effectiveness and a loss of motivation to study. Furthermore, when parents or teachers are busy, there is also the problem that learners are unable to receive appropriate support. The present invention aims to solve these problems and provide a system that allows learners to instantly resolve their questions.

[1093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1094] In this invention, the server includes means for receiving a question entered by a user, means for converting the received question into a data format, means for transmitting the converted data to the server, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for transmitting the generated explanation to the user, and means for displaying the explanation transmitted to the user. This makes it possible to provide quick and accurate explanations to questions that users may have while studying.

[1095] "User" refers to an individual who uses the system to receive learning support.

[1096] "Terminal" refers to an electronic device that allows a user to access and input data into the system.

[1097] "Question content" refers to information that expresses doubts that arise during the user's studies or problems that the user wants to solve.

[1098] "Data format" refers to the method by which data is organized and converted into an appropriate form for transmission from a device to a server.

[1099] "Server" refers to a central computer that receives, analyzes, and processes data sent from the terminals.

[1100] "Analysis" refers to the process by which the server breaks down and understands the data it receives and extracts specific information.

[1101] "Applicable questions" refer to specific study tasks related to the question submitted by the user.

[1102] A "generative AI model" refers to artificial intelligence that uses natural language processing technology to generate explanations for questions.

[1103] "Explanation" refers to the explanation and specific solution steps provided by the generative AI model for an identified problem.

[1104] "Data transmission" refers to the process of sending information from a terminal to a server or from a server to a terminal.

[1105] "Display" refers to the terminal providing the received commentary data to the user visually or audibly.

[1106] This invention relates to a learning support system that allows users to instantly resolve questions they may have while studying. This system receives questions entered by users and provides appropriate explanations via a server. Specific embodiments of this system are described below.

[1107] The user inputs the question using their own device (e.g., PC, smartphone, tablet). For example, they might input, "I don't understand page 30(2) of the math drill." The device then displays this input in a form and asks the user to confirm it.

[1108] The terminal converts the questions entered by the user into a data format such as JSON. For example, it converts the questions into JSON data like this:

[1109] json

[1110] {

[1111] "Material name": "Calculation drill",

[1112] "Page Number": 30,

[1113] "Problem Number": 2,

[1114] "Question": "I don't understand page 30(2)"

[1115] }

[1116] The converted data is sent over a network connection to a specific endpoint on the server, which receives the data.

[1117] The server parses the received JSON data. During parsing, it extracts information such as the teaching material name, page number, and question number. For example, it searches the database on the server using the condition "teaching material name = arithmetic drill AND page number = 30 AND question number = 2" to identify the corresponding question.

[1118] After the problem is identified, the server uses a generative AI model (e.g., a model using natural language processing technology) to generate an explanation for the problem. An example of a prompt that the server inputs to the generative AI model is, "Please provide an explanation for page 30(2) of the calculation drill."

[1119] The generated explanation includes specific calculation steps and important points to note. For example, the explanation generated might say, "To solve the calculation 3 / 4 + 2 / 3, you first need to find a common denominator. The least common multiple of 4 and 3 is 12."

[1120] After the description is generated, the server repackages it into JSON format and sends it to the device using an HTTP POST request. The device then parses the received description data and converts it into a user-readable HTML format. It can also provide audio description using speech synthesis software if needed.

[1121] This system allows users to instantly resolve any questions that arise during their studies, improving the quality of their learning and effectively supporting home study. A specific example of a prompt sentence is "I don't understand page 30(2) of the math drill," and real-time explanations are provided.

[1122] In this way, the present invention is expected to increase learners' motivation to learn and improve their learning effectiveness by quickly and accurately resolving the learning problems that users have. The overall configuration and operation of the system is realized based on the cooperation between terminals and a server.

[1123] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1124] Step 1:

[1125] If a user has a question while studying, they can input it using a device (e.g., PC, smartphone, tablet). An example of the input content is "I don't understand page 30(2) of the calculation drill." The device displays this input in a form for the user to confirm. Input: User's question. Output: Confirmed question.

[1126] Step 2:

[1127] The terminal converts the questions the user has confirmed into JSON format. For example, it converts it into JSON data like this:

[1128] json

[1129] {

[1130] "Material name": "Calculation drill",

[1131] "Page Number": 30,

[1132] "Problem Number": 2,

[1133] "Question": "I don't understand page 30(2)"

[1134] }

[1135] In this conversion process, each element of the question (teaching material name, page number, question number, question content) is extracted and put into a JSON structure. Input: Confirmed question content. Output: Data in JSON format.

[1136] Step 3:

[1137] The device sends the converted JSON data to the server using an HTTP POST request, using a network connection to securely send the data to a specific endpoint on the server. Input: JSON format data. Output: Data sent to the server.

[1138] Step 4:

[1139] The server parses the received JSON data. During parsing, it extracts information such as the learning material name, page number, and question number. For example, it parses the data as follows:

[1140] Material name: Calculation drill

[1141] Page number: 30

[1142] Issue Number: 2

[1143] Input: JSON formatted data. Output: Extracted question elements.

[1144] Step 5:

[1145] The server uses the extracted information to search its internal database and identify the relevant questions. Specifically, it searches using the query "Material name = Calculation drill AND Page number = 30 AND Question number = 2". Input: Extracted question elements. Output: Identification of the relevant questions.

[1146] Step 6:

[1147] The server generates an explanation for the identified problem using a generative AI model (e.g., a model using natural language processing technology). The prompt sentence that the server inputs to the generative AI model is, "Please provide an explanation for page 30(2) of the calculation drill." The generative AI model returns a specific explanation. Input: Prompt sentence. Output: Generated explanation.

[1148] Step 7:

[1149] The server then packages the generated description back into JSON, for example into the following JSON data:

[1150] json

[1151] {

[1152] "Explanation": "To solve the equation 3 / 4 + 2 / 3, we first need to find a common denominator. The least common multiple of 4 and 3 is 12..."

[1153] }

[1154] Input: Generated description. Output: Description data in JSON format.

[1155] Step 8:

[1156] The server sends the packaged description data to the device using an HTTP POST request. In this process, data is transferred to the device via the network. Input: Description data in JSON format. Output: Data sent to the device.

[1157] Step 9:

[1158] The device analyzes the explanatory data in JSON format received from the server. The analyzed data is converted into HTML format and displayed on the screen in a format that is easy for the user to read. Input: Explanation data in JSON format. Output: Explanation displayed in HTML format.

[1159] Step 10:

[1160] If necessary, the device will use speech synthesis software to provide audio descriptions, allowing users to hear as well as see the information. Input: Description data in HTML format. Output: Audio description.

[1161] In this way, the questions entered by the user are processed through a number of steps and an explanation is quickly generated and displayed. By providing immediate answers to questions during learning, learners can deepen their understanding and increase their motivation to learn.

[1162] (Application example 1)

[1163] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1164] In factories, when operational problems occur with machines or robots, it is difficult for engineers to respond quickly and resolve them. In particular, if specific explanations about operation are lacking, it takes time to resolve the problem, resulting in a decrease in production efficiency. The present invention aims to provide technology for quickly and appropriately resolving operational problems and machine issues encountered by workers and engineers in factories.

[1165] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1166] In this invention, the server includes means for receiving a question entered by a user, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for sending the generated explanation to the user, means for displaying the explanation sent to the user, means for receiving and analyzing a question about an operation problem of a factory robot and generating an explanation using a generative AI model, and means for sending and displaying the generated explanation to the display and audio output means of the factory robot, thereby enabling workers and engineers in a factory to quickly and appropriately solve operational problems and issues with machines and robots.

[1167] "Users" refer to factory workers and engineers who use the system.

[1168] "Questions" refer to inquiries about operational problems and machine issues entered by factory workers and engineers through the system.

[1169] "Means for receiving" refers to a method or device that allows the system to acquire and store the content of the question entered by the user.

[1170] "Means for analyzing" refers to a method or device that analyzes the content of the received query and identifies the problem based on the information contained therein.

[1171] "Means of identification" refers to the methods and devices used to identify specific operational problems or machine issues from the analyzed questions.

[1172] A "generative AI model" refers to an algorithm or program that uses machine learning and natural language processing techniques to generate explanations for identified problems.

[1173] "Means for generating explanations" refers to a method or device that utilizes a generative AI model to create detailed explanations and solutions to an identified problem.

[1174] "Means for transmitting to a user" refers to a method or device for transmitting the generated commentary from the system to a user.

[1175] "Means for displaying" refers to a method or device for displaying the transmitted commentary in a form that can be understood by the user.

[1176] "Factory robot" refers to an industrial robot on which the present system is installed.

[1177] "Operational trouble" refers to problems or malfunctions that occur during the operation of factory robots and other related machinery.

[1178] "Display" refers to a screen or monitor for visually displaying the generated commentary.

[1179] The "audio output means" refers to a speaker or a voice synthesizer for transmitting the generated commentary to the user by voice.

[1180] "Data format" refers to the standard data structure and format used when communicating question content and explanation data between servers and systems.

[1181] The system program that realizes this application example is designed to solve operational problems of factory robots. Below we explain how this system works.

[1182] Hardware and software used

[1183] The system mainly uses the following hardware and software:

[1184] Hardware: Factory robot body, touch panel, speaker, central server

[1185] Software: Question receiving program, data analysis program, generative AI model (e.g., OpenAI's GPT series), data transmission and reception program

[1186] User operations and question input

[1187] Workers and engineers (users) use the factory robot's touch panel or voice input function to input questions about operational problems into the system. For example, they can input a question like, "I don't know why the robot arm on line 3 isn't moving."

[1188] Sending data of question content

[1189] The question is converted to JSON format and sent to a central server. The server uses an analysis program to analyze the received question and identify the problem. For example, the JSON-formatted question data may contain information such as "Question: 'The robot arm on line 3 is not moving.'"

[1190] Server-based problem analysis and explanation generation

[1191] The server uses an analysis program to analyze the question and generates an explanation for the identified problem using a generative AI model. For example, after analyzing the question, the generative AI model generates an explanation such as, "It is highly likely that a foreign object is stuck in the sensor. Please check the sensor first."

[1192] Data transmission and display of commentary

[1193] The generated explanation is then packaged again in JSON format and sent to the factory robot, which then displays the explanation on its display and, if necessary, outputs it aloud, allowing workers and technicians to quickly understand how to solve the problem.

[1194] Examples and prompts

[1195] For example, a factory worker may input a question through a display, such as, "A sensor error is displayed. What should I do?" This question data is converted into JSON format and sent to a central server for analysis. The generative AI model generates a specific explanation, such as, "The possible cause of the sensor error is that a foreign object is stuck in the sensor. First, check the sensor and see if there is a foreign object." The explanation is sent to a factory robot, which then provides the explanation on the display and in voice.

[1196] Example prompt sentence:

[1197] "Please tell me why the machine isn't working. For example, the robot arm on line 3 has stopped working."

[1198] In this way, the system helps factory workers and engineers quickly and efficiently resolve operational problems they encounter.

[1199] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1200] Step 1:

[1201] A user uses the factory robot's touch panel or voice input function to input questions about operational problems. This input is in the form of a string, such as "I don't know why the robot arm on line 3 isn't moving."

[1202] Step 2:

[1203] The terminal converts the question entered by the user into JSON format. It parses the entered string and converts it into JSON data in the format of, for example, "Question: 'The robot arm on line 3 is not moving.'" This data format conversion results in structured data.

[1204] Step 3:

[1205] The device sends the converted JSON-formatted question data to the central server, which receives the data sent from the device via an API endpoint, thereby obtaining JSON data containing the question content.

[1206] Step 4:

[1207] The server analyzes the received JSON data and identifies the relevant problem from the question. Specifically, it analyzes the "question" field of the received data and extracts information about the relevant machine and the problem. For example, it performs an analysis based on the information, "The robot arm on line 3 is not moving."

[1208] Step 5:

[1209] The server uses a generative AI model to generate an explanation based on the analyzed question. Using a generative AI model (e.g., OpenAI's GPT series), the server generates an explanation based on the analysis results. For example, it creates an explanation such as, "There is a high possibility that a foreign object is stuck in the sensor. Please check the sensor first."

[1210] Step 6:

[1211] The server then packages the generated explanation into JSON format again and sends it to the device. This operation sends explanation data from the server to the device. Specifically, JSON data in the format "Explanation: 'There is a high possibility that a foreign object is stuck in the sensor. Please check the sensor first.'" is sent.

[1212] Step 7:

[1213] The device analyzes the received JSON-formatted explanatory data and converts it into a displayable format. Based on the analysis results, the explanation is converted into text or HTML format. This data conversion makes it possible to provide information to the user visually and audibly.

[1214] Step 8:

[1215] The device displays the converted commentary on the display and outputs it as audio if necessary. The user can check the commentary on the display and, in some cases, listen to the commentary via audio guidance. For example, the display may say, "There is a high possibility that a foreign object is stuck in the sensor. Please check the sensor first."

[1216] By following the steps explained in this manual, users can quickly and accurately identify the cause and solution of operational problems, thereby improving production efficiency and reducing the time required to resolve problems.

[1217] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1218] The present invention combines a system that solves the "I don't understand" problem that users experience during the learning process with an emotion engine that recognizes the user's emotions and adjusts the response content. The purpose of this invention is to improve the learning experience by taking into account the user's emotional state as well as providing appropriate explanations.

[1219] User input of question

[1220] When a user has a question about a problem, they enter the question into the terminal. For example, they might say, "I don't understand page 30(2) of the calculation drill." The terminal receives the user's input and processes it as properly formatted data.

[1221] Sending data from the device to the server

[1222] The terminal converts the question entered by the user into a data format (for example, JSON format) and sends it to the specified endpoint of the server. This allows the server to receive the user's question and prepare to move on to the next analysis process.

[1223] Server analysis and identification of the problem

[1224] The server analyzes the received question data and identifies the relevant question from the question content. Specifically, it extracts information such as the name of the learning material, page number, and question number from the received data and searches the database on the server for the relevant question.

[1225] Server-generated description

[1226] After identifying the problem, the server uses a generative AI model to generate a solution for the problem. The server creates a detailed solution that includes step-by-step instructions, important points to note, and background information on the problem. For example, it might say, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator."

[1227] Emotion recognition by emotion engine

[1228] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's facial expressions, tone of voice, typing speed, etc. to identify the emotion the user is feeling. For example, it detects whether the user is feeling anxious or irritated.

[1229] Adjustment of commentary content

[1230] The server adjusts the content and tone of the generated commentary based on the user's emotions recognized by the emotion engine. For example, if the user is feeling anxious, the server changes the content of the commentary to be more friendly and encouraging.

[1231] Sending and displaying commentary

[1232] The server packages the generated commentary in an appropriate data format (for example, JSON or HTML format) and sends it to the terminal. The terminal analyzes the commentary data received from the server and displays it on the screen in a format that is easy for the user to understand. If necessary, the terminal can also provide audio commentary using audio output means.

[1233] Specific examples

[1234] User input

[1235] User: "I don't understand page 30(2) of the math drill."

[1236] Terminal: (Display the input contents in the form and confirm)

[1237] Sending data from the device to the server

[1238] Terminal: (Convert input to JSON and send to server)

[1239] Server analysis and identification of the problem

[1240] Server: (Analysis of "I don't understand page 30(2) of the calculation drill")

[1241] Server: (Identifies the problem in an internal database)

[1242] Server-generated description

[1243] Server: (Generates explanations using AI models)

[1244] Server: (Configure generated commentary)

[1245] Emotion recognition by emotion engine

[1246] Server: (Analyzes user emotions and detects anxiety)

[1247] Adjustment of commentary content

[1248] Server: (adjusts commentary based on perceived emotion)

[1249] Sending and displaying commentary

[1250] Server: (Package the explanation in JSON format)

[1251] Server: (Sent to the device)

[1252] Display explanation

[1253] Terminal: (Analyzes received data and converts it to HTML)

[1254] Terminal: (Explanation displayed on screen)

[1255] Device: (Audio description available if needed)

[1256] In this way, the system can improve the learning experience by providing accurate explanations in real time to any questions or anxieties the user may have while studying, and by providing optimal responses based on the user's emotional state.

[1257] The processing flow will be explained below.

[1258] Step 1: User enters question

[1259] If a user has a question about a problem, they can enter the question using the terminal. For example, they can enter "I don't understand page 30(2) of the calculation drill."

[1260] Step 2: Check input on the terminal and convert to data format

[1261] The device checks the user's input and converts it into an appropriate format (e.g., JSON), checking that the input has been converted correctly into the data format.

[1262] Step 3: Send to the server

[1263] The terminal sends formatted data to the server, which then sends the data to the specified endpoint while checking the stability of the communication and the integrity of the data.

[1264] Step 4: Data reception and analysis by the server

[1265] The server receives the data sent from the terminal, analyzes the question content based on the received data, and extracts information such as the name of the learning material, page number, and question number.

[1266] Step 5: Identify the problem

[1267] Based on the information analyzed by the server, the server searches and identifies the relevant problem in the internal teaching material database. For example, it retrieves the problem on page 30(2) of the arithmetic drill from the database.

[1268] Step 6: Server-generated description

[1269] The server uses a generative AI model to generate a detailed explanation for the identified problem, including the background to the problem, the steps to solving it, and common mistakes.

[1270] Step 7: Emotion Recognition with the Emotion Engine

[1271] The server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial expressions, tone of voice, typing speed, etc. to determine whether the user is feeling anxious or annoyed.

[1272] Step 8: Adjust the commentary

[1273] The server adjusts the content and tone of the generated commentary based on the user's emotions recognized by the emotion engine. For example, if the user is feeling anxious, the server changes the commentary to be more friendly and encouraging.

[1274] Step 9: Converting the description to a data format

[1275] The server packages the adjusted description into an appropriate format (e.g., JSON or HTML), and the data is organized in a format that can be displayed on the user's device.

[1276] Step 10: Send data to your device

[1277] The server sends the formatted explanatory data to the terminal, and checks the integrity of the data while ensuring the stability and security of the communication.

[1278] Step 11: Receiving and analyzing data on the device

[1279] The terminal receives the commentary data sent from the server, analyzes the received data, and converts it into a format that can be displayed to the user (for example, HTML or text).

[1280] Step 12: Display the description

[1281] The device displays instructions on the user's screen, and optionally provides audio instructions using a voice output means. For example, detailed instructions such as "First, find the common denominator, then add the fractions."

[1282] Specific examples

[1283] 1. User: "I don't understand page 30(2) of the math drill."

[1284] 2. Terminal: (Converts input into JSON and sends it to the server)

[1285] 3. Server: (Analysis of "I don't understand page 30(2) of the math drill")

[1286] 4. Server: (Identify the problem in the internal database)

[1287] 5. Server: (Generates explanations using AI models)

[1288] 6. Server: (Analyze user emotions and detect anxiety)

[1289] 7. Server: (adjusts commentary based on perceived emotion)

[1290] 8. Server: (Package the description in JSON format)

[1291] 9. Server: (Send to device)

[1292] 10. Terminal: (Analyzes received data and converts it to HTML)

[1293] 11. Terminal: (Displays explanation on the screen)

[1294] 12. Terminal: (Audio description provided if needed)

[1295] This allows users to receive real-time explanations that resolve their questions and improve their learning experience by responding to their emotions.

[1296] Example 2

[1297] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1298] Conventional learning support systems have limited functionality for providing solutions to users' "unclear" problems in real time, and have difficulty providing appropriate explanations that take the user's emotional state into account. Furthermore, in order to provide accurate explanations to users' questions, it is necessary to analyze the user's input in detail and accurately recognize the user's emotional state. In this context, the present invention aims to realize a system that not only responds quickly and accurately to users' questions, but also provides optimal explanations based on the user's emotional state.

[1299] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1300] In this invention, the server includes means for receiving a question entered by a user, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for recognizing the user's emotion, means for adjusting the content of the generated explanation based on the recognized user emotion, means for sending the adjusted explanation to the user, and means for displaying the explanation sent to the user. This makes it possible to provide accurate and prompt answers to questions that the user has during the learning process, and further to provide optimal explanations according to the user's emotional state.

[1301] A "user" is a subject who uses the system to learn, enter questions, and receive explanations.

[1302] A "terminal" is a device that a user uses to input questions and that sends and receives information.

[1303] The "server" is a central system that analyzes the received question, generates an explanation, and transmits the explanation to the user.

[1304] "Question content" refers to a specific problem or question that a user wants to solve while studying.

[1305] The "receiving means" is a system for taking into the server the content of the question entered by the user through the terminal.

[1306] "Analysis means" refers to procedures and techniques for analyzing the content of the received question and identifying the relevant problem.

[1307] A "generative AI model" is an artificial intelligence model used to generate appropriate explanations for identified problems.

[1308] "Explanation" is information that includes answers and explanations to the user's questions.

[1309] "Emotion recognition means" refers to technology or systems for analyzing and identifying a user's emotional state.

[1310] "Adjustment means" refers to techniques or methods for adjusting the content and tone of generated commentary based on the perceived user sentiment.

[1311] "Transmission means" refers to a system or technology for transmitting the adjusted commentary to the user.

[1312] The "display means" refers to a system or function for displaying the commentary sent to the user on the screen.

[1313] This invention is a system that solves the "I don't understand" problem that users experience during the learning process, and combines it with an emotion engine that recognizes the user's emotions and adjusts the response content. The purpose of this invention is to improve the learning experience by providing appropriate explanations and taking into account the user's emotional state.

[1314] Hardware and software used

[1315] The main components of the system are:

[1316] 1. Terminal: The device through which the user enters their question, such as a PC, tablet, or smartphone.

[1317] 2. Server: A central system that analyzes questions, generates explanations, and recognizes user emotions. This can be a cloud server or an on-premise server.

[1318] 3. Generative AI model: An artificial intelligence model used to generate an appropriate explanation for an identified problem.

[1319] 4. Emotion engine: Software for recognizing the user's emotional state. This includes facial expression recognition software and voice analysis software.

[1320] Data processing and calculation

[1321] 1. Entering and receiving questions

[1322] When a user has a question while studying, they can enter it into the device. For example, they might write, "I don't understand page 30(2) of the math drill." The device will then check the input and convert it into the appropriate data format.

[1323] 2. Data Transmission

[1324] The device converts the input question into JSON format and sends it to the server, to an endpoint pre-specified within the system.

[1325] 3. Question analysis and identification

[1326] The server extracts the learning material name, page number, and question number from the received JSON data and searches the database for the corresponding question. Once the question is identified, it uses that information to generate a prompt for the generative AI model.

[1327] 4. Generating Explanations

[1328] The server uses a generative AI model to generate a commentary. Enter the following prompt:

[1329] "A user says he doesn't understand page 30(2) of the math drill. Please provide him with a step-by-step explanation starting with finding the common denominator. Also include an encouraging message to ease any anxiety he may be feeling."

[1330] The generative AI model generates step-by-step instructions and provides detailed explanations.

[1331] 5. Emotion recognition

[1332] The server uses an emotion engine to analyze the user's facial expression and voice data received from the device and identify the user's emotional state, for example, determining whether the user is feeling anxious or irritated.

[1333] 6. Adjustment of explanation content

[1334] The server adjusts the generated commentary based on information obtained from the emotion engine: for example, if the user is feeling anxious, it makes the commentary more friendly and adds encouraging elements.

[1335] 7. Sending and Displaying Commentary

[1336] The server converts the adjusted commentary into JSON or HTML format and sends it to the device. The device analyzes the received data and displays it in a format that is easy for the user to read. It also outputs audio if necessary.

[1337] Specific examples

[1338] The user enters "I don't understand page 30(2) of the calculation drill."

[1339] Terminal: Display input and ask user for confirmation

[1340] Terminal: Convert to JSON format and send to server

[1341] The server receives and analyzes the data

[1342] Server: Extract the teaching material name, page number, and question number

[1343] Server: Search database for matching problem

[1344] Generate explanations

[1345] Server: Enter prompt text using generative AI model: "A user says he doesn't understand page 30(2) of the math drill. Please provide him with a step-by-step explanation, starting with finding the common denominator. Include an encouraging message to ease any anxiety he may be feeling."

[1346] Generative AI model: Generate detailed explanations

[1347] emotion recognition

[1348] Server: Analyzes the user's facial expression and voice data to determine their emotional state

[1349] Adjustment of commentary content

[1350] Server: Adjust the tone and content of commentary depending on the emotional state

[1351] Sending and displaying commentary

[1352] Server: Converts the adjusted commentary into JSON or HTML format and sends it to the device.

[1353] Terminal: Analyzes received data and displays it on the screen. Also outputs audio if necessary.

[1354] In this way, the system can provide accurate explanations in real time to address any questions or concerns users may have while studying, providing an optimal learning experience.

[1355] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1356] Program processing flow

[1357] Step 1: Enter your question

[1358] If a user has a question while studying, they can enter it into the text field using the terminal. For example, they can enter "I don't understand page 30(2) of the calculation drill."

[1359] Input: Questions entered by the user

[1360] Output: The question displayed on the terminal

[1361] Specific behavior:

[1362] A user opens a learning app and writes a question in a text input field.

[1363] The terminal displays the entered text as is for the user to review.

[1364] Step 2: Data format conversion

[1365] The device converts the questions entered by the user into JSON format, which prepares the data in a suitable format for transmission.

[1366] Input: Text data entered by the user

[1367] Output: Data converted to JSON format

[1368] Specific behavior:

[1369] The terminal converts the input information into JSON format, dividing it into fields such as "teaching material name," "page number," and "question number."

[1370] The converted JSON data is passed to the next processing step.

[1371] Step 3: Send data

[1372] The device sends the question converted into JSON format to the server, to an endpoint pre-specified within the system.

[1373] Input: Data converted to JSON format

[1374] Output: Data sent to the server

[1375] Specific behavior:

[1376] The device sends the JSON data as an HTTPS request to the server's URL.

[1377] Step 4: Analyze and identify questions

[1378] The server parses the received JSON data, extracts the learning material name, page number, and question number, and uses this information to search and identify the corresponding question in its internal database.

[1379] Input: JSON data sent to the server

[1380] Output: Identification of the problem

[1381] Specific behavior:

[1382] The server extracts the fields "teaching material name," "page number," and "question number" from the JSON data.

[1383] Based on the extracted information, a query is run against the database to search for the relevant problem.

[1384] Step 5: Generate a description

[1385] The server uses a generative AI model to create a prompt based on the identified problem, which is then input into the AI ​​model to generate an explanation.

[1386] Input: Identified problem information

[1387] Output: Commentary by the generative AI model

[1388] Specific behavior:

[1389] The server creates a prompt sentence and inputs it into the generative AI model.

[1390] The generative AI model generates step-by-step instructions based on the prompt.

[1391] Example prompt statement:

[1392] "A user says he doesn't understand page 30(2) of the math drill. Please provide him with a step-by-step explanation starting with finding the common denominator. Also include an encouraging message to ease any anxiety he may be feeling."

[1393] Step 6: Recognize emotions

[1394] The server uses an emotion engine to analyze the user's emotions, analyzing the user's facial expressions, tone of voice, typing speed, etc. to identify the emotion the user is feeling.

[1395] Input: User's facial expression data, voice data

[1396] Output: User's emotional state

[1397] Specific behavior:

[1398] The server passes the webcam and microphone data received from the device to the emotion engine.

[1399] The emotion engine analyzes this data to determine the user's emotional state.

[1400] Step 7: Adjust the commentary

[1401] The server then adjusts the generated commentary based on the information obtained through emotion recognition, for example adding encouraging elements to the commentary if the user is feeling anxious.

[1402] Input: Emotion recognition data, generated commentary

[1403] Output: Adjusted description

[1404] Specific behavior:

[1405] The server reconstructs the generated commentary based on the emotional state.

[1406] The adjusted description is passed to the next processing step.

[1407] Step 8: Send and view your commentary

[1408] The post-adjustment explanation is converted into JSON or HTML format and sent to the device. The device analyzes the received data and displays it in a format that is easy for the user to read. It also outputs audio if necessary.

[1409] Input: Adjusted description

[1410] Output: The description displayed to the user

[1411] Specific behavior:

[1412] The server converts the adjusted description into JSON or HTML format.

[1413] The server sends this content to the terminal as an HTTPS response.

[1414] The device analyzes the received data and displays commentary on the screen, and also provides audio commentary if necessary.

[1415] In this way, the system can provide accurate explanations in real time to any questions or concerns users may have while studying, providing an optimal learning experience.

[1416] (Application example 2)

[1417] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1418] The problem that this invention aims to solve is to realize an effective learning experience by resolving the doubts and anxieties that users feel during learning in real time while providing emotionally sensitive explanations. Conventional learning support systems provide explanations without taking the user's emotional state into consideration, which has the problem of not being able to alleviate the anxiety and frustration that users feel during the learning process. This poses a risk of reducing the user's motivation to learn and reducing their level of understanding.

[1419] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1420] In this invention, the server includes means for receiving a question entered by a user, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for sending the generated explanation to the user, means for displaying the explanation sent to the user, means for analyzing the user's emotions, and means for adjusting the content of the explanation based on the analyzed emotions. This makes it possible to provide appropriate and friendly explanations while taking the user's emotional state into consideration.

[1421] The "means for receiving question content" refers to a hardware or software function for receiving a question entered by a user via a device.

[1422] "Means for analyzing the content of the question and identifying the relevant problem" refers to technology that analyzes the content of the received question using natural language processing and database search, and identifies the specific problem to be solved within the teaching materials or problem collection.

[1423] "Means for generating explanations using generative AI models" refers to technology that uses artificial intelligence to generate appropriate explanations and answers for identified problems.

[1424] The "means for transmitting the generated commentary to the user" refers to a communication means or protocol for transmitting the commentary generated on the server side to the user's device.

[1425] The "means for displaying the commentary sent to the user" refers to a screen display function or interface for visually displaying the commentary on the user's device.

[1426] "Means for analyzing user emotions" refers to technology that analyzes the user's facial expressions, tone of voice, input speed, etc., to identify the emotions the user is feeling.

[1427] The "means for adjusting the content of the commentary based on the analyzed emotions" is a technology for changing the tone and content of the commentary according to the user's emotional state, and adding friendly explanations and encouraging words.

[1428] This invention is a system that solves the problem of "I don't understand" that users experience during the learning process, and in particular, it combines an "emotion engine" that recognizes the user's emotions and adjusts the response accordingly. This system aims to improve the learning experience by taking into account the user's emotional state while providing appropriate explanations.

[1429] User input of question

[1430] If a user has a question while studying, they can input it into a device such as a smartphone or head-mounted display. For example, they can input a question into the device, such as, "I don't understand page 30(2) of the math drill." The device receives the user's input and processes it as appropriately formatted data.

[1431] Sending data from the device to the server

[1432] The terminal converts the question entered by the user into a data format (for example, JSON format) and sends it to the specified endpoint of the server. The server receives the user's question and is ready to move on to the next analysis process.

[1433] Server analysis and identification of the problem

[1434] The server analyzes the received question data and identifies the relevant question from the question content. Specifically, it extracts information such as the name of the learning material, page number, and question number from the received data and searches the database on the server for the relevant question.

[1435] Server-generated description

[1436] After identifying the problem, the server uses a generative AI model to generate a solution for the problem. The server creates a detailed solution that includes step-by-step instructions, important points to note, and background information on the problem. For example, it might say, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator."

[1437] Emotion recognition by emotion engine

[1438] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's facial expressions, tone of voice, typing speed, etc. to identify the emotion the user is feeling. For example, it detects whether the user is feeling anxious or irritated.

[1439] Adjustment of commentary content

[1440] The server adjusts the content and tone of the generated commentary based on the user's emotions recognized by the emotion engine. For example, if the user is feeling anxious, the server changes the content of the commentary to be more friendly and encouraging.

[1441] Sending and displaying commentary

[1442] The server packages the generated commentary in an appropriate data format (for example, JSON or HTML format) and sends it to the terminal. The terminal analyzes the commentary data received from the server and displays it on the screen in a format that is easy for the user to understand. If necessary, the terminal can also provide audio commentary using audio output means.

[1443] Specific examples of hardware and software used

[1444] Hardware used: smartphone, head-mounted display (HMD)

[1445] Software used:

[1446] Emotion Recognizer: Identifies emotions by analyzing the user's facial expressions and tone of voice.

[1447] Generative AI model (Explanation Generator): Generates appropriate explanations for the user's questions.

[1448] Text to Speech Engine (Text to Speech): Provides the generated commentary in audio format.

[1449] Virtual Display Engine (VirtualDisplay): Visually displays the generated commentary to the user.

[1450] Specific examples

[1451] User input

[1452] User: "I don't know how to add fractions."

[1453] Terminal: (Display the input contents in the form and confirm)

[1454] Sending data from the device to the server

[1455] Terminal: (Convert input to JSON and send to server)

[1456] Server analysis and identification of the problem

[1457] Server: ("I don't know how to add fractions" analysis)

[1458] Server: (Identifies the problem in an internal database)

[1459] Server-generated description

[1460] Server: (Generate explanations using generative AI models)

[1461] Server: (Configure generated commentary)

[1462] Emotion recognition by emotion engine

[1463] Server: (Analyzes user emotions and detects anxiety)

[1464] Adjustment of commentary content

[1465] Server: (adjusts commentary based on perceived emotion)

[1466] Sending and displaying commentary

[1467] Server: (Package the explanation in JSON format)

[1468] Server: (Sent to the device)

[1469] Display explanation

[1470] Terminal: (Analyzes received data and converts it to HTML)

[1471] Terminal: (Explanation displayed on screen)

[1472] Device: (Audio description available if needed)

[1473] Prompt Sentence Examples

[1474] If the user types "I don't know how to add fractions":

[1475] (Prompt for generative AI model)

[1476] "The user feels that they don't understand how to add fractions. Please explain carefully, especially emphasizing the step of finding the 'common denominator.' The user seems confused, so please explain slowly and clearly."

[1477] In this way, the system can improve the learning experience by providing accurate explanations in real time to any questions or anxieties the user may have while studying, and by providing optimal responses based on the user's emotional state.

[1478] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1479] Step 1:

[1480] User enters question

[1481] Input: The user uses a device (smartphone or head-mounted display) to input a question, such as "I don't know how to add fractions," using text or voice.

[1482] Specific operation: The terminal receives the user's input and displays it in a form for confirmation.

[1483] Step 2:

[1484] Sending data from the device to the server

[1485] Input: The question typed by the user

[1486] Data processing: The terminal converts the input content into JSON format.

[1487] Output: JSON data sent to the server

[1488] Specific operation: The terminal sends the converted JSON data to the specified endpoint on the server.

[1489] Step 3:

[1490] Server analysis and identification of the problem

[1491] Input: JSON data of the received question

[1492] Data calculation: The server analyzes the received question data, extracts information such as the name of the teaching material, page number, and question number from the question content, and performs a database search.

[1493] Output: Identifying the problem in question

[1494] Specific operation: The server searches the database for the corresponding problem and identifies detailed information about the problem.

[1495] Step 4:

[1496] Server-generated description

[1497] Input: Details of the problem in question

[1498] Data Computation: Uses a generative AI model to generate a step-by-step explanation for the problem.

[1499] Output: Generated commentary

[1500] How it works: The server uses a generative AI model to create an explanation, such as "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator."

[1501] Step 5:

[1502] Emotion recognition by emotion engine

[1503] Input: Data such as the user's facial expression, tone of voice, and typing speed

[1504] Data Calculation: An emotion engine is used to analyze this data and identify the emotions the user is feeling.

[1505] Output: Recognized user emotion information

[1506] Specific operation: The server analyzes the user's emotions and detects feelings such as anxiety or irritation.

[1507] Step 6:

[1508] Adjustment of commentary content

[1509] Input: Generated commentary and user's emotional information

[1510] Data arithmetic: Adjust the tone and content of your commentary based on emotion, for example, using friendly language if the user is feeling anxious.

[1511] Output: Adjusted commentary

[1512] Specific operation: The server generates explanatory content that takes the user's feelings into consideration, such as, "Don't worry. Let's take it step by step. To find the common denominator..."

[1513] Step 7:

[1514] Sending and displaying commentary

[1515] Input: Adjusted commentary

[1516] Data processing: The generated commentary is packaged into an appropriate data format (JSON or HTML format) and sent to the terminal.

[1517] Output: Descriptive data converted into a displayable format

[1518] Specific operation: The device analyzes the commentary data received from the server and displays it in a format that is easy for the user to understand. Audio commentary is also provided if necessary.

[1519] Through the above steps, the system provides accurate explanations in real time to any questions or anxieties the user may have while studying, and responds optimally according to the user's emotional state.

[1520] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1521] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1522] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1523] [Fourth embodiment]

[1524] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1525] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1526] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1527] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1528] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1529] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1530] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1531] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1532] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1533] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1534] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1535] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1536] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1537] The present invention is a system for solving "unclear" problems that a user encounters in the course of his / her own learning. Specific embodiments will now be described.

[1538] User input

[1539] When a user has a question about a problem, they enter the question into the terminal. For example, they might say, "I don't understand page 30(2) of the calculation drill." The terminal receives the user's input and processes it as properly formatted data.

[1540] Sending data from the device to the server

[1541] The terminal converts the question entered by the user into a data format (for example, JSON format) and sends it to the specified endpoint of the server. This allows the server to receive the user's question and prepare to move on to the next analysis process.

[1542] Server analysis and identification of the problem

[1543] The server analyzes the received question data and identifies the relevant question from the question content. Specifically, it extracts information such as the name of the learning material, page number, and question number from the received data and searches the database on the server for the relevant question.

[1544] Server-generated description

[1545] After identifying the problem, the server uses a generative AI model to generate a solution for the problem. The server creates a detailed solution that includes step-by-step instructions, important points to note, and background information on the problem. For example, the solution might include, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator. The least common multiple of 4 and 3 is 12."

[1546] Sending and displaying commentary

[1547] The server packages the generated commentary in an appropriate data format (for example, JSON or HTML format) and sends it to the terminal. The terminal analyzes the commentary data received from the server and displays it on the screen in a format that is easy for the user to understand. If necessary, the terminal can also provide audio commentary using audio output means.

[1548] Specific examples

[1549] User input

[1550] User: "I don't understand page 30(2) of the math drill."

[1551] Terminal: (Display the input contents in the form and confirm)

[1552] Sending data from the device to the server

[1553] Terminal: (Convert input to JSON and send to server)

[1554] Server analysis and identification of the problem

[1555] Server: (Analysis of "I don't understand page 30(2) of the calculation drill")

[1556] Server: (Identifies the problem in an internal database)

[1557] Server-generated description

[1558] Server: (Generates explanations using AI models)

[1559] Server: (Configure generated commentary)

[1560] Sending and displaying commentary

[1561] Server: (Package the explanation in JSON format)

[1562] Server: (Sent to the device)

[1563] Display explanation

[1564] Terminal: (Analyzes received data and converts it to HTML)

[1565] Terminal: (Explanation displayed on screen)

[1566] Device: (Audio description available if needed)

[1567] In this way, this system can provide real-time explanations to users' (especially learners') questions and effectively support home learning. It can be used as a tool to properly support children's learning even when parents are busy, and is expected to improve the quality of home learning.

[1568] The processing flow will be explained below.

[1569] Step 1: User enters question

[1570] If a user has a question about a homework assignment or problem, they can use the terminal to enter their question, for example, "I don't understand page 30(2) of the calculation drill."

[1571] Step 2: Check input on the terminal and convert to data format

[1572] The device checks the user's input and converts it into an appropriate format (e.g., JSON), checking that the input has been converted correctly into the data format.

[1573] Step 3: Send to the server

[1574] The terminal sends formatted data to the server, which then sends the data to the specified endpoint while checking the stability of the communication and the integrity of the data.

[1575] Step 4: Data reception and analysis by the server

[1576] The server receives the data sent from the terminal, analyzes the question content based on the received data, and extracts information such as the name of the learning material, page number, and question number.

[1577] Step 5: Identify the problem

[1578] Based on the information analyzed by the server, the server searches and identifies the relevant problem in the internal teaching material database. For example, it retrieves the problem on page 30(2) of the arithmetic drill from the database.

[1579] Step 6: Server-generated description

[1580] The server uses a generative AI model to generate a detailed explanation for the identified problem, including the background to the problem, the steps to solving it, and common mistakes.

[1581] Step 7: Converting the description to a data format

[1582] The server converts the generated description into an appropriate format (e.g., JSON or HTML), and the data is organized in a format that can be displayed on the user's device.

[1583] Step 8: Send data to your device

[1584] The server sends the formatted explanatory data to the terminal, and checks the integrity of the data while ensuring the stability and security of the communication.

[1585] Step 9: Data reception and analysis by the device

[1586] The terminal receives the commentary data sent from the server, analyzes the received data, and converts it into a format that can be displayed to the user (for example, HTML or text).

[1587] Step 10: Display the description

[1588] The device displays instructions on the user's screen, and optionally provides audio instructions using a voice output means. For example, detailed instructions such as "First, find the common denominator, then add the fractions."

[1589] This process allows users to receive explanations that resolve their questions in real time, enabling them to study effectively.

[1590] Example 1

[1591] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1592] Conventional learning support systems lack the means to instantly resolve questions users have while studying. This can lead to a decline in learning effectiveness and a loss of motivation to study. Furthermore, when parents or teachers are busy, there is also the problem that learners are unable to receive appropriate support. The present invention aims to solve these problems and provide a system that allows learners to instantly resolve their questions.

[1593] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1594] In this invention, the server includes means for receiving a question entered by a user, means for converting the received question into a data format, means for transmitting the converted data to the server, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for transmitting the generated explanation to the user, and means for displaying the explanation transmitted to the user. This makes it possible to provide quick and accurate explanations to questions that users may have while studying.

[1595] "User" refers to an individual who uses the system to receive learning support.

[1596] "Terminal" refers to an electronic device that allows a user to access and input data into the system.

[1597] "Question content" refers to information that expresses doubts that arise during the user's studies or problems that the user wants to solve.

[1598] "Data format" refers to the method by which data is organized and converted into an appropriate form for transmission from a device to a server.

[1599] "Server" refers to a central computer that receives, analyzes, and processes data sent from the terminals.

[1600] "Analysis" refers to the process by which the server breaks down and understands the data it receives and extracts specific information.

[1601] "Applicable questions" refer to specific study tasks related to the question submitted by the user.

[1602] A "generative AI model" refers to artificial intelligence that uses natural language processing technology to generate explanations for questions.

[1603] "Explanation" refers to the explanation and specific solution steps provided by the generative AI model for an identified problem.

[1604] "Data transmission" refers to the process of sending information from a terminal to a server or from a server to a terminal.

[1605] "Display" refers to the terminal providing the received commentary data to the user visually or audibly.

[1606] This invention relates to a learning support system that allows users to instantly resolve questions they may have while studying. This system receives questions entered by users and provides appropriate explanations via a server. Specific embodiments of this system are described below.

[1607] The user inputs the question using their own device (e.g., PC, smartphone, tablet). For example, they might input, "I don't understand page 30(2) of the math drill." The device then displays this input in a form and asks the user to confirm it.

[1608] The terminal converts the questions entered by the user into a data format such as JSON. For example, it converts the questions into JSON data like this:

[1609] json

[1610] {

[1611] "Material name": "Calculation drill",

[1612] "Page Number": 30,

[1613] "Problem Number": 2,

[1614] "Question": "I don't understand page 30(2)"

[1615] }

[1616] The converted data is sent over a network connection to a specific endpoint on the server, which receives the data.

[1617] The server parses the received JSON data. During parsing, it extracts information such as the teaching material name, page number, and question number. For example, it searches the database on the server using the condition "teaching material name = arithmetic drill AND page number = 30 AND question number = 2" to identify the corresponding question.

[1618] After the problem is identified, the server uses a generative AI model (e.g., a model using natural language processing technology) to generate an explanation for the problem. An example of a prompt that the server inputs to the generative AI model is, "Please provide an explanation for page 30(2) of the calculation drill."

[1619] The generated explanation includes specific calculation steps and important points to note. For example, the explanation generated might say, "To solve the calculation 3 / 4 + 2 / 3, you first need to find a common denominator. The least common multiple of 4 and 3 is 12."

[1620] After the description is generated, the server repackages it into JSON format and sends it to the device using an HTTP POST request. The device then parses the received description data and converts it into a user-readable HTML format. It can also provide audio description using speech synthesis software if needed.

[1621] This system allows users to instantly resolve any questions that arise during their studies, improving the quality of their learning and effectively supporting home study. A specific example of a prompt sentence is "I don't understand page 30(2) of the math drill," and real-time explanations are provided.

[1622] In this way, the present invention is expected to increase learners' motivation to learn and improve their learning effectiveness by quickly and accurately resolving the learning problems that users have. The overall configuration and operation of the system is realized based on the cooperation between terminals and a server.

[1623] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1624] Step 1:

[1625] If a user has a question while studying, they can input it using a device (e.g., PC, smartphone, tablet). An example of the input content is "I don't understand page 30(2) of the calculation drill." The device displays this input in a form for the user to confirm. Input: User's question. Output: Confirmed question.

[1626] Step 2:

[1627] The terminal converts the questions the user has confirmed into JSON format. For example, it converts it into JSON data like this:

[1628] json

[1629] {

[1630] "Material name": "Calculation drill",

[1631] "Page Number": 30,

[1632] "Problem Number": 2,

[1633] "Question": "I don't understand page 30(2)"

[1634] }

[1635] In this conversion process, each element of the question (teaching material name, page number, question number, question content) is extracted and put into a JSON structure. Input: Confirmed question content. Output: Data in JSON format.

[1636] Step 3:

[1637] The device sends the converted JSON data to the server using an HTTP POST request, using a network connection to securely send the data to a specific endpoint on the server. Input: JSON format data. Output: Data sent to the server.

[1638] Step 4:

[1639] The server parses the received JSON data. During parsing, it extracts information such as the learning material name, page number, and question number. For example, it parses the data as follows:

[1640] Material name: Calculation drill

[1641] Page number: 30

[1642] Issue Number: 2

[1643] Input: JSON formatted data. Output: Extracted question elements.

[1644] Step 5:

[1645] The server uses the extracted information to search its internal database and identify the relevant questions. Specifically, it searches using the query "Material name = Calculation drill AND Page number = 30 AND Question number = 2". Input: Extracted question elements. Output: Identification of the relevant questions.

[1646] Step 6:

[1647] The server generates an explanation for the identified problem using a generative AI model (e.g., a model using natural language processing technology). The prompt sentence that the server inputs to the generative AI model is, "Please provide an explanation for page 30(2) of the calculation drill." The generative AI model returns a specific explanation. Input: Prompt sentence. Output: Generated explanation.

[1648] Step 7:

[1649] The server then packages the generated description back into JSON, for example into the following JSON data:

[1650] json

[1651] {

[1652] "Explanation": "To solve the equation 3 / 4 + 2 / 3, we first need to find a common denominator. The least common multiple of 4 and 3 is 12..."

[1653] }

[1654] Input: Generated description. Output: Description data in JSON format.

[1655] Step 8:

[1656] The server sends the packaged description data to the device using an HTTP POST request. In this process, data is transferred to the device via the network. Input: Description data in JSON format. Output: Data sent to the device.

[1657] Step 9:

[1658] The device analyzes the explanatory data in JSON format received from the server. The analyzed data is converted into HTML format and displayed on the screen in a format that is easy for the user to read. Input: Explanation data in JSON format. Output: Explanation displayed in HTML format.

[1659] Step 10:

[1660] If necessary, the device will use speech synthesis software to provide audio descriptions, allowing users to hear as well as see the information. Input: Description data in HTML format. Output: Audio description.

[1661] In this way, the questions entered by the user are processed through a number of steps and an explanation is quickly generated and displayed. By providing immediate answers to questions during learning, learners can deepen their understanding and increase their motivation to learn.

[1662] (Application example 1)

[1663] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1664] In factories, when operational problems occur with machines or robots, it is difficult for engineers to respond quickly and resolve them. In particular, if specific explanations about operation are lacking, it takes time to resolve the problem, resulting in a decrease in production efficiency. The present invention aims to provide technology for quickly and appropriately resolving operational problems and machine issues encountered by workers and engineers in factories.

[1665] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1666] In this invention, the server includes means for receiving a question entered by a user, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for sending the generated explanation to the user, means for displaying the explanation sent to the user, means for receiving and analyzing a question about an operation problem of a factory robot and generating an explanation using a generative AI model, and means for sending and displaying the generated explanation to the display and audio output means of the factory robot, thereby enabling workers and engineers in a factory to quickly and appropriately solve operational problems and issues with machines and robots.

[1667] "Users" refer to factory workers and engineers who use the system.

[1668] "Questions" refer to inquiries about operational problems and machine issues entered by factory workers and engineers through the system.

[1669] "Means for receiving" refers to a method or device that allows the system to acquire and store the content of the question entered by the user.

[1670] "Means for analyzing" refers to a method or device that analyzes the content of the received query and identifies the problem based on the information contained therein.

[1671] "Means of identification" refers to the methods and devices used to identify specific operational problems or machine issues from the analyzed questions.

[1672] A "generative AI model" refers to an algorithm or program that uses machine learning and natural language processing techniques to generate explanations for identified problems.

[1673] "Means for generating explanations" refers to a method or device that utilizes a generative AI model to create detailed explanations and solutions to an identified problem.

[1674] "Means for transmitting to a user" refers to a method or device for transmitting the generated commentary from the system to a user.

[1675] "Means for displaying" refers to a method or device for displaying the transmitted commentary in a form that can be understood by the user.

[1676] "Factory robot" refers to an industrial robot on which the present system is installed.

[1677] "Operational trouble" refers to problems or malfunctions that occur during the operation of factory robots and other related machinery.

[1678] "Display" refers to a screen or monitor for visually displaying the generated commentary.

[1679] The "audio output means" refers to a speaker or a voice synthesizer for transmitting the generated commentary to the user by voice.

[1680] "Data format" refers to the standard data structure and format used when communicating question content and explanation data between servers and systems.

[1681] The system program that realizes this application example is designed to solve operational problems of factory robots. Below we explain how this system works.

[1682] Hardware and software used

[1683] The system mainly uses the following hardware and software:

[1684] Hardware: Factory robot body, touch panel, speaker, central server

[1685] Software: Question receiving program, data analysis program, generative AI model (e.g., OpenAI's GPT series), data transmission and reception program

[1686] User operations and question input

[1687] Workers and engineers (users) use the factory robot's touch panel or voice input function to input questions about operational problems into the system. For example, they can input a question like, "I don't know why the robot arm on line 3 isn't moving."

[1688] Sending data of question content

[1689] The question is converted to JSON format and sent to a central server. The server uses an analysis program to analyze the received question and identify the problem. For example, the JSON-formatted question data may contain information such as "Question: 'The robot arm on line 3 is not moving.'"

[1690] Server-based problem analysis and explanation generation

[1691] The server uses an analysis program to analyze the question and generates an explanation for the identified problem using a generative AI model. For example, after analyzing the question, the generative AI model generates an explanation such as, "It is highly likely that a foreign object is stuck in the sensor. Please check the sensor first."

[1692] Data transmission and display of commentary

[1693] The generated explanation is then packaged again in JSON format and sent to the factory robot, which then displays the explanation on its display and, if necessary, outputs it aloud, allowing workers and technicians to quickly understand how to solve the problem.

[1694] Examples and prompts

[1695] For example, a factory worker may input a question through a display, such as, "A sensor error is displayed. What should I do?" This question data is converted into JSON format and sent to a central server for analysis. The generative AI model generates a specific explanation, such as, "The possible cause of the sensor error is that a foreign object is stuck in the sensor. First, check the sensor and see if there is a foreign object." The explanation is sent to a factory robot, which then provides the explanation on the display and in voice.

[1696] Example prompt sentence:

[1697] "Please tell me why the machine isn't working. For example, the robot arm on line 3 has stopped working."

[1698] In this way, the system helps factory workers and engineers quickly and efficiently resolve operational problems they encounter.

[1699] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1700] Step 1:

[1701] A user uses the factory robot's touch panel or voice input function to input questions about operational problems. This input is in the form of a string, such as "I don't know why the robot arm on line 3 isn't moving."

[1702] Step 2:

[1703] The terminal converts the question entered by the user into JSON format. It parses the entered string and converts it into JSON data in the format of, for example, "Question: 'The robot arm on line 3 is not moving.'" This data format conversion results in structured data.

[1704] Step 3:

[1705] The device sends the converted JSON-formatted question data to the central server, which receives the data sent from the device via an API endpoint, thereby obtaining JSON data containing the question content.

[1706] Step 4:

[1707] The server analyzes the received JSON data and identifies the relevant problem from the question. Specifically, it analyzes the "question" field of the received data and extracts information about the relevant machine and the problem. For example, it performs an analysis based on the information, "The robot arm on line 3 is not moving."

[1708] Step 5:

[1709] The server uses a generative AI model to generate an explanation based on the analyzed question. Using a generative AI model (e.g., OpenAI's GPT series), the server generates an explanation based on the analysis results. For example, it creates an explanation such as, "There is a high possibility that a foreign object is stuck in the sensor. Please check the sensor first."

[1710] Step 6:

[1711] The server then packages the generated explanation into JSON format again and sends it to the device. This operation sends explanation data from the server to the device. Specifically, JSON data in the format "Explanation: 'There is a high possibility that a foreign object is stuck in the sensor. Please check the sensor first.'" is sent.

[1712] Step 7:

[1713] The device analyzes the received JSON-formatted explanatory data and converts it into a displayable format. Based on the analysis results, the explanation is converted into text or HTML format. This data conversion makes it possible to provide information to the user visually and audibly.

[1714] Step 8:

[1715] The device displays the converted commentary on the display and outputs it as audio if necessary. The user can check the commentary on the display and, in some cases, listen to the commentary via audio guidance. For example, the display may say, "There is a high possibility that a foreign object is stuck in the sensor. Please check the sensor first."

[1716] By following the steps explained in this manual, users can quickly and accurately identify the cause and solution of operational problems, thereby improving production efficiency and reducing the time required to resolve problems.

[1717] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1718] The present invention combines a system that solves the "I don't understand" problem that users experience during the learning process with an emotion engine that recognizes the user's emotions and adjusts the response content. The purpose of this invention is to improve the learning experience by taking into account the user's emotional state as well as providing appropriate explanations.

[1719] User input of question

[1720] When a user has a question about a problem, they enter the question into the terminal. For example, they might say, "I don't understand page 30(2) of the calculation drill." The terminal receives the user's input and processes it as properly formatted data.

[1721] Sending data from the device to the server

[1722] The terminal converts the question entered by the user into a data format (for example, JSON format) and sends it to the specified endpoint of the server. This allows the server to receive the user's question and prepare for the next analysis process.

[1723] Server analysis and identification of the problem

[1724] The server analyzes the received question data and identifies the relevant question from the question content. Specifically, it extracts information such as the name of the learning material, page number, and question number from the received data and searches the database on the server for the relevant question.

[1725] Server-generated description

[1726] After identifying the problem, the server uses a generative AI model to generate a solution for the problem. The server creates a detailed solution that includes step-by-step instructions, important points to note, and background information on the problem. For example, it might say, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator."

[1727] Emotion recognition by emotion engine

[1728] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's facial expressions, tone of voice, typing speed, etc. to identify the emotion the user is feeling. For example, it detects whether the user is feeling anxious or irritated.

[1729] Adjustment of commentary content

[1730] The server adjusts the content and tone of the generated commentary based on the user's emotions recognized by the emotion engine. For example, if the user is feeling anxious, the server changes the content of the commentary to be more friendly and encouraging.

[1731] Sending and displaying commentary

[1732] The server packages the generated commentary in an appropriate data format (for example, JSON or HTML format) and sends it to the terminal. The terminal analyzes the commentary data received from the server and displays it on the screen in a format that is easy for the user to understand. If necessary, the terminal can also provide audio commentary using audio output means.

[1733] Specific examples

[1734] User input

[1735] User: "I don't understand page 30(2) of the math drill."

[1736] Terminal: (Display the input contents in the form and confirm)

[1737] Sending data from the device to the server

[1738] Terminal: (Convert input to JSON and send to server)

[1739] Server analysis and identification of the problem

[1740] Server: (Analysis of "I don't understand page 30(2) of the calculation drill")

[1741] Server: (Identifies the problem in an internal database)

[1742] Server-generated description

[1743] Server: (Generates explanations using AI models)

[1744] Server: (Configure generated commentary)

[1745] Emotion recognition by emotion engine

[1746] Server: (Analyzes user emotions and detects anxiety)

[1747] Adjustment of commentary content

[1748] Server: (adjusts commentary based on perceived emotion)

[1749] Sending and displaying commentary

[1750] Server: (Package the explanation in JSON format)

[1751] Server: (Sent to the device)

[1752] Display explanation

[1753] Terminal: (Analyzes received data and converts it to HTML)

[1754] Terminal: (Explanation displayed on screen)

[1755] Device: (Audio description available if needed)

[1756] In this way, the system can improve the learning experience by providing accurate explanations in real time to any questions or anxieties the user may have while studying, and by providing optimal responses based on the user's emotional state.

[1757] The processing flow will be explained below.

[1758] Step 1: User enters question

[1759] If a user has a question about a problem, they can enter the question using the terminal. For example, they can enter "I don't understand page 30(2) of the calculation drill."

[1760] Step 2: Check input on the terminal and convert to data format

[1761] The device checks the user's input and converts it into an appropriate format (e.g., JSON), checking that the input has been converted correctly into the data format.

[1762] Step 3: Send to the server

[1763] The terminal sends formatted data to the server, which then sends the data to the specified endpoint while checking the stability of the communication and the integrity of the data.

[1764] Step 4: Data reception and analysis by the server

[1765] The server receives the data sent from the terminal, analyzes the question content based on the received data, and extracts information such as the name of the learning material, page number, and question number.

[1766] Step 5: Identify the problem

[1767] Based on the information analyzed by the server, the server searches and identifies the relevant problem in the internal teaching material database. For example, it retrieves the problem on page 30(2) of the arithmetic drill from the database.

[1768] Step 6: Server-generated description

[1769] The server uses a generative AI model to generate a detailed explanation for the identified problem, including the background to the problem, the steps to solving it, and common mistakes.

[1770] Step 7: Emotion Recognition with the Emotion Engine

[1771] The server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial expressions, tone of voice, typing speed, etc. to determine whether the user is feeling anxious or annoyed.

[1772] Step 8: Adjust the commentary

[1773] The server adjusts the content and tone of the generated commentary based on the user's emotions recognized by the emotion engine. For example, if the user is feeling anxious, the server changes the commentary to be more friendly and encouraging.

[1774] Step 9: Converting the description to a data format

[1775] The server packages the adjusted description into an appropriate format (e.g., JSON or HTML), and the data is organized in a format that can be displayed on the user's device.

[1776] Step 10: Send data to your device

[1777] The server sends the formatted explanatory data to the terminal, and checks the integrity of the data while ensuring the stability and security of the communication.

[1778] Step 11: Receiving and analyzing data on the device

[1779] The terminal receives the commentary data sent from the server, analyzes the received data, and converts it into a format that can be displayed to the user (for example, HTML or text).

[1780] Step 12: Display the description

[1781] The device displays instructions on the user's screen, and optionally provides audio instructions using a voice output means. For example, detailed instructions such as "First, find the common denominator, then add the fractions."

[1782] Specific examples

[1783] 1. User: "I don't understand page 30(2) of the math drill."

[1784] 2. Terminal: (Converts input into JSON and sends it to the server)

[1785] 3. Server: (Analysis of "I don't understand page 30(2) of the math drill")

[1786] 4. Server: (Identify the problem in the internal database)

[1787] 5. Server: (Generates explanations using AI models)

[1788] 6. Server: (Analyze user emotions and detect anxiety)

[1789] 7. Server: (adjusts commentary based on perceived emotion)

[1790] 8. Server: (Package the description in JSON format)

[1791] 9. Server: (Send to device)

[1792] 10. Terminal: (Analyzes received data and converts it to HTML)

[1793] 11. Terminal: (Displays explanation on the screen)

[1794] 12. Terminal: (Audio description provided if needed)

[1795] This allows users to receive real-time explanations that resolve their questions and improve their learning experience by responding to their emotions.

[1796] Example 2

[1797] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1798] Conventional learning support systems have limited functionality for providing solutions to users' "unclear" problems in real time, and have difficulty providing appropriate explanations that take the user's emotional state into account. Furthermore, in order to provide accurate explanations to users' questions, it is necessary to analyze the user's input in detail and accurately recognize the user's emotional state. In this context, the present invention aims to realize a system that not only responds quickly and accurately to users' questions, but also provides optimal explanations based on the user's emotional state.

[1799] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1800] In this invention, the server includes means for receiving a question entered by a user, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for recognizing the user's emotion, means for adjusting the content of the generated explanation based on the recognized user emotion, means for sending the adjusted explanation to the user, and means for displaying the explanation sent to the user. This makes it possible to provide accurate and prompt answers to questions that the user has during the learning process, and further to provide optimal explanations according to the user's emotional state.

[1801] A "user" is a subject who uses the system to learn, enter questions, and receive explanations.

[1802] A "terminal" is a device that a user uses to input questions and that sends and receives information.

[1803] The "server" is a central system that analyzes the received question, generates an explanation, and transmits the explanation to the user.

[1804] "Question content" refers to a specific problem or question that a user wants to solve while studying.

[1805] The "receiving means" is a system for taking into the server the content of the question entered by the user through the terminal.

[1806] "Analysis means" refers to procedures and techniques for analyzing the content of the received question and identifying the relevant problem.

[1807] A "generative AI model" is an artificial intelligence model used to generate appropriate explanations for identified problems.

[1808] "Explanation" is information that includes answers and explanations to the user's questions.

[1809] "Emotion recognition means" refers to technology or systems for analyzing and identifying a user's emotional state.

[1810] "Adjustment means" refers to techniques or methods for adjusting the content and tone of generated commentary based on the perceived user sentiment.

[1811] "Transmission means" refers to a system or technology for transmitting the adjusted commentary to the user.

[1812] The "display means" refers to a system or function for displaying the commentary sent to the user on the screen.

[1813] This invention is a system that solves the "I don't understand" problem that users experience during the learning process, and combines it with an emotion engine that recognizes the user's emotions and adjusts the response content. The purpose of this invention is to improve the learning experience by providing appropriate explanations and taking into account the user's emotional state.

[1814] Hardware and software used

[1815] The main components of the system are:

[1816] 1. Terminal: The device through which the user enters their question, such as a PC, tablet, or smartphone.

[1817] 2. Server: A central system that analyzes questions, generates explanations, and recognizes user emotions. This can be a cloud server or an on-premise server.

[1818] 3. Generative AI model: An artificial intelligence model used to generate an appropriate explanation for an identified problem.

[1819] 4. Emotion engine: Software for recognizing the user's emotional state. This includes facial expression recognition software and voice analysis software.

[1820] Data processing and calculation

[1821] 1. Entering and receiving questions

[1822] When a user has a question while studying, they can enter it into the device. For example, they might write, "I don't understand page 30(2) of the math drill." The device will then check the input and convert it into the appropriate data format.

[1823] 2. Data Transmission

[1824] The device converts the input question into JSON format and sends it to the server, to an endpoint pre-specified within the system.

[1825] 3. Question analysis and identification

[1826] The server extracts the learning material name, page number, and question number from the received JSON data and searches the database for the corresponding question. Once the question is identified, it uses that information to generate a prompt for the generative AI model.

[1827] 4. Generating Explanations

[1828] The server uses a generative AI model to generate a commentary. Enter the following prompt:

[1829] "A user says he doesn't understand page 30(2) of the math drill. Please provide him with a step-by-step explanation starting with finding the common denominator. Also include an encouraging message to ease any anxiety he may be feeling."

[1830] The generative AI model generates step-by-step instructions and provides detailed explanations.

[1831] 5. Emotion recognition

[1832] The server uses an emotion engine to analyze the user's facial expression and voice data received from the device and identify the user's emotional state, for example, determining whether the user is feeling anxious or irritated.

[1833] 6. Adjustment of explanation content

[1834] The server adjusts the generated commentary based on information obtained from the emotion engine: for example, if the user is feeling anxious, it makes the commentary more friendly and adds encouraging elements.

[1835] 7. Sending and Displaying Commentary

[1836] The server converts the adjusted commentary into JSON or HTML format and sends it to the device. The device analyzes the received data and displays it in a format that is easy for the user to read. It also outputs audio if necessary.

[1837] Specific examples

[1838] The user enters "I don't understand page 30(2) of the calculation drill."

[1839] Terminal: Display input and ask user for confirmation

[1840] Terminal: Convert to JSON format and send to server

[1841] The server receives and analyzes the data

[1842] Server: Extract the teaching material name, page number, and question number

[1843] Server: Search database for matching problem

[1844] Generate explanations

[1845] Server: Enter prompt text using generative AI model: "A user says he doesn't understand page 30(2) of the math drill. Please provide him with a step-by-step explanation, starting with finding the common denominator. Include an encouraging message to ease any anxiety he may be feeling."

[1846] Generative AI model: Generate detailed explanations

[1847] emotion recognition

[1848] Server: Analyzes the user's facial expression and voice data to determine their emotional state

[1849] Adjustment of commentary content

[1850] Server: Adjust the tone and content of commentary depending on the emotional state

[1851] Sending and displaying commentary

[1852] Server: Converts the adjusted commentary into JSON or HTML format and sends it to the device.

[1853] Terminal: Analyzes received data and displays it on the screen. Also outputs audio if necessary.

[1854] In this way, the system can provide accurate explanations in real time to address any questions or concerns users may have while studying, providing an optimal learning experience.

[1855] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1856] Program processing flow

[1857] Step 1: Enter your question

[1858] If a user has a question while studying, they can enter it into the text field using the terminal. For example, they can enter "I don't understand page 30(2) of the calculation drill."

[1859] Input: Questions entered by the user

[1860] Output: The question displayed on the terminal

[1861] Specific behavior:

[1862] A user opens a learning app and writes a question in a text input field.

[1863] The terminal displays the entered text as is for the user to review.

[1864] Step 2: Data format conversion

[1865] The device converts the questions entered by the user into JSON format, which prepares the data in a suitable format for transmission.

[1866] Input: Text data entered by the user

[1867] Output: Data converted to JSON format

[1868] Specific behavior:

[1869] The terminal converts the input information into JSON format, dividing it into fields such as "teaching material name," "page number," and "question number."

[1870] The converted JSON data is passed to the next processing step.

[1871] Step 3: Send data

[1872] The device sends the question converted into JSON format to the server, to an endpoint pre-specified within the system.

[1873] Input: Data converted to JSON format

[1874] Output: Data sent to the server

[1875] Specific behavior:

[1876] The device sends the JSON data as an HTTPS request to the server's URL.

[1877] Step 4: Analyze and identify questions

[1878] The server parses the received JSON data, extracts the learning material name, page number, and question number, and uses this information to search and identify the corresponding question in its internal database.

[1879] Input: JSON data sent to the server

[1880] Output: Identification of the problem

[1881] Specific behavior:

[1882] The server extracts the fields "teaching material name," "page number," and "question number" from the JSON data.

[1883] Based on the extracted information, a query is run against the database to search for the relevant problem.

[1884] Step 5: Generate a description

[1885] The server uses a generative AI model to create a prompt based on the identified problem, which is then input into the AI ​​model to generate an explanation.

[1886] Input: Identified problem information

[1887] Output: Commentary by the generative AI model

[1888] Specific behavior:

[1889] The server creates a prompt sentence and inputs it into the generative AI model.

[1890] The generative AI model generates step-by-step instructions based on the prompt.

[1891] Example prompt statement:

[1892] "A user says he doesn't understand page 30(2) of the math drill. Please provide him with a step-by-step explanation starting with finding the common denominator. Also include an encouraging message to ease any anxiety he may be feeling."

[1893] Step 6: Recognize emotions

[1894] The server uses an emotion engine to analyze the user's emotions, analyzing the user's facial expressions, tone of voice, typing speed, etc. to identify the emotion the user is feeling.

[1895] Input: User's facial expression data, voice data

[1896] Output: User's emotional state

[1897] Specific behavior:

[1898] The server passes the webcam and microphone data received from the device to the emotion engine.

[1899] The emotion engine analyzes this data to determine the user's emotional state.

[1900] Step 7: Adjust the commentary

[1901] The server then adjusts the generated commentary based on the information obtained through emotion recognition, for example adding encouraging elements to the commentary if the user is feeling anxious.

[1902] Input: Emotion recognition data, generated commentary

[1903] Output: Adjusted description

[1904] Specific behavior:

[1905] The server reconstructs the generated commentary based on the emotional state.

[1906] The adjusted description is passed to the next processing step.

[1907] Step 8: Send and view your commentary

[1908] The post-adjustment explanation is converted into JSON or HTML format and sent to the device. The device analyzes the received data and displays it in a format that is easy for the user to read. It also outputs audio if necessary.

[1909] Input: Adjusted description

[1910] Output: The description displayed to the user

[1911] Specific behavior:

[1912] The server converts the adjusted description into JSON or HTML format.

[1913] The server sends this content to the terminal as an HTTPS response.

[1914] The device analyzes the received data and displays commentary on the screen, and also provides audio commentary if necessary.

[1915] In this way, the system can provide accurate explanations in real time to any questions or concerns users may have while studying, providing an optimal learning experience.

[1916] (Application example 2)

[1917] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1918] The problem that this invention aims to solve is to realize an effective learning experience by resolving the doubts and anxieties that users feel during learning in real time while providing emotionally sensitive explanations. Conventional learning support systems provide explanations without taking the user's emotional state into consideration, which has the problem of not being able to alleviate the anxiety and frustration that users feel during the learning process. This poses a risk of reducing the user's motivation to learn and reducing their level of understanding.

[1919] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1920] In this invention, the server includes means for receiving a question entered by a user, means for analyzing the received question and identifying a corresponding problem, means for generating an explanation for the identified problem using a generative AI model, means for sending the generated explanation to the user, means for displaying the explanation sent to the user, means for analyzing the user's emotions, and means for adjusting the content of the explanation based on the analyzed emotions. This makes it possible to provide appropriate and friendly explanations while taking the user's emotional state into consideration.

[1921] The "means for receiving question content" refers to a hardware or software function for receiving a question entered by a user via a device.

[1922] "Means for analyzing the content of the question and identifying the relevant problem" refers to technology that analyzes the content of the received question using natural language processing and database search, and identifies the specific problem to be solved within the teaching materials or problem collection.

[1923] "Means for generating explanations using generative AI models" refers to technology that uses artificial intelligence to generate appropriate explanations and answers for identified problems.

[1924] The "means for transmitting the generated commentary to the user" refers to a communication means or protocol for transmitting the commentary generated on the server side to the user's device.

[1925] The "means for displaying the commentary sent to the user" refers to a screen display function or interface for visually displaying the commentary on the user's device.

[1926] "Means for analyzing user emotions" refers to technology that analyzes the user's facial expressions, tone of voice, input speed, etc., to identify the emotions the user is feeling.

[1927] The "means for adjusting the content of the commentary based on the analyzed emotions" is a technology for changing the tone and content of the commentary according to the user's emotional state, and adding friendly explanations and encouraging words.

[1928] This invention is a system that solves the problem of "I don't understand" that users experience during the learning process, and in particular, it combines an "emotion engine" that recognizes the user's emotions and adjusts the response accordingly. This system aims to improve the learning experience by taking into account the user's emotional state while providing appropriate explanations.

[1929] User input of question

[1930] If a user has a question while studying, they can input it into a device such as a smartphone or head-mounted display. For example, they can input a question into the device, such as, "I don't understand page 30(2) of the math drill." The device receives the user's input and processes it as appropriately formatted data.

[1931] Sending data from the device to the server

[1932] The terminal converts the question entered by the user into a data format (for example, JSON format) and sends it to the specified endpoint of the server. The server receives the user's question and is ready to move on to the next analysis process.

[1933] Server analysis and identification of the problem

[1934] The server analyzes the received question data and identifies the relevant question from the question content. Specifically, it extracts information such as the name of the learning material, page number, and question number from the received data and searches the database on the server for the relevant question.

[1935] Server-generated description

[1936] After identifying the problem, the server uses a generative AI model to generate a solution for the problem. The server creates a detailed solution that includes step-by-step instructions, important points to note, and background information on the problem. For example, it might say, "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator."

[1937] Emotion recognition by emotion engine

[1938] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's facial expressions, tone of voice, typing speed, etc. to identify the emotion the user is feeling. For example, it detects whether the user is feeling anxious or irritated.

[1939] Adjustment of commentary content

[1940] The server adjusts the content and tone of the generated commentary based on the user's emotions recognized by the emotion engine. For example, if the user is feeling anxious, the server changes the content of the commentary to be more friendly and encouraging.

[1941] Sending and displaying commentary

[1942] The server packages the generated commentary in an appropriate data format (for example, JSON or HTML format) and sends it to the terminal. The terminal analyzes the commentary data received from the server and displays it on the screen in a format that is easy for the user to understand. If necessary, the terminal can also provide audio commentary using audio output means.

[1943] Specific examples of hardware and software used

[1944] Hardware used: smartphone, head-mounted display (HMD)

[1945] Software used:

[1946] Emotion Recognizer: Identifies emotions by analyzing the user's facial expressions and tone of voice.

[1947] Generative AI model (Explanation Generator): Generates appropriate explanations for the user's questions.

[1948] Text to Speech Engine (Text to Speech): Provides the generated commentary in audio format.

[1949] Virtual Display Engine (VirtualDisplay): Visually displays the generated commentary to the user.

[1950] Specific examples

[1951] User input

[1952] User: "I don't know how to add fractions."

[1953] Terminal: (Display the input contents in the form and confirm)

[1954] Sending data from the device to the server

[1955] Terminal: (Convert input to JSON and send to server)

[1956] Server analysis and identification of the problem

[1957] Server: ("I don't know how to add fractions" analysis)

[1958] Server: (Identifies the problem in an internal database)

[1959] Server-generated description

[1960] Server: (Generate explanations using generative AI models)

[1961] Server: (Configure generated commentary)

[1962] Emotion recognition by emotion engine

[1963] Server: (Analyzes user emotions and detects anxiety)

[1964] Adjustment of commentary content

[1965] Server: (adjusts commentary based on perceived emotion)

[1966] Sending and displaying commentary

[1967] Server: (Package the explanation in JSON format)

[1968] Server: (Sent to the device)

[1969] Display explanation

[1970] Terminal: (Analyzes received data and converts it to HTML)

[1971] Terminal: (Explanation displayed on screen)

[1972] Device: (Audio description available if needed)

[1973] Prompt Sentence Examples

[1974] If the user types "I don't know how to add fractions":

[1975] (Prompt for generative AI model)

[1976] "The user feels that they don't understand how to add fractions. Please explain carefully, especially emphasizing the step of finding the 'common denominator.' The user seems confused, so please explain slowly and clearly."

[1977] In this way, the system can improve the learning experience by providing accurate explanations in real time to any questions or anxieties the user may have while studying, and by providing optimal responses based on the user's emotional state.

[1978] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1979] Step 1:

[1980] User enters question

[1981] Input: The user uses a device (smartphone or head-mounted display) to input a question, such as "I don't know how to add fractions," using text or voice.

[1982] Specific operation: The terminal receives the user's input and displays it in a form for confirmation.

[1983] Step 2:

[1984] Sending data from the device to the server

[1985] Input: The question typed by the user

[1986] Data processing: The terminal converts the input content into JSON format.

[1987] Output: JSON data sent to the server

[1988] Specific operation: The terminal sends the converted JSON data to the specified endpoint on the server.

[1989] Step 3:

[1990] Server analysis and identification of the problem

[1991] Input: JSON data of the received question

[1992] Data calculation: The server analyzes the received question data, extracts information such as the name of the teaching material, page number, and question number from the question content, and performs a database search.

[1993] Output: Identifying the problem in question

[1994] Specific operation: The server searches the database for the corresponding problem and identifies detailed information about the problem.

[1995] Step 4:

[1996] Server-generated description

[1997] Input: Details of the problem in question

[1998] Data Computation: Uses a generative AI model to generate a step-by-step explanation for the problem.

[1999] Output: Generated commentary

[2000] How it works: The server uses a generative AI model to create an explanation, such as "To solve the equation 3 / 4 + 2 / 3, you first need to find a common denominator."

[2001] Step 5:

[2002] Emotion recognition by emotion engine

[2003] Input: Data such as the user's facial expression, tone of voice, and typing speed

[2004] Data Calculation: An emotion engine is used to analyze this data and identify the emotions the user is feeling.

[2005] Output: Recognized user emotion information

[2006] Specific operation: The server analyzes the user's emotions and detects feelings such as anxiety or irritation.

[2007] Step 6:

[2008] Adjustment of commentary content

[2009] Input: Generated commentary and user's emotional information

[2010] Data arithmetic: Adjust the tone and content of your commentary based on emotion, for example, using friendly language if the user is feeling anxious.

[2011] Output: Adjusted commentary

[2012] Specific operation: The server generates explanatory content that takes the user's feelings into consideration, such as, "Don't worry. Let's take it step by step. To find the common denominator..."

[2013] Step 7:

[2014] Sending and displaying commentary

[2015] Input: Adjusted commentary

[2016] Data processing: The generated commentary is packaged into an appropriate data format (JSON or HTML format) and sent to the terminal.

[2017] Output: Descriptive data converted into a displayable format

[2018] Specific operation: The device analyzes the commentary data received from the server and displays it in a format that is easy for the user to understand. Audio commentary is also provided if necessary.

[2019] Through the above steps, the system provides accurate explanations in real time to any questions or anxieties the user may have while studying, and responds optimally according to the user's emotional state.

[2020] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2021] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2022] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2023] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2024] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2025] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2026] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2027] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2028] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2029] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2030] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2031] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2032] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2033] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2034] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2035] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2036] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2037] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2038] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2039] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2040] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2041] The following is further disclosed regarding the above embodiment.

[2042] (Claim 1)

[2043] means for receiving a question input by a user;

[2044] A means for analyzing the received question and identifying the relevant problem;

[2045] A means for generating an explanation for the identified problem using a generative AI model; and

[2046] means for transmitting the generated commentary to a user;

[2047] means for displaying the commentary sent to the user;

[2048] A system including:

[2049] (Claim 2)

[2050] means for converting the received question content into a data format;

[2051] means for transmitting the converted data to a server;

[2052] means for analyzing the commentary data received from the server and converting it into a displayable format;

[2053] 10. The system of claim 1, comprising:

[2054] (Claim 3)

[2055] 2. The system according to claim 1, wherein the explanation displayed to the user is accompanied by an audio output means.

[2056] "Example 1"

[2057] (Claim 1)

[2058] means for receiving a question input by a user;

[2059] means for converting the received question content into a data format;

[2060] means for transmitting the converted data to a server;

[2061] A means for analyzing the received question and identifying the relevant problem;

[2062] A means for generating an explanation for the identified problem using a generative AI model; and

[2063] means for transmitting the generated commentary to a user;

[2064] means for displaying the commentary sent to the user;

[2065] A system including:

[2066] (Claim 2)

[2067] means for analyzing the commentary data received from the server and converting it into a displayable format;

[2068] audio output means for outputting the commentary data as audio;

[2069] 10. The system of claim 1.

[2070] (Claim 3)

[2071] means for using the prompt sentence as input to the generative AI model;

[2072] 10. The system of claim 1.

[2073] "Application Example 1"

[2074] (Claim 1)

[2075] means for receiving a question input by a user;

[2076] A means for analyzing the received question and identifying the relevant problem;

[2077] A means for generating an explanation for the identified problem using a generative AI model; and

[2078] means for transmitting the generated commentary to a user;

[2079] means for displaying the commentary sent to the user;

[2080] A means for receiving and analyzing questions about operational problems with factory robots and generating explanations using a generative AI model;

[2081] a means for transmitting the generated commentary to a display and a voice output means of the factory robot and displaying the commentary;

[2082] A system including:

[2083] (Claim 2)

[2084] means for converting the received question content into a data format;

[2085] means for transmitting the converted data to a server;

[2086] means for analyzing the commentary data received from the server and converting it into a displayable format;

[2087] means for converting the content of the query received by the factory robot into a data format;

[2088] means for transmitting the converted data to a server and receiving commentary data;

[2089] a means for displaying and outputting the received commentary data on a display and a voice output means of the factory robot;

[2090] 10. The system of claim 1, comprising:

[2091] (Claim 3)

[2092] 2. The system according to claim 1, wherein the explanation displayed to the user is accompanied by audio output means and is an explanation regarding the operation of a factory robot.

[2093] "Example 2: Combining Emotion Engines"

[2094] (Claim 1)

[2095] means for receiving a question input by a user;

[2096] A means for analyzing the received question and identifying the relevant problem;

[2097] A means for generating an explanation for the identified problem using a generative AI model; and

[2098] means for recognizing a user's emotion;

[2099] means for adjusting the content of the generated commentary based on the recognized user sentiment;

[2100] means for transmitting the adjusted commentary to the user;

[2101] means for displaying the commentary sent to the user;

[2102] A system including:

[2103] (Claim 2)

[2104] means for converting the received question content into a data format;

[2105] means for transmitting the converted data to a server;

[2106] means for analyzing the commentary data received from the server and converting it into a displayable format;

[2107] 10. The system of claim 1, comprising:

[2108] (Claim 3)

[2109] 2. The system according to claim 1, wherein the explanation displayed to the user is accompanied by an audio output means.

[2110] "Application example 2 when combining emotion engines"

[2111] (Claim 1)

[2112] means for receiving a question input by a user;

[2113] A means for analyzing the received question and identifying the relevant problem;

[2114] A means for generating an explanation for the identified problem using a generative AI model; and

[2115] means for transmitting the generated commentary to a user;

[2116] means for displaying the commentary sent to the user;

[2117] means for analyzing user emotions;

[2118] A means for adjusting the content of the commentary based on the analyzed emotions;

[2119] A system including:

[2120] (Claim 2)

[2121] means for converting the received question content into a data format;

[2122] means for transmitting the converted data to a server;

[2123] means for analyzing the commentary data received from the server and converting it into a displayable format;

[2124] A means for adjusting the tone and method of commentary according to the user's emotions;

[2125] 10. The system of claim 1, comprising:

[2126] (Claim 3)

[2127] 2. The system according to claim 1, wherein the explanation displayed to the user is accompanied by an audio output means. [Explanation of symbols]

[2128] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving a question input by a user; A means for analyzing the received question and identifying the relevant problem; A means for generating an explanation for the identified problem using a generative AI model; and means for transmitting the generated commentary to a user; means for displaying the commentary sent to the user; A system including:

2. means for converting the received question content into a data format; means for transmitting the converted data to a server; means for analyzing the commentary data received from the server and converting it into a displayable format; The system of claim 1 , comprising:

3. 2. The system of claim 1, wherein the explanation displayed to the user is accompanied by audio output means.

Images