system

Abstract

We provide the system. [Solution] A means of receiving questions from specific users, A means of analyzing received questions and extracting keywords and topics related to the content of the questions, A means of generating appropriate answers using generative artificial intelligence based on extracted keywords and topics, A means of sending the generated response back to a specific user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Describe "Problems to be Solved by the Invention" and "Means for Solving the Problems".

[0005] In current online learning platforms, it is difficult for students to obtain prompt and appropriate answers even when they post questions or doubts. As a result, there are problems such as a decline in students' learning efficiency and a decrease in motivation for learning. In addition, the method of providing answers manually takes time for response and is difficult for individual response. Therefore, there is a demand for providing a learning support system that can respond quickly and appropriately on an individual basis.

Means for Solving the Problems

[0006] The present invention solves the above problems by a system that includes means for receiving questions from a specific user, analyzing the received questions to extract keywords and topics, generating appropriate answers using generative artificial intelligence based on these extraction results, and returning the generated answers to the specific user. This system enables quick and accurate responses to questions, thereby improving students' learning efficiency. Furthermore, by using natural language processing technology, more advanced question analysis and answer generation can be achieved, and individualized learning support can be facilitated smoothly.

[0007] "Users" refer to students or learners who use online learning platforms.

[0008] A "question" refers to the content of problems or inquiries that users post through the learning platform.

[0009] "Means of receiving questions" refers to the hardware and software functions for obtaining questions submitted by users.

[0010] "Methods for analyzing questions" refers to text analysis techniques used to break down the content of received questions and understand their meaning.

[0011] "Keywords and topics" refer to words or themes that are important for understanding the content of the question.

[0012] "Generative artificial intelligence" refers to artificial intelligence technology that generates appropriate answers based on questions.

[0013] "Means for generating answers" refers to a function that uses generative artificial intelligence to create appropriate answers to user questions.

[0014] "Means of returning responses" refers to hardware and software functions that send generated responses to users and allow them to review them.

[0015] "The means of shaping" refers to the function for adjusting the display format when returning the generated answer to the user.

Brief Explanation of Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. <动 [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [[ID=*5]] [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when combined with an emotion engine.

Embodiments for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention relates to a learning support system that receives questions from a specific user, analyzes them, generates appropriate answers, and returns them to the user. This system consists of multiple components, including the user's terminal, a server, and generative artificial intelligence (AI).

[0038] 1. User actions

[0039] User: For example, a high school student logs into an online learning platform at home. The user asks questions about things they are unsure of or problems they cannot solve during their studies through the "User Interface (UI)". For example, they might type a question like, "Please explain how to solve quadratic equations."

[0040] 2. Receiving the question

[0041] Terminal: When the user clicks the submit button, the terminal converts the question content into a data format such as JSON, issues an HTTP request, and sends it to the server.

[0042] Server: The server receives this HTTP request and retrieves the question content. It also stores data containing user information in the database to identify which user the question is from.

[0043] 3. Analysis of the Question

[0044] Server: Next, the server passes the received question to the text analysis module. This module uses natural language processing (NLP) techniques to analyze the question and extract important keywords and topics. For example, the keywords "quadratic equation" and "solution method" might be extracted.

[0045] 4. Generating the answer

[0046] Server: Based on the extracted keywords and topics, the server sends the analysis results to a generative AI engine. The generative AI engine understands the question and generates an appropriate answer. In this case, a specific answer is generated: "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0047] 5. Return your response

[0048] Server: Receives the generated response and formats it for return to the user. For example, it formats it to be easily viewable on the UI, and then sends an HTTP response containing the formatted data to the device.

[0049] Terminal: Analyzes response data received from the server and displays it on the user interface. Users can check the responses displayed on the terminal screen.

[0050] Specific example

[0051] For example, consider the case where a student asks, "Could you tell me the equation of a straight line?"

[0052] 1. User: A student types and submits the question, "Please tell me the equation of a straight line."

[0053] 2. Terminal: Converts the question content into JSON format and sends an HTTP request to the server.

[0054] 3. Server: Receives an HTTP request and passes the question "equation of a straight line" to the text analysis module.

[0055] 4. Text analysis module: Extracts keywords such as "straight line" and "equation," and sends the results to a generative AI engine.

[0056] 5. Generative AI engine: Generates the answer "The equation of a straight line is y = mx + b" and returns it to the server.

[0057] 6. Server: Formats the generated response into a user-friendly format for the UI and sends it to the device as an HTTP response.

[0058] 7. Terminal: Analyzes the HTTP response and displays the answer on the student's screen.

[0059] 8. User: You can check the answer displayed on the screen, "The equation of a straight line is y = mx + b," and use it to further your learning.

[0060] This invention enables effective personalized learning support because the process from when a user posts a question to when they receive an answer is carried out quickly and appropriately.

[0061] The following describes the processing flow.

[0062] Step 1:

[0063] User: After logging into the learning platform's user interface, the user enters their question in the question submission field and clicks the submit button.

[0064] Step 2:

[0065] Terminal: Retrieves the input question content and converts it to an appropriate data format such as JSON. Then, it sends this data to the server as an HTTP request.

[0066] Step 3:

[0067] Server: Receives HTTP requests and saves question data and user information to the database. Then, passes the received question content to the text analysis module.

[0068] Step 4:

[0069] Text Analysis Module: Analyzes question text using natural language processing (NLP) techniques to extract important keywords and topics. For example, it can extract keywords such as "quadratic equation" and "solution method."

[0070] Step 5:

[0071] Server: Based on extracted keywords and topics, it sends the analysis results to the generative AI engine.

[0072] Step 6:

[0073] Generative AI Engine: The generative AI engine generates appropriate answers from the analysis results it receives. For example, it can generate a specific answer such as, "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0074] Step 7:

[0075] Server: Receives the generated response and formats it for return to the user. For example, it formats it to be easily viewable on the UI and sends the HTTP response containing the formatted data to the device.

[0076] Step 8:

[0077] Terminal: Analyzes response data received from the server and displays it on the user interface.

[0078] Step 9:

[0079] User: Check the answer displayed on the device screen. If the question is resolved, continue learning. If you have further questions, return to step 1 and enter a new question.

[0080] The above outlines the specific processing steps involved in providing answers to user questions on an online learning platform.

[0081] (Example 1)

[0082] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0083] Conventional learning support systems suffered from problems such as delays in responding to user questions and the inability to obtain appropriate answers. This resulted in reduced learning efficiency and difficulty for users to quickly obtain the necessary information.

[0084] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0085] In this invention, the server includes means for receiving questions from specific end users, means for converting the received questions into a data format and sending them to the server, means for the server to store the received question data in a database and pass it to a text analysis module, means for the text analysis module to analyze the question content using natural language processing technology and extract important keywords and topics, means for generating appropriate answers using generative artificial intelligence based on the extracted keywords and topics, and means for formatting the generated answers into a user-friendly format on the user interface and returning them to the end user. This makes it possible for users to quickly and accurately resolve questions and problems they cannot solve during learning, thereby improving learning efficiency.

[0086] An "end user" refers to a user who uses the system to ask questions.

[0087] A "server" refers to a central system that receives and processes requests from clients.

[0088] "Data format" refers to a format in which data is encoded according to certain rules. For example, the JSON format is one such format.

[0089] A "database" refers to a system that systematically stores data and allows it to be retrieved as needed.

[0090] A "text analysis module" refers to a program that uses natural language processing technology to analyze text data and extract its meaning.

[0091] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human language.

[0092] "Keywords" refer to important words or phrases extracted from the question.

[0093] "Topic" refers to the subject or theme of the question.

[0094] "Generative artificial intelligence" refers to artificial intelligence that has the function of generating appropriate answers to user questions.

[0095] "User interface" refers to the screens and input methods that end users use to interact with a system.

[0096] "Formatting" refers to the process of converting data into a format that is easy to read. This includes, for example, using HTML and CSS.

[0097] Modes for carrying out the invention

[0098] This invention relates to a learning support system that receives questions from specific end users, analyzes them, generates appropriate answers, and returns them to the user. This system consists of multiple components, including the end user's terminal, a server, and generative artificial intelligence (AI).

[0099] End users log in to the online learning platform from their homes, schools, or other locations. Users input questions and problems they encounter during their studies through the user interface (UI). For example, they might input a question like, "Please explain how to solve quadratic equations."

[0100] The terminal converts the user's input into a data format (such as JSON) and sends it to the server as an HTTP request. This conversion uses the JavaScript® JSON.stringify method. The request is sent using the HTTP POST method.

[0101] The server receives an HTTP request and retrieves the question data from the request body. The Node.js express framework is used for this process. The server then stores user information in a database (e.g., MySQL®) to identify which user the question originated from. The database records the question content and associated user information.

[0102] Next, the server passes the acquired question data to the text analysis module. The text analysis module uses natural language processing (NLP) techniques to analyze the question content and extract important keywords and topics. Specifically, it uses Google's NLP API to extract keywords such as "quadratic equation" and "how to solve it."

[0103] The extracted keywords and topics are sent to a generative AI engine. This AI engine uses models such as OpenAI's GPT-3 and GPT-4. The AI ​​engine understands the question and generates an appropriate answer. In this case, a specific answer is generated: "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0104] The generated response is formatted on the server. HTML and CSS are used for formatting, converting it into a format that is easy to read on the UI. The formatted data is then sent to the terminal as an HTTP response.

[0105] The terminal parses the response data received from the server. The JavaScript JSON.parse method is used again for this parsing. After parsing, the response is displayed in the user interface. The user can then verify the response displayed on the terminal screen.

[0106] Specific example

[0107] For example, consider a case where a student asks, "What is the equation of a straight line?" The student types the question, "What is the equation of a straight line?", and clicks the submit button. The device converts the question into JSON format and sends it to the server as an HTTP request. The server receives this request and passes the question, "What is the equation of a straight line?", to a text analysis module. The text analysis module extracts keywords such as "straight line" and "equation", and sends the result to a generative AI engine. The generative AI engine generates the answer, "The equation of a straight line is y = mx + b", and returns it to the server. The server formats the generated answer into a format that is easy to read on the UI and sends it to the device as an HTTP response. The device analyzes the HTTP response and displays the answer on the student's screen. The student can then check the answer, "The equation of a straight line is y = mx + b", displayed on the screen and use it to aid in their learning.

[0108] Example of a prompt

[0109] Please answer the following question: How do I solve a quadratic equation?

[0110] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0111] Step 1:

[0112] User: The end user logs into the online learning platform and enters questions or problems they cannot solve. For example, they might type "Please explain how to solve quadratic equations" into the text box on the user interface and click the submit button. The input data is the question in text format.

[0113] Step 2:

[0114] Terminal: The terminal converts the user-entered question content into JSON format. This conversion process uses the JavaScript JSON.stringify method. The converted data is sent to the server as an HTTP POST request. The input is the user's question text, and the output is the question data in JSON format.

[0115] Step 3:

[0116] Server: The server receives HTTP requests and retrieves question data from the request body. This process uses the Node.js express framework. It also stores data containing user information in a database (e.g., MySQL) to identify which user the question is from. The input is question data in JSON format, and the output is the user information and question data stored in the database.

[0117] Step 4:

[0118] Server: The server passes the acquired question data to the text analysis module. The text analysis module uses Google's NLP API to analyze the question content and extract important keywords and topics. The input is the question data passed from the server, and the output is a list of extracted keywords and topics.

[0119] Step 5:

[0120] Server: Based on the extracted keywords and topics, the server sends the analysis results to a generative AI engine. This generative AI engine uses models such as OpenAI's GPT-3 and GPT-4. The AI ​​engine understands the question in the format of a prompt example such as "Please answer the following question: How do you solve a quadratic equation?" and generates an appropriate answer. The input is a list of extracted keywords and topics, and the output is the generated answer text.

[0121] Step 6:

[0122] Server: Formats the generated response into a user-friendly format for the user interface. This formatting process uses HTML and CSS to convert it into a readable format. It sends the formatted data to the terminal as an HTTP response. The input is the generated response text, and the output is formatted data in HTML format.

[0123] Step 7:

[0124] Terminal: Parses the response data received from the server. This parsing process uses the JavaScript JSON.parse method. The response is then displayed in the user interface. Specifically, it displays the response using certain HTML elements (e.g., Insert the answer text into the tag. The input is answer data in JSON format, and the output is the answer text displayed in the user interface.

[0125] Step 8:

[0126] User: Checks the answer displayed on the device screen. The user can learn by looking at answers such as, for example, "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a". The input is the text displayed on the device, and the output is the user's understanding and learning effect.

[0127] (Application Example 1)

[0128] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0129] Conventional learning support systems often struggled to provide quick and accurate answers to user questions. Furthermore, e-commerce sites, in particular, require rapid responses to product inquiries, necessitating a system that enhances the user experience. This invention aims to solve these problems by providing a system that quickly and appropriately analyzes user questions and generates and provides accurate answers using generative artificial intelligence.

[0130] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0131] In this invention, the server includes means for receiving questions from specific users, means for analyzing the received questions and extracting keywords and topics related to the question content, and means for generating appropriate answers using generative artificial intelligence based on the extracted keywords and topics. This makes it possible to provide quick and accurate answers to questions from users. Furthermore, the invention also includes means for converting the received question content into a data format and transmitting it to an information processing device, and means for providing quick and accurate answers to questions about products, which can significantly improve the user experience, especially on e-commerce sites.

[0132] "Specific users" refers to users who access the system and enter questions.

[0133] "Means for receiving questions" refers to the methods or devices used by the system to acquire questions entered by users.

[0134] "Means for extracting keywords and topics related to the content of a question" refers to methods or devices for identifying and extracting important words and themes from a received question.

[0135] "Means of generating appropriate answers using generative artificial intelligence" refers to methods or devices that use generative AI to create answers to users' questions based on extracted keywords and topics.

[0136] "Means of sending generated answers back to specific users" refers to methods or devices for sending generated answers to users.

[0137] "Information processing equipment" refers to computers and servers used for processing data.

[0138] "Means of providing prompt and accurate answers to questions about products" refers to methods or devices that provide prompt and accurate answers to user inquiries about goods or products.

[0139] "Means of converting to data format" refers to methods or devices for converting received questions and answers into a standardized data format such as JSON.

[0140] This invention relates to a learning support system that receives questions from specific users, analyzes them, generates appropriate answers, and sends them back to the users, as well as an answer generation system for e-commerce websites. This system consists of multiple components, including the user's terminal, a server, and generative artificial intelligence (AI).

[0141] 1. User actions

[0142] User: For example, a user of an online shopping site uses a smartphone app to input a product-related question into a chatbot. The user inputs a question through the "User Interface (UI)," such as, "Please tell me how to set up the remote control for this TV."

[0143] 2. Receiving a question

[0144] Terminal: When the user clicks the submit button, the terminal converts the question content into a data format such as JSON, issues an HTTP request, and sends it to the server.

[0145] Server: The server receives this HTTP request and retrieves the question content. It also stores data, including user information, in the database to identify which user the question is from.

[0146] 3. Analysis of the Question

[0147] Server: Next, the server passes the received question to the text analysis module. This module uses natural language processing (NLP) techniques to analyze the question and extract important keywords and topics. For example, the keywords "television" and "remote control settings" might be extracted.

[0148] 4. Generating the answer

[0149] Server: Based on the extracted keywords and topics, the server sends the analysis results to the generative AI engine. The generative AI engine understands the question and generates an appropriate answer. In this case, a specific answer is generated: "To set up the remote control, press and hold the button on the back of the remote control for 3 seconds, then follow the instructions displayed on the TV screen."

[0150] 5. Return your response

[0151] Server: Receives the generated response and formats it for return to the user. For example, it formats it to be easily viewable on the UI, and then sends an HTTP response containing the formatted data to the device.

[0152] Terminal: Analyzes response data received from the server and displays it on the user interface. Users can check the responses displayed on the terminal screen.

[0153] Hardware and software to be used

[0154] Hardware: Smartphone (user device), Server (API server)

[0155] Software: Flask (Python web framework), NLP module, generative AI engine

[0156] Data processing and data calculation

[0157] NLP Module: Analyzes the question content using natural language processing techniques and extracts important keywords. Specifically, it uses libraries such as spaCy and NLTK.

[0158] Generative AI Engine: Based on extracted keywords, it utilizes generative AI models such as GPT-3 to generate answers. This ensures that answers are provided quickly and accurately.

[0159] Specific example

[0160] For example, if a user of an online shopping site asks, "What is the capacity of this refrigerator?", the answer will be generated using the following steps.

[0161] User: Type "What is the capacity of this refrigerator?" into the chatbot on their smartphone and send it.

[0162] Terminal: Sends the question to the server.

[0163] Server: Receives an HTTP request and extracts the keywords "refrigerator" and "capacity".

[0164] Generative AI engine: Generates the answer "This refrigerator has a capacity of 300 liters."

[0165] Server: Send response.

[0166] Terminal: Display the answer.

[0167] Example of a prompt

[0168] "Please tell me how to set up the remote control for this TV."

[0169] "What's the capacity of this refrigerator?"

[0170] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0171] Step 1: Enter user questions

[0172] Subject: User

[0173] Specific operation: The user uses a smartphone app to type a question into the chatbot and clicks the send button.

[0174] Input: Example question entered by the user: "How do I set up the remote control for this TV?"

[0175] Output: Data processed internally by the device based on the questions entered by the user.

[0176] Step 2: Format and send the question data

[0177] Subject: terminal

[0178] Specific operation: The terminal converts the question entered by the user into JSON format and sends it to the server as an HTTP request.

[0179] Input: Question data entered by the user

[0180] Data processing: Encode the question data into JSON format.

[0181] Output: Question data in HTTP request format

[0182] Step 3: Receiving and analyzing questions

[0183] Subject: Server

[0184] Specific operation: The server receives an HTTP request and retrieves the question content. Then, it passes the question content to a text analysis module, which uses natural language processing (NLP) techniques to extract important keywords and topics.

[0185] Input: Question data in JSON format

[0186] Data processing: Extract question content as text data and perform NLP analysis.

[0187] Output: Keywords such as "TV" and "remote control settings"

[0188] Step 4: Generating the answer

[0189] Subject: Server

[0190] Specific operation: The server sends the extracted keywords to a generative AI engine, which then generates an appropriate response.

[0191] Input: Extracted keywords

[0192] Data processing: Answer generation using generative AI models (e.g., GPT-3)

[0193] Output: Response data: "To set up the remote control, press and hold the button on the back of the remote control for 3 seconds, then follow the instructions displayed on the TV screen."

[0194] Step 5: Format and return the response data.

[0195] Subject: Server

[0196] Specific operation: The server receives the generated response, formats it into a format easily displayed in the user interface, and then sends it to the terminal as an HTTP response.

[0197] Input: Generated response data

[0198] Data processing: Formatting of response data (converting to a format suitable for the UI)

[0199] Output: HTTP response containing formatted response data

[0200] Step 6: Display the answer

[0201] Subject: terminal

[0202] Specific operation: The terminal analyzes the response data received from the server and displays it on the user interface. The user can then check the response displayed on the terminal screen.

[0203] Input: HTTP response containing formatted response data

[0204] Data processing: Decoding response data and converting it for UI display.

[0205] Output: The answer displayed on the user's smartphone screen.

[0206] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0207] This invention relates to a learning support system that receives questions from a specific user, analyzes them, generates appropriate answers, and sends them back to the user. This system consists of multiple components, including the user's terminal, a server, generative artificial intelligence (AI), and an emotion engine. The system can also recognize the user's emotions and provide appropriate responses based on those emotions.

[0208] 1. User actions

[0209] User: The user logs into the learning platform. The user asks questions about things they are unsure of or problems they cannot solve during their studies through the "User Interface (UI)". For example, they might type a question like, "Please explain how to solve quadratic equations."

[0210] 2. Receiving questions and recognizing emotions

[0211] Terminal: When the user clicks the send button, the terminal retrieves the question and uses its built-in camera and microphone to analyze the user's emotions from their facial expressions and voice. Facial analysis determines, for example, whether the user is troubled or nervous.

[0212] 3. Sending questions and sentiment data

[0213] Terminal: Converts the question content into an appropriate data format such as JSON, and sends it to the server by issuing an HTTP request along with sentiment data.

[0214] Server: The server receives HTTP requests and stores question data, sentiment data, and user information in the database. It then passes the received question content to the text analysis module.

[0215] 4. Analysis of the Questions

[0216] Text Analysis Module: This module analyzes the question text using natural language processing (NLP) techniques to extract important keywords and topics. For example, it might extract keywords such as "quadratic equation" and "solution method."

[0217] 5. Generating the answer

[0218] Server: Based on extracted keywords and topics, it sends analysis results and sentiment data to the generative AI engine.

[0219] Generative AI Engine: The generative AI engine generates appropriate answers from the analysis results it receives. It also adjusts the answers based on sentiment data. For example, if the user is confused, it will generate a helpful answer such as, "Please calm down, I will explain slowly. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0220] 6. Return your response

[0221] Server: Receives the generated response and formats it for return to the user. For example, it formats it into a format that is easy to read on the UI and sends an HTTP response containing the formatted data to the device.

[0222] 7. Display the answer

[0223] Terminal: Analyzes response data received from the server and displays it on the user interface. Users can check the responses displayed on the terminal screen.

[0224] Specific example

[0225] For example, consider a case where a student asks, "Could you please explain the equation of a straight line?" while simultaneously looking troubled.

[0226] User: A student types and submits the question, "Please tell me the equation of a straight line."

[0227] Terminal: It retrieves the question content and analyzes the student's facial expressions using the built-in camera. If the student appears to be struggling, the analyzed data is retrieved. An HTTP request containing this data is sent to the server.

[0228] Server: Receives HTTP requests, stores question and sentiment data, and passes it to the text analysis module.

[0229] Text analysis module: Extracts the keywords "straight line" and "equation" and sends the results to a generative AI engine.

[0230] Generative AI engine: Based on the question and sentiment data, it generates a helpful answer such as, "Don't worry, the equation of a straight line is y = mx + b."

[0231] Server: Formats the generated response and sends it to the terminal as an HTTP response.

[0232] Terminal: Analyzes the HTTP response and displays the answer on the student's screen.

[0233] User: You can check the answer displayed on the screen, "Don't worry. The equation of a straight line is y = mx + b," and use that information to further your learning.

[0234] This invention enables a swift and appropriate process from the time a user posts a question until they receive an answer, thereby effectively providing personalized learning support in an emotionally resonant manner.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] User: Log in to the learning platform's user interface. The user enters the question "Please tell me the quadratic formula" in the question submission field and clicks the submit button.

[0238] Step 2:

[0239] Terminal: It acquires the input content of questions and collects emotional data from the user's facial expressions and voice using the built-in camera and microphone. For example, it determines whether the user is confused or nervous through image analysis and voice emotion analysis.

[0240] Step 3:

[0241] Terminal: Converts the question content and sentiment data into JSON format and sends it to the server as an HTTP request.

[0242] Step 4:

[0243] Server: Receives HTTP requests and stores question data, sentiment data, and user information in the database. Passes the received question content to the text analysis module.

[0244] Step 5:

[0245] Text Analysis Module: Analyzes question text using natural language processing (NLP) techniques to extract important keywords and topics. For example, it extracts keywords such as "quadratic equation" and "quadratic formula."

[0246] Step 6:

[0247] Server: Based on extracted keywords, topics, and sentiment data, it sends the analysis results to a generative AI engine.

[0248] Step 7:

[0249] Generative AI Engine: Generates appropriate responses from the input analysis results and sentiment data. For example, if it determines that the user is confused, it will supplement the response with kind words. It might generate a response like, "Please calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0250] Step 8:

[0251] Server: Receives the generated response and formats it for return to the user. For example, it formats it into a format that is easy to read on the UI and sends an HTTP response containing the formatted data to the device.

[0252] Step 9:

[0253] Terminal: Analyzes response data received from the server and displays it on the user interface.

[0254] Step 10:

[0255] User: Checks the answer displayed on the device screen. Seeing the answer, "Calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a," the user's question is resolved and they can continue learning.

[0256] The above outlines the specific processing steps from questioning to answer presentation in an online learning support system that incorporates an emotion engine.

[0257] (Example 2)

[0258] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0259] Conventional learning support systems often suffered from inconsistent accuracy in answering user-submitted questions and difficulty in considering user emotions. As a result, users may not receive adequate support when they are struggling or stressed, potentially leading to decreased learning effectiveness. Furthermore, providing quick and accurate answers to complex problems was challenging.

[0260] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a question from a specific user, means for analyzing the content of the received question and extracting keywords and topics related to the question, means for generating an appropriate answer using natural language processing technology based on the extracted keywords and topics, means for analyzing the user's facial expressions and voice to acquire emotional data, and means for returning the generated answer to the specific user. This makes it possible to provide a quick and accurate answer in a way that is sensitive to the user's emotions.

[0261] "Specific user" refers to an individual user who can be uniquely identified.

[0262] "Analyzing the question content" refers to the process of breaking down the question received from the user and understanding its meaning.

[0263] "Keyword and topic extraction" refers to identifying important words, phrases, and themes from a question.

[0264] "Natural language processing technology" is the technology that enables computers to understand, analyze, and generate human language.

[0265] "Generating appropriate answers" means creating accurate and useful answers to user questions based on extracted keywords and topics.

[0266] "Analyzing user facial expressions and voice to acquire emotional data" means using cameras and microphones to perform feature analysis on the user's facial expressions and voice to identify their emotional state.

[0267] "Returning the answer to a specific user" means sending the generated answer to the relevant user.

[0268] "Adjusting responses based on emotional data" means optimizing the content and tone of responses according to the analyzed emotional state.

[0269] "Formatting answers" means formatting answers into a user-friendly format, making them visually easy to read.

[0270] This invention relates to a learning support system that receives questions from a specific user, analyzes them, generates appropriate answers, and sends them back to the user. This system consists of multiple components, including the user's terminal, a server, generative artificial intelligence (AI), and an emotion engine. The system can also recognize the user's emotions and provide appropriate responses based on those emotions.

[0271] The following describes specific embodiments for carrying out the invention.

[0272] Hardware and software configuration

[0273] 1. User's device:

[0274] Built-in camera, microphone, user interface (UI)

[0275] Software: OpenCV (for facial expression analysis), TENSORFLOW (registered trademark) (for voice analysis)

[0276] 2. Server:

[0277] Database (for storing question data and sentiment data)

[0278] Text analysis module (using natural language processing technology)

[0279] Generative AI Engine (using a generative AI model)

[0280] System Operation

[0281] The detailed operation of this system is as follows.

[0282] User Operation

[0283] User: The user logs in to the learning platform and enters a question through the "User Interface (UI)". For example, enter "Please teach me how to solve quadratic equations" and click the send button. By this operation, the user's expression and voice are recorded simultaneously with the question content.

[0284] Question Reception and Emotion Recognition

[0285] Terminal: The terminal detects the click of the send button and obtains the question content. Also, using the built-in camera and microphone, it analyzes the user's expression and voice in real time. For example, it uses OpenCV to recognize the user's expression and extracts emotion data from the voice using TensorFlow.

[0286] Question and Emotion Data Transmission

[0287] Terminal: The terminal converts the question content into JSON format and sends it to the server as an HTTP request in a form that includes emotion data. At this time, the question content, expression data, and voice analysis results are combined into one JSON object.

[0288] Question Analysis

[0289] Server: The server receives the HTTP request, saves the question data, emotion data, and user information in the database. Next, it analyzes the question content using a text analysis module and extracts important keywords and topics. For example, it extracts keywords such as "quadratic equation" and "solution method".

[0290] Answer Generation

[0291] Server: Sends extracted keywords, topics, and sentiment data to a generative AI engine.

[0292] Generative AI Engine: The generative AI engine generates appropriate responses based on analysis results and sentiment data. Taking sentiment data into consideration, for example, if the user is distressed, it will generate a helpful response such as, "Please calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0293] Send back your response

[0294] Server: Receives the generated response and formats it into a format that is easy to read on the UI. Sends the HTTP response containing the formatted data to the terminal.

[0295] Display the answer

[0296] Terminal: The terminal analyzes the response data received from the server and displays it on the user interface. Users can review the responses displayed on the terminal screen and use them for learning.

[0297] Specific example

[0298] For example, consider a case where a student asks, "Could you please explain the equation of a straight line?" while simultaneously looking troubled.

[0299] Example of a prompt:

[0300] "Please tell me the equation of a straight line."

[0301] In this case, the system will operate as follows:

[0302] User: A student types and submits the question, "Please tell me the equation of a straight line."

[0303] Terminal: Obtain the question content and analyze the expression of the student in distress using the built-in camera. Send an HTTP request to the server in a form that includes the data.

[0304] Server: Receive the HTTP request, save the question and emotion data, and let the text analysis module analyze keywords such as "straight line" and "equation".

[0305] Generative AI engine: Based on the question content and emotion data, generate a kind answer like "Please don't worry. The equation of a straight line is y = mx + b."

[0306] Server: Format the generated answer and send it to the terminal as an HTTP response.

[0307] Terminal: Analyze the HTTP response and display the answer on the student's screen.

[0308] User: The user can confirm the answer "Please don't worry. The equation of a straight line is y = mx + b." displayed on the screen and apply it to learning.

[0309] Thus, by using the present invention, it is possible for the user to receive a quick and accurate answer in a form that empathizes with emotions, and the learning effect can be enhanced. [[ID=2​​​​​​​​​​​​​​​​​Output: Login session and question content.

[0315] Step 2:

[0316] Terminal: The terminal detects the click of the send button and retrieves the question content. Simultaneously, it records the user's facial expressions and voice using the built-in camera and microphone, and performs facial expression analysis and sentiment analysis. For example, it uses OpenCV to analyze the user's facial expressions in real time and TensorFlow to extract sentiment data from the audio.

[0317] Input: Question content and user's facial expressions and voice data.

[0318] Output: Question content and emotional data (confusion, anxiety, etc.).

[0319] Step 3:

[0320] Terminal: Converts the question content and emotion data into JSON format and sends it to the server via an HTTP request. At this time, the question content, facial expression data, and voice analysis results are combined into a single JSON object.

[0321] Input: Question content and sentiment data.

[0322] Output: Data in JSON format.

[0323] Step 4:

[0324] Server: The server receives HTTP requests and stores question data, sentiment data, and user information in the database.

[0325] Input: Data in JSON format.

[0326] Output: Question data and sentiment data stored in the database.

[0327] Step 5:

[0328] Text Analysis Module: The text analysis module analyzes the received question content and extracts important keywords and topics. For example, it might extract the keywords "quadratic equation" and "solution method" from the question text.

[0329] Input: Question content.

[0330] Output: Extracted keywords and topics.

[0331] Step 6:

[0332] Server: Sends extracted keywords, topics, and sentiment data to a generative AI engine.

[0333] Input: Extracted keywords, topics, and sentiment data.

[0334] Output: API call to a generative AI engine.

[0335] Step 7:

[0336] Generative AI Engine: The generative AI engine generates appropriate responses based on the received analysis results and sentiment data. For example, if the sentiment data indicates "confused," it will generate a helpful response such as, "Please calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0337] Input: Keywords, topics, sentiment data.

[0338] Output: The generated answer.

[0339] Step 8:

[0340] Server: Receives the generated response and formats it for return to the user. It uses HTML and CSS to format it into a user-friendly format and sends the HTTP response containing the formatted data to the device.

[0341] Input: Generated answer.

[0342] Output: HTTP response containing formatted response data.

[0343] Step 9:

[0344] Terminal: The terminal analyzes the response data received from the server and displays it on the user interface. Users can review the responses displayed on the terminal screen and use them for learning.

[0345] Input: HTTP response from the server.

[0346] Output: The answer displayed in the user interface.

[0347] (Application Example 2)

[0348] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0349] In today's educational environment, there is a need to quickly and accurately resolve questions and anxieties that arise as learners progress at their own pace. Especially with the increase in online learning, immediate question support and emotionally resonant support are crucial. However, conventional systems only provide standardized answers to user questions, failing to consider the user's emotional state. As a result, learning effectiveness has been reduced when learners experience anxiety or confusion.

[0350] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for receiving questions from a specific user, means for analyzing the received questions and user emotion data extracted from the built-in camera and voice input to extract keywords and topics related to the question content, means for generating an appropriate response according to the emotion using generative artificial intelligence based on the extracted keywords and topics and the user emotion data, and means for returning the generated response to the specific user. This enables rapid and accurate learning support that takes into account the user's emotional state.

[0351] "Means for receiving questions" refers to a device or software module for acquiring user inquiries and questions as digital data.

[0352] A "camera" is an optical device used to record a user's facial expressions and movements as video.

[0353] "Voice input" refers to a device or software module that recognizes and analyzes sounds emitted by a user as digital signals.

[0354] "Emotional data" refers to digital data that indicates the emotional state of a user, analyzed from information such as their facial expressions and voice.

[0355] "Means of analysis" refers to a device or software module used to perform analysis based on acquired data and extract meaningful information.

[0356] "Means for extracting keywords and topics" refers to devices or software modules that use natural language processing (NLP) techniques to find important words or themes from the content of a question.

[0357] "Generative artificial intelligence" refers to algorithms or models that automatically generate natural language responses or suggestions based on received data.

[0358] An "emotionally appropriate response" is a response that takes the user's emotional state into consideration and is generated in a kind and comforting tone and content.

[0359] "Means for returning the generated response" refers to a communication method or software module for delivering the generated response to the user.

[0360] "Means for formatting into a format suitable for a display device" refers to a device or software module that converts the format of the response sent back to the user so that it can be displayed clearly on the user's device.

[0361] This invention relates to a learning support system using a smart device that can instantly respond to user questions and generate and return answers tailored to the user's emotions. This system consists of multiple components, including the user's terminal, a server, generative artificial intelligence (AI), and an emotion engine.

[0362] 1. User actions

[0363] The user uses smart glasses, which have a built-in camera and microphone. The user inputs questions by voice. For example, if the user asks, "Please tell me how to solve a quadratic equation," the voice is captured by the smart glasses' microphone. The camera simultaneously analyzes the user's facial expressions and acquires emotional data.

[0364] 2. Questions and acquisition of sentiment data

[0365] The questions are converted to text using speech recognition technology (e.g., Google Cloud Speech-to-Text API). The camera uses facial recognition technology (e.g., OpenFace) to analyze the user's facial expressions. This allows for the acquisition of emotional data, such as whether the user is distressed or nervous.

[0366] 3. Sending questions and sentiment data

[0367] The terminal sends the text questions and sentiment data, converted from speech, to the server in an appropriate data format such as JSON. The server receives this and stores it in a database. Next, a text analysis module analyzes the questions using natural language processing (NLP) techniques and extracts important keywords and topics (e.g., "quadratic equation," "how to solve").

[0368] 4. Generating the answer

[0369] The server sends analysis results and sentiment data to a generative AI engine (e.g., OpenAI GPT-4) based on extracted keywords and topics. The generative AI engine generates appropriate responses based on the analysis results and further adjusts the tone and content of the responses based on the sentiment data. For example, if the user is in distress, a helpful response such as "Please calm down, I will explain slowly. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a" is generated.

[0370] 5. Return and display of responses

[0371] The generated response is sent back to the device from the server. The device formats the received response into an appropriate format and displays it on the smart glasses' display. This allows the user to see the emotionally resonant response in real time.

[0372] Specific example

[0373] For example, consider a situation where a student is solving a math problem and asks, "Can you teach me how to solve a quadratic equation?" while simultaneously looking troubled.

[0374] Example of a prompt message (input to the generation AI engine):

[0375] User's question: "How do I solve a quadratic equation?"

[0376] User's emotion: "Confused"

[0377] Question: "Explanation of how to solve quadratic equations"

[0378] Emotion-based response: "Explain in a kind and reassuring tone."

[0379] The following is an example of a response from a generative AI engine.

[0380] "Please stay calm. I will explain how to solve quadratic equations. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0381] This invention allows users to instantly receive responses appropriate to their emotional state, thereby effectively supporting their learning.

[0382] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0383] Step 1:

[0384] The user inputs a question using smart glasses. The user asks a question by voice, such as "Tell me how to solve a quadratic equation." This voice is captured by the microphone in the smart glasses. At the same time, the camera records the user's facial expressions and acquires the video data. Thus, audio data and video data are obtained as input.

[0385] Step 2:

[0386] The device uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to convert captured audio into text. This process generates text data such as "Tell me how to solve a quadratic equation." Additionally, facial recognition technology (e.g., OpenFace) is used to analyze the user's facial expressions from the camera footage, and emotion data (e.g., confused) is extracted.

[0387] Step 3:

[0388] The terminal converts text data and sentiment data into an appropriate data format such as JSON and sends it to the server as an HTTP request. The input consists of question text and sentiment data, and the output is JSON data. The server receives this and stores it in its database.

[0389] Step 4:

[0390] On the server, a text analysis module extracts question content from JSON-formatted data and analyzes it using natural language processing (NLP) techniques. For example, keywords such as "quadratic equation" and "solution method" might be extracted. The input for this process is question data and sentiment data, and the output is the analyzed keywords and topics.

[0391] Step 5:

[0392] The server sends the analysis results and sentiment data to a generative AI engine (e.g., OpenAI GPT-4) based on the analyzed keywords and topics. The generative AI engine receives this data, constructs a prompt, and generates an appropriate response. The input consists of keywords, topics, and sentiment data, and the output is the generated response. For example, a response like "Calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a" might be generated.

[0393] Step 6:

[0394] The server receives the generated response and sends it back to the terminal. The terminal then formats the received response into a format suitable for the user's smart glasses. After the output data has been properly formatted, it is displayed on the user's screen.

[0395] Step 7:

[0396] The device displays the final answer on the smart glasses' display. This allows the user to see an emotionally resonant response in real time. The output is the displayed answer, where the user can confirm the information.

[0397] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0398] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0399] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0400] [Second Embodiment]

[0401] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0402] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0403] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0404] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0405] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0406] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0407] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0408] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0409] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0410] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0411] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0412] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0413] This invention relates to a learning support system that receives questions from a specific user, analyzes them, generates appropriate answers, and returns them to the user. This system consists of multiple components, including the user's terminal, a server, and generative artificial intelligence (AI).

[0414] 1. User actions

[0415] User: For example, a high school student logs into an online learning platform at home. The user asks questions about things they are unsure of or problems they cannot solve during their studies through the "User Interface (UI)". For example, they might type a question like, "Please explain how to solve quadratic equations."

[0416] 2. Receiving a question

[0417] Terminal: When the user clicks the submit button, the terminal converts the question content into a data format such as JSON, issues an HTTP request, and sends it to the server.

[0418] Server: The server receives this HTTP request and retrieves the question content. It also stores data containing user information in the database to identify which user the question is from.

[0419] 3. Analysis of the Question

[0420] Server: Next, the server passes the received question to the text analysis module. This module uses natural language processing (NLP) techniques to analyze the question and extract important keywords and topics. For example, the keywords "quadratic equation" and "solution method" might be extracted.

[0421] 4. Generating the answer

[0422] Server: Based on the extracted keywords and topics, the server sends the analysis results to a generative AI engine. The generative AI engine understands the question and generates an appropriate answer. In this case, a specific answer is generated: "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0423] 5. Return your response

[0424] Server: Receives the generated response and formats it for return to the user. For example, it formats it to be easily viewable on the UI, and then sends an HTTP response containing the formatted data to the device.

[0425] Terminal: Analyzes response data received from the server and displays it on the user interface. Users can check the responses displayed on the terminal screen.

[0426] Specific example

[0427] For example, consider the case where a student asks, "Could you tell me the equation of a straight line?"

[0428] 1. User: A student types and submits the question, "Please tell me the equation of a straight line."

[0429] 2. Terminal: Converts the question content into JSON format and sends an HTTP request to the server.

[0430] 3. Server: Receives an HTTP request and passes the question "equation of a straight line" to the text analysis module.

[0431] 4. Text analysis module: Extracts keywords such as "straight line" and "equation," and sends the results to a generative AI engine.

[0432] 5. Generative AI engine: Generates the answer "The equation of a straight line is y = mx + b" and returns it to the server.

[0433] 6. Server: Formats the generated response into a user-friendly format for the UI and sends it to the device as an HTTP response.

[0434] 7. Terminal: Analyzes the HTTP response and displays the answer on the student's screen.

[0435] 8. User: You can check the answer displayed on the screen, "The equation of a straight line is y = mx + b," and use it to further your learning.

[0436] This invention enables effective personalized learning support because the process from when a user posts a question to when they receive an answer is carried out quickly and appropriately.

[0437] The following describes the processing flow.

[0438] Step 1:

[0439] User: After logging into the learning platform's user interface, the user enters their question in the question submission field and clicks the submit button.

[0440] Step 2:

[0441] Terminal: Retrieves the input question content and converts it to an appropriate data format such as JSON. Then, it sends this data to the server as an HTTP request.

[0442] Step 3:

[0443] Server: Receives HTTP requests and saves question data and user information to the database. Then, passes the received question content to the text analysis module.

[0444] Step 4:

[0445] Text Analysis Module: Analyzes question text using natural language processing (NLP) techniques to extract important keywords and topics. For example, it can extract keywords such as "quadratic equation" and "solution method."

[0446] Step 5:

[0447] Server: Based on extracted keywords and topics, it sends the analysis results to the generative AI engine.

[0448] Step 6:

[0449] Generative AI Engine: The generative AI engine generates appropriate answers from the analysis results it receives. For example, it can generate a specific answer such as, "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0450] Step 7:

[0451] Server: Receives the generated response and formats it for return to the user. For example, it formats it to be easily viewable on the UI and sends the HTTP response containing the formatted data to the device.

[0452] Step 8:

[0453] Terminal: Analyzes response data received from the server and displays it on the user interface.

[0454] Step 9:

[0455] User: Check the answer displayed on the device screen. If the question is resolved, continue learning. If you have further questions, return to step 1 and enter a new question.

[0456] The above outlines the specific processing steps involved in providing answers to user questions on an online learning platform.

[0457] (Example 1)

[0458] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0459] Conventional learning support systems suffered from problems such as delays in responding to user questions and the inability to obtain appropriate answers. This resulted in reduced learning efficiency and difficulty for users to quickly obtain the necessary information.

[0460] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0461] In this invention, the server includes means for receiving questions from specific end users, means for converting the received questions into a data format and sending them to the server, means for the server to store the received question data in a database and pass it to a text analysis module, means for the text analysis module to analyze the question content using natural language processing technology and extract important keywords and topics, means for generating appropriate answers using generative artificial intelligence based on the extracted keywords and topics, and means for formatting the generated answers into a user-friendly format on the user interface and returning them to the end user. This makes it possible for users to quickly and accurately resolve questions and problems they cannot solve during learning, thereby improving learning efficiency.

[0462] An "end user" refers to a user who uses the system to ask questions.

[0463] A "server" refers to a central system that receives and processes requests from clients.

[0464] "Data format" refers to a format in which data is encoded according to certain rules. For example, the JSON format is one such format.

[0465] A "database" refers to a system that systematically stores data and allows it to be retrieved as needed.

[0466] A "text analysis module" refers to a program that uses natural language processing technology to analyze text data and extract its meaning.

[0467] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human language.

[0468] "Keywords" refer to important words or phrases extracted from the question.

[0469] "Topic" refers to the subject or theme of the question.

[0470] "Generative artificial intelligence" refers to artificial intelligence that has the function of generating appropriate answers to user questions.

[0471] "User interface" refers to the screens and input methods that end users use to interact with a system.

[0472] "Formatting" refers to the process of converting data into a format that is easy to read. This includes, for example, using HTML and CSS.

[0473] Modes for carrying out the invention

[0474] This invention relates to a learning support system that receives questions from specific end users, analyzes them, generates appropriate answers, and returns them to the user. This system consists of multiple components, including the end user's terminal, a server, and generative artificial intelligence (AI).

[0475] End users log in to the online learning platform from their homes, schools, or other locations. Users input questions and problems they encounter during their studies through the user interface (UI). For example, they might input a question like, "Please explain how to solve quadratic equations."

[0476] The terminal converts the user's input into a data format (such as JSON) and sends it to the server as an HTTP request. The JavaScript JSON.stringify method is used for this conversion. The request is sent using the HTTP POST method.

[0477] The server receives an HTTP request and retrieves the question data from the request body. The Node.js express framework is used for this process. The server then stores user information in a database (e.g., MySQL) to identify which user the question originated from. The database records the question content and associated user information.

[0478] Next, the server passes the acquired question data to the text analysis module. The text analysis module uses natural language processing (NLP) techniques to analyze the question content and extract important keywords and topics. Specifically, it uses Google's NLP API to extract keywords such as "quadratic equation" and "how to solve it."

[0479] The extracted keywords and topics are sent to a generative AI engine. This AI engine uses models such as OpenAI's GPT-3 and GPT-4. The AI ​​engine understands the question and generates an appropriate answer. In this case, a specific answer is generated: "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0480] The generated response is formatted on the server. HTML and CSS are used for formatting, converting it into a format that is easy to read on the UI. The formatted data is then sent to the terminal as an HTTP response.

[0481] The terminal parses the response data received from the server. The JavaScript JSON.parse method is used again for this parsing. After parsing, the response is displayed in the user interface. The user can then verify the response displayed on the terminal screen.

[0482] Specific example

[0483] For example, consider a case where a student asks, "What is the equation of a straight line?" The student types the question, "What is the equation of a straight line?", and clicks the submit button. The device converts the question into JSON format and sends it to the server as an HTTP request. The server receives this request and passes the question, "What is the equation of a straight line?", to a text analysis module. The text analysis module extracts keywords such as "straight line" and "equation", and sends the result to a generative AI engine. The generative AI engine generates the answer, "The equation of a straight line is y = mx + b", and returns it to the server. The server formats the generated answer into a format that is easy to read on the UI and sends it to the device as an HTTP response. The device analyzes the HTTP response and displays the answer on the student's screen. The student can then check the answer, "The equation of a straight line is y = mx + b", displayed on the screen and use it to aid in their learning.

[0484] Example of a prompt

[0485] Please answer the following question: How do I solve a quadratic equation?

[0486] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0487] Step 1:

[0488] User: The end user logs into the online learning platform and enters questions or problems they cannot solve. For example, they might type "Please explain how to solve quadratic equations" into the text box on the user interface and click the submit button. The input data is the question in text format.

[0489] Step 2:

[0490] Terminal: The terminal converts the user-entered question content into JSON format. This conversion process uses the JavaScript JSON.stringify method. The converted data is sent to the server as an HTTP POST request. The input is the user's question text, and the output is the question data in JSON format.

[0491] Step 3:

[0492] Server: The server receives HTTP requests and retrieves question data from the request body. This process uses the Node.js express framework. It also stores data containing user information in a database (e.g., MySQL) to identify which user the question is from. The input is question data in JSON format, and the output is the user information and question data stored in the database.

[0493] Step 4:

[0494] Server: The server passes the acquired question data to the text analysis module. The text analysis module uses Google's NLP API to analyze the question content and extract important keywords and topics. The input is the question data passed from the server, and the output is a list of extracted keywords and topics.

[0495] Step 5:

[0496] Server: Based on the extracted keywords and topics, the server sends the analysis results to a generative AI engine. This generative AI engine uses models such as OpenAI's GPT-3 and GPT-4. The AI ​​engine understands the question in the format of a prompt example such as "Please answer the following question: How do you solve a quadratic equation?" and generates an appropriate answer. The input is a list of extracted keywords and topics, and the output is the generated answer text.

[0497] Step 6:

[0498] Server: Formats the generated response into a user-friendly format for the user interface. This formatting process uses HTML and CSS to convert it into a readable format. It sends the formatted data to the terminal as an HTTP response. The input is the generated response text, and the output is formatted data in HTML format.

[0499] Step 7:

[0500] Terminal: Parses the response data received from the server. This parsing process uses the JavaScript JSON.parse method. The response is then displayed in the user interface. Specifically, it displays the response using certain HTML elements (e.g., Insert the answer text into the tag. The input is answer data in JSON format, and the output is the answer text displayed in the user interface.

[0501] Step 8:

[0502] User: Checks the answer displayed on the device screen. The user can learn by looking at answers such as, for example, "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a". The input is the text displayed on the device, and the output is the user's understanding and learning effect.

[0503] (Application Example 1)

[0504] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0505] Conventional learning support systems often struggled to provide quick and accurate answers to user questions. Furthermore, e-commerce sites, in particular, require rapid responses to product inquiries, necessitating a system that enhances the user experience. This invention aims to solve these problems by providing a system that quickly and appropriately analyzes user questions and generates and provides accurate answers using generative artificial intelligence.

[0506] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0507] In this invention, the server includes means for receiving questions from specific users, means for analyzing the received questions and extracting keywords and topics related to the question content, and means for generating appropriate answers using generative artificial intelligence based on the extracted keywords and topics. This makes it possible to provide quick and accurate answers to questions from users. Furthermore, the invention also includes means for converting the received question content into a data format and transmitting it to an information processing device, and means for providing quick and accurate answers to questions about products, which can significantly improve the user experience, especially on e-commerce sites.

[0508] "Specific users" refers to users who access the system and enter questions.

[0509] "Means for receiving questions" refers to the methods or devices used by the system to acquire questions entered by users.

[0510] "Means for extracting keywords and topics related to the content of a question" refers to methods or devices for identifying and extracting important words and themes from a received question.

[0511] "Means of generating appropriate answers using generative artificial intelligence" refers to methods or devices that use generative AI to create answers to users' questions based on extracted keywords and topics.

[0512] "Means of sending generated answers back to specific users" refers to methods or devices for sending generated answers to users.

[0513] "Information processing equipment" refers to computers and servers used for processing data.

[0514] "Means of providing prompt and accurate answers to questions about products" refers to methods or devices that provide prompt and accurate answers to user inquiries about goods or products.

[0515] "Means of converting to data format" refers to methods or devices for converting received questions and answers into a standardized data format such as JSON.

[0516] This invention relates to a learning support system that receives questions from specific users, analyzes them, generates appropriate answers, and sends them back to the users, as well as an answer generation system for e-commerce websites. This system consists of multiple components, including the user's terminal, a server, and generative artificial intelligence (AI).

[0517] 1. User actions

[0518] User: For example, a user of an online shopping site uses a smartphone app to input a product-related question into a chatbot. The user inputs a question through the "User Interface (UI)," such as, "Please tell me how to set up the remote control for this TV."

[0519] 2. Receiving a question

[0520] Terminal: When the user clicks the submit button, the terminal converts the question content into a data format such as JSON, issues an HTTP request, and sends it to the server.

[0521] Server: The server receives this HTTP request and retrieves the question content. It also stores data, including user information, in the database to identify which user the question is from.

[0522] 3. Analysis of the Question

[0523] Server: Next, the server passes the received question to the text analysis module. This module uses natural language processing (NLP) techniques to analyze the question and extract important keywords and topics. For example, the keywords "television" and "remote control settings" might be extracted.

[0524] 4. Generating the answer

[0525] Server: Based on the extracted keywords and topics, the server sends the analysis results to the generative AI engine. The generative AI engine understands the question and generates an appropriate answer. In this case, a specific answer is generated: "To set up the remote control, press and hold the button on the back of the remote control for 3 seconds, then follow the instructions displayed on the TV screen."

[0526] 5. Return your response

[0527] Server: Receives the generated response and formats it for return to the user. For example, it formats it to be easily viewable on the UI, and then sends an HTTP response containing the formatted data to the device.

[0528] Terminal: Analyzes response data received from the server and displays it on the user interface. Users can check the responses displayed on the terminal screen.

[0529] Hardware and software to be used

[0530] Hardware: Smartphone (user device), Server (API server)

[0531] Software: Flask (Python web framework), NLP module, generative AI engine

[0532] Data processing and data calculation

[0533] NLP Module: Analyzes the question content using natural language processing techniques and extracts important keywords. Specifically, it uses libraries such as spaCy and NLTK.

[0534] Generative AI Engine: Based on extracted keywords, it utilizes generative AI models such as GPT-3 to generate answers. This ensures that answers are provided quickly and accurately.

[0535] Specific example

[0536] For example, if a user of an online shopping site asks, "What is the capacity of this refrigerator?", the answer will be generated using the following steps.

[0537] User: Type "What is the capacity of this refrigerator?" into the chatbot on their smartphone and send it.

[0538] Terminal: Sends the question to the server.

[0539] Server: Receives an HTTP request and extracts the keywords "refrigerator" and "capacity".

[0540] Generative AI engine: Generates the answer "This refrigerator has a capacity of 300 liters."

[0541] Server: Send response.

[0542] Terminal: Display the answer.

[0543] Example of a prompt

[0544] "Please tell me how to set up the remote control for this TV."

[0545] "What's the capacity of this refrigerator?"

[0546] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0547] Step 1: Enter user questions

[0548] Subject: User

[0549] Specific operation: The user uses a smartphone app to type a question into the chatbot and clicks the send button.

[0550] Input: Example question entered by the user: "How do I set up the remote control for this TV?"

[0551] Output: Data processed internally by the device based on the questions entered by the user.

[0552] Step 2: Format and send the question data

[0553] Subject: terminal

[0554] Specific operation: The terminal converts the question entered by the user into JSON format and sends it to the server as an HTTP request.

[0555] Input: Question data entered by the user

[0556] Data processing: Encode the question data into JSON format.

[0557] Output: Question data in HTTP request format

[0558] Step 3: Receiving and analyzing questions

[0559] Subject: Server

[0560] Specific operation: The server receives an HTTP request and retrieves the question content. Then, it passes the question content to a text analysis module, which uses natural language processing (NLP) techniques to extract important keywords and topics.

[0561] Input: Question data in JSON format

[0562] Data processing: Extract question content as text data and perform NLP analysis.

[0563] Output: Keywords such as "TV" and "remote control settings"

[0564] Step 4: Generating the answer

[0565] Subject: Server

[0566] Specific operation: The server sends the extracted keywords to a generative AI engine, which then generates an appropriate response.

[0567] Input: Extracted keywords

[0568] Data processing: Answer generation using generative AI models (e.g., GPT-3)

[0569] Output: Response data: "To set up the remote control, press and hold the button on the back of the remote control for 3 seconds, then follow the instructions displayed on the TV screen."

[0570] Step 5: Format and return the response data.

[0571] Subject: Server

[0572] Specific operation: The server receives the generated response, formats it into a format easily displayed in the user interface, and then sends it to the terminal as an HTTP response.

[0573] Input: Generated response data

[0574] Data processing: Formatting of response data (converting to a format suitable for the UI)

[0575] Output: HTTP response containing formatted response data

[0576] Step 6: Display the answer

[0577] Subject: terminal

[0578] Specific operation: The terminal analyzes the response data received from the server and displays it on the user interface. The user can then check the response displayed on the terminal screen.

[0579] Input: HTTP response containing formatted response data

[0580] Data processing: Decoding response data and converting it for UI display.

[0581] Output: The answer displayed on the user's smartphone screen.

[0582] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0583] This invention relates to a learning support system that receives questions from a specific user, analyzes them, generates appropriate answers, and sends them back to the user. This system consists of multiple components, including the user's terminal, a server, generative artificial intelligence (AI), and an emotion engine. The system can also recognize the user's emotions and provide appropriate responses based on those emotions.

[0584] 1. User actions

[0585] User: The user logs into the learning platform. The user asks questions about things they are unsure of or problems they cannot solve during their studies through the "User Interface (UI)". For example, they might type a question like, "Please explain how to solve quadratic equations."

[0586] 2. Receiving questions and recognizing emotions

[0587] Terminal: When the user clicks the send button, the terminal retrieves the question and uses its built-in camera and microphone to analyze the user's emotions from their facial expressions and voice. Facial analysis determines, for example, whether the user is troubled or nervous.

[0588] 3. Sending questions and sentiment data

[0589] Terminal: Converts the question content into an appropriate data format such as JSON, and sends it to the server by issuing an HTTP request along with sentiment data.

[0590] Server: The server receives HTTP requests and stores question data, sentiment data, and user information in the database. It then passes the received question content to the text analysis module.

[0591] 4. Analysis of the Questions

[0592] Text Analysis Module: This module analyzes the question text using natural language processing (NLP) techniques to extract important keywords and topics. For example, it might extract keywords such as "quadratic equation" and "solution method."

[0593] 5. Generating the answer

[0594] Server: Based on extracted keywords and topics, it sends analysis results and sentiment data to the generative AI engine.

[0595] Generative AI Engine: The generative AI engine generates appropriate answers from the analysis results it receives. It also adjusts the answers based on sentiment data. For example, if the user is confused, it will generate a helpful answer such as, "Please calm down, I will explain slowly. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0596] 6. Return your response

[0597] Server: Receives the generated response and formats it for return to the user. For example, it formats it into a format that is easy to read on the UI and sends an HTTP response containing the formatted data to the device.

[0598] 7. Display the answer

[0599] Terminal: Analyzes response data received from the server and displays it on the user interface. Users can check the responses displayed on the terminal screen.

[0600] Specific example

[0601] For example, consider a case where a student asks, "Could you please explain the equation of a straight line?" while simultaneously looking troubled.

[0602] User: A student types and submits the question, "Please tell me the equation of a straight line."

[0603] Terminal: It retrieves the question content and analyzes the student's facial expressions using the built-in camera. If the student appears to be struggling, the analyzed data is retrieved. An HTTP request containing this data is sent to the server.

[0604] Server: Receives HTTP requests, stores question and sentiment data, and passes it to the text analysis module.

[0605] Text analysis module: Extracts the keywords "straight line" and "equation" and sends the results to a generative AI engine.

[0606] Generative AI engine: Based on the question and sentiment data, it generates a helpful answer such as, "Don't worry, the equation of a straight line is y = mx + b."

[0607] Server: Formats the generated response and sends it to the terminal as an HTTP response.

[0608] Terminal: Analyzes the HTTP response and displays the answer on the student's screen.

[0609] User: You can check the answer displayed on the screen, "Don't worry. The equation of a straight line is y = mx + b," and use that information to further your learning.

[0610] This invention enables a swift and appropriate process from the time a user posts a question until they receive an answer, thereby effectively providing personalized learning support in an emotionally resonant manner.

[0611] The following describes the processing flow.

[0612] Step 1:

[0613] User: Log in to the learning platform's user interface. The user enters the question "Please tell me the quadratic formula" in the question submission field and clicks the submit button.

[0614] Step 2:

[0615] Terminal: It acquires the input content of questions and collects emotional data from the user's facial expressions and voice using the built-in camera and microphone. For example, it determines whether the user is confused or nervous through image analysis and voice emotion analysis.

[0616] Step 3:

[0617] Terminal: Converts the question content and sentiment data into JSON format and sends it to the server as an HTTP request.

[0618] Step 4:

[0619] Server: Receives HTTP requests and stores question data, sentiment data, and user information in the database. Passes the received question content to the text analysis module.

[0620] Step 5:

[0621] Text Analysis Module: Analyzes question text using natural language processing (NLP) techniques to extract important keywords and topics. For example, it extracts keywords such as "quadratic equation" and "quadratic formula."

[0622] Step 6:

[0623] Server: Based on extracted keywords, topics, and sentiment data, it sends the analysis results to a generative AI engine.

[0624] Step 7:

[0625] Generative AI Engine: Generates appropriate responses from the input analysis results and sentiment data. For example, if it determines that the user is confused, it will supplement the response with kind words. It might generate a response like, "Please calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0626] Step 8:

[0627] Server: Receives the generated response and formats it for return to the user. For example, it formats it into a format that is easy to read on the UI and sends an HTTP response containing the formatted data to the device.

[0628] Step 9:

[0629] Terminal: Analyzes response data received from the server and displays it on the user interface.

[0630] Step 10:

[0631] User: Checks the answer displayed on the device screen. Seeing the answer, "Calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a," the user's question is resolved and they can continue learning.

[0632] The above outlines the specific processing steps from questioning to answer presentation in an online learning support system that incorporates an emotion engine.

[0633] (Example 2)

[0634] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0635] Conventional learning support systems often suffered from inconsistent accuracy in answering user-submitted questions and difficulty in considering user emotions. As a result, users may not receive adequate support when they are struggling or stressed, potentially leading to decreased learning effectiveness. Furthermore, providing quick and accurate answers to complex problems was challenging.

[0636] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a question from a specific user, means for analyzing the content of the received question and extracting keywords and topics related to the question, means for generating an appropriate answer using natural language processing technology based on the extracted keywords and topics, means for analyzing the user's facial expressions and voice to acquire emotional data, and means for returning the generated answer to the specific user. This makes it possible to provide a quick and accurate answer in a way that is sensitive to the user's emotions.

[0637] "Specific user" refers to an individual user who can be uniquely identified.

[0638] "Analyzing the question content" refers to the process of breaking down the question received from the user and understanding its meaning.

[0639] "Keyword and topic extraction" refers to identifying important words, phrases, and themes from a question.

[0640] "Natural language processing technology" is the technology that enables computers to understand, analyze, and generate human language.

[0641] "Generating appropriate answers" means creating accurate and useful answers to user questions based on extracted keywords and topics.

[0642] "Analyzing user facial expressions and voice to acquire emotional data" means using cameras and microphones to perform feature analysis on the user's facial expressions and voice to identify their emotional state.

[0643] "Returning the answer to a specific user" means sending the generated answer to the relevant user.

[0644] "Adjusting responses based on emotional data" means optimizing the content and tone of responses according to the analyzed emotional state.

[0645] "Formatting answers" means formatting answers into a user-friendly format, making them visually easy to read.

[0646] This invention relates to a learning support system that receives questions from a specific user, analyzes them, generates appropriate answers, and sends them back to the user. This system consists of multiple components, including the user's terminal, a server, generative artificial intelligence (AI), and an emotion engine. The system can also recognize the user's emotions and provide appropriate responses based on those emotions.

[0647] The following describes specific embodiments for carrying out the invention.

[0648] Hardware and software configuration

[0649] 1. User's device:

[0650] Built-in camera, microphone, user interface (UI)

[0651] Software: OpenCV (for facial expression analysis), TensorFlow (for speech analysis)

[0652] 2. Server:

[0653] Database (for storing question data and sentiment data)

[0654] Text analysis module (using natural language processing technology)

[0655] Generative AI engine (uses a generative AI model)

[0656] System operation

[0657] The detailed operation of this system is as follows:

[0658] User actions

[0659] User: The user logs into the learning platform and enters a question through the "User Interface (UI)". For example, they might enter "Please teach me how to solve quadratic equations" and click the submit button. This action records the question content along with the user's facial expressions and voice.

[0660] Receiving questions and recognizing emotions

[0661] Terminal: The terminal detects the click of the send button and retrieves the question content. It also uses its built-in camera and microphone to analyze the user's facial expressions and voice in real time. For example, it uses OpenCV to recognize the user's facial expressions and TensorFlow to extract emotion data from the audio.

[0662] Sending questions and sentiment data

[0663] Terminal: Converts the question content into JSON format and sends it to the server as an HTTP request, including emotion data. At this time, the question content, facial expression data, and voice analysis results are combined into a single JSON object.

[0664] Question analysis

[0665] Server: The server receives HTTP requests and stores question data, sentiment data, and user information in the database. Next, it analyzes the question content using a text analysis module and extracts important keywords and topics. For example, it might extract keywords such as "quadratic equation" and "how to solve it."

[0666] Generating an answer

[0667] Server: Sends extracted keywords, topics, and sentiment data to a generative AI engine.

[0668] Generative AI Engine: The generative AI engine generates appropriate responses based on analysis results and sentiment data. Taking sentiment data into consideration, for example, if the user is distressed, it will generate a helpful response such as, "Please calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0669] Send back your response

[0670] Server: Receives the generated response and formats it into a format that is easy to read on the UI. Sends the HTTP response containing the formatted data to the terminal.

[0671] Display the answer

[0672] Terminal: The terminal analyzes the response data received from the server and displays it on the user interface. Users can review the responses displayed on the terminal screen and use them for learning.

[0673] Specific example

[0674] For example, consider a case where a student asks, "Could you please explain the equation of a straight line?" while simultaneously looking troubled.

[0675] Example of a prompt:

[0676] "Please tell me the equation of a straight line."

[0677] In this case, the system will operate as follows:

[0678] User: A student types and submits the question, "Please tell me the equation of a straight line."

[0679] Terminal: It retrieves the question content and analyzes the student's facial expressions, including any signs of distress, using its built-in camera. It then sends an HTTP request to the server containing this data.

[0680] Server: Receives HTTP requests, stores question and sentiment data, and has a text analysis module analyze the keywords "straight line" and "equation".

[0681] Generative AI engine: Based on the question content and sentiment data, it generates a helpful answer such as, "Don't worry. The equation of a straight line is y = mx + b."

[0682] Server: Formats the generated response and sends it to the terminal as an HTTP response.

[0683] Terminal: Analyzes the HTTP response and displays the answer on the student's screen.

[0684] User: You can check the answer displayed on the screen, "Don't worry. The equation of a straight line is y = mx + b," and use that information to further your learning.

[0685] As a result, by using this invention, users can receive quick and accurate answers that are empathetic to their emotions, thereby enhancing the learning effect.

[0686] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0687] Step 1:

[0688] User: The user logs into the learning platform. After logging in, they enter their question through the user interface (UI). For example, they might enter "Please teach me how to solve quadratic equations" and click the submit button.

[0689] Input: User ID and question.

[0690] Output: Login session and question content.

[0691] Step 2:

[0692] Terminal: The terminal detects the click of the send button and retrieves the question content. Simultaneously, it records the user's facial expressions and voice using the built-in camera and microphone, and performs facial expression analysis and sentiment analysis. For example, it uses OpenCV to analyze the user's facial expressions in real time and TensorFlow to extract sentiment data from the audio.

[0693] Input: Question content and user's facial expressions and voice data.

[0694] Output: Question content and emotional data (confusion, anxiety, etc.).

[0695] Step 3:

[0696] Terminal: Converts the question content and emotion data into JSON format and sends it to the server via an HTTP request. At this time, the question content, facial expression data, and voice analysis results are combined into a single JSON object.

[0697] Input: Question content and sentiment data.

[0698] Output: Data in JSON format.

[0699] Step 4:

[0700] Server: The server receives HTTP requests and stores question data, sentiment data, and user information in the database.

[0701] Input: Data in JSON format.

[0702] Output: Question data and sentiment data stored in the database.

[0703] Step 5:

[0704] Text Analysis Module: The text analysis module analyzes the received question content and extracts important keywords and topics. For example, it might extract the keywords "quadratic equation" and "solution method" from the question text.

[0705] Input: Question content.

[0706] Output: Extracted keywords and topics.

[0707] Step 6:

[0708] Server: Sends extracted keywords, topics, and sentiment data to a generative AI engine.

[0709] Input: Extracted keywords, topics, and sentiment data.

[0710] Output: API call to a generative AI engine.

[0711] Step 7:

[0712] Generative AI Engine: The generative AI engine generates appropriate responses based on the received analysis results and sentiment data. For example, if the sentiment data indicates "confused," it will generate a helpful response such as, "Please calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0713] Input: Keywords, topics, sentiment data.

[0714] Output: The generated answer.

[0715] Step 8:

[0716] Server: Receives the generated response and formats it for return to the user. It uses HTML and CSS to format it into a user-friendly format and sends the HTTP response containing the formatted data to the device.

[0717] Input: Generated answer.

[0718] Output: HTTP response containing formatted response data.

[0719] Step 9:

[0720] Terminal: The terminal analyzes the response data received from the server and displays it on the user interface. Users can review the responses displayed on the terminal screen and use them for learning.

[0721] Input: HTTP response from the server.

[0722] Output: The answer displayed in the user interface.

[0723] (Application Example 2)

[0724] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0725] In today's educational environment, there is a need to quickly and accurately resolve questions and anxieties that arise as learners progress at their own pace. Especially with the increase in online learning, immediate question support and emotionally resonant support are crucial. However, conventional systems only provide standardized answers to user questions, failing to consider the user's emotional state. As a result, learning effectiveness has been reduced when learners experience anxiety or confusion.

[0726] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for receiving questions from a specific user, means for analyzing the received questions and user emotion data extracted from the built-in camera and voice input to extract keywords and topics related to the question content, means for generating an appropriate response according to the emotion using generative artificial intelligence based on the extracted keywords and topics and the user emotion data, and means for returning the generated response to the specific user. This enables rapid and accurate learning support that takes into account the user's emotional state.

[0727] "Means for receiving questions" refers to a device or software module for acquiring user inquiries and questions as digital data.

[0728] A "camera" is an optical device used to record a user's facial expressions and movements as video.

[0729] "Voice input" refers to a device or software module that recognizes and analyzes sounds emitted by a user as digital signals.

[0730] "Emotional data" refers to digital data that indicates the emotional state of a user, analyzed from information such as their facial expressions and voice.

[0731] "Means of analysis" refers to a device or software module used to perform analysis based on acquired data and extract meaningful information.

[0732] "Means for extracting keywords and topics" refers to devices or software modules that use natural language processing (NLP) techniques to find important words or themes from the content of a question.

[0733] "Generative artificial intelligence" refers to algorithms or models that automatically generate natural language responses or suggestions based on received data.

[0734] An "emotionally appropriate response" is a response that takes the user's emotional state into consideration and is generated in a kind and comforting tone and content.

[0735] "Means for returning the generated response" refers to a communication method or software module for delivering the generated response to the user.

[0736] "Means for formatting into a format suitable for a display device" refers to a device or software module that converts the format of the response sent back to the user so that it can be displayed clearly on the user's device.

[0737] This invention relates to a learning support system using a smart device that can instantly respond to user questions and generate and return answers tailored to the user's emotions. This system consists of multiple components, including the user's terminal, a server, generative artificial intelligence (AI), and an emotion engine.

[0738] 1. User actions

[0739] The user uses smart glasses, which have a built-in camera and microphone. The user inputs questions by voice. For example, if the user asks, "Please tell me how to solve a quadratic equation," the voice is captured by the smart glasses' microphone. The camera simultaneously analyzes the user's facial expressions and acquires emotional data.

[0740] 2. Questions and acquisition of sentiment data

[0741] The questions are converted to text using speech recognition technology (e.g., Google Cloud Speech-to-Text API). The camera uses facial recognition technology (e.g., OpenFace) to analyze the user's facial expressions. This allows for the acquisition of emotional data, such as whether the user is distressed or nervous.

[0742] 3. Sending questions and sentiment data

[0743] The terminal sends the text questions and sentiment data, converted from speech, to the server in an appropriate data format such as JSON. The server receives this and stores it in a database. Next, a text analysis module analyzes the questions using natural language processing (NLP) techniques and extracts important keywords and topics (e.g., "quadratic equation," "how to solve").

[0744] 4. Generating the answer

[0745] The server sends analysis results and sentiment data to a generative AI engine (e.g., OpenAI GPT-4) based on extracted keywords and topics. The generative AI engine generates appropriate responses based on the analysis results and further adjusts the tone and content of the responses based on the sentiment data. For example, if the user is in distress, a helpful response such as "Please calm down, I will explain slowly. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a" is generated.

[0746] 5. Return and display of responses

[0747] The generated response is sent back to the device from the server. The device formats the received response into an appropriate format and displays it on the smart glasses' display. This allows the user to see the emotionally resonant response in real time.

[0748] Specific example

[0749] For example, consider a situation where a student is solving a math problem and asks, "Can you teach me how to solve a quadratic equation?" while simultaneously looking troubled.

[0750] Example of a prompt message (input to the generation AI engine):

[0751] User's question: "How do I solve a quadratic equation?"

[0752] User's emotion: "Confused"

[0753] Question: "Explanation of how to solve quadratic equations"

[0754] Emotion-based response: "Explain in a kind and reassuring tone."

[0755] The following is an example of a response from a generative AI engine.

[0756] "Please stay calm. I will explain how to solve quadratic equations. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0757] This invention allows users to instantly receive responses appropriate to their emotional state, thereby effectively supporting their learning.

[0758] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0759] Step 1:

[0760] The user inputs a question using smart glasses. The user asks a question by voice, such as "Tell me how to solve a quadratic equation." This voice is captured by the microphone in the smart glasses. At the same time, the camera records the user's facial expressions and acquires the video data. Thus, audio data and video data are obtained as input.

[0761] Step 2:

[0762] The device uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to convert captured audio into text. This process generates text data such as "Tell me how to solve a quadratic equation." Additionally, facial recognition technology (e.g., OpenFace) is used to analyze the user's facial expressions from the camera footage, and emotion data (e.g., confused) is extracted.

[0763] Step 3:

[0764] The terminal converts text data and sentiment data into an appropriate data format such as JSON and sends it to the server as an HTTP request. The input consists of question text and sentiment data, and the output is JSON data. The server receives this and stores it in its database.

[0765] Step 4:

[0766] On the server, a text analysis module extracts question content from JSON-formatted data and analyzes it using natural language processing (NLP) techniques. For example, keywords such as "quadratic equation" and "solution method" might be extracted. The input for this process is question data and sentiment data, and the output is the analyzed keywords and topics.

[0767] Step 5:

[0768] The server sends the analysis results and sentiment data to a generative AI engine (e.g., OpenAI GPT-4) based on the analyzed keywords and topics. The generative AI engine receives this data, constructs a prompt, and generates an appropriate response. The input consists of keywords, topics, and sentiment data, and the output is the generated response. For example, a response like "Calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a" might be generated.

[0769] Step 6:

[0770] The server receives the generated response and sends it back to the terminal. The terminal then formats the received response into a format suitable for the user's smart glasses. After the output data has been properly formatted, it is displayed on the user's screen.

[0771] Step 7:

[0772] The device displays the final answer on the smart glasses' display. This allows the user to see an emotionally resonant response in real time. The output is the displayed answer, where the user can confirm the information.

[0773] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0774] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0775] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0776] [Third Embodiment]

[0777] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0778] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0779] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0780] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0781] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0782] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0783] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0784] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0785] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0786] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0787] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0788] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0789] This invention relates to a learning support system that receives questions from a specific user, analyzes them, generates appropriate answers, and returns them to the user. This system consists of multiple components, including the user's terminal, a server, and generative artificial intelligence (AI).

[0790] 1. User actions

[0791] User: For example, a high school student logs into an online learning platform at home. The user asks questions about things they are unsure of or problems they cannot solve during their studies through the "User Interface (UI)". For example, they might type a question like, "Please explain how to solve quadratic equations."

[0792] 2. Receiving a question

[0793] Terminal: When the user clicks the submit button, the terminal converts the question content into a data format such as JSON, issues an HTTP request, and sends it to the server.

[0794] Server: The server receives this HTTP request and retrieves the question content. It also stores data containing user information in the database to identify which user the question is from.

[0795] 3. Analysis of the Question

[0796] Server: Next, the server passes the received question to the text analysis module. This module uses natural language processing (NLP) techniques to analyze the question and extract important keywords and topics. For example, the keywords "quadratic equation" and "solution method" might be extracted.

[0797] 4. Generating the answer

[0798] Server: Based on the extracted keywords and topics, the server sends the analysis results to a generative AI engine. The generative AI engine understands the question and generates an appropriate answer. In this case, a specific answer is generated: "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0799] 5. Return your response

[0800] Server: Receives the generated response and formats it for return to the user. For example, it formats it to be easily viewable on the UI, and then sends an HTTP response containing the formatted data to the device.

[0801] Terminal: Analyzes response data received from the server and displays it on the user interface. Users can check the responses displayed on the terminal screen.

[0802] Specific example

[0803] For example, consider the case where a student asks, "Could you tell me the equation of a straight line?"

[0804] 1. User: A student types and submits the question, "Please tell me the equation of a straight line."

[0805] 2. Terminal: Converts the question content into JSON format and sends an HTTP request to the server.

[0806] 3. Server: Receives an HTTP request and passes the question "equation of a straight line" to the text analysis module.

[0807] 4. Text analysis module: Extracts keywords such as "straight line" and "equation," and sends the results to a generative AI engine.

[0808] 5. Generative AI engine: Generates the answer "The equation of a straight line is y = mx + b" and returns it to the server.

[0809] 6. Server: Formats the generated response into a user-friendly format for the UI and sends it to the device as an HTTP response.

[0810] 7. Terminal: Analyzes the HTTP response and displays the answer on the student's screen.

[0811] 8. User: You can check the answer displayed on the screen, "The equation of a straight line is y = mx + b," and use it to further your learning.

[0812] This invention enables effective personalized learning support because the process from when a user posts a question to when they receive an answer is carried out quickly and appropriately.

[0813] The following describes the processing flow.

[0814] Step 1:

[0815] User: After logging into the learning platform's user interface, the user enters their question in the question submission field and clicks the submit button.

[0816] Step 2:

[0817] Terminal: Retrieves the input question content and converts it to an appropriate data format such as JSON. Then, it sends this data to the server as an HTTP request.

[0818] Step 3:

[0819] Server: Receives HTTP requests and saves question data and user information to the database. Then, passes the received question content to the text analysis module.

[0820] Step 4:

[0821] Text Analysis Module: Analyzes question text using natural language processing (NLP) techniques to extract important keywords and topics. For example, it can extract keywords such as "quadratic equation" and "solution method."

[0822] Step 5:

[0823] Server: Based on extracted keywords and topics, it sends the analysis results to the generative AI engine.

[0824] Step 6:

[0825] Generative AI Engine: The generative AI engine generates appropriate answers from the analysis results it receives. For example, it can generate a specific answer such as, "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0826] Step 7:

[0827] Server: Receives the generated response and formats it for return to the user. For example, it formats it to be easily viewable on the UI and sends the HTTP response containing the formatted data to the device.

[0828] Step 8:

[0829] Terminal: Analyzes response data received from the server and displays it on the user interface.

[0830] Step 9:

[0831] User: Check the answer displayed on the device screen. If the question is resolved, continue learning. If you have further questions, return to step 1 and enter a new question.

[0832] The above outlines the specific processing steps involved in providing answers to user questions on an online learning platform.

[0833] (Example 1)

[0834] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0835] Conventional learning support systems suffered from problems such as delays in responding to user questions and the inability to obtain appropriate answers. This resulted in reduced learning efficiency and difficulty for users to quickly obtain the necessary information.

[0836] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0837] In this invention, the server includes means for receiving questions from specific end users, means for converting the received questions into a data format and sending them to the server, means for the server to store the received question data in a database and pass it to a text analysis module, means for the text analysis module to analyze the question content using natural language processing technology and extract important keywords and topics, means for generating appropriate answers using generative artificial intelligence based on the extracted keywords and topics, and means for formatting the generated answers into a user-friendly format on the user interface and returning them to the end user. This makes it possible for users to quickly and accurately resolve questions and problems they cannot solve during learning, thereby improving learning efficiency.

[0838] An "end user" refers to a user who uses the system to ask questions.

[0839] A "server" refers to a central system that receives and processes requests from clients.

[0840] "Data format" refers to a format in which data is encoded according to certain rules. For example, the JSON format is one such format.

[0841] A "database" refers to a system that systematically stores data and allows it to be retrieved as needed.

[0842] A "text analysis module" refers to a program that uses natural language processing technology to analyze text data and extract its meaning.

[0843] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human language.

[0844] "Keywords" refer to important words or phrases extracted from the question.

[0845] "Topic" refers to the subject or theme of the question.

[0846] "Generative artificial intelligence" refers to artificial intelligence that has the function of generating appropriate answers to user questions.

[0847] "User interface" refers to the screens and input methods that end users use to interact with a system.

[0848] "Formatting" refers to the process of converting data into a format that is easy to read. This includes, for example, using HTML and CSS.

[0849] Modes for carrying out the invention

[0850] This invention relates to a learning support system that receives questions from specific end users, analyzes them, generates appropriate answers, and returns them to the user. This system consists of multiple components, including the end user's terminal, a server, and generative artificial intelligence (AI).

[0851] End users log in to the online learning platform from their homes, schools, or other locations. Users input questions and problems they encounter during their studies through the user interface (UI). For example, they might input a question like, "Please explain how to solve quadratic equations."

[0852] The terminal converts the user's input into a data format (such as JSON) and sends it to the server as an HTTP request. The JavaScript JSON.stringify method is used for this conversion. The request is sent using the HTTP POST method.

[0853] The server receives an HTTP request and retrieves the question data from the request body. The Node.js express framework is used for this process. The server then stores user information in a database (e.g., MySQL) to identify which user the question originated from. The database records the question content and associated user information.

[0854] Next, the server passes the acquired question data to the text analysis module. The text analysis module uses natural language processing (NLP) techniques to analyze the question content and extract important keywords and topics. Specifically, it uses Google's NLP API to extract keywords such as "quadratic equation" and "how to solve it."

[0855] The extracted keywords and topics are sent to a generative AI engine. This AI engine uses models such as OpenAI's GPT-3 and GPT-4. The AI ​​engine understands the question and generates an appropriate answer. In this case, a specific answer is generated: "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0856] The generated response is formatted on the server. HTML and CSS are used for formatting, converting it into a format that is easy to read on the UI. The formatted data is then sent to the terminal as an HTTP response.

[0857] The terminal parses the response data received from the server. The JavaScript JSON.parse method is used again for this parsing. After parsing, the response is displayed in the user interface. The user can then verify the response displayed on the terminal screen.

[0858] Specific example

[0859] For example, consider a case where a student asks, "What is the equation of a straight line?" The student types the question, "What is the equation of a straight line?", and clicks the submit button. The device converts the question into JSON format and sends it to the server as an HTTP request. The server receives this request and passes the question, "What is the equation of a straight line?", to a text analysis module. The text analysis module extracts keywords such as "straight line" and "equation", and sends the result to a generative AI engine. The generative AI engine generates the answer, "The equation of a straight line is y = mx + b", and returns it to the server. The server formats the generated answer into a format that is easy to read on the UI and sends it to the device as an HTTP response. The device analyzes the HTTP response and displays the answer on the student's screen. The student can then check the answer, "The equation of a straight line is y = mx + b", displayed on the screen and use it to aid in their learning.

[0860] Example of a prompt

[0861] Please answer the following question: How do I solve a quadratic equation?

[0862] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0863] Step 1:

[0864] User: The end user logs into the online learning platform and enters questions or problems they cannot solve. For example, they might type "Please explain how to solve quadratic equations" into the text box on the user interface and click the submit button. The input data is the question in text format.

[0865] Step 2:

[0866] Terminal: The terminal converts the user-entered question content into JSON format. This conversion process uses the JavaScript JSON.stringify method. The converted data is sent to the server as an HTTP POST request. The input is the user's question text, and the output is the question data in JSON format.

[0867] Step 3:

[0868] Server: The server receives HTTP requests and retrieves question data from the request body. This process uses the Node.js express framework. It also stores data containing user information in a database (e.g., MySQL) to identify which user the question is from. The input is question data in JSON format, and the output is the user information and question data stored in the database.

[0869] Step 4:

[0870] Server: The server passes the acquired question data to the text analysis module. The text analysis module uses Google's NLP API to analyze the question content and extract important keywords and topics. The input is the question data passed from the server, and the output is a list of extracted keywords and topics.

[0871] Step 5:

[0872] Server: Based on the extracted keywords and topics, the server sends the analysis results to a generative AI engine. This generative AI engine uses models such as OpenAI's GPT-3 and GPT-4. The AI ​​engine understands the question in the format of a prompt example such as "Please answer the following question: How do you solve a quadratic equation?" and generates an appropriate answer. The input is a list of extracted keywords and topics, and the output is the generated answer text.

[0873] Step 6:

[0874] Server: Formats the generated response into a user-friendly format for the user interface. This formatting process uses HTML and CSS to convert it into a readable format. It sends the formatted data to the terminal as an HTTP response. The input is the generated response text, and the output is formatted data in HTML format.

[0875] Step 7:

[0876] Terminal: Parses the response data received from the server. This parsing process uses the JavaScript JSON.parse method. The response is then displayed in the user interface. Specifically, it displays the response using certain HTML elements (e.g., Insert the answer text into the tag. The input is answer data in JSON format, and the output is the answer text displayed in the user interface.

[0877] Step 8:

[0878] User: Checks the answer displayed on the device screen. The user can learn by looking at answers such as, for example, "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a". The input is the text displayed on the device, and the output is the user's understanding and learning effect.

[0879] (Application Example 1)

[0880] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0881] Conventional learning support systems often struggled to provide quick and accurate answers to user questions. Furthermore, e-commerce sites, in particular, require rapid responses to product inquiries, necessitating a system that enhances the user experience. This invention aims to solve these problems by providing a system that quickly and appropriately analyzes user questions and generates and provides accurate answers using generative artificial intelligence.

[0882] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0883] In this invention, the server includes means for receiving questions from specific users, means for analyzing the received questions and extracting keywords and topics related to the question content, and means for generating appropriate answers using generative artificial intelligence based on the extracted keywords and topics. This makes it possible to provide quick and accurate answers to questions from users. Furthermore, the invention also includes means for converting the received question content into a data format and transmitting it to an information processing device, and means for providing quick and accurate answers to questions about products, which can significantly improve the user experience, especially on e-commerce sites.

[0884] "Specific users" refers to users who access the system and enter questions.

[0885] "Means for receiving questions" refers to the methods or devices used by the system to acquire questions entered by users.

[0886] "Means for extracting keywords and topics related to the content of a question" refers to methods or devices for identifying and extracting important words and themes from a received question.

[0887] "Means of generating appropriate answers using generative artificial intelligence" refers to methods or devices that use generative AI to create answers to users' questions based on extracted keywords and topics.

[0888] "Means of sending generated answers back to specific users" refers to methods or devices for sending generated answers to users.

[0889] "Information processing equipment" refers to computers and servers used for processing data.

[0890] "Means of providing prompt and accurate answers to questions about products" refers to methods or devices that provide prompt and accurate answers to user inquiries about goods or products.

[0891] "Means of converting to data format" refers to methods or devices for converting received questions and answers into a standardized data format such as JSON.

[0892] This invention relates to a learning support system that receives questions from specific users, analyzes them, generates appropriate answers, and sends them back to the users, as well as an answer generation system for e-commerce websites. This system consists of multiple components, including the user's terminal, a server, and generative artificial intelligence (AI).

[0893] 1. User actions

[0894] User: For example, a user of an online shopping site uses a smartphone app to input a product-related question into a chatbot. The user inputs a question through the "User Interface (UI)," such as, "Please tell me how to set up the remote control for this TV."

[0895] 2. Receiving a question

[0896] Terminal: When the user clicks the submit button, the terminal converts the question content into a data format such as JSON, issues an HTTP request, and sends it to the server.

[0897] Server: The server receives this HTTP request and retrieves the question content. It also stores data, including user information, in the database to identify which user the question is from.

[0898] 3. Analysis of the Question

[0899] Server: Next, the server passes the received question to the text analysis module. This module uses natural language processing (NLP) techniques to analyze the question and extract important keywords and topics. For example, the keywords "television" and "remote control settings" might be extracted.

[0900] 4. Generating the answer

[0901] Server: Based on the extracted keywords and topics, the server sends the analysis results to the generative AI engine. The generative AI engine understands the question and generates an appropriate answer. In this case, a specific answer is generated: "To set up the remote control, press and hold the button on the back of the remote control for 3 seconds, then follow the instructions displayed on the TV screen."

[0902] 5. Return your response

[0903] Server: Receives the generated response and formats it for return to the user. For example, it formats it to be easily viewable on the UI, and then sends an HTTP response containing the formatted data to the device.

[0904] Terminal: Analyzes response data received from the server and displays it on the user interface. Users can check the responses displayed on the terminal screen.

[0905] Hardware and software to be used

[0906] Hardware: Smartphone (user device), Server (API server)

[0907] Software: Flask (Python web framework), NLP module, generative AI engine

[0908] Data processing and data calculation

[0909] NLP Module: Analyzes the question content using natural language processing techniques and extracts important keywords. Specifically, it uses libraries such as spaCy and NLTK.

[0910] Generative AI Engine: Based on extracted keywords, it utilizes generative AI models such as GPT-3 to generate answers. This ensures that answers are provided quickly and accurately.

[0911] Specific example

[0912] For example, if a user of an online shopping site asks, "What is the capacity of this refrigerator?", the answer will be generated using the following steps.

[0913] User: Type "What is the capacity of this refrigerator?" into the chatbot on their smartphone and send it.

[0914] Terminal: Sends the question to the server.

[0915] Server: Receives an HTTP request and extracts the keywords "refrigerator" and "capacity".

[0916] Generative AI engine: Generates the answer "This refrigerator has a capacity of 300 liters."

[0917] Server: Send response.

[0918] Terminal: Display the answer.

[0919] Example of a prompt

[0920] "Please tell me how to set up the remote control for this TV."

[0921] "What's the capacity of this refrigerator?"

[0922] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0923] Step 1: Enter user questions

[0924] Subject: User

[0925] Specific operation: The user uses a smartphone app to type a question into the chatbot and clicks the send button.

[0926] Input: Example question entered by the user: "How do I set up the remote control for this TV?"

[0927] Output: Data processed internally by the device based on the questions entered by the user.

[0928] Step 2: Format and send the question data

[0929] Subject: terminal

[0930] Specific operation: The terminal converts the question entered by the user into JSON format and sends it to the server as an HTTP request.

[0931] Input: Question data entered by the user

[0932] Data processing: Encode the question data into JSON format.

[0933] Output: Question data in HTTP request format

[0934] Step 3: Receiving and analyzing questions

[0935] Subject: Server

[0936] Specific operation: The server receives an HTTP request and retrieves the question content. Then, it passes the question content to a text analysis module, which uses natural language processing (NLP) techniques to extract important keywords and topics.

[0937] Input: Question data in JSON format

[0938] Data processing: Extract question content as text data and perform NLP analysis.

[0939] Output: Keywords such as "TV" and "remote control settings"

[0940] Step 4: Generating the answer

[0941] Subject: Server

[0942] Specific operation: The server sends the extracted keywords to a generative AI engine, which then generates an appropriate response.

[0943] Input: Extracted keywords

[0944] Data processing: Answer generation using generative AI models (e.g., GPT-3)

[0945] Output: Response data: "To set up the remote control, press and hold the button on the back of the remote control for 3 seconds, then follow the instructions displayed on the TV screen."

[0946] Step 5: Format and return the response data.

[0947] Subject: Server

[0948] Specific operation: The server receives the generated response, formats it into a format easily displayed in the user interface, and then sends it to the terminal as an HTTP response.

[0949] Input: Generated response data

[0950] Data processing: Formatting of response data (converting to a format suitable for the UI)

[0951] Output: HTTP response containing formatted response data

[0952] Step 6: Display the answer

[0953] Subject: terminal

[0954] Specific operation: The terminal analyzes the response data received from the server and displays it on the user interface. The user can then check the response displayed on the terminal screen.

[0955] Input: HTTP response containing formatted response data

[0956] Data processing: Decoding response data and converting it for UI display.

[0957] Output: The answer displayed on the user's smartphone screen.

[0958] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0959] This invention relates to a learning support system that receives questions from a specific user, analyzes them, generates appropriate answers, and sends them back to the user. This system consists of multiple components, including the user's terminal, a server, generative artificial intelligence (AI), and an emotion engine. The system can also recognize the user's emotions and provide appropriate responses based on those emotions.

[0960] 1. User actions

[0961] User: The user logs into the learning platform. The user asks questions about things they are unsure of or problems they cannot solve during their studies through the "User Interface (UI)". For example, they might type a question like, "Please explain how to solve quadratic equations."

[0962] 2. Receiving questions and recognizing emotions

[0963] Terminal: When the user clicks the send button, the terminal retrieves the question and uses its built-in camera and microphone to analyze the user's emotions from their facial expressions and voice. Facial analysis determines, for example, whether the user is troubled or nervous.

[0964] 3. Sending questions and sentiment data

[0965] Terminal: Converts the question content into an appropriate data format such as JSON, and sends it to the server by issuing an HTTP request along with sentiment data.

[0966] Server: The server receives HTTP requests and stores question data, sentiment data, and user information in the database. It then passes the received question content to the text analysis module.

[0967] 4. Analysis of the Questions

[0968] Text Analysis Module: This module analyzes the question text using natural language processing (NLP) techniques to extract important keywords and topics. For example, it might extract keywords such as "quadratic equation" and "solution method."

[0969] 5. Generating the answer

[0970] Server: Based on extracted keywords and topics, it sends analysis results and sentiment data to the generative AI engine.

[0971] Generative AI Engine: The generative AI engine generates appropriate answers from the analysis results it receives. It also adjusts the answers based on sentiment data. For example, if the user is confused, it will generate a helpful answer such as, "Please calm down, I will explain slowly. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[0972] 6. Return your response

[0973] Server: Receives the generated response and formats it for return to the user. For example, it formats it into a format that is easy to read on the UI and sends an HTTP response containing the formatted data to the device.

[0974] 7. Display the answer

[0975] Terminal: Analyzes response data received from the server and displays it on the user interface. Users can check the responses displayed on the terminal screen.

[0976] Specific example

[0977] For example, consider a case where a student asks, "Could you please explain the equation of a straight line?" while simultaneously looking troubled.

[0978] User: A student types and submits the question, "Please tell me the equation of a straight line."

[0979] Terminal: It retrieves the question content and analyzes the student's facial expressions using the built-in camera. If the student appears to be struggling, the analyzed data is retrieved. An HTTP request containing this data is sent to the server.

[0980] Server: Receives HTTP requests, stores question and sentiment data, and passes it to the text analysis module.

[0981] Text analysis module: Extracts the keywords "straight line" and "equation" and sends the results to a generative AI engine.

[0982] Generative AI engine: Based on the question and sentiment data, it generates a helpful answer such as, "Don't worry, the equation of a straight line is y = mx + b."

[0983] Server: Formats the generated response and sends it to the terminal as an HTTP response.

[0984] Terminal: Analyzes the HTTP response and displays the answer on the student's screen.

[0985] User: You can check the answer displayed on the screen, "Don't worry. The equation of a straight line is y = mx + b," and use that information to further your learning.

[0986] This invention enables a swift and appropriate process from the time a user posts a question until they receive an answer, thereby effectively providing personalized learning support in an emotionally resonant manner.

[0987] The following describes the processing flow.

[0988] Step 1:

[0989] User: Log in to the learning platform's user interface. The user enters the question "Please tell me the quadratic formula" in the question submission field and clicks the submit button.

[0990] Step 2:

[0991] Terminal: It acquires the input content of questions and collects emotional data from the user's facial expressions and voice using the built-in camera and microphone. For example, it determines whether the user is confused or nervous through image analysis and voice emotion analysis.

[0992] Step 3:

[0993] Terminal: Converts the question content and sentiment data into JSON format and sends it to the server as an HTTP request.

[0994] Step 4:

[0995] Server: Receives HTTP requests and stores question data, sentiment data, and user information in the database. Passes the received question content to the text analysis module.

[0996] Step 5:

[0997] Text Analysis Module: Analyzes question text using natural language processing (NLP) techniques to extract important keywords and topics. For example, it extracts keywords such as "quadratic equation" and "quadratic formula."

[0998] Step 6:

[0999] Server: Based on extracted keywords, topics, and sentiment data, it sends the analysis results to a generative AI engine.

[1000] Step 7:

[1001] Generative AI Engine: Generates appropriate responses from the input analysis results and sentiment data. For example, if it determines that the user is confused, it will supplement the response with kind words. It might generate a response like, "Please calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[1002] Step 8:

[1003] Server: Receives the generated response and formats it for return to the user. For example, it formats it into a format that is easy to read on the UI and sends an HTTP response containing the formatted data to the device.

[1004] Step 9:

[1005] Terminal: Analyzes response data received from the server and displays it on the user interface.

[1006] Step 10:

[1007] User: Checks the answer displayed on the device screen. Seeing the answer, "Calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a," the user's question is resolved and they can continue learning.

[1008] The above outlines the specific processing steps from questioning to answer presentation in an online learning support system that incorporates an emotion engine.

[1009] (Example 2)

[1010] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1011] Conventional learning support systems often suffered from inconsistent accuracy in answering user-submitted questions and difficulty in considering user emotions. As a result, users may not receive adequate support when they are struggling or stressed, potentially leading to decreased learning effectiveness. Furthermore, providing quick and accurate answers to complex problems was challenging.

[1012] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a question from a specific user, means for analyzing the content of the received question and extracting keywords and topics related to the question, means for generating an appropriate answer using natural language processing technology based on the extracted keywords and topics, means for analyzing the user's facial expressions and voice to acquire emotional data, and means for returning the generated answer to the specific user. This makes it possible to provide a quick and accurate answer in a way that is sensitive to the user's emotions.

[1013] "Specific user" refers to an individual user who can be uniquely identified.

[1014] "Analyzing the question content" refers to the process of breaking down the question received from the user and understanding its meaning.

[1015] "Keyword and topic extraction" refers to identifying important words, phrases, and themes from a question.

[1016] "Natural language processing technology" is the technology that enables computers to understand, analyze, and generate human language.

[1017] "Generating appropriate answers" means creating accurate and useful answers to user questions based on extracted keywords and topics.

[1018] "Analyzing user facial expressions and voice to acquire emotional data" means using cameras and microphones to perform feature analysis on the user's facial expressions and voice to identify their emotional state.

[1019] "Returning the answer to a specific user" means sending the generated answer to the relevant user.

[1020] "Adjusting responses based on emotional data" means optimizing the content and tone of responses according to the analyzed emotional state.

[1021] "Formatting answers" means formatting answers into a user-friendly format, making them visually easy to read.

[1022] This invention relates to a learning support system that receives questions from a specific user, analyzes them, generates appropriate answers, and sends them back to the user. This system consists of multiple components, including the user's terminal, a server, generative artificial intelligence (AI), and an emotion engine. The system can also recognize the user's emotions and provide appropriate responses based on those emotions.

[1023] The following describes specific embodiments for carrying out the invention.

[1024] Hardware and software configuration

[1025] 1. User's device:

[1026] Built-in camera, microphone, user interface (UI)

[1027] Software: OpenCV (for facial expression analysis), TensorFlow (for speech analysis)

[1028] 2. Server:

[1029] Database (for storing question data and sentiment data)

[1030] Text analysis module (using natural language processing technology)

[1031] Generative AI engine (uses a generative AI model)

[1032] System operation

[1033] The detailed operation of this system is as follows:

[1034] User actions

[1035] User: The user logs into the learning platform and enters a question through the "User Interface (UI)". For example, they might enter "Please teach me how to solve quadratic equations" and click the submit button. This action records the question content along with the user's facial expressions and voice.

[1036] Receiving questions and recognizing emotions

[1037] Terminal: The terminal detects the click of the send button and retrieves the question content. It also uses its built-in camera and microphone to analyze the user's facial expressions and voice in real time. For example, it uses OpenCV to recognize the user's facial expressions and TensorFlow to extract emotion data from the audio.

[1038] Sending questions and sentiment data

[1039] Terminal: Converts the question content into JSON format and sends it to the server as an HTTP request, including emotion data. At this time, the question content, facial expression data, and voice analysis results are combined into a single JSON object.

[1040] Question analysis

[1041] Server: The server receives HTTP requests and stores question data, sentiment data, and user information in the database. Next, it analyzes the question content using a text analysis module and extracts important keywords and topics. For example, it might extract keywords such as "quadratic equation" and "how to solve it."

[1042] Generating an answer

[1043] Server: Sends extracted keywords, topics, and sentiment data to a generative AI engine.

[1044] Generative AI Engine: The generative AI engine generates appropriate responses based on analysis results and sentiment data. Taking sentiment data into consideration, for example, if the user is distressed, it will generate a helpful response such as, "Please calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[1045] Send back your response

[1046] Server: Receives the generated response and formats it into a format that is easy to read on the UI. Sends the HTTP response containing the formatted data to the terminal.

[1047] Display the answer

[1048] Terminal: The terminal analyzes the response data received from the server and displays it on the user interface. Users can review the responses displayed on the terminal screen and use them for learning.

[1049] Specific example

[1050] For example, consider a case where a student asks, "Could you please explain the equation of a straight line?" while simultaneously looking troubled.

[1051] Example of a prompt:

[1052] "Please tell me the equation of a straight line."

[1053] In this case, the system will operate as follows:

[1054] User: A student types and submits the question, "Please tell me the equation of a straight line."

[1055] Terminal: It retrieves the question content and analyzes the student's facial expressions, including any signs of distress, using its built-in camera. It then sends an HTTP request to the server containing this data.

[1056] Server: Receives HTTP requests, stores question and sentiment data, and has a text analysis module analyze the keywords "straight line" and "equation".

[1057] Generative AI engine: Based on the question content and sentiment data, it generates a helpful answer such as, "Don't worry. The equation of a straight line is y = mx + b."

[1058] Server: Formats the generated response and sends it to the terminal as an HTTP response.

[1059] Terminal: Analyzes the HTTP response and displays the answer on the student's screen.

[1060] User: You can check the answer displayed on the screen, "Don't worry. The equation of a straight line is y = mx + b," and use that information to further your learning.

[1061] As a result, by using this invention, users can receive quick and accurate answers that are empathetic to their emotions, thereby enhancing the learning effect.

[1062] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1063] Step 1:

[1064] User: The user logs into the learning platform. After logging in, they enter their question through the user interface (UI). For example, they might enter "Please teach me how to solve quadratic equations" and click the submit button.

[1065] Input: User ID and question.

[1066] Output: Login session and question content.

[1067] Step 2:

[1068] Terminal: The terminal detects the click of the send button and retrieves the question content. Simultaneously, it records the user's facial expressions and voice using the built-in camera and microphone, and performs facial expression analysis and sentiment analysis. For example, it uses OpenCV to analyze the user's facial expressions in real time and TensorFlow to extract sentiment data from the audio.

[1069] Input: Question content and user's facial expressions and voice data.

[1070] Output: Question content and emotional data (confusion, anxiety, etc.).

[1071] Step 3:

[1072] Terminal: Converts the question content and emotion data into JSON format and sends it to the server via an HTTP request. At this time, the question content, facial expression data, and voice analysis results are combined into a single JSON object.

[1073] Input: Question content and sentiment data.

[1074] Output: Data in JSON format.

[1075] Step 4:

[1076] Server: The server receives HTTP requests and stores question data, sentiment data, and user information in the database.

[1077] Input: Data in JSON format.

[1078] Output: Question data and sentiment data stored in the database.

[1079] Step 5:

[1080] Text Analysis Module: The text analysis module analyzes the received question content and extracts important keywords and topics. For example, it might extract the keywords "quadratic equation" and "solution method" from the question text.

[1081] Input: Question content.

[1082] Output: Extracted keywords and topics.

[1083] Step 6:

[1084] Server: Sends extracted keywords, topics, and sentiment data to a generative AI engine.

[1085] Input: Extracted keywords, topics, and sentiment data.

[1086] Output: API call to a generative AI engine.

[1087] Step 7:

[1088] Generative AI Engine: The generative AI engine generates appropriate responses based on the received analysis results and sentiment data. For example, if the sentiment data indicates "confused," it will generate a helpful response such as, "Please calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[1089] Input: Keywords, topics, sentiment data.

[1090] Output: The generated answer.

[1091] Step 8:

[1092] Server: Receives the generated response and formats it for return to the user. It uses HTML and CSS to format it into a user-friendly format and sends the HTTP response containing the formatted data to the device.

[1093] Input: Generated answer.

[1094] Output: HTTP response containing formatted response data.

[1095] Step 9:

[1096] Terminal: The terminal analyzes the response data received from the server and displays it on the user interface. Users can review the responses displayed on the terminal screen and use them for learning.

[1097] Input: HTTP response from the server.

[1098] Output: The answer displayed in the user interface.

[1099] (Application Example 2)

[1100] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1101] In today's educational environment, there is a need to quickly and accurately resolve questions and anxieties that arise as learners progress at their own pace. Especially with the increase in online learning, immediate question support and emotionally resonant support are crucial. However, conventional systems only provide standardized answers to user questions, failing to consider the user's emotional state. As a result, learning effectiveness has been reduced when learners experience anxiety or confusion.

[1102] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for receiving questions from a specific user, means for analyzing the received questions and user emotion data extracted from the built-in camera and voice input to extract keywords and topics related to the question content, means for generating an appropriate response according to the emotion using generative artificial intelligence based on the extracted keywords and topics and the user emotion data, and means for returning the generated response to the specific user. This enables rapid and accurate learning support that takes into account the user's emotional state.

[1103] "Means for receiving questions" refers to a device or software module for acquiring user inquiries and questions as digital data.

[1104] A "camera" is an optical device used to record a user's facial expressions and movements as video.

[1105] "Voice input" refers to a device or software module that recognizes and analyzes sounds emitted by a user as digital signals.

[1106] "Emotional data" refers to digital data that indicates the emotional state of a user, analyzed from information such as their facial expressions and voice.

[1107] "Means of analysis" refers to a device or software module used to perform analysis based on acquired data and extract meaningful information.

[1108] "Means for extracting keywords and topics" refers to devices or software modules that use natural language processing (NLP) techniques to find important words or themes from the content of a question.

[1109] "Generative artificial intelligence" refers to algorithms or models that automatically generate natural language responses or suggestions based on received data.

[1110] An "emotionally appropriate response" is a response that takes the user's emotional state into consideration and is generated in a kind and comforting tone and content.

[1111] "Means for returning the generated response" refers to a communication method or software module for delivering the generated response to the user.

[1112] "Means for formatting into a format suitable for a display device" refers to a device or software module that converts the format of the response sent back to the user so that it can be displayed clearly on the user's device.

[1113] This invention relates to a learning support system using a smart device that can instantly respond to user questions and generate and return answers tailored to the user's emotions. This system consists of multiple components, including the user's terminal, a server, generative artificial intelligence (AI), and an emotion engine.

[1114] 1. User actions

[1115] The user uses smart glasses, which have a built-in camera and microphone. The user inputs questions by voice. For example, if the user asks, "Please tell me how to solve a quadratic equation," the voice is captured by the smart glasses' microphone. The camera simultaneously analyzes the user's facial expressions and acquires emotional data.

[1116] 2. Questions and acquisition of sentiment data

[1117] The questions are converted to text using speech recognition technology (e.g., Google Cloud Speech-to-Text API). The camera uses facial recognition technology (e.g., OpenFace) to analyze the user's facial expressions. This allows for the acquisition of emotional data, such as whether the user is distressed or nervous.

[1118] 3. Sending questions and sentiment data

[1119] The terminal sends the text questions and sentiment data, converted from speech, to the server in an appropriate data format such as JSON. The server receives this and stores it in a database. Next, a text analysis module analyzes the questions using natural language processing (NLP) techniques and extracts important keywords and topics (e.g., "quadratic equation," "how to solve").

[1120] 4. Generating the answer

[1121] The server sends analysis results and sentiment data to a generative AI engine (e.g., OpenAI GPT-4) based on extracted keywords and topics. The generative AI engine generates appropriate responses based on the analysis results and further adjusts the tone and content of the responses based on the sentiment data. For example, if the user is in distress, a helpful response such as "Please calm down, I will explain slowly. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a" is generated.

[1122] 5. Return and display of responses

[1123] The generated response is sent back to the device from the server. The device formats the received response into an appropriate format and displays it on the smart glasses' display. This allows the user to see the emotionally resonant response in real time.

[1124] Specific example

[1125] For example, consider a situation where a student is solving a math problem and asks, "Can you teach me how to solve a quadratic equation?" while simultaneously looking troubled.

[1126] Example of a prompt message (input to the generation AI engine):

[1127] User's question: "How do I solve a quadratic equation?"

[1128] User's emotion: "Confused"

[1129] Question: "Explanation of how to solve quadratic equations"

[1130] Emotion-based response: "Explain in a kind and reassuring tone."

[1131] The following is an example of a response from a generative AI engine.

[1132] "Please stay calm. I will explain how to solve quadratic equations. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[1133] This invention allows users to instantly receive responses appropriate to their emotional state, thereby effectively supporting their learning.

[1134] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1135] Step 1:

[1136] The user inputs a question using smart glasses. The user asks a question by voice, such as "Tell me how to solve a quadratic equation." This voice is captured by the microphone in the smart glasses. At the same time, the camera records the user's facial expressions and acquires the video data. Thus, audio data and video data are obtained as input.

[1137] Step 2:

[1138] The device uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to convert captured audio into text. This process generates text data such as "Tell me how to solve a quadratic equation." Additionally, facial recognition technology (e.g., OpenFace) is used to analyze the user's facial expressions from the camera footage, and emotion data (e.g., confused) is extracted.

[1139] Step 3:

[1140] The terminal converts text data and sentiment data into an appropriate data format such as JSON and sends it to the server as an HTTP request. The input consists of question text and sentiment data, and the output is JSON data. The server receives this and stores it in its database.

[1141] Step 4:

[1142] On the server, a text analysis module extracts question content from JSON-formatted data and analyzes it using natural language processing (NLP) techniques. For example, keywords such as "quadratic equation" and "solution method" might be extracted. The input for this process is question data and sentiment data, and the output is the analyzed keywords and topics.

[1143] Step 5:

[1144] The server sends the analysis results and sentiment data to a generative AI engine (e.g., OpenAI GPT-4) based on the analyzed keywords and topics. The generative AI engine receives this data, constructs a prompt, and generates an appropriate response. The input consists of keywords, topics, and sentiment data, and the output is the generated response. For example, a response like "Calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a" might be generated.

[1145] Step 6:

[1146] The server receives the generated response and sends it back to the terminal. The terminal then formats the received response into a format suitable for the user's smart glasses. After the output data has been properly formatted, it is displayed on the user's screen.

[1147] Step 7:

[1148] The device displays the final answer on the smart glasses' display. This allows the user to see an emotionally resonant response in real time. The output is the displayed answer, where the user can confirm the information.

[1149] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1150] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1151] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1152] [Fourth Embodiment]

[1153] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1154] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1155] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1156] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1157] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1158] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1159] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1160] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1161] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1162] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1163] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1164] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1165] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1166] This invention relates to a learning support system that receives questions from a specific user, analyzes them, generates appropriate answers, and returns them to the user. This system consists of multiple components, including the user's terminal, a server, and generative artificial intelligence (AI).

[1167] 1. User actions

[1168] User: For example, a high school student logs into an online learning platform at home. The user asks questions about things they are unsure of or problems they cannot solve during their studies through the "User Interface (UI)". For example, they might type a question like, "Please explain how to solve quadratic equations."

[1169] 2. Receiving a question

[1170] Terminal: When the user clicks the submit button, the terminal converts the question content into a data format such as JSON, issues an HTTP request, and sends it to the server.

[1171] Server: The server receives this HTTP request and retrieves the question content. It also stores data containing user information in the database to identify which user the question is from.

[1172] 3. Analysis of the Question

[1173] Server: Next, the server passes the received question to the text analysis module. This module uses natural language processing (NLP) techniques to analyze the question and extract important keywords and topics. For example, the keywords "quadratic equation" and "solution method" might be extracted.

[1174] 4. Generating the answer

[1175] Server: Based on the extracted keywords and topics, the server sends the analysis results to a generative AI engine. The generative AI engine understands the question and generates an appropriate answer. In this case, a specific answer is generated: "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[1176] 5. Return your response

[1177] Server: Receives the generated response and formats it for return to the user. For example, it formats it to be easily viewable on the UI, and then sends an HTTP response containing the formatted data to the device.

[1178] Terminal: Analyzes response data received from the server and displays it on the user interface. Users can check the responses displayed on the terminal screen.

[1179] Specific example

[1180] For example, consider the case where a student asks, "Could you tell me the equation of a straight line?"

[1181] 1. User: A student types and submits the question, "Please tell me the equation of a straight line."

[1182] 2. Terminal: Converts the question content into JSON format and sends an HTTP request to the server.

[1183] 3. Server: Receives an HTTP request and passes the question "equation of a straight line" to the text analysis module.

[1184] 4. Text analysis module: Extracts keywords such as "straight line" and "equation," and sends the results to a generative AI engine.

[1185] 5. Generative AI engine: Generates the answer "The equation of a straight line is y = mx + b" and returns it to the server.

[1186] 6. Server: Formats the generated response into a user-friendly format for the UI and sends it to the device as an HTTP response.

[1187] 7. Terminal: Analyzes the HTTP response and displays the answer on the student's screen.

[1188] 8. User: You can check the answer displayed on the screen, "The equation of a straight line is y = mx + b," and use it to further your learning.

[1189] This invention enables effective personalized learning support because the process from when a user posts a question to when they receive an answer is carried out quickly and appropriately.

[1190] The following describes the processing flow.

[1191] Step 1:

[1192] User: After logging into the learning platform's user interface, the user enters their question in the question submission field and clicks the submit button.

[1193] Step 2:

[1194] Terminal: Retrieves the input question content and converts it to an appropriate data format such as JSON. Then, it sends this data to the server as an HTTP request.

[1195] Step 3:

[1196] Server: Receives HTTP requests and saves question data and user information to the database. Then, passes the received question content to the text analysis module.

[1197] Step 4:

[1198] Text Analysis Module: Analyzes question text using natural language processing (NLP) techniques to extract important keywords and topics. For example, it can extract keywords such as "quadratic equation" and "solution method."

[1199] Step 5:

[1200] Server: Based on extracted keywords and topics, it sends the analysis results to the generative AI engine.

[1201] Step 6:

[1202] Generative AI Engine: The generative AI engine generates appropriate answers from the analysis results it receives. For example, it can generate a specific answer such as, "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[1203] Step 7:

[1204] Server: Receives the generated response and formats it for return to the user. For example, it formats it to be easily viewable on the UI and sends the HTTP response containing the formatted data to the device.

[1205] Step 8:

[1206] Terminal: Analyzes response data received from the server and displays it on the user interface.

[1207] Step 9:

[1208] User: Check the answer displayed on the device screen. If the question is resolved, continue learning. If you have further questions, return to step 1 and enter a new question.

[1209] The above outlines the specific processing steps involved in providing answers to user questions on an online learning platform.

[1210] (Example 1)

[1211] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1212] Conventional learning support systems suffered from problems such as delays in responding to user questions and the inability to obtain appropriate answers. This resulted in reduced learning efficiency and difficulty for users to quickly obtain the necessary information.

[1213] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1214] In this invention, the server includes means for receiving questions from specific end users, means for converting the received questions into a data format and sending them to the server, means for the server to store the received question data in a database and pass it to a text analysis module, means for the text analysis module to analyze the question content using natural language processing technology and extract important keywords and topics, means for generating appropriate answers using generative artificial intelligence based on the extracted keywords and topics, and means for formatting the generated answers into a user-friendly format on the user interface and returning them to the end user. This makes it possible for users to quickly and accurately resolve questions and problems they cannot solve during learning, thereby improving learning efficiency.

[1215] An "end user" refers to a user who uses the system to ask questions.

[1216] A "server" refers to a central system that receives and processes requests from clients.

[1217] "Data format" refers to a format in which data is encoded according to certain rules. For example, the JSON format is one such format.

[1218] A "database" refers to a system that systematically stores data and allows it to be retrieved as needed.

[1219] A "text analysis module" refers to a program that uses natural language processing technology to analyze text data and extract its meaning.

[1220] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human language.

[1221] "Keywords" refer to important words or phrases extracted from the question.

[1222] "Topic" refers to the subject or theme of the question.

[1223] "Generative artificial intelligence" refers to artificial intelligence that has the function of generating appropriate answers to user questions.

[1224] "User interface" refers to the screens and input methods that end users use to interact with a system.

[1225] "Formatting" refers to the process of converting data into a format that is easy to read. This includes, for example, using HTML and CSS.

[1226] Modes for carrying out the invention

[1227] This invention relates to a learning support system that receives questions from specific end users, analyzes them, generates appropriate answers, and returns them to the user. This system consists of multiple components, including the end user's terminal, a server, and generative artificial intelligence (AI).

[1228] End users log in to the online learning platform from their homes, schools, or other locations. Users input questions and problems they encounter during their studies through the user interface (UI). For example, they might input a question like, "Please explain how to solve quadratic equations."

[1229] The terminal converts the user's input into a data format (such as JSON) and sends it to the server as an HTTP request. The JavaScript JSON.stringify method is used for this conversion. The request is sent using the HTTP POST method.

[1230] The server receives an HTTP request and retrieves the question data from the request body. The Node.js express framework is used for this process. The server then stores user information in a database (e.g., MySQL) to identify which user the question originated from. The database records the question content and associated user information.

[1231] Next, the server passes the acquired question data to the text analysis module. The text analysis module uses natural language processing (NLP) techniques to analyze the question content and extract important keywords and topics. Specifically, it uses Google's NLP API to extract keywords such as "quadratic equation" and "how to solve it."

[1232] The extracted keywords and topics are sent to a generative AI engine. This AI engine uses models such as OpenAI's GPT-3 and GPT-4. The AI ​​engine understands the question and generates an appropriate answer. In this case, a specific answer is generated: "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[1233] The generated response is formatted on the server. HTML and CSS are used for formatting, converting it into a format that is easy to read on the UI. The formatted data is then sent to the terminal as an HTTP response.

[1234] The terminal parses the response data received from the server. The JavaScript JSON.parse method is used again for this parsing. After parsing, the response is displayed in the user interface. The user can then verify the response displayed on the terminal screen.

[1235] Specific example

[1236] For example, consider a case where a student asks, "What is the equation of a straight line?" The student types the question, "What is the equation of a straight line?", and clicks the submit button. The device converts the question into JSON format and sends it to the server as an HTTP request. The server receives this request and passes the question, "What is the equation of a straight line?", to a text analysis module. The text analysis module extracts keywords such as "straight line" and "equation", and sends the result to a generative AI engine. The generative AI engine generates the answer, "The equation of a straight line is y = mx + b", and returns it to the server. The server formats the generated answer into a format that is easy to read on the UI and sends it to the device as an HTTP response. The device analyzes the HTTP response and displays the answer on the student's screen. The student can then check the answer, "The equation of a straight line is y = mx + b", displayed on the screen and use it to aid in their learning.

[1237] Example of a prompt

[1238] Please answer the following question: How do I solve a quadratic equation?

[1239] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1240] Step 1:

[1241] User: The end user logs into the online learning platform and enters questions or problems they cannot solve. For example, they might type "Please explain how to solve quadratic equations" into the text box on the user interface and click the submit button. The input data is the question in text format.

[1242] Step 2:

[1243] Terminal: The terminal converts the user-entered question content into JSON format. This conversion process uses the JavaScript JSON.stringify method. The converted data is sent to the server as an HTTP POST request. The input is the user's question text, and the output is the question data in JSON format.

[1244] Step 3:

[1245] Server: The server receives HTTP requests and retrieves question data from the request body. This process uses the Node.js express framework. It also stores data containing user information in a database (e.g., MySQL) to identify which user the question is from. The input is question data in JSON format, and the output is the user information and question data stored in the database.

[1246] Step 4:

[1247] Server: The server passes the acquired question data to the text analysis module. The text analysis module uses Google's NLP API to analyze the question content and extract important keywords and topics. The input is the question data passed from the server, and the output is a list of extracted keywords and topics.

[1248] Step 5:

[1249] Server: Based on the extracted keywords and topics, the server sends the analysis results to a generative AI engine. This generative AI engine uses models such as OpenAI's GPT-3 and GPT-4. The AI ​​engine understands the question in the format of a prompt example such as "Please answer the following question: How do you solve a quadratic equation?" and generates an appropriate answer. The input is a list of extracted keywords and topics, and the output is the generated answer text.

[1250] Step 6:

[1251] Server: Formats the generated response into a user-friendly format for the user interface. This formatting process uses HTML and CSS to convert it into a readable format. It sends the formatted data to the terminal as an HTTP response. The input is the generated response text, and the output is formatted data in HTML format.

[1252] Step 7:

[1253] Terminal: Parses the response data received from the server. This parsing process uses the JavaScript JSON.parse method. The response is then displayed in the user interface. Specifically, it displays the response using certain HTML elements (e.g., Insert the answer text into the tag. The input is answer data in JSON format, and the output is the answer text displayed in the user interface.

[1254] Step 8:

[1255] User: Checks the answer displayed on the device screen. The user can learn by looking at answers such as, for example, "The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a". The input is the text displayed on the device, and the output is the user's understanding and learning effect.

[1256] (Application Example 1)

[1257] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1258] Conventional learning support systems often struggled to provide quick and accurate answers to user questions. Furthermore, e-commerce sites, in particular, require rapid responses to product inquiries, necessitating a system that enhances the user experience. This invention aims to solve these problems by providing a system that quickly and appropriately analyzes user questions and generates and provides accurate answers using generative artificial intelligence.

[1259] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1260] In this invention, the server includes means for receiving questions from specific users, means for analyzing the received questions and extracting keywords and topics related to the question content, and means for generating appropriate answers using generative artificial intelligence based on the extracted keywords and topics. This makes it possible to provide quick and accurate answers to questions from users. Furthermore, the invention also includes means for converting the received question content into a data format and transmitting it to an information processing device, and means for providing quick and accurate answers to questions about products, which can significantly improve the user experience, especially on e-commerce sites.

[1261] "Specific users" refers to users who access the system and enter questions.

[1262] "Means for receiving questions" refers to the methods or devices used by the system to acquire questions entered by users.

[1263] "Means for extracting keywords and topics related to the content of a question" refers to methods or devices for identifying and extracting important words and themes from a received question.

[1264] "Means of generating appropriate answers using generative artificial intelligence" refers to methods or devices that use generative AI to create answers to users' questions based on extracted keywords and topics.

[1265] "Means of sending generated answers back to specific users" refers to methods or devices for sending generated answers to users.

[1266] "Information processing equipment" refers to computers and servers used for processing data.

[1267] "Means of providing prompt and accurate answers to questions about products" refers to methods or devices that provide prompt and accurate answers to user inquiries about goods or products.

[1268] "Means of converting to data format" refers to methods or devices for converting received questions and answers into a standardized data format such as JSON.

[1269] This invention relates to a learning support system that receives questions from specific users, analyzes them, generates appropriate answers, and sends them back to the users, as well as an answer generation system for e-commerce websites. This system consists of multiple components, including the user's terminal, a server, and generative artificial intelligence (AI).

[1270] 1. User actions

[1271] User: For example, a user of an online shopping site uses a smartphone app to input a product-related question into a chatbot. The user inputs a question through the "User Interface (UI)," such as, "Please tell me how to set up the remote control for this TV."

[1272] 2. Receiving a question

[1273] Terminal: When the user clicks the submit button, the terminal converts the question content into a data format such as JSON, issues an HTTP request, and sends it to the server.

[1274] Server: The server receives this HTTP request and retrieves the question content. It also stores data, including user information, in the database to identify which user the question is from.

[1275] 3. Analysis of the Question

[1276] Server: Next, the server passes the received question to the text analysis module. This module uses natural language processing (NLP) techniques to analyze the question and extract important keywords and topics. For example, the keywords "television" and "remote control settings" might be extracted.

[1277] 4. Generating the answer

[1278] Server: Based on the extracted keywords and topics, the server sends the analysis results to the generative AI engine. The generative AI engine understands the question and generates an appropriate answer. In this case, a specific answer is generated: "To set up the remote control, press and hold the button on the back of the remote control for 3 seconds, then follow the instructions displayed on the TV screen."

[1279] 5. Return your response

[1280] Server: Receives the generated response and formats it for return to the user. For example, it formats it to be easily viewable on the UI, and then sends an HTTP response containing the formatted data to the device.

[1281] Terminal: Analyzes response data received from the server and displays it on the user interface. Users can check the responses displayed on the terminal screen.

[1282] Hardware and software to be used

[1283] Hardware: Smartphone (user device), Server (API server)

[1284] Software: Flask (Python web framework), NLP module, generative AI engine

[1285] Data processing and data calculation

[1286] NLP Module: Analyzes the question content using natural language processing techniques and extracts important keywords. Specifically, it uses libraries such as spaCy and NLTK.

[1287] Generative AI Engine: Based on extracted keywords, it utilizes generative AI models such as GPT-3 to generate answers. This ensures that answers are provided quickly and accurately.

[1288] Specific example

[1289] For example, if a user of an online shopping site asks, "What is the capacity of this refrigerator?", the answer will be generated using the following steps.

[1290] User: Type "What is the capacity of this refrigerator?" into the chatbot on their smartphone and send it.

[1291] Terminal: Sends the question to the server.

[1292] Server: Receives an HTTP request and extracts the keywords "refrigerator" and "capacity".

[1293] Generative AI engine: Generates the answer "This refrigerator has a capacity of 300 liters."

[1294] Server: Send response.

[1295] Terminal: Display the answer.

[1296] Example of a prompt

[1297] "Please tell me how to set up the remote control for this TV."

[1298] "What's the capacity of this refrigerator?"

[1299] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1300] Step 1: Enter user questions

[1301] Subject: User

[1302] Specific operation: The user uses a smartphone app to type a question into the chatbot and clicks the send button.

[1303] Input: Example question entered by the user: "How do I set up the remote control for this TV?"

[1304] Output: Data processed internally by the device based on the questions entered by the user.

[1305] Step 2: Format and send the question data

[1306] Subject: terminal

[1307] Specific operation: The terminal converts the question entered by the user into JSON format and sends it to the server as an HTTP request.

[1308] Input: Question data entered by the user

[1309] Data processing: Encode the question data into JSON format.

[1310] Output: Question data in HTTP request format

[1311] Step 3: Receiving and analyzing questions

[1312] Subject: Server

[1313] Specific operation: The server receives an HTTP request and retrieves the question content. Then, it passes the question content to a text analysis module, which uses natural language processing (NLP) techniques to extract important keywords and topics.

[1314] Input: Question data in JSON format

[1315] Data processing: Extract question content as text data and perform NLP analysis.

[1316] Output: Keywords such as "TV" and "remote control settings"

[1317] Step 4: Generating the answer

[1318] Subject: Server

[1319] Specific operation: The server sends the extracted keywords to a generative AI engine, which then generates an appropriate response.

[1320] Input: Extracted keywords

[1321] Data processing: Answer generation using generative AI models (e.g., GPT-3)

[1322] Output: Response data: "To set up the remote control, press and hold the button on the back of the remote control for 3 seconds, then follow the instructions displayed on the TV screen."

[1323] Step 5: Format and return the response data.

[1324] Subject: Server

[1325] Specific operation: The server receives the generated response, formats it into a format easily displayed in the user interface, and then sends it to the terminal as an HTTP response.

[1326] Input: Generated response data

[1327] Data processing: Formatting of response data (converting to a format suitable for the UI)

[1328] Output: HTTP response containing formatted response data

[1329] Step 6: Display the answer

[1330] Subject: terminal

[1331] Specific operation: The terminal analyzes the response data received from the server and displays it on the user interface. The user can then check the response displayed on the terminal screen.

[1332] Input: HTTP response containing formatted response data

[1333] Data processing: Decoding response data and converting it for UI display.

[1334] Output: The answer displayed on the user's smartphone screen.

[1335] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1336] This invention relates to a learning support system that receives questions from a specific user, analyzes them, generates appropriate answers, and sends them back to the user. This system consists of multiple components, including the user's terminal, a server, generative artificial intelligence (AI), and an emotion engine. The system can also recognize the user's emotions and provide appropriate responses based on those emotions.

[1337] 1. User actions

[1338] User: The user logs into the learning platform. The user asks questions about things they are unsure of or problems they cannot solve during their studies through the "User Interface (UI)". For example, they might type a question like, "Please explain how to solve quadratic equations."

[1339] 2. Receiving questions and recognizing emotions

[1340] Terminal: When the user clicks the send button, the terminal retrieves the question and uses its built-in camera and microphone to analyze the user's emotions from their facial expressions and voice. Facial analysis determines, for example, whether the user is troubled or nervous.

[1341] 3. Sending questions and sentiment data

[1342] Terminal: Converts the question content into an appropriate data format such as JSON, and sends it to the server by issuing an HTTP request along with sentiment data.

[1343] Server: The server receives HTTP requests and stores question data, sentiment data, and user information in the database. It then passes the received question content to the text analysis module.

[1344] 4. Analysis of the Questions

[1345] Text Analysis Module: This module analyzes the question text using natural language processing (NLP) techniques to extract important keywords and topics. For example, it might extract keywords such as "quadratic equation" and "solution method."

[1346] 5. Generating the answer

[1347] Server: Based on extracted keywords and topics, it sends analysis results and sentiment data to the generative AI engine.

[1348] Generative AI Engine: The generative AI engine generates appropriate answers from the analysis results it receives. It also adjusts the answers based on sentiment data. For example, if the user is confused, it will generate a helpful answer such as, "Please calm down, I will explain slowly. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[1349] 6. Return your response

[1350] Server: Receives the generated response and formats it for return to the user. For example, it formats it into a format that is easy to read on the UI and sends an HTTP response containing the formatted data to the device.

[1351] 7. Display the answer

[1352] Terminal: Analyzes response data received from the server and displays it on the user interface. Users can check the responses displayed on the terminal screen.

[1353] Specific example

[1354] For example, consider a case where a student asks, "Could you please explain the equation of a straight line?" while simultaneously looking troubled.

[1355] User: A student types and submits the question, "Please tell me the equation of a straight line."

[1356] Terminal: It retrieves the question content and analyzes the student's facial expressions using the built-in camera. If the student appears to be struggling, the analyzed data is retrieved. An HTTP request containing this data is sent to the server.

[1357] Server: Receives HTTP requests, stores question and sentiment data, and passes it to the text analysis module.

[1358] Text analysis module: Extracts the keywords "straight line" and "equation" and sends the results to a generative AI engine.

[1359] Generative AI engine: Based on the question and sentiment data, it generates a helpful answer such as, "Don't worry, the equation of a straight line is y = mx + b."

[1360] Server: Formats the generated response and sends it to the terminal as an HTTP response.

[1361] Terminal: Analyzes the HTTP response and displays the answer on the student's screen.

[1362] User: You can check the answer displayed on the screen, "Don't worry. The equation of a straight line is y = mx + b," and use that information to further your learning.

[1363] This invention enables a swift and appropriate process from the time a user posts a question until they receive an answer, thereby effectively providing personalized learning support in an emotionally resonant manner.

[1364] The following describes the processing flow.

[1365] Step 1:

[1366] User: Log in to the learning platform's user interface. The user enters the question "Please tell me the quadratic formula" in the question submission field and clicks the submit button.

[1367] Step 2:

[1368] Terminal: It acquires the input content of questions and collects emotional data from the user's facial expressions and voice using the built-in camera and microphone. For example, it determines whether the user is confused or nervous through image analysis and voice emotion analysis.

[1369] Step 3:

[1370] Terminal: Converts the question content and sentiment data into JSON format and sends it to the server as an HTTP request.

[1371] Step 4:

[1372] Server: Receives HTTP requests and stores question data, sentiment data, and user information in the database. Passes the received question content to the text analysis module.

[1373] Step 5:

[1374] Text Analysis Module: Analyzes question text using natural language processing (NLP) techniques to extract important keywords and topics. For example, it extracts keywords such as "quadratic equation" and "quadratic formula."

[1375] Step 6:

[1376] Server: Based on extracted keywords, topics, and sentiment data, it sends the analysis results to a generative AI engine.

[1377] Step 7:

[1378] Generative AI Engine: Generates appropriate responses from the input analysis results and sentiment data. For example, if it determines that the user is confused, it will supplement the response with kind words. It might generate a response like, "Please calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[1379] Step 8:

[1380] Server: Receives the generated response and formats it for return to the user. For example, it formats it into a format that is easy to read on the UI and sends an HTTP response containing the formatted data to the device.

[1381] Step 9:

[1382] Terminal: Analyzes response data received from the server and displays it on the user interface.

[1383] Step 10:

[1384] User: Checks the answer displayed on the device screen. Seeing the answer, "Calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a," the user's question is resolved and they can continue learning.

[1385] The above outlines the specific processing steps from questioning to answer presentation in an online learning support system that incorporates an emotion engine.

[1386] (Example 2)

[1387] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1388] Conventional learning support systems often suffered from inconsistent accuracy in answering user-submitted questions and difficulty in considering user emotions. As a result, users may not receive adequate support when they are struggling or stressed, potentially leading to decreased learning effectiveness. Furthermore, providing quick and accurate answers to complex problems was challenging.

[1389] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a question from a specific user, means for analyzing the content of the received question and extracting keywords and topics related to the question, means for generating an appropriate answer using natural language processing technology based on the extracted keywords and topics, means for analyzing the user's facial expressions and voice to acquire emotional data, and means for returning the generated answer to the specific user. This makes it possible to provide a quick and accurate answer in a way that is sensitive to the user's emotions.

[1390] "Specific user" refers to an individual user who can be uniquely identified.

[1391] "Analyzing the question content" refers to the process of breaking down the question received from the user and understanding its meaning.

[1392] "Keyword and topic extraction" refers to identifying important words, phrases, and themes from a question.

[1393] "Natural language processing technology" is the technology that enables computers to understand, analyze, and generate human language.

[1394] "Generating appropriate answers" means creating accurate and useful answers to user questions based on extracted keywords and topics.

[1395] "Analyzing user facial expressions and voice to acquire emotional data" means using cameras and microphones to perform feature analysis on the user's facial expressions and voice to identify their emotional state.

[1396] "Returning the answer to a specific user" means sending the generated answer to the relevant user.

[1397] "Adjusting responses based on emotional data" means optimizing the content and tone of responses according to the analyzed emotional state.

[1398] "Formatting answers" means formatting answers into a user-friendly format, making them visually easy to read.

[1399] This invention relates to a learning support system that receives questions from a specific user, analyzes them, generates appropriate answers, and sends them back to the user. This system consists of multiple components, including the user's terminal, a server, generative artificial intelligence (AI), and an emotion engine. The system can also recognize the user's emotions and provide appropriate responses based on those emotions.

[1400] The following describes specific embodiments for carrying out the invention.

[1401] Hardware and software configuration

[1402] 1. User's device:

[1403] Built-in camera, microphone, user interface (UI)

[1404] Software: OpenCV (for facial expression analysis), TensorFlow (for speech analysis)

[1405] 2. Server:

[1406] Database (for storing question data and sentiment data)

[1407] Text analysis module (using natural language processing technology)

[1408] Generative AI engine (uses a generative AI model)

[1409] System operation

[1410] The detailed operation of this system is as follows:

[1411] User actions

[1412] User: The user logs into the learning platform and enters a question through the "User Interface (UI)". For example, they might enter "Please teach me how to solve quadratic equations" and click the submit button. This action records the question content along with the user's facial expressions and voice.

[1413] Receiving questions and recognizing emotions

[1414] Terminal: The terminal detects the click of the send button and retrieves the question content. It also uses its built-in camera and microphone to analyze the user's facial expressions and voice in real time. For example, it uses OpenCV to recognize the user's facial expressions and TensorFlow to extract emotion data from the audio.

[1415] Sending questions and sentiment data

[1416] Terminal: Converts the question content into JSON format and sends it to the server as an HTTP request, including emotion data. At this time, the question content, facial expression data, and voice analysis results are combined into a single JSON object.

[1417] Question analysis

[1418] Server: The server receives HTTP requests and stores question data, sentiment data, and user information in the database. Next, it analyzes the question content using a text analysis module and extracts important keywords and topics. For example, it might extract keywords such as "quadratic equation" and "how to solve it."

[1419] Generating an answer

[1420] Server: Sends extracted keywords, topics, and sentiment data to a generative AI engine.

[1421] Generative AI Engine: The generative AI engine generates appropriate responses based on analysis results and sentiment data. Taking sentiment data into consideration, for example, if the user is distressed, it will generate a helpful response such as, "Please calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[1422] Send back your response

[1423] Server: Receives the generated response and formats it into a format that is easy to read on the UI. Sends the HTTP response containing the formatted data to the terminal.

[1424] Display the answer

[1425] Terminal: The terminal analyzes the response data received from the server and displays it on the user interface. Users can review the responses displayed on the terminal screen and use them for learning.

[1426] Specific example

[1427] For example, consider a case where a student asks, "Could you please explain the equation of a straight line?" while simultaneously looking troubled.

[1428] Example of a prompt:

[1429] "Please tell me the equation of a straight line."

[1430] In this case, the system will operate as follows:

[1431] User: A student types and submits the question, "Please tell me the equation of a straight line."

[1432] Terminal: It retrieves the question content and analyzes the student's facial expressions, including any signs of distress, using its built-in camera. It then sends an HTTP request to the server containing this data.

[1433] Server: Receives HTTP requests, stores question and sentiment data, and has a text analysis module analyze the keywords "straight line" and "equation".

[1434] Generative AI engine: Based on the question content and sentiment data, it generates a helpful answer such as, "Don't worry. The equation of a straight line is y = mx + b."

[1435] Server: Formats the generated response and sends it to the terminal as an HTTP response.

[1436] Terminal: Analyzes the HTTP response and displays the answer on the student's screen.

[1437] User: You can check the answer displayed on the screen, "Don't worry. The equation of a straight line is y = mx + b," and use that information to further your learning.

[1438] As a result, by using this invention, users can receive quick and accurate answers that are empathetic to their emotions, thereby enhancing the learning effect.

[1439] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1440] Step 1:

[1441] User: The user logs into the learning platform. After logging in, they enter their question through the user interface (UI). For example, they might enter "Please teach me how to solve quadratic equations" and click the submit button.

[1442] Input: User ID and question.

[1443] Output: Login session and question content.

[1444] Step 2:

[1445] Terminal: The terminal detects the click of the send button and retrieves the question content. Simultaneously, it records the user's facial expressions and voice using the built-in camera and microphone, and performs facial expression analysis and sentiment analysis. For example, it uses OpenCV to analyze the user's facial expressions in real time and TensorFlow to extract sentiment data from the audio.

[1446] Input: Question content and user's facial expressions and voice data.

[1447] Output: Question content and emotional data (confusion, anxiety, etc.).

[1448] Step 3:

[1449] Terminal: Converts the question content and emotion data into JSON format and sends it to the server via an HTTP request. At this time, the question content, facial expression data, and voice analysis results are combined into a single JSON object.

[1450] Input: Question content and sentiment data.

[1451] Output: Data in JSON format.

[1452] Step 4:

[1453] Server: The server receives HTTP requests and stores question data, sentiment data, and user information in the database.

[1454] Input: Data in JSON format.

[1455] Output: Question data and sentiment data stored in the database.

[1456] Step 5:

[1457] Text Analysis Module: The text analysis module analyzes the received question content and extracts important keywords and topics. For example, it might extract the keywords "quadratic equation" and "solution method" from the question text.

[1458] Input: Question content.

[1459] Output: Extracted keywords and topics.

[1460] Step 6:

[1461] Server: Sends extracted keywords, topics, and sentiment data to a generative AI engine.

[1462] Input: Extracted keywords, topics, and sentiment data.

[1463] Output: API call to a generative AI engine.

[1464] Step 7:

[1465] Generative AI Engine: The generative AI engine generates appropriate responses based on the received analysis results and sentiment data. For example, if the sentiment data indicates "confused," it will generate a helpful response such as, "Please calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[1466] Input: Keywords, topics, sentiment data.

[1467] Output: The generated answer.

[1468] Step 8:

[1469] Server: Receives the generated response and formats it for return to the user. It uses HTML and CSS to format it into a user-friendly format and sends the HTTP response containing the formatted data to the device.

[1470] Input: Generated answer.

[1471] Output: HTTP response containing formatted response data.

[1472] Step 9:

[1473] Terminal: The terminal analyzes the response data received from the server and displays it on the user interface. Users can review the responses displayed on the terminal screen and use them for learning.

[1474] Input: HTTP response from the server.

[1475] Output: The answer displayed in the user interface.

[1476] (Application Example 2)

[1477] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1478] In today's educational environment, there is a need to quickly and accurately resolve questions and anxieties that arise as learners progress at their own pace. Especially with the increase in online learning, immediate question support and emotionally resonant support are crucial. However, conventional systems only provide standardized answers to user questions, failing to consider the user's emotional state. As a result, learning effectiveness has been reduced when learners experience anxiety or confusion.

[1479] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for receiving questions from a specific user, means for analyzing the received questions and user emotion data extracted from the built-in camera and voice input to extract keywords and topics related to the question content, means for generating an appropriate response according to the emotion using generative artificial intelligence based on the extracted keywords and topics and the user emotion data, and means for returning the generated response to the specific user. This enables rapid and accurate learning support that takes into account the user's emotional state.

[1480] "Means for receiving questions" refers to a device or software module for acquiring user inquiries and questions as digital data.

[1481] A "camera" is an optical device used to record a user's facial expressions and movements as video.

[1482] "Voice input" refers to a device or software module that recognizes and analyzes sounds emitted by a user as digital signals.

[1483] "Emotional data" refers to digital data that indicates the emotional state of a user, analyzed from information such as their facial expressions and voice.

[1484] "Means of analysis" refers to a device or software module used to perform analysis based on acquired data and extract meaningful information.

[1485] "Means for extracting keywords and topics" refers to devices or software modules that use natural language processing (NLP) techniques to find important words or themes from the content of a question.

[1486] "Generative artificial intelligence" refers to algorithms or models that automatically generate natural language responses or suggestions based on received data.

[1487] An "emotionally appropriate response" is a response that takes the user's emotional state into consideration and is generated in a kind and comforting tone and content.

[1488] "Means for returning the generated response" refers to a communication method or software module for delivering the generated response to the user.

[1489] "Means for formatting into a format suitable for a display device" refers to a device or software module that converts the format of the response sent back to the user so that it can be displayed clearly on the user's device.

[1490] This invention relates to a learning support system using a smart device that can instantly respond to user questions and generate and return answers tailored to the user's emotions. This system consists of multiple components, including the user's terminal, a server, generative artificial intelligence (AI), and an emotion engine.

[1491] 1. User actions

[1492] The user uses smart glasses, which have a built-in camera and microphone. The user inputs questions by voice. For example, if the user asks, "Please tell me how to solve a quadratic equation," the voice is captured by the smart glasses' microphone. The camera simultaneously analyzes the user's facial expressions and acquires emotional data.

[1493] 2. Questions and acquisition of sentiment data

[1494] The questions are converted to text using speech recognition technology (e.g., Google Cloud Speech-to-Text API). The camera uses facial recognition technology (e.g., OpenFace) to analyze the user's facial expressions. This allows for the acquisition of emotional data, such as whether the user is distressed or nervous.

[1495] 3. Sending questions and sentiment data

[1496] The terminal sends the text questions and sentiment data, converted from speech, to the server in an appropriate data format such as JSON. The server receives this and stores it in a database. Next, a text analysis module analyzes the questions using natural language processing (NLP) techniques and extracts important keywords and topics (e.g., "quadratic equation," "how to solve").

[1497] 4. Generating the answer

[1498] The server sends analysis results and sentiment data to a generative AI engine (e.g., OpenAI GPT-4) based on extracted keywords and topics. The generative AI engine generates appropriate responses based on the analysis results and further adjusts the tone and content of the responses based on the sentiment data. For example, if the user is in distress, a helpful response such as "Please calm down, I will explain slowly. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a" is generated.

[1499] 5. Return and display of responses

[1500] The generated response is sent back to the device from the server. The device formats the received response into an appropriate format and displays it on the smart glasses' display. This allows the user to see the emotionally resonant response in real time.

[1501] Specific example

[1502] For example, consider a situation where a student is solving a math problem and asks, "Can you teach me how to solve a quadratic equation?" while simultaneously looking troubled.

[1503] Example of a prompt message (input to the generation AI engine):

[1504] User's question: "How do I solve a quadratic equation?"

[1505] User's emotion: "Confused"

[1506] Question: "Explanation of how to solve quadratic equations"

[1507] Emotion-based response: "Explain in a kind and reassuring tone."

[1508] The following is an example of a response from a generative AI engine.

[1509] "Please stay calm. I will explain how to solve quadratic equations. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a."

[1510] This invention allows users to instantly receive responses appropriate to their emotional state, thereby effectively supporting their learning.

[1511] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1512] Step 1:

[1513] The user inputs a question using smart glasses. The user asks a question by voice, such as "Tell me how to solve a quadratic equation." This voice is captured by the microphone in the smart glasses. At the same time, the camera records the user's facial expressions and acquires the video data. Thus, audio data and video data are obtained as input.

[1514] Step 2:

[1515] The device uses speech recognition technology (e.g., Google Cloud Speech-to-Text API) to convert captured audio into text. This process generates text data such as "Tell me how to solve a quadratic equation." Additionally, facial recognition technology (e.g., OpenFace) is used to analyze the user's facial expressions from the camera footage, and emotion data (e.g., confused) is extracted.

[1516] Step 3:

[1517] The terminal converts text data and sentiment data into an appropriate data format such as JSON and sends it to the server as an HTTP request. The input consists of question text and sentiment data, and the output is JSON data. The server receives this and stores it in its database.

[1518] Step 4:

[1519] On the server, a text analysis module extracts question content from JSON-formatted data and analyzes it using natural language processing (NLP) techniques. For example, keywords such as "quadratic equation" and "solution method" might be extracted. The input for this process is question data and sentiment data, and the output is the analyzed keywords and topics.

[1520] Step 5:

[1521] The server sends the analysis results and sentiment data to a generative AI engine (e.g., OpenAI GPT-4) based on the analyzed keywords and topics. The generative AI engine receives this data, constructs a prompt, and generates an appropriate response. The input consists of keywords, topics, and sentiment data, and the output is the generated response. For example, a response like "Calm down. The quadratic formula is x = (-b ± √(b^2-4ac)) / 2a" might be generated.

[1522] Step 6:

[1523] The server receives the generated response and sends it back to the terminal. The terminal then formats the received response into a format suitable for the user's smart glasses. After the output data has been properly formatted, it is displayed on the user's screen.

[1524] Step 7:

[1525] The device displays the final answer on the smart glasses' display. This allows the user to see an emotionally resonant response in real time. The output is the displayed answer, where the user can confirm the information.

[1526] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1527] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1528] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1529] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1530] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1531] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1532] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1533] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1534] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1535] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1536] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1537] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1538] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1539] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1540] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1541] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1542] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1543] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1544] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1545] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1546] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1547] The following is further disclosed regarding the embodiments described above.

[1548] (Claim 1)

[1549] A means of receiving questions from specific users,

[1550] A means of analyzing received questions and extracting keywords and topics related to the content of the questions,

[1551] A means of generating appropriate answers using generative artificial intelligence based on extracted keywords and topics,

[1552] A means of sending the generated response back to a specific user,

[1553] A system that includes this.

[1554] (Claim 2)

[1555] The system according to claim 1, wherein the generative artificial intelligence analyzes the content of a question using natural language processing technology.

[1556] (Claim 3)

[1557] The system according to claim 1, further comprising means for formatting the generated response when returning the generated response to a specific user.

[1558] "Example 1"

[1559] (Claim 1)

[1560] A means of receiving questions from specific end users,

[1561] A means of converting the received question into a data format and sending it to the server,

[1562] A means of saving the received question data to a database and passing it to a text analysis module,

[1563] The text analysis module uses natural language processing techniques to analyze the question content and extract important keywords and topics.

[1564] A means of generating appropriate answers using generative artificial intelligence based on extracted keywords and topics,

[1565] A means of formatting the generated responses into a user-friendly format on the user interface and sending them back to the end user,

[1566] A system that includes this.

[1567] (Claim 2)

[1568] The system according to claim 1, wherein the generative artificial intelligence uses a generative AI model to understand the content of a question and generate an answer.

[1569] (Claim 3)

[1570] The system according to claim 1, wherein the text analysis module analyzes a question using natural language processing techniques.

[1571] "Application Example 1"

[1572] (Claim 1)

[1573] A means of receiving questions from specific users,

[1574] A means of analyzing received questions and extracting keywords and topics related to the content of the questions,

[1575] A means of generating appropriate answers using generative artificial intelligence based on extracted keywords and topics,

[1576] A means of returning the generated response to a specific user,

[1577] A means for converting the received question content into a data format and transmitting it to an information processing device,

[1578] A means of providing quick and accurate answers to questions about products,

[1579] A system that includes this.

[1580] (Claim 2)

[1581] The system according to claim 1, wherein the generative artificial intelligence analyzes the content of a question using natural language processing technology.

[1582] (Claim 3)

[1583] The system according to claim 1, further comprising means for formatting the generated response when returning the generated response to a specific user.

[1584] "Example 2 of combining an emotion engine"

[1585] (Claim 1)

[1586] A means of receiving questions from specific users,

[1587] A means of analyzing the received question content and extracting keywords and topics related to the question,

[1588] A means of generating appropriate answers using natural language processing technology based on extracted keywords and topics,

[1589] A method for obtaining emotional data by analyzing the user's facial expressions and voice,

[1590] A means of sending the generated response back to a specific user,

[1591] A system that includes this.

[1592] (Claim 2)

[1593] The system according to claim 1, further comprising means for adjusting the response based on sentiment data when generating the generated response.

[1594] (Claim 3)

[1595] The system according to claim 1, further comprising means for formatting the generated response when returning the generated response to a specific user.

[1596] "Application example 2 of combining emotional engines"

[1597] (Claim 1)

[1598] A means of receiving questions from specific users,

[1599] A means for analyzing user sentiment data extracted from received questions and the built-in camera and voice input, and for extracting keywords and topics related to the content of the questions,

[1600] A means of generating appropriate responses that respond to emotions using generative artificial intelligence based on extracted keywords, topics, and user sentiment data,

[1601] A means of sending the generated response back to a specific user,

[1602] A system that includes this.

[1603] (Claim 2)

[1604] The system according to claim 1, characterized in that the generative artificial intelligence analyzes the content of a question using natural language processing technology and adjusts the tone and content of the response based on the user's sentiment data.

[1605] (Claim 3)

[1606] The system according to claim 1, further comprising means for formatting the generated response into a format suitable for a display device when returning it to a specific user. [Explanation of symbols]

[1607] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of receiving questions from specific users, A means of analyzing received questions and extracting keywords and topics related to the content of the questions, A means of generating appropriate answers using generative artificial intelligence based on extracted keywords and topics, A means of sending the generated response back to a specific user, A system that includes this.

2. The system according to claim 1, wherein the generative artificial intelligence analyzes the content of a question using natural language processing technology.

3. The system according to claim 1, further comprising means for formatting the generated response when returning the generated response to a specific user.

Images