System

Abstract

A system is provided.SOLUTION: The system includes a means for receiving a question about a language from a user, a means for transmitting the received question to a server, a means for causing a generation artificial intelligence to generate an answer based on the question, a means for causing the server to receive the generated answer and returning the answer to the terminal, and a means for displaying the answer received by the terminal to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional language learning methods often rely on textbooks and instructor-led lessons, making it difficult to adapt to individual learners' needs. Furthermore, there are limited ways for learners to ask questions at their own pace and receive immediate answers. This can result in a lack of motivation and limited learning outcomes. Furthermore, it is difficult to find conversation practice partners, resulting in a lack of opportunities for practical language use. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides a language learning support system that cooperates between a user, a terminal, and a server. Specifically, the system includes a means for receiving language-related questions from a user and sending them to a server, a means for having a generative artificial intelligence generate an answer based on the question, a means for the server to receive the generated answer and return it to the terminal, and a means for the terminal to display the answer received by the user. This system allows users to ask questions at their own convenience and receive appropriate answers instantly. Furthermore, by using generative artificial intelligence, it is possible to provide not only advice on grammar and expressions but also scenarios for conversation practice, thereby realizing a practical language learning environment.

[0006] A "question" is text entered by a user requesting information or advice about a language.

[0007] "Terminal" refers to a device operated by a user, including smartphones, tablets, PCs, etc.

[0008] A "server" is a computer system that receives a user's question, sends the question to a generative artificial intelligence, and transmits the generated answer to a terminal.

[0009] "Generative AI" is an AI technology that uses natural language processing to generate appropriate answers to user questions.

[0010] "Answer" refers to advice or information generated by generative artificial intelligence based on a user's question.

[0011] "Display" refers to the act of the device visually providing the user with the answer obtained from the generative artificial intelligence.

[0012] A "scenario" is a combination of dialogue-style sentences generated by generative artificial intelligence for conversation practice. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention provides a system for supporting language learning through collaboration between a user, a terminal, a server, and a generative artificial intelligence. The system includes a process in which a user asks a language-related question and a generative artificial intelligence provides an answer to the question.

[0035] System configuration

[0036] Users operate language learning apps using devices such as smartphones, tablets, and PCs. When a user inputs a language-related question, the question is sent from the device to a server. The server analyzes the question, queries a generative artificial intelligence (e.g., a natural language processing model), and generates an answer. The generated answer is sent back to the device via the server, and the device displays it to the user.

[0037] Specific examples

[0038] For example, consider a user asking a question about English grammar: "How do I say 'I saw a movie yesterday' in English?"

[0039] 1. User enters question

[0040] Users enter their question into the app's text box and press the send button.

[0041] 2. Send the question to the server

[0042] The terminal receives the entered question, converts it into a specific format (e.g., JSON format), and sends an HTTP request to the server.

[0043] 3. The server analyzes the question and queries the generative AI

[0044] The server analyzes the received question and sends it to the generative AI in an appropriate format. The generative AI receives the question in a form such as "How do you say 'I watched a movie yesterday' in English?"

[0045] 4. Generative AI generates answers

[0046] Generative AI generates answers based on the information you submit, for example, "You can say 'I watched a movie yesterday.'"

[0047] 5. Send the answer back to the device

[0048] The server receives the response from the generative AI, converts it into a format that is easy for the user to understand if necessary, and sends it back to the device.

[0049] 6. Show the answer to the user

[0050] The device displays the answer received from the server to the user, specifically, "You can say 'I watched a movie yesterday.'"

[0051] The system of the present invention allows users to ask language-related questions anytime, anywhere and get instant, relevant answers. It also provides grammar and expression advice as well as dialogue scenarios for conversation practice. This feature allows users to experience a practical language learning environment.

[0052] The processing flow will be explained below.

[0053] Step 1:

[0054] A user opens the app and types in a language-related question, such as "How do I say 'I saw a movie yesterday' in English?"

[0055] Step 2:

[0056] The device receives the user's input and detects that the send button has been pressed.

[0057] Step 3:

[0058] The device sends the entered question to the server in a specific format (e.g., JSON format). For example, the JSON data {"question": "How should I say 'I saw a movie yesterday' in English?"} is sent via an HTTP POST request.

[0059] Step 4:

[0060] The server receives the request from the device and analyzes the question, converting it into an easy-to-understand form.

[0061] Step 5:

[0062] The server sends the analyzed question to the generative AI. For example, the server sends a question to the generative AI in the form of "How do you say 'I watched a movie yesterday' in English?"

[0063] Step 6:

[0064] The generative AI generates an answer based on the question received from the server. An example of a generated answer would be "You can say 'I watched a movie yesterday.'"

[0065] Step 7:

[0066] The server receives the generated answer and formats the answer as needed, including translating the answer into the user's language.

[0067] Step 8:

[0068] The server returns a formatted answer to the device. For example, it sends the JSON data {"answer": "Just say 'I watched a movie yesterday'"} as an HTTP response.

[0069] Step 9:

[0070] The device analyzes the response it receives from the server and prepares it for display to the user, converting it into a format suitable for display on the screen.

[0071] Step 10:

[0072] The device displays the analyzed answer in the user interface. For example, it displays "You can say 'I watched a movie yesterday.'"

[0073] Example 1

[0074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0075] Current language learning systems have a complex process for quickly and accurately solving language-related questions, resulting in a poor user experience. Furthermore, when using generative AI to provide answers, there is a lack of a way to accurately analyze the user's input and generate appropriate prompts, resulting in issues with the quality of the generated answers. Furthermore, there is a lack of a way to convert the generated answers into a format that is easily understandable to the user. Therefore, improvements are needed to promote effective and efficient language learning.

[0076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0077] In this invention, the server includes a device for receiving language-related questions from a user, a device for converting the received questions into a specific data format and sending them to the server, a device for analyzing the received questions, generating prompt sentences based on a generative AI model, and sending them to the generative AI model, a device for receiving answers from the generative AI model, converting them into a format that is easy for the user to understand, and returning the answers to the terminal, and a device for displaying the answers received by the terminal to the user. This makes it possible to accurately analyze the content of the questions entered by the user and generate appropriate prompt sentences. Furthermore, by converting the generated answers into a format that is easy for the user to understand and providing them, it is possible to improve the user experience and enable effective language learning.

[0078] A "user" is an entity that utilizes the system to enter language-related questions and receive answers.

[0079] A "terminal" is a device that a user operates to input questions and display answers, and includes common electronic devices such as smartphones, tablets, and PCs.

[0080] A "server" is a device that receives a question sent from a terminal, analyzes the question, generates an answer using a generative AI model, and returns the answer to the terminal.

[0081] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to generate appropriate answers to user questions.

[0082] A "prompt sentence" is an input sentence used to generate an answer for a generative AI model, and is an appropriate translation of the user's question.

[0083] "Device" is a general term that refers to a hardware or software component that performs a particular function or role.

[0084] A "data format" is a way of representing information based on a specific structure or rules, and includes specific examples such as JSON and XML.

[0085] An "answer" is information generated by a generative AI model and provided in response to a user's question.

[0086] This invention is a system that supports language learning through collaboration between a user, a terminal, a server, and a generative AI model. The operation of each component and the overall system process are described in detail below.

[0087] System Overview

[0088] Users operate language learning apps using devices such as smartphones, tablets, and PCs. The devices receive user questions, convert them into a specific data format (e.g., JSON format), and send them to a server. The server analyzes the received questions, generates prompts based on a generative AI model, and sends them to the generative AI model. The generative AI model generates answers based on the sent prompts and sends them back to the server. The server converts the generated answers into a format that is easy for the user to understand and sends them back to the device, where the device displays the answers to the user.

[0089] Hardware and software used

[0090] Device: A device on which a user enters questions and displays answers. Examples include smartphones, tablets, and PCs.

[0091] Server: A back-end system that receives questions, analyzes them, and sends prompts to the generative AI model. The server is implemented in a common programming language such as Python or Java (registered trademark).

[0092] Generative AI model: A natural language processing model that generates answers to user questions. Examples include OpenAI® GPT-3®.

[0093] Communication method: A method for exchanging data between the device and the server, and between the server and the generative AI model. HTTP / HTTPS protocols are commonly used.

[0094] Specific operation explanation

[0095] 1. User enters language question

[0096] A user launches a language learning app and logs in. For example, they enter a question about English learning into the text box and press the "Submit" button. For example, they might enter, "How do I say 'I saw a movie yesterday' in English?"

[0097] 2. The device sends the question to the server

[0098] The terminal converts the question entered by the user into JSON format and sends it to the server using an HTTP POST request.

[0099] 3. The server analyzes the question and sends it to the generative AI model as a prompt.

[0100] The server analyzes the data received from the device and extracts the question. It then converts the extracted question into an appropriate format (e.g., English) and generates a prompt for the generative AI model. For example, it generates a prompt such as, "How do you say 'I watched a movie yesterday' in English?"

[0101] 4. Generative AI model generates answers

[0102] The generative AI model generates an answer based on the prompt it receives, for example, "You can say 'I watched a movie yesterday.'"

[0103] 5. The server returns the answer to the device

[0104] The server receives the answer from the generative AI model, formats it into a form that is easy for the user to understand, for example, "You can say 'I watched a movie yesterday'," and sends it back to the device.

[0105] 6. The device displays the answer to the user

[0106] The device displays the answer received from the server to the user. Specifically, it displays "You can say 'I watched a movie yesterday'" in the text area on the screen.

[0107] Prompt Sentence Examples

[0108] How do you say 'I watched a movie yesterday' in English?

[0109] In this way, the system of the present invention allows users to ask language-related questions anytime and anywhere and receive appropriate answers immediately, allowing users to progress with language learning effectively and efficiently.

[0110] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0111] Step 1:

[0112] Process Description: User enters a language question.

[0113] Specific operation: The user starts the application, logs in, enters a question in the text box, and presses the "Submit" button.

[0114] Input: User question (e.g., "How do I say 'I saw a movie yesterday' in English?")

[0115] Output: The action of the user pressing the submit button

[0116] Step 2:

[0117] Process description: The device sends a query to the server.

[0118] Specific operation: The terminal converts the text entered by the user into JSON format and sends it to the server via an HTTP POST request.

[0119] Input: User question

[0120] Output: JSON data sent to the server (e.g., {"question": "How do I say 'I saw a movie yesterday' in English?"})

[0121] Step 3:

[0122] Process description: The server parses the question and sends a prompt to the generative AI model.

[0123] How it works: The server parses the received JSON data, extracts the question, converts it into an appropriate format (e.g., generates an English prompt, "How do you say 'I watched a movie yesterday' in English?"), and creates a request to send to the generative AI model.

[0124] Input: JSON data sent from the terminal

[0125] Output: The prompt sent to the generative AI model (e.g., "How do you say 'I watched a movie yesterday' in English?")

[0126] Step 4:

[0127] Process Description: The generative AI model generates an answer.

[0128] How it works: The generative AI model receives a prompt as input and uses natural language processing techniques to generate an answer, which is then sent back to the server.

[0129] Input: Prompt text sent from the server

[0130] Output: The answer returned by the generative AI model (e.g., "You can say 'I watched a movie yesterday.'")

[0131] Step 5:

[0132] Process description: The server sends a response back to the device.

[0133] Specific operation: The server receives the answer from the generative AI model, converts it into a format that is easy for the user to understand, generates JSON data containing the converted answer, and sends it to the terminal as an HTTP response.

[0134] Input: The answer returned by the generative AI model

[0135] Output: JSON data sent to the device (e.g., {"answer": "You can say 'I watched a movie yesterday'."})

[0136] Step 6:

[0137] Process Description: The device displays the answer to the user.

[0138] Specific operation: The device parses the JSON data received from the server, extracts the answer, and displays it in the text area on the screen.

[0139] Input: JSON data sent from the server

[0140] Output: The answer that is displayed to the user (e.g., "You can say 'I watched a movie yesterday'.")

[0141] These steps will help users progress through language learning efficiently and effectively.

[0142] (Application example 1)

[0143] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0144] In modern content distribution services, when users encounter a word or phrase they don't understand while watching, they need a way to quickly ask questions and receive answers on the spot. However, existing systems lack the functionality to answer questions in real time while watching, which reduces user learning efficiency. Furthermore, it is difficult to implement advanced natural language processing technology that can provide appropriate answers to questions.

[0145] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0146] In this invention, the server includes means for receiving questions about information from users, means for transmitting the received questions to a generative artificial intelligence, means for the generative artificial intelligence to generate answers based on the questions, means for the server to receive the answers and return them to the display device, means for the display device to display the received answers to the user, and means for the user to ask questions about the content being displayed in real time, thereby enabling users to instantly resolve any questions that arise while viewing, and to learn efficiently.

[0147] "User" refers to any individual or entity that uses the System.

[0148] "Information" refers to the knowledge and data that users ask about, such as language, grammar, and methods of expression.

[0149] "Question" refers to the inquiry a user enters into the system to confirm the information they are seeking.

[0150] "Network equipment" refers to devices used to send and receive data, such as servers and routers.

[0151] "Generative artificial intelligence" refers to a system that has natural language processing capabilities and generates appropriate answers to input questions.

[0152] "Answer" refers to appropriate information or data generated by generative artificial intelligence in response to a user's question.

[0153] "Display device" refers to a device on which the user can view the answers, such as a smartphone, tablet, or computer.

[0154] "Content" refers to media for viewing provided by a distribution service, such as movies, dramas, and news.

[0155] "Real-time" refers to minimal delay between the moment a user asks a question and the moment a response is received.

[0156] "Learning" refers to the process by which a user acquires new knowledge, such as language or grammar.

[0157] The present invention relates to a language learning system for content distribution services that allows users to ask questions in real time about words or phrases they do not understand while watching, and provides answers using generative artificial intelligence.

[0158] This system is configured as follows:

[0159] System configuration

[0160] A user inputs a question via their device (smartphone, tablet, PC, etc.) and sends it to the server. At this time, network equipment is responsible for sending and receiving the data. The question received by the server is sent to a generative artificial intelligence, which generates an appropriate answer. The generated answer is then sent back to the display device via the server and displayed to the user.

[0161] Program processing

[0162] The hardware and software used are as follows:

[0163] Hardware: Servers (e.g., AWS (registered trademark) EC2), user devices (smartphones, tablets, PCs)

[0164] Software: Flask (server), external natural language processing API (e.g., GPT-3)

[0165] The flow of data processing and data calculation is as follows:

[0166] 1. The user inputs a question from the device. For example, "The user asked a question about a specific scene while watching a movie. Question: 'What is the concept of "dream within a dream" in the movie "Inception"?'"

[0167] 2. The device sends the entered question to the server via a network device. The data is sent in a standard format such as JSON.

[0168] 3. The server sends the received question to the generative AI, which generates an answer based on the received question. For example, it generates an answer such as, "The concept of 'dream within a dream' refers to multiple layers of dreams where each layer is a dream within the previous one."

[0169] 4. The server receives the generated answer and, if necessary, converts it into a user-friendly format.

[0170] 5. The converted answer is sent back from the server to the display device and displayed to the user. Specifically, it is displayed in the form of "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."

[0171] Adding specific examples

[0172] For example, a system that answers questions about scenes in a movie a user is watching in real time might use the following prompt:

[0173] "A user is watching a movie and has a question about a particular scene. Please answer the following question: Question: 'What is the concept of "dream within a dream" in the movie "Inception"?'"

[0174] This invention allows users to instantly resolve any questions they may have while watching, providing an efficient learning experience.

[0175] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0176] Step 1:

[0177] The user inputs a question from the device. The user enters a question about the content they are watching in text format into the device's input form and presses the send button. For example, the user might input a question such as, "What is the concept of 'dream within a dream' in the movie 'Inception'?" The input at this time is treated as text data.

[0178] Step 2:

[0179] The device sends the entered question to the server via a network device. The device converts the question into JSON format and sends the data to the server using an HTTP request. Specifically, data in the format {"question": "What is the concept of 'dream within a dream' in the movie 'Inception'?"} is sent.

[0180] Step 3:

[0181] The server parses the received question and sends it to the generative AI in an appropriate format. The server receives the JSON-formatted data, extracts the contents of the "question" field, and sends an API request to the generative AI. At this time, a prompt is formed, and the body of the API request includes a prompt such as, "A user asked a question about a specific scene while watching a movie. Please answer the following question. Question: 'What is the concept of "dream within a dream" in the movie "Inception"?'"

[0182] Step 4:

[0183] The generative AI analyzes the received query and generates an answer to the question. The generative AI uses a natural language processing model to analyze the prompt sentence and generate an appropriate response. For example, an answer may be generated in the form of "The concept of 'dream within a dream' refers to multiple layers of dreams where each layer is a dream within the previous one." This answer is sent back to the server in JSON format.

[0184] Step 5:

[0185] The server receives the generated answer and, if necessary, converts it into a format that is easy for the user to understand. The server analyzes the answer received from the generative AI and converts it into language and expressions that are easy for the user to understand. Specifically, it performs processing such as translating the generated English answer into Japanese. As a result, for example, the answer may be converted into a form such as, "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."

[0186] Step 6:

[0187] The server returns the converted answer to the device. The server then converts the answer back into JSON format and sends it to the device as an HTTP response. For example, data such as {"answer": "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."} is sent.

[0188] Step 7:

[0189] The device displays the received answer to the user. The device parses the received JSON data, extracts the contents of the "answer" field, and displays it on the screen. This allows the user to instantly resolve any questions they may have while watching. For example, the screen displays the following: "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."

[0190] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0191] The present invention is a system for supporting language learning through collaboration between a user, a terminal, a server, a generative artificial intelligence, and an emotion engine. The system includes a process in which a user asks a language-related question, a generative artificial intelligence provides an answer to the question, and the system further adjusts the answer by recognizing the user's emotion.

[0192] System configuration

[0193] A user operates a language learning app using a device such as a smartphone, tablet, or PC. When the user enters a language-related question, the question is sent from the device to a server. The server analyzes the question and recognizes the user's emotion using an emotion engine. The analyzed question and the recognized emotion are sent to a generative artificial intelligence (e.g., a natural language processing model) to generate an answer. The answer is sent back to the device via the server, and the device displays it to the user.

[0194] Specific examples

[0195] For example, consider a case where a user has a question about English grammar. The user might type, "How do I say 'I saw a movie yesterday' in English?" The emotion engine then analyzes the emotion from the text and voice input by the user.

[0196] 1. User enters question

[0197] The user enters a question into the app's text box and presses the submit button. For example, "How do I say 'I saw a movie yesterday' in English?" The emotion engine simultaneously analyzes the user's input text and identifies the emotion.

[0198] 2. Send the question to the server

[0199] The terminal receives the entered question, converts it into a specific format (e.g., JSON format), adds the analysis results of the emotion engine, and sends an HTTP request to the server.

[0200] 3. The server queries the generative AI with the question and emotion

[0201] The server sends the received question and the emotion analysis results to the generative AI, which receives the form "How do you say 'I watched a movie yesterday' in English?" and the user's emotional state.

[0202] 4. Generative AI generates answers

[0203] Generative AI generates answers based on the information sent. For example, if there is emotional information that the user is confused, the answer will be gentler. An example of a generated answer would be a response that includes gentle advice such as, "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!"

[0204] 5. Send the answer back to the device

[0205] The server receives the answer from the generative AI, converts it into a format that is easy for the user to understand if necessary, and sends it back to the device. For example, it sends JSON data such as {"answer": "Just say 'I watched a movie yesterday'. Don't worry, this is easy."} as an HTTP response.

[0206] 6. Show the answer to the user

[0207] The device analyzes the answer received from the server and displays it to the user. Specifically, it displays the message "Just say 'I watched a movie yesterday'. Don't worry, it's easy."

[0208] The system of the present invention allows users to ask language-related questions anytime, anywhere and receive immediate, appropriate answers. Furthermore, by using generative artificial intelligence, the system can provide advice on grammar and expressions, as well as provide answers that take the user's emotions into consideration by using an emotion engine. This reduces learning stress for users and allows them to learn languages ​​more effectively.

[0209] The processing flow will be explained below.

[0210] Step 1:

[0211] A user opens the app and types in a language-related question, such as "How do I say 'I saw a movie yesterday' in English?"

[0212] Step 2:

[0213] The device receives the user's input and detects when the send button is pressed. At the same time, the device sends the user's input text or voice data to the emotion engine to analyze the user's emotions.

[0214] Step 3:

[0215] The device sends the entered question to the server in a specific format (e.g., JSON format). For example, the following JSON data is sent via an HTTP POST request: {"question": "How should I say 'I saw a movie yesterday' in English?", "emotion": "confused"}

[0216] Step 4:

[0217] The server receives requests from the device and analyzes the question and emotional information, converting the user's question and emotional state into an easy-to-understand format.

[0218] Step 5:

[0219] The server sends the analyzed question and emotional information to the generative AI. For example, it sends the question in the form "How do you say 'I watched a movie yesterday' in English?" along with the emotional information "confused."

[0220] Step 6:

[0221] The generative AI generates an answer based on the question and emotional information received from the server. For example, if the user is in a "confused" emotional state, it generates an answer with gentle advice in the form of "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!"

[0222] Step 7:

[0223] The server receives the generated answer and formats it as needed, for example translating it into the user's language to make it easier to understand.

[0224] Step 8:

[0225] The server returns a formatted answer to the device. For example, it sends the following JSON data as an HTTP response: {"answer": "Just say 'I watched a movie yesterday'. Don't worry, this is easy."}

[0226] Step 9:

[0227] The device parses the response it receives from the server and prepares it for display to the user, converting it into the appropriate format for display on the screen.

[0228] Step 10:

[0229] The device will then display the parsed answer in a user interface, for example, "Just say 'I watched a movie yesterday.' Don't worry, this is easy."

[0230] Example 2

[0231] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0232] Conventional language learning systems simply provide answers to questions entered by users, making it difficult to provide appropriate instruction or feedback that takes into account the user's emotional state. Therefore, there is a need for a system that can flexibly respond to learners' emotional needs, reduce learning stress, and provide effective education.

[0233] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0234] In this invention, the server includes means for receiving language-related questions from a user, means for the terminal to convert the received question into a specific format and transmit it to the server, means for an emotion engine built into the terminal to analyze the question and the user's emotion, means for the server to transmit the received question and emotional state to a generative artificial intelligence and make a query in the form of a prompt sentence, means for the generative artificial intelligence to generate an answer taking the emotional state into consideration, means for the server to transmit the generated answer to the terminal, and means for the terminal to display the answer received to the user. This makes it possible to provide an appropriate answer according to the user's emotional state, reduce learning stress, and enable effective language learning.

[0235] "User" refers to an individual who uses the language learning system.

[0236] "Terminal" refers to devices operated by users, such as smartphones, tablets, and PCs.

[0237] "Server" refers to a central device that communicates with terminals via a network and processes questions and answers.

[0238] "Emotion engine" refers to a software component that analyzes a user's emotional state from input information.

[0239] "Generative artificial intelligence" refers to an algorithm or model that uses natural language processing to generate appropriate answers from a given prompt.

[0240] "Specific format" refers to a standardized data format (e.g., JSON format) used by a terminal when sending data to a server.

[0241] A "prompt sentence" refers to an input sentence used to give a specific question or command to a generative artificial intelligence.

[0242] The present invention is a system for supporting language learning through collaboration between a user, a terminal, a server, a generative artificial intelligence, and an emotion engine. The system includes a process in which a user asks a language-related question, a generative artificial intelligence provides an answer to the question, and the system further recognizes the user's emotion and adjusts the answer.

[0243] System configuration

[0244] A user operates a language learning app using a device such as a smartphone, tablet, or PC. When the user enters a language-related question, the question is sent from the device to a server. The server analyzes the question and recognizes the user's emotion using an emotion engine. The analyzed question and the recognized emotion are sent to a generative artificial intelligence (e.g., a natural language processing model) to generate an answer. The answer is sent back to the device via the server, and the device displays it to the user.

[0245] Hardware and software used

[0246] Devices: Smartphones, tablets, PCs, etc.

[0247] Server: A cloud-based server or an on-premise server.

[0248] Emotion engine: Software that analyzes emotions from user input information.

[0249] Generative AI: Uses natural language processing models (e.g., GPT-3).

[0250] Data processing and calculation

[0251] Data entry: The user enters a question into the device by text or voice.

[0252] Sentiment Analysis: The emotion engine analyzes user input to identify their emotional state.

[0253] Data transmission: The question and sentiment analysis results are converted into JSON format and sent to the server.

[0254] Use of generative artificial intelligence: Generative artificial intelligence generates answers based on the question and emotional state.

[0255] Specific examples

[0256] For example, consider a case where a user has a question about English grammar. The user inputs, "How should I say 'I saw a movie yesterday' in English?" At this time, the emotion engine analyzes the emotion from the text and voice input by the user.

[0257] 1. User enters a question

[0258] The user types a question into the app's text box and presses the send button. For example, they might type, "How do I say 'I saw a movie yesterday' in English?"

[0259] 2. Emotion analysis

[0260] An emotion engine within the device analyzes the input text and recognizes the user's emotional state. For example, it may determine that the user is confused.

[0261] 3. Send the question and emotion to the server

[0262] The device converts the question and sentiment analysis results into JSON format and sends them to the server.

[0263] 4. The server queries the generative AI

[0264] The server sends this data to the generative AI in the form of a prompt: "How do you say 'I watched a movie yesterday' in English?" and the user seems confused."

[0265] 5. Generative AI generates answers

[0266] Generative AI generates answers based on the information submitted, such as "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!"

[0267] 6. Returning and Displaying Answers

[0268] The server receives the generated answer and sends it back to the device. The device then analyzes the answer and displays it to the user. Specifically, the screen displays the message, "Just say 'I watched a movie yesterday.' Don't worry, this is easy."

[0269] This system allows users to ask language-related questions anytime, anywhere and receive instant, appropriate answers. It also uses generative AI to provide advice on grammar and expressions, and an emotion engine to provide answers that take the user's emotions into consideration. This reduces learning stress for users and allows them to learn languages ​​more effectively.

[0270] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0271] Step 1:

[0272] The user inputs a language-related question. Using a device such as a smartphone or PC, the user operates a language learning app, inputting "How do I say 'I saw a movie yesterday' in English?" and pressing the send button. The input text is saved in the device's internal memory.

[0273] Step 2:

[0274] The device sends the question and the emotion analysis results to the server. The emotion engine analyzes the text entered by the user and determines the user's emotional state (for example, confusion). The analysis results and the question text are converted into JSON format and an HTTP request is sent to the server. The input is the user's question and emotional state, and the output is the JSON data sent to the server.

[0275] Step 3:

[0276] The server queries the generative AI for the question and the result of the sentiment analysis. The server analyzes the received JSON data, extracts the question content and the user's emotional state, and converts it into a prompt format. The specific prompt is "How do you say 'I watched a movie yesterday' in English? The user seems confused." The server sends the prompt to the generative AI. The input is JSON data, and the output is the prompt text for the generative AI.

[0277] Step 4:

[0278] A generative AI generates an answer. A generative AI (e.g., GPT-3) receives a prompt and generates an appropriate answer. For example, it generates the answer "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!" The input is the prompt, and the output is the generated answer text.

[0279] Step 5:

[0280] The server returns the generated answer to the device. The server receives the answer from the generative AI and converts it into a format that is easy for the user to understand. The converted data is converted into JSON format and sent to the device as an HTTP response. The input is the answer from the generative AI, and the output is the JSON data sent to the device.

[0281] Step 6:

[0282] The terminal displays the answer to the user. The terminal parses the received JSON data and displays the answer on the screen. Specifically, it displays "Just say 'I watched a movie yesterday'. Don't worry, this is easy." The input is the JSON data received from the server, and the output is the text displayed on the user's screen.

[0283] (Application example 2)

[0284] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0285] Conventional electronic payment systems do not adequately address situations where users feel confused or uneasy about their operations. Therefore, there is a need for a method to alleviate users' anxiety and provide a smooth operational experience.

[0286] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for processing user data, means for analyzing the user's emotions, and means for causing a generative artificial intelligence to generate advice based on the emotional information. This makes it possible to provide appropriate operation guidance and reassuring messages based on the user's emotional information.

[0287] "User Data" means information that a User inputs into the System, including text, audio, images, etc.

[0288] An "analyzing means" is a method or device for analyzing received data and extracting specific information.

[0289] "Means for analyzing emotions" refers to methods or techniques for identifying a user's emotional state from input user data and generating emotional information.

[0290] A "server" is a computer system that stores, processes, and distributes data over a network.

[0291] "Generative AI" is an AI technology that generates appropriate responses and advice based on input data.

[0292] "Advice" or "operation guide" means instructions or explanations provided to a user to enable them to operate the system correctly.

[0293] "Reassuring messages" are reliable information provided to reduce users' anxiety and confusion and to facilitate smooth operation.

[0294] "Terminal" means an electronic device used by a User to interact with the System, including a smartphone, tablet, or PC.

[0295] The "displaying means" refers to a method or device for visually or audibly presenting the generated advice or message to the user.

[0296] The present invention provides a system that provides emotion-based advice and operation guidance to users when they make electronic payments. An embodiment of this system will be described in detail below.

[0297] 1. System Configuration

[0298] The system includes a user terminal, a server, generative artificial intelligence, and an emotion analysis engine to analyze the user's emotions and generate emotion-based operation guides.

[0299] User terminals are common electronic devices such as smartphones, tablets, PCs, etc. These devices are responsible for inputting and displaying data from users.

[0300] The server is a computer system that processes, stores, and distributes data. It analyzes the data received from the user's device, sends the results to the generative AI, and returns the generated advice to the device.

[0301] Generative artificial intelligence (e.g., OpenAI's GPT-4 (registered trademark)) generates appropriate advice and operation guides based on emotional information.

[0302] An emotion analysis engine (e.g., IBM Watson® Emotion Analysis) analyzes data received from users and extracts emotional information.

[0303] 2. Data Flow

[0304] To initiate the electronic payment process, users interact with a smartphone app and enter data, such as text or voice instructions like, "I want to complete my payment, but I don't know what to do next."

[0305] The device sends the input data to the server. At the same time, the emotion analysis engine analyzes the user's emotions and sends the results to the server in JSON format.

[0306] The server then sends the received data and emotional information to a generative artificial intelligence (AI) system, which then generates appropriate advice and operational guidance based on the emotion. For example, it generates a specific guidance message such as, "Don't worry, you're just one step away from completing the payment!"

[0307] The generated advice is sent back to the user terminal via the server and displayed visually or audibly to the user.

[0308] 3. Hardware and Software Used

[0309] This system uses the following hardware and software:

[0310] User Device: Electronic devices such as smartphones, tablets, and PCs.

[0311] Server: A computer system that processes, stores, and delivers data.

[0312] Generative artificial intelligence: OpenAI GPT-4, etc.

[0313] Sentiment analysis engines: such as IBM Watson Emotion Analysis.

[0314] 4. Specific Examples

[0315] For example, if a user says, "I want to complete a payment, but I don't know what to do next," the sentiment analysis engine will recognize this "confusion" and send it to the generative AI, which will then provide specific operational guidance such as, "Don't worry, just press the button to complete the payment!"

[0316] Examples of prompt statements

[0317] User is feeling confused. Provide clear and comforting instructions for completing an electronic payment.

[0318] This allows users to receive appropriate advice on the spot, reducing their anxiety and enabling them to complete electronic payments smoothly.

[0319] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0320] Step 1:

[0321] A user provides input data to a device (such as a smartphone) to initiate an electronic payment. The input data can include text or voice instructions. For example, a user might say, "I want to complete the payment, but I don't know what to do next." This data is given to the device as input.

[0322] Step 2:

[0323] The device sends the input data to the emotion analysis engine. The emotion analysis engine analyzes the input data and identifies the user's emotional state. The analysis results in the user's emotion (e.g., confusion, anxiety, etc.). This is the output emotion information.

[0324] Step 3:

[0325] The device sends JSON-formatted data, including input data and emotional information output from the emotion analysis engine, to the server. The server analyzes the received data and prepares it for transmission to the generative AI. During this process, the data format is converted and necessary information is extracted.

[0326] Step 4:

[0327] The server sends the input data along with the emotion information to a generative AI (e.g., OpenAI GPT-4). The generative AI generates appropriate advice and operation guides based on the emotion information. For example, it outputs a generated guide such as, "Don't worry, just press the payment completion button!"

[0328] Step 5:

[0329] The server receives the advice generated by the generative AI and returns it to the device. This return is often in JSON format. The server converts the advice it receives into an appropriate format and sends it to the device.

[0330] Step 6:

[0331] The device analyzes the advice received from the server and displays it to the user. Specifically, it is presented as a visual display (text displayed on the screen) or audio guidance (voice output by speech synthesis). The user looks at this advice and continues the operation.

[0332] This allows users to receive appropriate guidance according to their emotional state, allowing them to smoothly proceed through the electronic payment process. The overall flow of the system also aims to reduce user anxiety.

[0333] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0334] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0335] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0336] [Second embodiment]

[0337] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0338] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0339] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0340] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0341] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0342] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0343] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0344] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0345] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0346] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0347] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0348] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0349] The present invention provides a system for supporting language learning through collaboration between a user, a terminal, a server, and a generative artificial intelligence. The system includes a process in which a user asks a language-related question and a generative artificial intelligence provides an answer to the question.

[0350] System configuration

[0351] Users operate language learning apps using devices such as smartphones, tablets, and PCs. When a user inputs a language-related question, the question is sent from the device to a server. The server analyzes the question, queries a generative artificial intelligence (e.g., a natural language processing model), and generates an answer. The generated answer is sent back to the device via the server, and the device displays it to the user.

[0352] Specific examples

[0353] For example, consider a user asking a question about English grammar: "How do I say 'I saw a movie yesterday' in English?"

[0354] 1. User enters question

[0355] Users enter their question into the app's text box and press the send button.

[0356] 2. Send the question to the server

[0357] The terminal receives the entered question, converts it into a specific format (e.g., JSON format), and sends an HTTP request to the server.

[0358] 3. The server analyzes the question and queries the generative AI

[0359] The server analyzes the received question and sends it to the generative AI in an appropriate format. The generative AI receives the question in a form such as "How do you say 'I watched a movie yesterday' in English?"

[0360] 4. Generative AI generates answers

[0361] Generative AI generates answers based on the information you submit, for example, "You can say 'I watched a movie yesterday.'"

[0362] 5. Send the answer back to the device

[0363] The server receives the response from the generative AI, converts it into a format that is easy for the user to understand if necessary, and sends it back to the device.

[0364] 6. Show the answer to the user

[0365] The device displays the answer received from the server to the user, specifically, "You can say 'I watched a movie yesterday.'"

[0366] The system of the present invention allows users to ask language-related questions anytime, anywhere and get instant, relevant answers. It also provides grammar and expression advice as well as dialogue scenarios for conversation practice. This feature allows users to experience a practical language learning environment.

[0367] The processing flow will be explained below.

[0368] Step 1:

[0369] A user opens the app and types in a language-related question, such as "How do I say 'I saw a movie yesterday' in English?"

[0370] Step 2:

[0371] The device receives the user's input and detects that the send button has been pressed.

[0372] Step 3:

[0373] The device sends the entered question to the server in a specific format (e.g., JSON format). For example, the JSON data {"question": "How should I say 'I saw a movie yesterday' in English?"} is sent via an HTTP POST request.

[0374] Step 4:

[0375] The server receives the request from the device and analyzes the question, converting it into an easy-to-understand form.

[0376] Step 5:

[0377] The server sends the analyzed question to the generative AI. For example, the server sends a question to the generative AI in the form of "How do you say 'I watched a movie yesterday' in English?"

[0378] Step 6:

[0379] The generative AI generates an answer based on the question received from the server. An example of a generated answer would be "You can say 'I watched a movie yesterday.'"

[0380] Step 7:

[0381] The server receives the generated answer and formats the answer as needed, including translating the answer into the user's language.

[0382] Step 8:

[0383] The server returns a formatted answer to the device. For example, it sends the JSON data {"answer": "Just say 'I watched a movie yesterday'"} as an HTTP response.

[0384] Step 9:

[0385] The device analyzes the response it receives from the server and prepares it for display to the user, converting it into a format suitable for display on the screen.

[0386] Step 10:

[0387] The device displays the analyzed answer in the user interface. For example, it displays "You can say 'I watched a movie yesterday.'"

[0388] Example 1

[0389] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0390] Current language learning systems have a complex process for quickly and accurately solving language-related questions, resulting in a poor user experience. Furthermore, when using generative AI to provide answers, there is a lack of a way to accurately analyze the user's input and generate appropriate prompts, resulting in issues with the quality of the generated answers. Furthermore, there is a lack of a way to convert the generated answers into a format that is easily understandable to the user. Therefore, improvements are needed to promote effective and efficient language learning.

[0391] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0392] In this invention, the server includes a device for receiving language-related questions from a user, a device for converting the received questions into a specific data format and sending them to the server, a device for analyzing the received questions, generating prompt sentences based on a generative AI model, and sending them to the generative AI model, a device for receiving answers from the generative AI model, converting them into a format that is easy for the user to understand, and returning the answers to the terminal, and a device for displaying the answers received by the terminal to the user. This makes it possible to accurately analyze the content of the questions entered by the user and generate appropriate prompt sentences. Furthermore, by converting the generated answers into a format that is easy for the user to understand and providing them, it is possible to improve the user experience and enable effective language learning.

[0393] A "user" is an entity that utilizes the system to enter language-related questions and receive answers.

[0394] A "terminal" is a device that a user operates to input questions and display answers, and includes common electronic devices such as smartphones, tablets, and PCs.

[0395] A "server" is a device that receives a question sent from a terminal, analyzes the question, generates an answer using a generative AI model, and returns the answer to the terminal.

[0396] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to generate appropriate answers to user questions.

[0397] A "prompt sentence" is an input sentence used to generate an answer for a generative AI model, and is an appropriate translation of the user's question.

[0398] "Device" is a general term that refers to a hardware or software component that performs a particular function or role.

[0399] A "data format" is a way of representing information based on a specific structure or rules, and includes specific examples such as JSON and XML.

[0400] An "answer" is information generated by a generative AI model and provided in response to a user's question.

[0401] This invention is a system that supports language learning through collaboration between a user, a terminal, a server, and a generative AI model. The operation of each component and the overall system process are described in detail below.

[0402] System Overview

[0403] Users operate language learning apps using devices such as smartphones, tablets, and PCs. The devices receive user questions, convert them into a specific data format (e.g., JSON format), and send them to a server. The server analyzes the received questions, generates prompts based on a generative AI model, and sends them to the generative AI model. The generative AI model generates answers based on the sent prompts and sends them back to the server. The server converts the generated answers into a format that is easy for the user to understand and sends them back to the device, where the device displays the answers to the user.

[0404] Hardware and software used

[0405] Device: A device on which a user enters questions and displays answers. Examples include smartphones, tablets, and PCs.

[0406] Server: A back-end system that receives questions, analyzes them, and sends prompts to the generative AI model. The server is implemented in a common programming language such as Python or Java.

[0407] Generative AI model: A natural language processing model that generates answers to user questions. Examples include OpenAI GPT-3.

[0408] Communication method: A method for exchanging data between the device and the server, and between the server and the generative AI model. HTTP / HTTPS protocols are commonly used.

[0409] Specific operation explanation

[0410] 1. User enters language question

[0411] A user launches a language learning app and logs in. For example, they enter a question about English learning into the text box and press the "Submit" button. For example, they might enter, "How do I say 'I saw a movie yesterday' in English?"

[0412] 2. The device sends the question to the server

[0413] The terminal converts the question entered by the user into JSON format and sends it to the server using an HTTP POST request.

[0414] 3. The server analyzes the question and sends it to the generative AI model as a prompt.

[0415] The server analyzes the data received from the device and extracts the question. It then converts the extracted question into an appropriate format (e.g., English) and generates a prompt for the generative AI model. For example, it generates a prompt such as, "How do you say 'I watched a movie yesterday' in English?"

[0416] 4. Generative AI model generates answers

[0417] The generative AI model generates an answer based on the prompt it receives, for example, "You can say 'I watched a movie yesterday.'"

[0418] 5. The server returns the answer to the device

[0419] The server receives the answer from the generative AI model, formats it into a form that is easy for the user to understand, for example, "You can say 'I watched a movie yesterday'," and sends it back to the device.

[0420] 6. The device displays the answer to the user

[0421] The device displays the answer received from the server to the user. Specifically, it displays "You can say 'I watched a movie yesterday'" in the text area on the screen.

[0422] Prompt Sentence Examples

[0423] How do you say 'I watched a movie yesterday' in English?

[0424] In this way, the system of the present invention allows users to ask language-related questions anytime and anywhere and receive appropriate answers immediately, allowing users to progress with language learning effectively and efficiently.

[0425] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0426] Step 1:

[0427] Process Description: User enters a language question.

[0428] Specific operation: The user starts the application, logs in, enters a question in the text box, and presses the "Submit" button.

[0429] Input: User question (e.g., "How do I say 'I saw a movie yesterday' in English?")

[0430] Output: The action of the user pressing the submit button

[0431] Step 2:

[0432] Process description: The device sends a query to the server.

[0433] Specific operation: The terminal converts the text entered by the user into JSON format and sends it to the server via an HTTP POST request.

[0434] Input: User question

[0435] Output: JSON data sent to the server (e.g., {"question": "How do I say 'I saw a movie yesterday' in English?"})

[0436] Step 3:

[0437] Process description: The server parses the question and sends a prompt to the generative AI model.

[0438] How it works: The server parses the received JSON data, extracts the question, converts it into an appropriate format (e.g., generates an English prompt, "How do you say 'I watched a movie yesterday' in English?"), and creates a request to send to the generative AI model.

[0439] Input: JSON data sent from the terminal

[0440] Output: The prompt sent to the generative AI model (e.g., "How do you say 'I watched a movie yesterday' in English?")

[0441] Step 4:

[0442] Process Description: The generative AI model generates an answer.

[0443] How it works: The generative AI model receives a prompt as input and uses natural language processing techniques to generate an answer, which is then sent back to the server.

[0444] Input: Prompt text sent from the server

[0445] Output: The answer returned by the generative AI model (e.g., "You can say 'I watched a movie yesterday.'")

[0446] Step 5:

[0447] Process description: The server sends a response back to the device.

[0448] Specific operation: The server receives the answer from the generative AI model, converts it into a format that is easy for the user to understand, generates JSON data containing the converted answer, and sends it to the terminal as an HTTP response.

[0449] Input: The answer returned by the generative AI model

[0450] Output: JSON data sent to the device (e.g., {"answer": "You can say 'I watched a movie yesterday'."})

[0451] Step 6:

[0452] Process Description: The device displays the answer to the user.

[0453] Specific operation: The device parses the JSON data received from the server, extracts the answer, and displays it in the text area on the screen.

[0454] Input: JSON data sent from the server

[0455] Output: The answer that is displayed to the user (e.g., "You can say 'I watched a movie yesterday'.")

[0456] These steps will help users progress through language learning efficiently and effectively.

[0457] (Application example 1)

[0458] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0459] In modern content distribution services, when users encounter a word or phrase they don't understand while watching, they need a way to quickly ask questions and receive answers on the spot. However, existing systems lack the functionality to answer questions in real time while watching, which reduces user learning efficiency. Furthermore, it is difficult to implement advanced natural language processing technology that can provide appropriate answers to questions.

[0460] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0461] In this invention, the server includes means for receiving questions about information from users, means for transmitting the received questions to a generative artificial intelligence, means for the generative artificial intelligence to generate answers based on the questions, means for the server to receive the answers and return them to the display device, means for the display device to display the received answers to the user, and means for the user to ask questions about the content being displayed in real time, thereby enabling users to instantly resolve any questions that arise while viewing, and to learn efficiently.

[0462] "User" refers to any individual or entity that uses the System.

[0463] "Information" refers to the knowledge and data that users ask about, such as language, grammar, and methods of expression.

[0464] "Question" refers to the inquiry a user enters into the system to confirm the information they are seeking.

[0465] "Network equipment" refers to devices used to send and receive data, such as servers and routers.

[0466] "Generative artificial intelligence" refers to a system that has natural language processing capabilities and generates appropriate answers to input questions.

[0467] "Answer" refers to appropriate information or data generated by generative artificial intelligence in response to a user's question.

[0468] "Display device" refers to a device on which the user can view the answers, such as a smartphone, tablet, or computer.

[0469] "Content" refers to media for viewing provided by a distribution service, such as movies, dramas, and news.

[0470] "Real-time" refers to minimal delay between the moment a user asks a question and the moment a response is received.

[0471] "Learning" refers to the process by which a user acquires new knowledge, such as language or grammar.

[0472] The present invention relates to a language learning system for content distribution services that allows users to ask questions in real time about words or phrases they do not understand while watching, and provides answers using generative artificial intelligence.

[0473] This system is configured as follows:

[0474] System configuration

[0475] A user inputs a question via their device (smartphone, tablet, PC, etc.) and sends it to the server. At this time, network equipment is responsible for sending and receiving the data. The question received by the server is sent to a generative artificial intelligence, which generates an appropriate answer. The generated answer is then sent back to the display device via the server and displayed to the user.

[0476] Program processing

[0477] The hardware and software used are as follows:

[0478] Hardware: Servers (e.g., AWS EC2), user devices (smartphones, tablets, PCs)

[0479] Software: Flask (server), external natural language processing API (e.g., GPT-3)

[0480] The flow of data processing and data calculation is as follows:

[0481] 1. The user inputs a question from the device. For example, "The user asked a question about a specific scene while watching a movie. Question: 'What is the concept of "dream within a dream" in the movie "Inception"?'"

[0482] 2. The device sends the entered question to the server via a network device. The data is sent in a standard format such as JSON.

[0483] 3. The server sends the received question to the generative AI, which generates an answer based on the received question. For example, it generates an answer such as, "The concept of 'dream within a dream' refers to multiple layers of dreams where each layer is a dream within the previous one."

[0484] 4. The server receives the generated answer and, if necessary, converts it into a user-friendly format.

[0485] 5. The converted answer is sent back from the server to the display device and displayed to the user. Specifically, it is displayed in the form of "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."

[0486] Adding specific examples

[0487] For example, a system that answers questions about scenes in a movie a user is watching in real time might use the following prompt:

[0488] "A user is watching a movie and has a question about a particular scene. Please answer the following question: Question: 'What is the concept of "dream within a dream" in the movie "Inception"?'"

[0489] This invention allows users to instantly resolve any questions they may have while watching, providing an efficient learning experience.

[0490] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0491] Step 1:

[0492] The user inputs a question from the device. The user enters a question about the content they are watching in text format into the device's input form and presses the send button. For example, the user might input a question such as, "What is the concept of 'dream within a dream' in the movie 'Inception'?" The input at this time is treated as text data.

[0493] Step 2:

[0494] The device sends the entered question to the server via a network device. The device converts the question into JSON format and sends the data to the server using an HTTP request. Specifically, data in the format {"question": "What is the concept of 'dream within a dream' in the movie 'Inception'?"} is sent.

[0495] Step 3:

[0496] The server parses the received question and sends it to the generative AI in an appropriate format. The server receives the JSON-formatted data, extracts the contents of the "question" field, and sends an API request to the generative AI. At this time, a prompt is formed, and the body of the API request includes a prompt such as, "A user asked a question about a specific scene while watching a movie. Please answer the following question. Question: 'What is the concept of "dream within a dream" in the movie "Inception"?'"

[0497] Step 4:

[0498] The generative AI analyzes the received query and generates an answer to the question. The generative AI uses a natural language processing model to analyze the prompt sentence and generate an appropriate response. For example, an answer may be generated in the form of "The concept of 'dream within a dream' refers to multiple layers of dreams where each layer is a dream within the previous one." This answer is sent back to the server in JSON format.

[0499] Step 5:

[0500] The server receives the generated answer and, if necessary, converts it into a format that is easy for the user to understand. The server analyzes the answer received from the generative AI and converts it into language and expressions that are easy for the user to understand. Specifically, it performs processing such as translating the generated English answer into Japanese. As a result, for example, the answer may be converted into a form such as, "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."

[0501] Step 6:

[0502] The server returns the converted answer to the device. The server then converts the answer back into JSON format and sends it to the device as an HTTP response. For example, data such as {"answer": "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."} is sent.

[0503] Step 7:

[0504] The device displays the received answer to the user. The device parses the received JSON data, extracts the contents of the "answer" field, and displays it on the screen. This allows the user to instantly resolve any questions they may have while watching. For example, the screen displays the following: "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."

[0505] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0506] The present invention is a system for supporting language learning through collaboration between a user, a terminal, a server, a generative artificial intelligence, and an emotion engine. The system includes a process in which a user asks a language-related question, a generative artificial intelligence provides an answer to the question, and the system further adjusts the answer by recognizing the user's emotion.

[0507] System configuration

[0508] A user operates a language learning app using a device such as a smartphone, tablet, or PC. When the user enters a language-related question, the question is sent from the device to a server. The server analyzes the question and recognizes the user's emotion using an emotion engine. The analyzed question and the recognized emotion are sent to a generative artificial intelligence (e.g., a natural language processing model) to generate an answer. The answer is sent back to the device via the server, and the device displays it to the user.

[0509] Specific examples

[0510] For example, consider a case where a user has a question about English grammar. The user might type, "How do I say 'I saw a movie yesterday' in English?" The emotion engine then analyzes the emotion from the text and voice input by the user.

[0511] 1. User enters question

[0512] The user enters a question into the app's text box and presses the submit button. For example, "How do I say 'I saw a movie yesterday' in English?" The emotion engine simultaneously analyzes the user's input text and identifies the emotion.

[0513] 2. Send the question to the server

[0514] The terminal receives the entered question, converts it into a specific format (e.g., JSON format), adds the analysis results of the emotion engine, and sends an HTTP request to the server.

[0515] 3. The server queries the generative AI with the question and emotion

[0516] The server sends the received question and the emotion analysis results to the generative AI, which receives the form "How do you say 'I watched a movie yesterday' in English?" and the user's emotional state.

[0517] 4. Generative AI generates answers

[0518] Generative AI generates answers based on the information sent. For example, if there is emotional information that the user is confused, the answer will be gentler. An example of a generated answer would be a response that includes gentle advice such as, "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!"

[0519] 5. Send the answer back to the device

[0520] The server receives the answer from the generative AI, converts it into a format that is easy for the user to understand if necessary, and sends it back to the device. For example, it sends JSON data such as {"answer": "Just say 'I watched a movie yesterday'. Don't worry, this is easy."} as an HTTP response.

[0521] 6. Show the answer to the user

[0522] The device analyzes the answer received from the server and displays it to the user. Specifically, it displays the message "Just say 'I watched a movie yesterday'. Don't worry, it's easy."

[0523] The system of the present invention allows users to ask language-related questions anytime, anywhere and receive immediate, appropriate answers. Furthermore, by using generative artificial intelligence, the system can provide advice on grammar and expressions, as well as provide answers that take the user's emotions into consideration by using an emotion engine. This reduces learning stress for users and allows them to learn languages ​​more effectively.

[0524] The processing flow will be explained below.

[0525] Step 1:

[0526] A user opens the app and types in a language-related question, such as "How do I say 'I saw a movie yesterday' in English?"

[0527] Step 2:

[0528] The device receives the user's input and detects when the send button is pressed. At the same time, the device sends the user's input text or voice data to the emotion engine to analyze the user's emotions.

[0529] Step 3:

[0530] The device sends the entered question to the server in a specific format (e.g., JSON format). For example, the following JSON data is sent via an HTTP POST request: {"question": "How should I say 'I saw a movie yesterday' in English?", "emotion": "confused"}

[0531] Step 4:

[0532] The server receives requests from the device and analyzes the question and emotional information, converting the user's question and emotional state into an easy-to-understand format.

[0533] Step 5:

[0534] The server sends the analyzed question and emotional information to the generative AI. For example, it sends the question in the form "How do you say 'I watched a movie yesterday' in English?" along with the emotional information "confused."

[0535] Step 6:

[0536] The generative AI generates an answer based on the question and emotional information received from the server. For example, if the user is in a "confused" emotional state, it generates an answer with gentle advice in the form of "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!"

[0537] Step 7:

[0538] The server receives the generated answer and formats it as needed, for example translating it into the user's language to make it easier to understand.

[0539] Step 8:

[0540] The server returns a formatted answer to the device. For example, it sends the following JSON data as an HTTP response: {"answer": "Just say 'I watched a movie yesterday'. Don't worry, this is easy."}

[0541] Step 9:

[0542] The device parses the response it receives from the server and prepares it for display to the user, converting it into the appropriate format for display on the screen.

[0543] Step 10:

[0544] The device will then display the parsed answer in a user interface, for example, "Just say 'I watched a movie yesterday.' Don't worry, this is easy."

[0545] Example 2

[0546] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0547] Conventional language learning systems simply provide answers to questions entered by users, making it difficult to provide appropriate instruction or feedback that takes into account the user's emotional state. Therefore, there is a need for a system that can flexibly respond to learners' emotional needs, reduce learning stress, and provide effective education.

[0548] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0549] In this invention, the server includes means for receiving language-related questions from a user, means for the terminal to convert the received question into a specific format and transmit it to the server, means for an emotion engine built into the terminal to analyze the question and the user's emotion, means for the server to transmit the received question and emotional state to a generative artificial intelligence and make a query in the form of a prompt sentence, means for the generative artificial intelligence to generate an answer taking the emotional state into consideration, means for the server to transmit the generated answer to the terminal, and means for the terminal to display the answer received to the user. This makes it possible to provide an appropriate answer according to the user's emotional state, reduce learning stress, and enable effective language learning.

[0550] "User" refers to an individual who uses the language learning system.

[0551] "Terminal" refers to devices operated by users, such as smartphones, tablets, and PCs.

[0552] "Server" refers to a central device that communicates with terminals via a network and processes questions and answers.

[0553] "Emotion engine" refers to a software component that analyzes a user's emotional state from input information.

[0554] "Generative artificial intelligence" refers to an algorithm or model that uses natural language processing to generate appropriate answers from a given prompt.

[0555] "Specific format" refers to a standardized data format (e.g., JSON format) used by a terminal when sending data to a server.

[0556] A "prompt sentence" refers to an input sentence used to give a specific question or command to a generative artificial intelligence.

[0557] The present invention is a system for supporting language learning through collaboration between a user, a terminal, a server, a generative artificial intelligence, and an emotion engine. The system includes a process in which a user asks a language-related question, a generative artificial intelligence provides an answer to the question, and the system further recognizes the user's emotion and adjusts the answer.

[0558] System configuration

[0559] A user operates a language learning app using a device such as a smartphone, tablet, or PC. When the user enters a language-related question, the question is sent from the device to a server. The server analyzes the question and recognizes the user's emotion using an emotion engine. The analyzed question and the recognized emotion are sent to a generative artificial intelligence (e.g., a natural language processing model) to generate an answer. The answer is sent back to the device via the server, and the device displays it to the user.

[0560] Hardware and software used

[0561] Devices: Smartphones, tablets, PCs, etc.

[0562] Server: A cloud-based server or an on-premise server.

[0563] Emotion engine: Software that analyzes emotions from user input information.

[0564] Generative AI: Uses natural language processing models (e.g., GPT-3).

[0565] Data processing and calculation

[0566] Data entry: The user enters a question into the device by text or voice.

[0567] Sentiment Analysis: The emotion engine analyzes user input to identify their emotional state.

[0568] Data transmission: The question and sentiment analysis results are converted into JSON format and sent to the server.

[0569] Use of generative artificial intelligence: Generative artificial intelligence generates answers based on the question and emotional state.

[0570] Specific examples

[0571] For example, consider a case where a user has a question about English grammar. The user inputs, "How should I say 'I saw a movie yesterday' in English?" At this time, the emotion engine analyzes the emotion from the text and voice input by the user.

[0572] 1. User enters a question

[0573] The user types a question into the app's text box and presses the send button. For example, they might type, "How do I say 'I saw a movie yesterday' in English?"

[0574] 2. Emotion analysis

[0575] An emotion engine within the device analyzes the input text and recognizes the user's emotional state. For example, it may determine that the user is confused.

[0576] 3. Send the question and emotion to the server

[0577] The device converts the question and sentiment analysis results into JSON format and sends them to the server.

[0578] 4. The server queries the generative AI

[0579] The server sends this data to the generative AI in the form of a prompt: "How do you say 'I watched a movie yesterday' in English?" and the user seems confused."

[0580] 5. Generative AI generates answers

[0581] Generative AI generates answers based on the information submitted, such as "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!"

[0582] 6. Returning and Displaying Answers

[0583] The server receives the generated answer and sends it back to the device. The device then analyzes the answer and displays it to the user. Specifically, the screen displays the message, "Just say 'I watched a movie yesterday.' Don't worry, this is easy."

[0584] This system allows users to ask language-related questions anytime, anywhere and receive instant, appropriate answers. It also uses generative AI to provide advice on grammar and expressions, and an emotion engine to provide answers that take the user's emotions into consideration. This reduces learning stress for users and allows them to learn languages ​​more effectively.

[0585] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0586] Step 1:

[0587] The user inputs a language-related question. Using a device such as a smartphone or PC, the user operates a language learning app, inputting "How do I say 'I saw a movie yesterday' in English?" and pressing the send button. The input text is saved in the device's internal memory.

[0588] Step 2:

[0589] The device sends the question and the emotion analysis results to the server. The emotion engine analyzes the text entered by the user and determines the user's emotional state (for example, confusion). The analysis results and the question text are converted into JSON format and an HTTP request is sent to the server. The input is the user's question and emotional state, and the output is the JSON data sent to the server.

[0590] Step 3:

[0591] The server queries the generative AI for the question and the result of the sentiment analysis. The server analyzes the received JSON data, extracts the question content and the user's emotional state, and converts it into a prompt format. The specific prompt is "How do you say 'I watched a movie yesterday' in English? The user seems confused." The server sends the prompt to the generative AI. The input is JSON data, and the output is the prompt text for the generative AI.

[0592] Step 4:

[0593] A generative AI generates an answer. A generative AI (e.g., GPT-3) receives a prompt and generates an appropriate answer. For example, it generates the answer "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!" The input is the prompt, and the output is the generated answer text.

[0594] Step 5:

[0595] The server returns the generated answer to the device. The server receives the answer from the generative AI and converts it into a format that is easy for the user to understand. The converted data is converted into JSON format and sent to the device as an HTTP response. The input is the answer from the generative AI, and the output is the JSON data sent to the device.

[0596] Step 6:

[0597] The terminal displays the answer to the user. The terminal parses the received JSON data and displays the answer on the screen. Specifically, it displays "Just say 'I watched a movie yesterday'. Don't worry, this is easy." The input is the JSON data received from the server, and the output is the text displayed on the user's screen.

[0598] (Application example 2)

[0599] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0600] Conventional electronic payment systems do not adequately address situations where users feel confused or uneasy about their operations. Therefore, there is a need for a method to alleviate users' anxiety and provide a smooth operational experience.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for processing user data, means for analyzing the user's emotions, and means for causing a generative artificial intelligence to generate advice based on the emotional information. This makes it possible to provide appropriate operation guidance and reassuring messages based on the user's emotional information.

[0602] "User Data" means information that a User inputs into the System, including text, audio, images, etc.

[0603] An "analyzing means" is a method or device for analyzing received data and extracting specific information.

[0604] "Means for analyzing emotions" refers to methods or techniques for identifying a user's emotional state from input user data and generating emotional information.

[0605] A "server" is a computer system that stores, processes, and distributes data over a network.

[0606] "Generative AI" is an AI technology that generates appropriate responses and advice based on input data.

[0607] "Advice" or "operation guide" means instructions or explanations provided to a user to enable them to operate the system correctly.

[0608] "Reassuring messages" are reliable information provided to reduce users' anxiety and confusion and to facilitate smooth operation.

[0609] "Terminal" means an electronic device used by a User to interact with the System, including a smartphone, tablet, or PC.

[0610] The "displaying means" refers to a method or device for visually or audibly presenting the generated advice or message to the user.

[0611] The present invention provides a system that provides emotion-based advice and operation guidance to users when they make electronic payments. An embodiment of this system will be described in detail below.

[0612] 1. System Configuration

[0613] The system includes a user terminal, a server, generative artificial intelligence, and an emotion analysis engine to analyze the user's emotions and generate emotion-based operation guides.

[0614] User terminals are common electronic devices such as smartphones, tablets, PCs, etc. These devices are responsible for inputting and displaying data from users.

[0615] The server is a computer system that processes, stores, and distributes data. It analyzes the data received from the user's device, sends the results to the generative AI, and returns the generated advice to the device.

[0616] Generative artificial intelligence (e.g., OpenAI's GPT-4) generates appropriate advice and operation guides based on emotional information.

[0617] An emotion analysis engine (e.g., IBM Watson Emotion Analysis) analyzes data received from users and extracts emotional information.

[0618] 2. Data Flow

[0619] To initiate the electronic payment process, users interact with a smartphone app and enter data, such as text or voice instructions like, "I want to complete my payment, but I don't know what to do next."

[0620] The device sends the input data to the server. At the same time, the emotion analysis engine analyzes the user's emotions and sends the results to the server in JSON format.

[0621] The server then sends the received data and emotional information to a generative artificial intelligence (AI) system, which then generates appropriate advice and operational guidance based on the emotion. For example, it generates a specific guidance message such as, "Don't worry, you're just one step away from completing the payment!"

[0622] The generated advice is sent back to the user terminal via the server and displayed visually or audibly to the user.

[0623] 3. Hardware and Software Used

[0624] This system uses the following hardware and software:

[0625] User Device: Electronic devices such as smartphones, tablets, and PCs.

[0626] Server: A computer system that processes, stores, and delivers data.

[0627] Generative artificial intelligence: OpenAI GPT-4, etc.

[0628] Sentiment analysis engines: such as IBM Watson Emotion Analysis.

[0629] 4. Specific Examples

[0630] For example, if a user says, "I want to complete a payment, but I don't know what to do next," the sentiment analysis engine will recognize this "confusion" and send it to the generative AI, which will then provide specific operational guidance such as, "Don't worry, just press the button to complete the payment!"

[0631] Examples of prompt statements

[0632] User is feeling confused. Provide clear and comforting instructions for completing an electronic payment.

[0633] This allows users to receive appropriate advice on the spot, reducing their anxiety and enabling them to complete electronic payments smoothly.

[0634] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0635] Step 1:

[0636] A user provides input data to a device (such as a smartphone) to initiate an electronic payment. The input data can include text or voice instructions. For example, a user might say, "I want to complete the payment, but I don't know what to do next." This data is given to the device as input.

[0637] Step 2:

[0638] The device sends the input data to the emotion analysis engine. The emotion analysis engine analyzes the input data and identifies the user's emotional state. The analysis results in the user's emotion (e.g., confusion, anxiety, etc.). This is the output emotion information.

[0639] Step 3:

[0640] The device sends JSON-formatted data, including input data and emotional information output from the emotion analysis engine, to the server. The server analyzes the received data and prepares it for transmission to the generative AI. During this process, the data format is converted and necessary information is extracted.

[0641] Step 4:

[0642] The server sends the input data along with the emotion information to a generative AI (e.g., OpenAI GPT-4). The generative AI generates appropriate advice and operation guides based on the emotion information. For example, it outputs a generated guide such as, "Don't worry, just press the payment completion button!"

[0643] Step 5:

[0644] The server receives the advice generated by the generative AI and returns it to the device. This return is often in JSON format. The server converts the advice it receives into an appropriate format and sends it to the device.

[0645] Step 6:

[0646] The device analyzes the advice received from the server and displays it to the user. Specifically, it is presented as a visual display (text displayed on the screen) or audio guidance (voice output by speech synthesis). The user looks at this advice and continues the operation.

[0647] This allows users to receive appropriate guidance according to their emotional state, allowing them to smoothly proceed through the electronic payment process. The overall flow of the system also aims to reduce user anxiety.

[0648] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0649] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0650] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0651] [Third embodiment]

[0652] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0653] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0654] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0655] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0656] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0657] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0658] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0659] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0660] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0661] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0662] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0663] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0664] The present invention provides a system for supporting language learning through collaboration between a user, a terminal, a server, and a generative artificial intelligence. The system includes a process in which a user asks a language-related question and a generative artificial intelligence provides an answer to the question.

[0665] System configuration

[0666] Users operate language learning apps using devices such as smartphones, tablets, and PCs. When a user inputs a language-related question, the question is sent from the device to a server. The server analyzes the question, queries a generative artificial intelligence (e.g., a natural language processing model), and generates an answer. The generated answer is sent back to the device via the server, and the device displays it to the user.

[0667] Specific examples

[0668] For example, consider a user asking a question about English grammar: "How do I say 'I saw a movie yesterday' in English?"

[0669] 1. User enters question

[0670] Users enter their question into the app's text box and press the send button.

[0671] 2. Send the question to the server

[0672] The terminal receives the entered question, converts it into a specific format (e.g., JSON format), and sends an HTTP request to the server.

[0673] 3. The server analyzes the question and queries the generative AI

[0674] The server analyzes the received question and sends it to the generative AI in an appropriate format. The generative AI receives the question in a form such as "How do you say 'I watched a movie yesterday' in English?"

[0675] 4. Generative AI generates answers

[0676] Generative AI generates answers based on the information you submit, for example, "You can say 'I watched a movie yesterday.'"

[0677] 5. Send the answer back to the device

[0678] The server receives the response from the generative AI, converts it into a format that is easy for the user to understand if necessary, and sends it back to the device.

[0679] 6. Show the answer to the user

[0680] The device displays the answer received from the server to the user, specifically, "You can say 'I watched a movie yesterday.'"

[0681] The system of the present invention allows users to ask language-related questions anytime, anywhere and get instant, relevant answers. It also provides grammar and expression advice as well as dialogue scenarios for conversation practice. This feature allows users to experience a practical language learning environment.

[0682] The processing flow will be explained below.

[0683] Step 1:

[0684] A user opens the app and types in a language-related question, such as "How do I say 'I saw a movie yesterday' in English?"

[0685] Step 2:

[0686] The device receives the user's input and detects that the send button has been pressed.

[0687] Step 3:

[0688] The device sends the entered question to the server in a specific format (e.g., JSON format). For example, the JSON data {"question": "How should I say 'I saw a movie yesterday' in English?"} is sent via an HTTP POST request.

[0689] Step 4:

[0690] The server receives the request from the device and analyzes the question, converting it into an easy-to-understand form.

[0691] Step 5:

[0692] The server sends the analyzed question to the generative AI. For example, the server sends a question to the generative AI in the form of "How do you say 'I watched a movie yesterday' in English?"

[0693] Step 6:

[0694] The generative AI generates an answer based on the question received from the server. An example of a generated answer would be "You can say 'I watched a movie yesterday.'"

[0695] Step 7:

[0696] The server receives the generated answer and formats the answer as needed, including translating the answer into the user's language.

[0697] Step 8:

[0698] The server returns a formatted answer to the device. For example, it sends the JSON data {"answer": "Just say 'I watched a movie yesterday'"} as an HTTP response.

[0699] Step 9:

[0700] The device analyzes the response it receives from the server and prepares it for display to the user, converting it into a format suitable for display on the screen.

[0701] Step 10:

[0702] The device displays the analyzed answer in the user interface. For example, it displays "You can say 'I watched a movie yesterday.'"

[0703] Example 1

[0704] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0705] Current language learning systems have a complex process for quickly and accurately solving language-related questions, resulting in a poor user experience. Furthermore, when using generative AI to provide answers, there is a lack of a way to accurately analyze the user's input and generate appropriate prompts, resulting in issues with the quality of the generated answers. Furthermore, there is a lack of a way to convert the generated answers into a format that is easily understandable to the user. Therefore, improvements are needed to promote effective and efficient language learning.

[0706] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0707] In this invention, the server includes a device for receiving language-related questions from a user, a device for converting the received questions into a specific data format and sending them to the server, a device for analyzing the received questions, generating prompt sentences based on a generative AI model, and sending them to the generative AI model, a device for receiving answers from the generative AI model, converting them into a format that is easy for the user to understand, and returning the answers to the terminal, and a device for displaying the answers received by the terminal to the user. This makes it possible to accurately analyze the content of the questions entered by the user and generate appropriate prompt sentences. Furthermore, by converting the generated answers into a format that is easy for the user to understand and providing them, it is possible to improve the user experience and enable effective language learning.

[0708] A "user" is an entity that utilizes the system to enter language-related questions and receive answers.

[0709] A "terminal" is a device that a user operates to input questions and display answers, and includes common electronic devices such as smartphones, tablets, and PCs.

[0710] A "server" is a device that receives a question sent from a terminal, analyzes the question, generates an answer using a generative AI model, and returns the answer to the terminal.

[0711] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to generate appropriate answers to user questions.

[0712] A "prompt sentence" is an input sentence used to generate an answer for a generative AI model, and is an appropriate translation of the user's question.

[0713] "Device" is a general term that refers to a hardware or software component that performs a particular function or role.

[0714] A "data format" is a way of representing information based on a specific structure or rules, and includes specific examples such as JSON and XML.

[0715] An "answer" is information generated by a generative AI model and provided in response to a user's question.

[0716] This invention is a system that supports language learning through collaboration between a user, a terminal, a server, and a generative AI model. The operation of each component and the overall system process are described in detail below.

[0717] System Overview

[0718] Users operate language learning apps using devices such as smartphones, tablets, and PCs. The devices receive user questions, convert them into a specific data format (e.g., JSON format), and send them to a server. The server analyzes the received questions, generates prompts based on a generative AI model, and sends them to the generative AI model. The generative AI model generates answers based on the sent prompts and sends them back to the server. The server converts the generated answers into a format that is easy for the user to understand and sends them back to the device, where the device displays the answers to the user.

[0719] Hardware and software used

[0720] Device: A device on which a user enters questions and displays answers. Examples include smartphones, tablets, and PCs.

[0721] Server: A back-end system that receives questions, analyzes them, and sends prompts to the generative AI model. The server is implemented in a common programming language such as Python or Java.

[0722] Generative AI model: A natural language processing model that generates answers to user questions. Examples include OpenAI GPT-3.

[0723] Communication method: A method for exchanging data between the device and the server, and between the server and the generative AI model. HTTP / HTTPS protocols are commonly used.

[0724] Specific operation explanation

[0725] 1. User enters language question

[0726] A user launches a language learning app and logs in. For example, they enter a question about English learning into the text box and press the "Submit" button. For example, they might enter, "How do I say 'I saw a movie yesterday' in English?"

[0727] 2. The device sends the question to the server

[0728] The terminal converts the question entered by the user into JSON format and sends it to the server using an HTTP POST request.

[0729] 3. The server analyzes the question and sends it to the generative AI model as a prompt.

[0730] The server analyzes the data received from the device and extracts the question. It then converts the extracted question into an appropriate format (e.g., English) and generates a prompt for the generative AI model. For example, it generates a prompt such as, "How do you say 'I watched a movie yesterday' in English?"

[0731] 4. Generative AI model generates answers

[0732] The generative AI model generates an answer based on the prompt it receives, for example, "You can say 'I watched a movie yesterday.'"

[0733] 5. The server returns the answer to the device

[0734] The server receives the answer from the generative AI model, formats it into a form that is easy for the user to understand, for example, "You can say 'I watched a movie yesterday'," and sends it back to the device.

[0735] 6. The device displays the answer to the user

[0736] The device displays the answer received from the server to the user. Specifically, it displays "You can say 'I watched a movie yesterday'" in the text area on the screen.

[0737] Prompt Sentence Examples

[0738] How do you say 'I watched a movie yesterday' in English?

[0739] In this way, the system of the present invention allows users to ask language-related questions anytime and anywhere and receive appropriate answers immediately, allowing users to progress with language learning effectively and efficiently.

[0740] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0741] Step 1:

[0742] Process Description: User enters a language question.

[0743] Specific operation: The user starts the application, logs in, enters a question in the text box, and presses the "Submit" button.

[0744] Input: User question (e.g., "How do I say 'I saw a movie yesterday' in English?")

[0745] Output: The action of the user pressing the submit button

[0746] Step 2:

[0747] Process description: The device sends a query to the server.

[0748] Specific operation: The terminal converts the text entered by the user into JSON format and sends it to the server via an HTTP POST request.

[0749] Input: User question

[0750] Output: JSON data sent to the server (e.g., {"question": "How do I say 'I saw a movie yesterday' in English?"})

[0751] Step 3:

[0752] Process description: The server parses the question and sends a prompt to the generative AI model.

[0753] How it works: The server parses the received JSON data, extracts the question, converts it into an appropriate format (e.g., generates an English prompt, "How do you say 'I watched a movie yesterday' in English?"), and creates a request to send to the generative AI model.

[0754] Input: JSON data sent from the terminal

[0755] Output: The prompt sent to the generative AI model (e.g., "How do you say 'I watched a movie yesterday' in English?")

[0756] Step 4:

[0757] Process Description: The generative AI model generates an answer.

[0758] How it works: The generative AI model receives a prompt as input and uses natural language processing techniques to generate an answer, which is then sent back to the server.

[0759] Input: Prompt text sent from the server

[0760] Output: The answer returned by the generative AI model (e.g., "You can say 'I watched a movie yesterday.'")

[0761] Step 5:

[0762] Process description: The server sends a response back to the device.

[0763] Specific operation: The server receives the answer from the generative AI model, converts it into a format that is easy for the user to understand, generates JSON data containing the converted answer, and sends it to the terminal as an HTTP response.

[0764] Input: The answer returned by the generative AI model

[0765] Output: JSON data sent to the device (e.g., {"answer": "You can say 'I watched a movie yesterday'."})

[0766] Step 6:

[0767] Process Description: The device displays the answer to the user.

[0768] Specific operation: The device parses the JSON data received from the server, extracts the answer, and displays it in the text area on the screen.

[0769] Input: JSON data sent from the server

[0770] Output: The answer that is displayed to the user (e.g., "You can say 'I watched a movie yesterday'.")

[0771] These steps will help users progress through language learning efficiently and effectively.

[0772] (Application example 1)

[0773] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0774] In modern content distribution services, when users encounter a word or phrase they don't understand while watching, they need a way to quickly ask questions and receive answers on the spot. However, existing systems lack the functionality to answer questions in real time while watching, which reduces user learning efficiency. Furthermore, it is difficult to implement advanced natural language processing technology that can provide appropriate answers to questions.

[0775] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0776] In this invention, the server includes means for receiving questions about information from users, means for transmitting the received questions to a generative artificial intelligence, means for the generative artificial intelligence to generate answers based on the questions, means for the server to receive the answers and return them to the display device, means for the display device to display the received answers to the user, and means for the user to ask questions about the content being displayed in real time, thereby enabling users to instantly resolve any questions that arise while viewing, and to learn efficiently.

[0777] "User" refers to any individual or entity that uses the System.

[0778] "Information" refers to the knowledge and data that users ask about, such as language, grammar, and methods of expression.

[0779] "Question" refers to the inquiry a user enters into the system to confirm the information they are seeking.

[0780] "Network equipment" refers to devices used to send and receive data, such as servers and routers.

[0781] "Generative artificial intelligence" refers to a system that has natural language processing capabilities and generates appropriate answers to input questions.

[0782] "Answer" refers to appropriate information or data generated by generative artificial intelligence in response to a user's question.

[0783] "Display device" refers to a device on which the user can view the answers, such as a smartphone, tablet, or computer.

[0784] "Content" refers to media for viewing provided by a distribution service, such as movies, dramas, and news.

[0785] "Real-time" refers to minimal delay between the moment a user asks a question and the moment a response is received.

[0786] "Learning" refers to the process by which a user acquires new knowledge, such as language or grammar.

[0787] The present invention relates to a language learning system for content distribution services that allows users to ask questions in real time about words or phrases they do not understand while watching, and provides answers using generative artificial intelligence.

[0788] This system is configured as follows:

[0789] System configuration

[0790] A user inputs a question via their device (smartphone, tablet, PC, etc.) and sends it to the server. At this time, network equipment is responsible for sending and receiving the data. The question received by the server is sent to a generative artificial intelligence, which generates an appropriate answer. The generated answer is then sent back to the display device via the server and displayed to the user.

[0791] Program processing

[0792] The hardware and software used are as follows:

[0793] Hardware: Servers (e.g., AWS EC2), user devices (smartphones, tablets, PCs)

[0794] Software: Flask (server), external natural language processing API (e.g., GPT-3)

[0795] The flow of data processing and data calculation is as follows:

[0796] 1. The user inputs a question from the device. For example, "The user asked a question about a specific scene while watching a movie. Question: 'What is the concept of "dream within a dream" in the movie "Inception"?'"

[0797] 2. The device sends the entered question to the server via a network device. The data is sent in a standard format such as JSON.

[0798] 3. The server sends the received question to the generative AI, which generates an answer based on the received question. For example, it generates an answer such as, "The concept of 'dream within a dream' refers to multiple layers of dreams where each layer is a dream within the previous one."

[0799] 4. The server receives the generated answer and, if necessary, converts it into a user-friendly format.

[0800] 5. The converted answer is sent back from the server to the display device and displayed to the user. Specifically, it is displayed in the form of "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."

[0801] Adding specific examples

[0802] For example, a system that answers questions about scenes in a movie a user is watching in real time might use the following prompt:

[0803] "A user is watching a movie and has a question about a particular scene. Please answer the following question: Question: 'What is the concept of "dream within a dream" in the movie "Inception"?'"

[0804] This invention allows users to instantly resolve any questions they may have while watching, providing an efficient learning experience.

[0805] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0806] Step 1:

[0807] The user inputs a question from the device. The user enters a question about the content they are watching in text format into the device's input form and presses the send button. For example, the user might input a question such as, "What is the concept of 'dream within a dream' in the movie 'Inception'?" The input at this time is treated as text data.

[0808] Step 2:

[0809] The device sends the entered question to the server via a network device. The device converts the question into JSON format and sends the data to the server using an HTTP request. Specifically, data in the format {"question": "What is the concept of 'dream within a dream' in the movie 'Inception'?"} is sent.

[0810] Step 3:

[0811] The server parses the received question and sends it to the generative AI in an appropriate format. The server receives the JSON-formatted data, extracts the contents of the "question" field, and sends an API request to the generative AI. At this time, a prompt is formed, and the body of the API request includes a prompt such as, "A user asked a question about a specific scene while watching a movie. Please answer the following question. Question: 'What is the concept of "dream within a dream" in the movie "Inception"?'"

[0812] Step 4:

[0813] The generative AI analyzes the received query and generates an answer to the question. The generative AI uses a natural language processing model to analyze the prompt sentence and generate an appropriate response. For example, an answer may be generated in the form of "The concept of 'dream within a dream' refers to multiple layers of dreams where each layer is a dream within the previous one." This answer is sent back to the server in JSON format.

[0814] Step 5:

[0815] The server receives the generated answer and, if necessary, converts it into a format that is easy for the user to understand. The server analyzes the answer received from the generative AI and converts it into language and expressions that are easy for the user to understand. Specifically, it performs processing such as translating the generated English answer into Japanese. As a result, for example, the answer may be converted into a form such as, "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."

[0816] Step 6:

[0817] The server returns the converted answer to the device. The server then converts the answer back into JSON format and sends it to the device as an HTTP response. For example, data such as {"answer": "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."} is sent.

[0818] Step 7:

[0819] The device displays the received answer to the user. The device parses the received JSON data, extracts the contents of the "answer" field, and displays it on the screen. This allows the user to instantly resolve any questions they may have while watching. For example, the screen displays the following: "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."

[0820] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0821] The present invention is a system for supporting language learning through collaboration between a user, a terminal, a server, a generative artificial intelligence, and an emotion engine. The system includes a process in which a user asks a language-related question, a generative artificial intelligence provides an answer to the question, and the system further adjusts the answer by recognizing the user's emotion.

[0822] System configuration

[0823] A user operates a language learning app using a device such as a smartphone, tablet, or PC. When the user enters a language-related question, the question is sent from the device to a server. The server analyzes the question and recognizes the user's emotion using an emotion engine. The analyzed question and the recognized emotion are sent to a generative artificial intelligence (e.g., a natural language processing model) to generate an answer. The answer is sent back to the device via the server, and the device displays it to the user.

[0824] Specific examples

[0825] For example, consider a case where a user has a question about English grammar. The user might type, "How do I say 'I saw a movie yesterday' in English?" The emotion engine then analyzes the emotion from the text and voice input by the user.

[0826] 1. User enters question

[0827] The user enters a question into the app's text box and presses the submit button. For example, "How do I say 'I saw a movie yesterday' in English?" The emotion engine simultaneously analyzes the user's input text and identifies the emotion.

[0828] 2. Send the question to the server

[0829] The terminal receives the entered question, converts it into a specific format (e.g., JSON format), adds the analysis results of the emotion engine, and sends an HTTP request to the server.

[0830] 3. The server queries the generative AI with the question and emotion

[0831] The server sends the received question and the emotion analysis results to the generative AI, which receives the form "How do you say 'I watched a movie yesterday' in English?" and the user's emotional state.

[0832] 4. Generative AI generates answers

[0833] Generative AI generates answers based on the information sent. For example, if there is emotional information that the user is confused, the answer will be gentler. An example of a generated answer would be a response that includes gentle advice such as, "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!"

[0834] 5. Send the answer back to the device

[0835] The server receives the answer from the generative AI, converts it into a format that is easy for the user to understand if necessary, and sends it back to the device. For example, it sends JSON data such as {"answer": "Just say 'I watched a movie yesterday'. Don't worry, this is easy."} as an HTTP response.

[0836] 6. Show the answer to the user

[0837] The device analyzes the answer received from the server and displays it to the user. Specifically, it displays the message "Just say 'I watched a movie yesterday'. Don't worry, it's easy."

[0838] The system of the present invention allows users to ask language-related questions anytime, anywhere and receive immediate, appropriate answers. Furthermore, by using generative artificial intelligence, the system can provide advice on grammar and expressions, as well as provide answers that take the user's emotions into consideration by using an emotion engine. This reduces learning stress for users and allows them to learn languages ​​more effectively.

[0839] The processing flow will be explained below.

[0840] Step 1:

[0841] A user opens the app and types in a language-related question, such as "How do I say 'I saw a movie yesterday' in English?"

[0842] Step 2:

[0843] The device receives the user's input and detects when the send button is pressed. At the same time, the device sends the user's input text or voice data to the emotion engine to analyze the user's emotions.

[0844] Step 3:

[0845] The device sends the entered question to the server in a specific format (e.g., JSON format). For example, the following JSON data is sent via an HTTP POST request: {"question": "How should I say 'I saw a movie yesterday' in English?", "emotion": "confused"}

[0846] Step 4:

[0847] The server receives requests from the device and analyzes the question and emotional information, converting the user's question and emotional state into an easy-to-understand format.

[0848] Step 5:

[0849] The server sends the analyzed question and emotional information to the generative AI. For example, it sends the question in the form "How do you say 'I watched a movie yesterday' in English?" along with the emotional information "confused."

[0850] Step 6:

[0851] The generative AI generates an answer based on the question and emotional information received from the server. For example, if the user is in a "confused" emotional state, it generates an answer with gentle advice in the form of "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!"

[0852] Step 7:

[0853] The server receives the generated answer and formats it as needed, for example translating it into the user's language to make it easier to understand.

[0854] Step 8:

[0855] The server returns a formatted answer to the device. For example, it sends the following JSON data as an HTTP response: {"answer": "Just say 'I watched a movie yesterday'. Don't worry, this is easy."}

[0856] Step 9:

[0857] The device parses the response it receives from the server and prepares it for display to the user, converting it into the appropriate format for display on the screen.

[0858] Step 10:

[0859] The device will then display the parsed answer in a user interface, for example, "Just say 'I watched a movie yesterday.' Don't worry, this is easy."

[0860] Example 2

[0861] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0862] Conventional language learning systems simply provide answers to questions entered by users, making it difficult to provide appropriate instruction or feedback that takes into account the user's emotional state. Therefore, there is a need for a system that can flexibly respond to learners' emotional needs, reduce learning stress, and provide effective education.

[0863] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0864] In this invention, the server includes means for receiving language-related questions from a user, means for the terminal to convert the received question into a specific format and transmit it to the server, means for an emotion engine built into the terminal to analyze the question and the user's emotion, means for the server to transmit the received question and emotional state to a generative artificial intelligence and make a query in the form of a prompt sentence, means for the generative artificial intelligence to generate an answer taking the emotional state into consideration, means for the server to transmit the generated answer to the terminal, and means for the terminal to display the answer received to the user. This makes it possible to provide an appropriate answer according to the user's emotional state, reduce learning stress, and enable effective language learning.

[0865] "User" refers to an individual who uses the language learning system.

[0866] "Terminal" refers to devices operated by users, such as smartphones, tablets, and PCs.

[0867] "Server" refers to a central device that communicates with terminals via a network and processes questions and answers.

[0868] "Emotion engine" refers to a software component that analyzes a user's emotional state from input information.

[0869] "Generative artificial intelligence" refers to an algorithm or model that uses natural language processing to generate appropriate answers from a given prompt.

[0870] "Specific format" refers to a standardized data format (e.g., JSON format) used by a terminal when sending data to a server.

[0871] A "prompt sentence" refers to an input sentence used to give a specific question or command to a generative artificial intelligence.

[0872] The present invention is a system for supporting language learning through collaboration between a user, a terminal, a server, a generative artificial intelligence, and an emotion engine. The system includes a process in which a user asks a language-related question, a generative artificial intelligence provides an answer to the question, and the system further recognizes the user's emotion and adjusts the answer.

[0873] System configuration

[0874] A user operates a language learning app using a device such as a smartphone, tablet, or PC. When the user enters a language-related question, the question is sent from the device to a server. The server analyzes the question and recognizes the user's emotion using an emotion engine. The analyzed question and the recognized emotion are sent to a generative artificial intelligence (e.g., a natural language processing model) to generate an answer. The answer is sent back to the device via the server, and the device displays it to the user.

[0875] Hardware and software used

[0876] Devices: Smartphones, tablets, PCs, etc.

[0877] Server: A cloud-based server or an on-premise server.

[0878] Emotion engine: Software that analyzes emotions from user input information.

[0879] Generative AI: Uses natural language processing models (e.g., GPT-3).

[0880] Data processing and calculation

[0881] Data entry: The user enters a question into the device by text or voice.

[0882] Sentiment Analysis: The emotion engine analyzes user input to identify their emotional state.

[0883] Data transmission: The question and sentiment analysis results are converted into JSON format and sent to the server.

[0884] Use of generative artificial intelligence: Generative artificial intelligence generates answers based on the question and emotional state.

[0885] Specific examples

[0886] For example, consider a case where a user has a question about English grammar. The user inputs, "How should I say 'I saw a movie yesterday' in English?" At this time, the emotion engine analyzes the emotion from the text and voice input by the user.

[0887] 1. User enters a question

[0888] The user types a question into the app's text box and presses the send button. For example, they might type, "How do I say 'I saw a movie yesterday' in English?"

[0889] 2. Emotion analysis

[0890] An emotion engine within the device analyzes the input text and recognizes the user's emotional state. For example, it may determine that the user is confused.

[0891] 3. Send the question and emotion to the server

[0892] The device converts the question and sentiment analysis results into JSON format and sends them to the server.

[0893] 4. The server queries the generative AI

[0894] The server sends this data to the generative AI in the form of a prompt: "How do you say 'I watched a movie yesterday' in English?" and the user seems confused."

[0895] 5. Generative AI generates answers

[0896] Generative AI generates answers based on the information submitted, such as "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!"

[0897] 6. Returning and Displaying Answers

[0898] The server receives the generated answer and sends it back to the device. The device then analyzes the answer and displays it to the user. Specifically, the screen displays the message, "Just say 'I watched a movie yesterday.' Don't worry, this is easy."

[0899] This system allows users to ask language-related questions anytime, anywhere and receive instant, appropriate answers. It also uses generative AI to provide advice on grammar and expressions, and an emotion engine to provide answers that take the user's emotions into consideration. This reduces learning stress for users and allows them to learn languages ​​more effectively.

[0900] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0901] Step 1:

[0902] The user inputs a language-related question. Using a device such as a smartphone or PC, the user operates a language learning app, inputting "How do I say 'I saw a movie yesterday' in English?" and pressing the send button. The input text is saved in the device's internal memory.

[0903] Step 2:

[0904] The device sends the question and the emotion analysis results to the server. The emotion engine analyzes the text entered by the user and determines the user's emotional state (for example, confusion). The analysis results and the question text are converted into JSON format and an HTTP request is sent to the server. The input is the user's question and emotional state, and the output is the JSON data sent to the server.

[0905] Step 3:

[0906] The server queries the generative AI for the question and the result of the sentiment analysis. The server analyzes the received JSON data, extracts the question content and the user's emotional state, and converts it into a prompt format. The specific prompt is "How do you say 'I watched a movie yesterday' in English? The user seems confused." The server sends the prompt to the generative AI. The input is JSON data, and the output is the prompt text for the generative AI.

[0907] Step 4:

[0908] A generative AI generates an answer. A generative AI (e.g., GPT-3) receives a prompt and generates an appropriate answer. For example, it generates the answer "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!" The input is the prompt, and the output is the generated answer text.

[0909] Step 5:

[0910] The server returns the generated answer to the device. The server receives the answer from the generative AI and converts it into a format that is easy for the user to understand. The converted data is converted into JSON format and sent to the device as an HTTP response. The input is the answer from the generative AI, and the output is the JSON data sent to the device.

[0911] Step 6:

[0912] The terminal displays the answer to the user. The terminal parses the received JSON data and displays the answer on the screen. Specifically, it displays "Just say 'I watched a movie yesterday'. Don't worry, this is easy." The input is the JSON data received from the server, and the output is the text displayed on the user's screen.

[0913] (Application example 2)

[0914] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0915] Conventional electronic payment systems do not adequately address situations where users feel confused or uneasy about their operations. Therefore, there is a need for a method to alleviate users' anxiety and provide a smooth operational experience.

[0916] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for processing user data, means for analyzing the user's emotions, and means for causing a generative artificial intelligence to generate advice based on the emotional information. This makes it possible to provide appropriate operation guidance and reassuring messages based on the user's emotional information.

[0917] "User Data" means information that a User inputs into the System, including text, audio, images, etc.

[0918] An "analyzing means" is a method or device for analyzing received data and extracting specific information.

[0919] "Means for analyzing emotions" refers to methods or techniques for identifying a user's emotional state from input user data and generating emotional information.

[0920] A "server" is a computer system that stores, processes, and distributes data over a network.

[0921] "Generative AI" is an AI technology that generates appropriate responses and advice based on input data.

[0922] "Advice" or "operation guide" means instructions or explanations provided to a user to enable them to operate the system correctly.

[0923] "Reassuring messages" are reliable information provided to reduce users' anxiety and confusion and to facilitate smooth operation.

[0924] "Terminal" means an electronic device used by a User to interact with the System, including a smartphone, tablet, or PC.

[0925] The "displaying means" refers to a method or device for visually or audibly presenting the generated advice or message to the user.

[0926] The present invention provides a system that provides emotion-based advice and operation guidance to users when they make electronic payments. An embodiment of this system will be described in detail below.

[0927] 1. System Configuration

[0928] The system includes a user terminal, a server, generative artificial intelligence, and an emotion analysis engine to analyze the user's emotions and generate emotion-based operation guides.

[0929] User terminals are common electronic devices such as smartphones, tablets, PCs, etc. These devices are responsible for inputting and displaying data from users.

[0930] The server is a computer system that processes, stores, and distributes data. It analyzes the data received from the user's device, sends the results to the generative AI, and returns the generated advice to the device.

[0931] Generative artificial intelligence (e.g., OpenAI's GPT-4) generates appropriate advice and operation guides based on emotional information.

[0932] An emotion analysis engine (e.g., IBM Watson Emotion Analysis) analyzes data received from users and extracts emotional information.

[0933] 2. Data Flow

[0934] To initiate the electronic payment process, users interact with a smartphone app and enter data, such as text or voice instructions like, "I want to complete my payment, but I don't know what to do next."

[0935] The device sends the input data to the server. At the same time, the emotion analysis engine analyzes the user's emotions and sends the results to the server in JSON format.

[0936] The server then sends the received data and emotional information to a generative artificial intelligence (AI) system, which then generates appropriate advice and operational guidance based on the emotion. For example, it generates a specific guidance message such as, "Don't worry, you're just one step away from completing the payment!"

[0937] The generated advice is sent back to the user terminal via the server and displayed visually or audibly to the user.

[0938] 3. Hardware and Software Used

[0939] This system uses the following hardware and software:

[0940] User Device: Electronic devices such as smartphones, tablets, and PCs.

[0941] Server: A computer system that processes, stores, and delivers data.

[0942] Generative artificial intelligence: OpenAI GPT-4, etc.

[0943] Sentiment analysis engines: such as IBM Watson Emotion Analysis.

[0944] 4. Specific Examples

[0945] For example, if a user says, "I want to complete a payment, but I don't know what to do next," the sentiment analysis engine will recognize this "confusion" and send it to the generative AI, which will then provide specific operational guidance such as, "Don't worry, just press the button to complete the payment!"

[0946] Examples of prompt statements

[0947] User is feeling confused. Provide clear and comforting instructions for completing an electronic payment.

[0948] This allows users to receive appropriate advice on the spot, reducing their anxiety and enabling them to complete electronic payments smoothly.

[0949] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0950] Step 1:

[0951] A user provides input data to a device (such as a smartphone) to initiate an electronic payment. The input data can include text or voice instructions. For example, a user might say, "I want to complete the payment, but I don't know what to do next." This data is given to the device as input.

[0952] Step 2:

[0953] The device sends the input data to the emotion analysis engine. The emotion analysis engine analyzes the input data and identifies the user's emotional state. The analysis results in the user's emotion (e.g., confusion, anxiety, etc.). This is the output emotion information.

[0954] Step 3:

[0955] The device sends JSON-formatted data, including input data and emotional information output from the emotion analysis engine, to the server. The server analyzes the received data and prepares it for transmission to the generative AI. During this process, the data format is converted and necessary information is extracted.

[0956] Step 4:

[0957] The server sends the input data along with the emotion information to a generative AI (e.g., OpenAI GPT-4). The generative AI generates appropriate advice and operation guides based on the emotion information. For example, it outputs a generated guide such as, "Don't worry, just press the payment completion button!"

[0958] Step 5:

[0959] The server receives the advice generated by the generative AI and returns it to the device. This return is often in JSON format. The server converts the advice it receives into an appropriate format and sends it to the device.

[0960] Step 6:

[0961] The device analyzes the advice received from the server and displays it to the user. Specifically, it is presented as a visual display (text displayed on the screen) or audio guidance (voice output by speech synthesis). The user looks at this advice and continues the operation.

[0962] This allows users to receive appropriate guidance according to their emotional state, allowing them to smoothly proceed through the electronic payment process. The overall flow of the system also aims to reduce user anxiety.

[0963] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0964] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0965] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0966] [Fourth embodiment]

[0967] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0968] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0969] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0970] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0971] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0972] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0973] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0974] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0975] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0976] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0977] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0978] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0979] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0980] The present invention provides a system for supporting language learning through collaboration between a user, a terminal, a server, and a generative artificial intelligence. The system includes a process in which a user asks a language-related question and a generative artificial intelligence provides an answer to the question.

[0981] System configuration

[0982] Users operate language learning apps using devices such as smartphones, tablets, and PCs. When a user inputs a language-related question, the question is sent from the device to a server. The server analyzes the question, queries a generative artificial intelligence (e.g., a natural language processing model), and generates an answer. The generated answer is sent back to the device via the server, and the device displays it to the user.

[0983] Specific examples

[0984] For example, consider a user asking a question about English grammar: "How do I say 'I saw a movie yesterday' in English?"

[0985] 1. User enters question

[0986] Users enter their question into the app's text box and press the send button.

[0987] 2. Send the question to the server

[0988] The terminal receives the entered question, converts it into a specific format (e.g., JSON format), and sends an HTTP request to the server.

[0989] 3. The server analyzes the question and queries the generative AI

[0990] The server analyzes the received question and sends it to the generative AI in an appropriate format. The generative AI receives the question in a form such as "How do you say 'I watched a movie yesterday' in English?"

[0991] 4. Generative AI generates answers

[0992] Generative AI generates answers based on the information you submit, for example, "You can say 'I watched a movie yesterday.'"

[0993] 5. Send the answer back to the device

[0994] The server receives the response from the generative AI, converts it into a format that is easy for the user to understand if necessary, and sends it back to the device.

[0995] 6. Show the answer to the user

[0996] The device displays the answer received from the server to the user, specifically, "You can say 'I watched a movie yesterday.'"

[0997] The system of the present invention allows users to ask language-related questions anytime, anywhere and get instant, relevant answers. It also provides grammar and expression advice as well as dialogue scenarios for conversation practice. This feature allows users to experience a practical language learning environment.

[0998] The processing flow will be explained below.

[0999] Step 1:

[1000] A user opens the app and types in a language-related question, such as "How do I say 'I saw a movie yesterday' in English?"

[1001] Step 2:

[1002] The device receives the user's input and detects that the send button has been pressed.

[1003] Step 3:

[1004] The device sends the entered question to the server in a specific format (e.g., JSON format). For example, the JSON data {"question": "How should I say 'I saw a movie yesterday' in English?"} is sent via an HTTP POST request.

[1005] Step 4:

[1006] The server receives the request from the device and analyzes the question, converting it into an easy-to-understand form.

[1007] Step 5:

[1008] The server sends the analyzed question to the generative AI. For example, the server sends a question to the generative AI in the form of "How do you say 'I watched a movie yesterday' in English?"

[1009] Step 6:

[1010] The generative AI generates an answer based on the question received from the server. An example of a generated answer would be "You can say 'I watched a movie yesterday.'"

[1011] Step 7:

[1012] The server receives the generated answer and formats the answer as needed, including translating the answer into the user's language.

[1013] Step 8:

[1014] The server returns a formatted answer to the device. For example, it sends the JSON data {"answer": "Just say 'I watched a movie yesterday'"} as an HTTP response.

[1015] Step 9:

[1016] The device analyzes the response it receives from the server and prepares it for display to the user, converting it into a format suitable for display on the screen.

[1017] Step 10:

[1018] The device displays the analyzed answer in the user interface. For example, it displays "You can say 'I watched a movie yesterday.'"

[1019] Example 1

[1020] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1021] Current language learning systems have a complex process for quickly and accurately solving language-related questions, resulting in a poor user experience. Furthermore, when using generative AI to provide answers, there is a lack of a way to accurately analyze the user's input and generate appropriate prompts, resulting in issues with the quality of the generated answers. Furthermore, there is a lack of a way to convert the generated answers into a format that is easily understandable to the user. Therefore, improvements are needed to promote effective and efficient language learning.

[1022] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1023] In this invention, the server includes a device for receiving language-related questions from a user, a device for converting the received questions into a specific data format and sending them to the server, a device for analyzing the received questions, generating prompt sentences based on a generative AI model, and sending them to the generative AI model, a device for receiving answers from the generative AI model, converting them into a format that is easy for the user to understand, and returning the answers to the terminal, and a device for displaying the answers received by the terminal to the user. This makes it possible to accurately analyze the content of the questions entered by the user and generate appropriate prompt sentences. Furthermore, by converting the generated answers into a format that is easy for the user to understand and providing them, it is possible to improve the user experience and enable effective language learning.

[1024] A "user" is an entity that utilizes the system to enter language-related questions and receive answers.

[1025] A "terminal" is a device that a user operates to input questions and display answers, and includes common electronic devices such as smartphones, tablets, and PCs.

[1026] A "server" is a device that receives a question sent from a terminal, analyzes the question, generates an answer using a generative AI model, and returns the answer to the terminal.

[1027] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to generate appropriate answers to user questions.

[1028] A "prompt sentence" is an input sentence used to generate an answer for a generative AI model, and is an appropriate translation of the user's question.

[1029] "Device" is a general term that refers to a hardware or software component that performs a particular function or role.

[1030] A "data format" is a way of representing information based on a specific structure or rules, and includes specific examples such as JSON and XML.

[1031] An "answer" is information generated by a generative AI model and provided in response to a user's question.

[1032] This invention is a system that supports language learning through collaboration between a user, a terminal, a server, and a generative AI model. The operation of each component and the overall system process are described in detail below.

[1033] System Overview

[1034] Users operate language learning apps using devices such as smartphones, tablets, and PCs. The devices receive user questions, convert them into a specific data format (e.g., JSON format), and send them to a server. The server analyzes the received questions, generates prompts based on a generative AI model, and sends them to the generative AI model. The generative AI model generates answers based on the sent prompts and sends them back to the server. The server converts the generated answers into a format that is easy for the user to understand and sends them back to the device, where the device displays the answers to the user.

[1035] Hardware and software used

[1036] Device: A device on which a user enters questions and displays answers. Examples include smartphones, tablets, and PCs.

[1037] Server: A back-end system that receives questions, analyzes them, and sends prompts to the generative AI model. The server is implemented in a common programming language such as Python or Java.

[1038] Generative AI model: A natural language processing model that generates answers to user questions. Examples include OpenAI GPT-3.

[1039] Communication method: A method for exchanging data between the device and the server, and between the server and the generative AI model. HTTP / HTTPS protocols are commonly used.

[1040] Specific operation explanation

[1041] 1. User enters language question

[1042] A user launches a language learning app and logs in. For example, they enter a question about English learning into the text box and press the "Submit" button. For example, they might enter, "How do I say 'I saw a movie yesterday' in English?"

[1043] 2. The device sends the question to the server

[1044] The terminal converts the question entered by the user into JSON format and sends it to the server using an HTTP POST request.

[1045] 3. The server analyzes the question and sends it to the generative AI model as a prompt.

[1046] The server analyzes the data received from the device and extracts the question. It then converts the extracted question into an appropriate format (e.g., English) and generates a prompt for the generative AI model. For example, it generates a prompt such as, "How do you say 'I watched a movie yesterday' in English?"

[1047] 4. Generative AI model generates answers

[1048] The generative AI model generates an answer based on the prompt it receives, for example, "You can say 'I watched a movie yesterday.'"

[1049] 5. The server returns the answer to the device

[1050] The server receives the answer from the generative AI model, formats it into a form that is easy for the user to understand, for example, "You can say 'I watched a movie yesterday'," and sends it back to the device.

[1051] 6. The device displays the answer to the user

[1052] The device displays the answer received from the server to the user. Specifically, it displays "You can say 'I watched a movie yesterday'" in the text area on the screen.

[1053] Prompt Sentence Examples

[1054] How do you say 'I watched a movie yesterday' in English?

[1055] In this way, the system of the present invention allows users to ask language-related questions anytime and anywhere and receive appropriate answers immediately, allowing users to progress with language learning effectively and efficiently.

[1056] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1057] Step 1:

[1058] Process Description: User enters a language question.

[1059] Specific operation: The user starts the application, logs in, enters a question in the text box, and presses the "Submit" button.

[1060] Input: User question (e.g., "How do I say 'I saw a movie yesterday' in English?")

[1061] Output: The action of the user pressing the submit button

[1062] Step 2:

[1063] Process description: The device sends a query to the server.

[1064] Specific operation: The terminal converts the text entered by the user into JSON format and sends it to the server via an HTTP POST request.

[1065] Input: User question

[1066] Output: JSON data sent to the server (e.g., {"question": "How do I say 'I saw a movie yesterday' in English?"})

[1067] Step 3:

[1068] Process description: The server parses the question and sends a prompt to the generative AI model.

[1069] How it works: The server parses the received JSON data, extracts the question, converts it into an appropriate format (e.g., generates an English prompt, "How do you say 'I watched a movie yesterday' in English?"), and creates a request to send to the generative AI model.

[1070] Input: JSON data sent from the terminal

[1071] Output: The prompt sent to the generative AI model (e.g., "How do you say 'I watched a movie yesterday' in English?")

[1072] Step 4:

[1073] Process Description: The generative AI model generates an answer.

[1074] How it works: The generative AI model receives a prompt as input and uses natural language processing techniques to generate an answer, which is then sent back to the server.

[1075] Input: Prompt text sent from the server

[1076] Output: The answer returned by the generative AI model (e.g., "You can say 'I watched a movie yesterday.'")

[1077] Step 5:

[1078] Process description: The server sends a response back to the device.

[1079] Specific operation: The server receives the answer from the generative AI model, converts it into a format that is easy for the user to understand, generates JSON data containing the converted answer, and sends it to the terminal as an HTTP response.

[1080] Input: The answer returned by the generative AI model

[1081] Output: JSON data sent to the device (e.g., {"answer": "You can say 'I watched a movie yesterday'."})

[1082] Step 6:

[1083] Process Description: The device displays the answer to the user.

[1084] Specific operation: The device parses the JSON data received from the server, extracts the answer, and displays it in the text area on the screen.

[1085] Input: JSON data sent from the server

[1086] Output: The answer that is displayed to the user (e.g., "You can say 'I watched a movie yesterday'.")

[1087] These steps will help users progress through language learning efficiently and effectively.

[1088] (Application example 1)

[1089] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1090] In modern content distribution services, when users encounter a word or phrase they don't understand while watching, they need a way to quickly ask questions and receive answers on the spot. However, existing systems lack the functionality to answer questions in real time while watching, which reduces user learning efficiency. Furthermore, it is difficult to implement advanced natural language processing technology that can provide appropriate answers to questions.

[1091] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1092] In this invention, the server includes means for receiving questions about information from users, means for transmitting the received questions to a generative artificial intelligence, means for the generative artificial intelligence to generate answers based on the questions, means for the server to receive the answers and return them to the display device, means for the display device to display the received answers to the user, and means for the user to ask questions about the content being displayed in real time, thereby enabling users to instantly resolve any questions that arise while viewing, and to learn efficiently.

[1093] "User" refers to any individual or entity that uses the System.

[1094] "Information" refers to the knowledge and data that users ask about, such as language, grammar, and methods of expression.

[1095] "Question" refers to the inquiry a user enters into the system to confirm the information they are seeking.

[1096] "Network equipment" refers to devices used to send and receive data, such as servers and routers.

[1097] "Generative artificial intelligence" refers to a system that has natural language processing capabilities and generates appropriate answers to input questions.

[1098] "Answer" refers to appropriate information or data generated by generative artificial intelligence in response to a user's question.

[1099] "Display device" refers to a device on which the user can view the answers, such as a smartphone, tablet, or computer.

[1100] "Content" refers to media for viewing provided by a distribution service, such as movies, dramas, and news.

[1101] "Real-time" refers to minimal delay between the moment a user asks a question and the moment a response is received.

[1102] "Learning" refers to the process by which a user acquires new knowledge, such as language or grammar.

[1103] The present invention relates to a language learning system for content distribution services that allows users to ask questions in real time about words or phrases they do not understand while watching, and provides answers using generative artificial intelligence.

[1104] This system is configured as follows:

[1105] System configuration

[1106] A user inputs a question via their device (smartphone, tablet, PC, etc.) and sends it to the server. At this time, network equipment is responsible for sending and receiving the data. The question received by the server is sent to a generative artificial intelligence, which generates an appropriate answer. The generated answer is then sent back to the display device via the server and displayed to the user.

[1107] Program processing

[1108] The hardware and software used are as follows:

[1109] Hardware: Servers (e.g., AWS EC2), user devices (smartphones, tablets, PCs)

[1110] Software: Flask (server), external natural language processing API (e.g., GPT-3)

[1111] The flow of data processing and data calculation is as follows:

[1112] 1. The user inputs a question from the device. For example, "The user asked a question about a specific scene while watching a movie. Question: 'What is the concept of "dream within a dream" in the movie "Inception"?'"

[1113] 2. The device sends the entered question to the server via a network device. The data is sent in a standard format such as JSON.

[1114] 3. The server sends the received question to the generative AI, which generates an answer based on the received question. For example, it generates an answer such as, "The concept of 'dream within a dream' refers to multiple layers of dreams where each layer is a dream within the previous one."

[1115] 4. The server receives the generated answer and, if necessary, converts it into a user-friendly format.

[1116] 5. The converted answer is sent back from the server to the display device and displayed to the user. Specifically, it is displayed in the form of "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."

[1117] Adding specific examples

[1118] For example, a system that answers questions about scenes in a movie a user is watching in real time might use the following prompt:

[1119] "A user is watching a movie and has a question about a particular scene. Please answer the following question: Question: 'What is the concept of "dream within a dream" in the movie "Inception"?'"

[1120] This invention allows users to instantly resolve any questions they may have while watching, providing an efficient learning experience.

[1121] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1122] Step 1:

[1123] The user inputs a question from the device. The user enters a question about the content they are watching in text format into the device's input form and presses the send button. For example, the user might input a question such as, "What is the concept of 'dream within a dream' in the movie 'Inception'?" The input at this time is treated as text data.

[1124] Step 2:

[1125] The device sends the entered question to the server via a network device. The device converts the question into JSON format and sends the data to the server using an HTTP request. Specifically, data in the format {"question": "What is the concept of 'dream within a dream' in the movie 'Inception'?"} is sent.

[1126] Step 3:

[1127] The server parses the received question and sends it to the generative AI in an appropriate format. The server receives the JSON-formatted data, extracts the contents of the "question" field, and sends an API request to the generative AI. At this time, a prompt is formed, and the body of the API request includes a prompt such as, "A user asked a question about a specific scene while watching a movie. Please answer the following question. Question: 'What is the concept of "dream within a dream" in the movie "Inception"?'"

[1128] Step 4:

[1129] The generative AI analyzes the received query and generates an answer to the question. The generative AI uses a natural language processing model to analyze the prompt sentence and generate an appropriate response. For example, an answer may be generated in the form of "The concept of 'dream within a dream' refers to multiple layers of dreams where each layer is a dream within the previous one." This answer is sent back to the server in JSON format.

[1130] Step 5:

[1131] The server receives the generated answer and, if necessary, converts it into a format that is easy for the user to understand. The server analyzes the answer received from the generative AI and converts it into language and expressions that are easy for the user to understand. Specifically, it performs processing such as translating the generated English answer into Japanese. As a result, for example, the answer may be converted into a form such as, "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."

[1132] Step 6:

[1133] The server returns the converted answer to the device. The server then converts the answer back into JSON format and sends it to the device as an HTTP response. For example, data such as {"answer": "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."} is sent.

[1134] Step 7:

[1135] The device displays the received answer to the user. The device parses the received JSON data, extracts the contents of the "answer" field, and displays it on the screen. This allows the user to instantly resolve any questions they may have while watching. For example, the screen displays the following: "The concept of 'dreams within dreams' refers to multiple layers of dreams, each layer existing within the previous dream."

[1136] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1137] The present invention is a system for supporting language learning through collaboration between a user, a terminal, a server, a generative artificial intelligence, and an emotion engine. The system includes a process in which a user asks a language-related question, a generative artificial intelligence provides an answer to the question, and the system further adjusts the answer by recognizing the user's emotion.

[1138] System configuration

[1139] A user operates a language learning app using a device such as a smartphone, tablet, or PC. When the user enters a language-related question, the question is sent from the device to a server. The server analyzes the question and recognizes the user's emotion using an emotion engine. The analyzed question and the recognized emotion are sent to a generative artificial intelligence (e.g., a natural language processing model) to generate an answer. The answer is sent back to the device via the server, and the device displays it to the user.

[1140] Specific examples

[1141] For example, consider a case where a user has a question about English grammar. The user might type, "How do I say 'I saw a movie yesterday' in English?" The emotion engine then analyzes the emotion from the text and voice input by the user.

[1142] 1. User enters question

[1143] The user enters a question into the app's text box and presses the submit button. For example, "How do I say 'I saw a movie yesterday' in English?" The emotion engine simultaneously analyzes the user's input text and identifies the emotion.

[1144] 2. Send the question to the server

[1145] The terminal receives the entered question, converts it into a specific format (e.g., JSON format), adds the analysis results of the emotion engine, and sends an HTTP request to the server.

[1146] 3. The server queries the generative AI with the question and emotion

[1147] The server sends the received question and the emotion analysis results to the generative AI, which receives the form "How do you say 'I watched a movie yesterday' in English?" and the user's emotional state.

[1148] 4. Generative AI generates answers

[1149] Generative AI generates answers based on the information sent. For example, if there is emotional information that the user is confused, the answer will be gentler. An example of a generated answer would be a response that includes gentle advice such as, "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!"

[1150] 5. Send the answer back to the device

[1151] The server receives the answer from the generative AI, converts it into a format that is easy for the user to understand if necessary, and sends it back to the device. For example, it sends JSON data such as {"answer": "Just say 'I watched a movie yesterday'. Don't worry, this is easy."} as an HTTP response.

[1152] 6. Show the answer to the user

[1153] The device analyzes the answer received from the server and displays it to the user. Specifically, it displays the message "Just say 'I watched a movie yesterday'. Don't worry, it's easy."

[1154] The system of the present invention allows users to ask language-related questions anytime, anywhere and receive immediate, appropriate answers. Furthermore, by using generative artificial intelligence, the system can provide advice on grammar and expressions, as well as provide answers that take the user's emotions into consideration by using an emotion engine. This reduces learning stress for users and allows them to learn languages ​​more effectively.

[1155] The processing flow will be explained below.

[1156] Step 1:

[1157] A user opens the app and types in a language-related question, such as "How do I say 'I saw a movie yesterday' in English?"

[1158] Step 2:

[1159] The device receives the user's input and detects when the send button is pressed. At the same time, the device sends the user's input text or voice data to the emotion engine to analyze the user's emotions.

[1160] Step 3:

[1161] The device sends the entered question to the server in a specific format (e.g., JSON format). For example, the following JSON data is sent via an HTTP POST request: {"question": "How should I say 'I saw a movie yesterday' in English?", "emotion": "confused"}

[1162] Step 4:

[1163] The server receives requests from the device and analyzes the question and emotional information, converting the user's question and emotional state into an easy-to-understand format.

[1164] Step 5:

[1165] The server sends the analyzed question and emotional information to the generative AI. For example, it sends the question in the form "How do you say 'I watched a movie yesterday' in English?" along with the emotional information "confused."

[1166] Step 6:

[1167] The generative AI generates an answer based on the question and emotional information received from the server. For example, if the user is in a "confused" emotional state, it generates an answer with gentle advice in the form of "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!"

[1168] Step 7:

[1169] The server receives the generated answer and formats it as needed, for example translating it into the user's language to make it easier to understand.

[1170] Step 8:

[1171] The server returns a formatted answer to the device. For example, it sends the following JSON data as an HTTP response: {"answer": "Just say 'I watched a movie yesterday'. Don't worry, this is easy."}

[1172] Step 9:

[1173] The device parses the response it receives from the server and prepares it for display to the user, converting it into the appropriate format for display on the screen.

[1174] Step 10:

[1175] The device will then display the parsed answer in a user interface, for example, "Just say 'I watched a movie yesterday.' Don't worry, this is easy."

[1176] Example 2

[1177] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1178] Conventional language learning systems simply provide answers to questions entered by users, making it difficult to provide appropriate instruction or feedback that takes into account the user's emotional state. Therefore, there is a need for a system that can flexibly respond to learners' emotional needs, reduce learning stress, and provide effective education.

[1179] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1180] In this invention, the server includes means for receiving language-related questions from a user, means for the terminal to convert the received question into a specific format and transmit it to the server, means for an emotion engine built into the terminal to analyze the question and the user's emotion, means for the server to transmit the received question and emotional state to a generative artificial intelligence and make a query in the form of a prompt sentence, means for the generative artificial intelligence to generate an answer taking the emotional state into consideration, means for the server to transmit the generated answer to the terminal, and means for the terminal to display the answer received to the user. This makes it possible to provide an appropriate answer according to the user's emotional state, reduce learning stress, and enable effective language learning.

[1181] "User" refers to an individual who uses the language learning system.

[1182] "Terminal" refers to devices operated by users, such as smartphones, tablets, and PCs.

[1183] "Server" refers to a central device that communicates with terminals via a network and processes questions and answers.

[1184] "Emotion engine" refers to a software component that analyzes a user's emotional state from input information.

[1185] "Generative artificial intelligence" refers to an algorithm or model that uses natural language processing to generate appropriate answers from a given prompt.

[1186] "Specific format" refers to a standardized data format (e.g., JSON format) used by a terminal when sending data to a server.

[1187] A "prompt sentence" refers to an input sentence used to give a specific question or command to a generative artificial intelligence.

[1188] The present invention is a system for supporting language learning through collaboration between a user, a terminal, a server, a generative artificial intelligence, and an emotion engine. The system includes a process in which a user asks a language-related question, a generative artificial intelligence provides an answer to the question, and the system further recognizes the user's emotion and adjusts the answer.

[1189] System configuration

[1190] A user operates a language learning app using a device such as a smartphone, tablet, or PC. When the user enters a language-related question, the question is sent from the device to a server. The server analyzes the question and recognizes the user's emotion using an emotion engine. The analyzed question and the recognized emotion are sent to a generative artificial intelligence (e.g., a natural language processing model) to generate an answer. The answer is sent back to the device via the server, and the device displays it to the user.

[1191] Hardware and software used

[1192] Devices: Smartphones, tablets, PCs, etc.

[1193] Server: A cloud-based server or an on-premise server.

[1194] Emotion engine: Software that analyzes emotions from user input information.

[1195] Generative AI: Uses natural language processing models (e.g., GPT-3).

[1196] Data processing and calculation

[1197] Data entry: The user enters a question into the device by text or voice.

[1198] Sentiment Analysis: The emotion engine analyzes user input to identify their emotional state.

[1199] Data transmission: The question and sentiment analysis results are converted into JSON format and sent to the server.

[1200] Use of generative artificial intelligence: Generative artificial intelligence generates answers based on the question and emotional state.

[1201] Specific examples

[1202] For example, consider a case where a user has a question about English grammar. The user inputs, "How should I say 'I saw a movie yesterday' in English?" At this time, the emotion engine analyzes the emotion from the text and voice input by the user.

[1203] 1. User enters a question

[1204] The user types a question into the app's text box and presses the send button. For example, they might type, "How do I say 'I saw a movie yesterday' in English?"

[1205] 2. Emotion analysis

[1206] An emotion engine within the device analyzes the input text and recognizes the user's emotional state. For example, it may determine that the user is confused.

[1207] 3. Send the question and emotion to the server

[1208] The device converts the question and sentiment analysis results into JSON format and sends them to the server.

[1209] 4. The server queries the generative AI

[1210] The server sends this data to the generative AI in the form of a prompt: "How do you say 'I watched a movie yesterday' in English?" and the user seems confused."

[1211] 5. Generative AI generates answers

[1212] Generative AI generates answers based on the information submitted, such as "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!"

[1213] 6. Returning and Displaying Answers

[1214] The server receives the generated answer and sends it back to the device. The device then analyzes the answer and displays it to the user. Specifically, the screen displays the message, "Just say 'I watched a movie yesterday.' Don't worry, this is easy."

[1215] This system allows users to ask language-related questions anytime, anywhere and receive instant, appropriate answers. It also uses generative AI to provide advice on grammar and expressions, and an emotion engine to provide answers that take the user's emotions into consideration. This reduces learning stress for users and allows them to learn languages ​​more effectively.

[1216] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1217] Step 1:

[1218] The user inputs a language-related question. Using a device such as a smartphone or PC, the user operates a language learning app, inputting "How do I say 'I saw a movie yesterday' in English?" and pressing the send button. The input text is saved in the device's internal memory.

[1219] Step 2:

[1220] The device sends the question and the emotion analysis results to the server. The emotion engine analyzes the text entered by the user and determines the user's emotional state (for example, confusion). The analysis results and the question text are converted into JSON format and an HTTP request is sent to the server. The input is the user's question and emotional state, and the output is the JSON data sent to the server.

[1221] Step 3:

[1222] The server queries the generative AI for the question and the result of the sentiment analysis. The server analyzes the received JSON data, extracts the question content and the user's emotional state, and converts it into a prompt format. The specific prompt is "How do you say 'I watched a movie yesterday' in English? The user seems confused." The server sends the prompt to the generative AI. The input is JSON data, and the output is the prompt text for the generative AI.

[1223] Step 4:

[1224] A generative AI generates an answer. A generative AI (e.g., GPT-3) receives a prompt and generates an appropriate answer. For example, it generates the answer "You can say 'I watched a movie yesterday.' Don't worry, it's quite simple!" The input is the prompt, and the output is the generated answer text.

[1225] Step 5:

[1226] The server returns the generated answer to the device. The server receives the answer from the generative AI and converts it into a format that is easy for the user to understand. The converted data is converted into JSON format and sent to the device as an HTTP response. The input is the answer from the generative AI, and the output is the JSON data sent to the device.

[1227] Step 6:

[1228] The terminal displays the answer to the user. The terminal parses the received JSON data and displays the answer on the screen. Specifically, it displays "Just say 'I watched a movie yesterday'. Don't worry, this is easy." The input is the JSON data received from the server, and the output is the text displayed on the user's screen.

[1229] (Application example 2)

[1230] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1231] Conventional electronic payment systems do not adequately address situations where users feel confused or uneasy about their operations. Therefore, there is a need for a method to alleviate users' anxiety and provide a smooth operational experience.

[1232] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for processing user data, means for analyzing the user's emotions, and means for causing a generative artificial intelligence to generate advice based on the emotional information. This makes it possible to provide appropriate operation guidance and reassuring messages based on the user's emotional information.

[1233] "User Data" means information that a User inputs into the System, including text, audio, images, etc.

[1234] An "analyzing means" is a method or device for analyzing received data and extracting specific information.

[1235] "Means for analyzing emotions" refers to methods or techniques for identifying a user's emotional state from input user data and generating emotional information.

[1236] A "server" is a computer system that stores, processes, and distributes data over a network.

[1237] "Generative AI" is an AI technology that generates appropriate responses and advice based on input data.

[1238] "Advice" or "operation guide" means instructions or explanations provided to a user to enable them to operate the system correctly.

[1239] "Reassuring messages" are reliable information provided to reduce users' anxiety and confusion and to facilitate smooth operation.

[1240] "Terminal" means an electronic device used by a User to interact with the System, including a smartphone, tablet, or PC.

[1241] The "displaying means" refers to a method or device for visually or audibly presenting the generated advice or message to the user.

[1242] The present invention provides a system that provides emotion-based advice and operation guidance to users when they make electronic payments. An embodiment of this system will be described in detail below.

[1243] 1. System Configuration

[1244] The system includes a user terminal, a server, generative artificial intelligence, and an emotion analysis engine to analyze the user's emotions and generate emotion-based operation guides.

[1245] User terminals are common electronic devices such as smartphones, tablets, PCs, etc. These devices are responsible for inputting and displaying data from users.

[1246] The server is a computer system that processes, stores, and distributes data. It analyzes the data received from the user's device, sends the results to the generative AI, and returns the generated advice to the device.

[1247] Generative artificial intelligence (e.g., OpenAI's GPT-4) generates appropriate advice and operation guides based on emotional information.

[1248] An emotion analysis engine (e.g., IBM Watson Emotion Analysis) analyzes data received from users and extracts emotional information.

[1249] 2. Data Flow

[1250] To initiate the electronic payment process, users interact with a smartphone app and enter data, such as text or voice instructions like, "I want to complete my payment, but I don't know what to do next."

[1251] The device sends the input data to the server. At the same time, the emotion analysis engine analyzes the user's emotions and sends the results to the server in JSON format.

[1252] The server then sends the received data and emotional information to a generative artificial intelligence (AI) system, which then generates appropriate advice and operational guidance based on the emotion. For example, it generates a specific guidance message such as, "Don't worry, you're just one step away from completing the payment!"

[1253] The generated advice is sent back to the user terminal via the server and displayed visually or audibly to the user.

[1254] 3. Hardware and Software Used

[1255] This system uses the following hardware and software:

[1256] User Device: Electronic devices such as smartphones, tablets, and PCs.

[1257] Server: A computer system that processes, stores, and delivers data.

[1258] Generative artificial intelligence: OpenAI GPT-4, etc.

[1259] Sentiment analysis engines: such as IBM Watson Emotion Analysis.

[1260] 4. Specific Examples

[1261] For example, if a user says, "I want to complete a payment, but I don't know what to do next," the sentiment analysis engine will recognize this "confusion" and send it to the generative AI, which will then provide specific operational guidance such as, "Don't worry, just press the button to complete the payment!"

[1262] Examples of prompt statements

[1263] User is feeling confused. Provide clear and comforting instructions for completing an electronic payment.

[1264] This allows users to receive appropriate advice on the spot, reducing their anxiety and enabling them to complete electronic payments smoothly.

[1265] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1266] Step 1:

[1267] A user provides input data to a device (such as a smartphone) to initiate an electronic payment. The input data can include text or voice instructions. For example, a user might say, "I want to complete the payment, but I don't know what to do next." This data is given to the device as input.

[1268] Step 2:

[1269] The device sends the input data to the emotion analysis engine. The emotion analysis engine analyzes the input data and identifies the user's emotional state. The analysis results in the user's emotion (e.g., confusion, anxiety, etc.). This is the output emotion information.

[1270] Step 3:

[1271] The device sends JSON-formatted data, including input data and emotional information output from the emotion analysis engine, to the server. The server analyzes the received data and prepares it for transmission to the generative AI. During this process, the data format is converted and necessary information is extracted.

[1272] Step 4:

[1273] The server sends the input data along with the emotion information to a generative AI (e.g., OpenAI GPT-4). The generative AI generates appropriate advice and operation guides based on the emotion information. For example, it outputs a generated guide such as, "Don't worry, just press the payment completion button!"

[1274] Step 5:

[1275] The server receives the advice generated by the generative AI and returns it to the device. This return is often in JSON format. The server converts the advice it receives into an appropriate format and sends it to the device.

[1276] Step 6:

[1277] The device analyzes the advice received from the server and displays it to the user. Specifically, it is presented as a visual display (text displayed on the screen) or audio guidance (voice output by speech synthesis). The user looks at this advice and continues the operation.

[1278] This allows users to receive appropriate guidance according to their emotional state, allowing them to smoothly proceed through the electronic payment process. The overall flow of the system also aims to reduce user anxiety.

[1279] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1280] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1281] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1282] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1283] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1284] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1285] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1286] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1287] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1288] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1289] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1290] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1291] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1292] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1293] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1294] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1295] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1296] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1297] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1298] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1299] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1300] The following is further disclosed regarding the above embodiment.

[1301] (Claim 1)

[1302] a means for receiving language-related questions from users;

[1303] means for transmitting the received query to a server;

[1304] A means for causing a generative artificial intelligence to generate an answer based on a question;

[1305] A means for the server to receive the generated answer and return it to the terminal;

[1306] The system includes means for displaying to the user the answers received by the terminal.

[1307] (Claim 2)

[1308] The system of claim 1, wherein the generative artificial intelligence provides advice on grammar and expression.

[1309] (Claim 3)

[1310] The system of claim 1, wherein the generative artificial intelligence generates a scenario for conversation practice.

[1311] "Example 1"

[1312] (Claim 1)

[1313] an apparatus for receiving a language question from a user;

[1314] a device that converts the received question into a specific data format and transmits it to a server;

[1315] a device that analyzes the received question, generates a prompt sentence based on the generative AI model, and transmits the prompt sentence to the generative AI model;

[1316] A device that receives the answer from the generative AI model, converts it into a format that is easy for the user to understand, and returns the answer to the terminal;

[1317] a device for displaying to the user the response received by the terminal;

[1318] A system including:

[1319] (Claim 2)

[1320] 10. The system of claim 1, wherein the generative AI model provides advice on grammar and expression.

[1321] (Claim 3)

[1322] 10. The system of claim 1, wherein the generative AI model generates scenarios for conversation practice.

[1323] "Application Example 1"

[1324] (Claim 1)

[1325] a means for receiving informational questions from users;

[1326] means for transmitting the received query to a network device;

[1327] A means for causing a generative artificial intelligence to generate an answer based on a question;

[1328] a means for receiving the generated response by the network device and returning it to the display device;

[1329] means for displaying to the user the answers received by the display device;

[1330] A system that includes a means for users to ask questions in real time about the content they are viewing.

[1331] (Claim 2)

[1332] The system of claim 1, wherein the generative artificial intelligence provides advice on grammar and expression.

[1333] (Claim 3)

[1334] The system of claim 1, wherein the generative artificial intelligence generates a scenario for conversation practice.

[1335] "Example 2: Combining Emotion Engines"

[1336] (Claim 1)

[1337] means for receiving language queries from a user;

[1338] A means for the terminal to convert the received question into a specific format and transmit it to the server;

[1339] A means for an emotion engine built into the device to analyze the user's emotion along with the question;

[1340] a means for transmitting the received question and emotional state to the generative artificial intelligence by the server in the form of a prompt;

[1341] A means for the generative artificial intelligence to generate an answer taking into account the emotional state;

[1342] A means for the server to transmit the generated answer to the terminal;

[1343] The system includes means for displaying to the user the answers received by the terminal.

[1344] (Claim 2)

[1345] The system of claim 1, wherein the generative artificial intelligence provides advice on grammar and expression.

[1346] (Claim 3)

[1347] The system of claim 1, wherein the generative artificial intelligence generates a scenario for conversation practice.

[1348] "Application example 2 when combining emotion engines"

[1349] (Claim 1)

[1350] A means of receiving data from users; and

[1351] means for processing the received data;

[1352] A means of analyzing user emotions,

[1353] means for transmitting the analyzed emotion information to a server;

[1354] A means for causing a generative artificial intelligence to generate advice based on emotion information;

[1355] A means for the server to receive the generated advice and return it to the terminal;

[1356] The system includes means for displaying the advice received by the terminal to the user.

[1357] (Claim 2)

[1358] The system of claim 1, wherein the generative artificial intelligence provides operation guidance based on emotional information.

[1359] (Claim 3)

[1360] The system of claim 1, wherein the generative artificial intelligence generates a reassuring message based on emotional information. [Explanation of symbols]

[1361] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for receiving language-related questions from users; means for transmitting the received query to a server; A means for causing a generative artificial intelligence to generate an answer based on a question; A means for the server to receive the generated answer and return it to the terminal; The system includes means for displaying to the user the answers received by the terminal.

2. The system of claim 1, wherein the generative artificial intelligence provides advice on grammar and expression.

3. The system according to claim 1, wherein the generative artificial intelligence generates a scenario for conversation practice.

Images