System

Abstract

A system is provided.SOLUTION: A learning system comprising: means for receiving an input of an alphabet; means for acquiring an English word based on the input alphabet; means for outputting the acquired English word by voice; and means for displaying an illustration corresponding to the English word.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] This invention relates to a learning system that allows children to learn English naturally while playing. In particular, it aims to provide a low-cost, fun, and efficient means of English learning to solve the problem of many parents and children feeling that they are not good at English due to the high cost of conventional English language learning materials and English classes. Another objective is to create an environment where parents and children can enjoy learning English together, creating opportunities for children to be exposed to English from an early age. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means. First, a means for accepting input of alphabetic characters is provided. Next, a means for retrieving English words from a database based on the input alphabetic characters is provided. Furthermore, a means for audibly outputting the retrieved English words and a means for displaying illustrations corresponding to the English words are provided. Furthermore, a means for generating dialogue-style questions based on the English words and presenting them to the user is provided, and a means for accepting answers from the user and generating the next dialogue is also added. Furthermore, by including a means for retrieving English words and illustrations from a database on the Internet, the latest vocabulary and illustrations can be easily updated. This allows children to learn English naturally while playing, providing a learning environment that parents and children can enjoy together.

[0006] The "alphabet" is the set of 26 basic English letters, from A to Z.

[0007] "Means for accepting input" refers to an interface or method by which a user inputs information such as alphabets through a terminal or device.

[0008] An "English word" is the smallest unit of the English vocabulary, made up of a combination of letters of the alphabet, and has a specific meaning.

[0009] A "database" is an information storage device or system that systematically stores a large amount of data and is designed to be easy to search and retrieve.

[0010] "Audio output means" refers to an engine or technology that converts text information into audio and plays it back, and utilizes speech synthesis technology.

[0011] An "illustration" is a picture or image that conveys information visually and provides a visual aid for a particular English word.

[0012] "Display means" refers to a technique or method for visually presenting text, illustrations, images, etc. on the device screen.

[0013] A "conversational question" is an interactive question between a user and a system, in which the next question or response is generated based on the user's input.

[0014] A "means for accepting responses from a user" is an interface or method for obtaining responses entered by a user to interactive questions.

[0015] A "means for generating next dialogue" is an algorithm or technology that generates new questions or comments based on user input.

[0016] An "Internet database" is a database that is accessible via the Internet and that can store and update the latest information. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention relates to a learning system that allows children to learn English naturally while playing. This system accepts alphabet input, acquires English words based on the input, outputs them aloud, and displays corresponding illustrations. It can also generate simple dialogues and generate the next dialogue based on the user's answers.

[0039] System configuration

[0040] This system consists of a terminal operated by the user (child), a server that manages data on English words and illustrations, and various program components for outputting voice and displaying illustrations.

[0041] Program processing

[0042] 1. User input of alphabet

[0043] User: Enters alphabets into the input interface of the terminal. For example, the user enters "A."

[0044] 2. Sending the alphabet and acquiring English words

[0045] Terminal: The input alphabet is acquired and sent to the server. The server then acquires the corresponding English word (e.g., "Apple") from the database based on the received alphabet.

[0046] Server: Returns the acquired English word and the URL of the illustration corresponding to that word to the device.

[0047] 3. English word pronunciation and illustrations

[0048] Terminal: Receives English words sent from the server, pronounces them using a speech synthesis engine, and displays the corresponding illustrations on the screen using the corresponding illustration URL.

[0049] Specific examples

[0050] For example, when a user types "A" into a device, the device sends this to the server. The server retrieves the English word "Apple" and an illustration URL from the database and sends them back to the device. The device pronounces "Apple" using a speech synthesis engine and displays the illustration on the screen.

[0051] Conversational conversation generation

[0052] 1. Selecting the interaction mode

[0053] User: Select the interaction mode.

[0054] 2. Request to start a conversation

[0055] Terminal: Sends a request to start a conversation to the server.

[0056] Server: Generates the initial question and sends it to the device.

[0057] 3. Pronunciation and display of questions

[0058] Device: The received question is pronounced by a speech synthesis engine and displayed on the screen. For example, the question "What's your favorite color?" is displayed and pronounced aloud.

[0059] 4. Accepting User Answers

[0060] User: Enters an answer to the question. For example, enters "Blue."

[0061] Terminal: Send this response to the server.

[0062] 5. Generating the next dialogue

[0063] Server: Generates the next question or comment based on the user's answer and sends it to the device.

[0064] Specific examples

[0065] For example, if a user selects "Interactive mode," the device requests the server to ask the question "What's your favorite color?" If the user answers "Blue," the server generates the next question, "Blue is a nice color! Do you like blue flowers or blue cars?" and sends it back to the device. The device then pronounces this aloud and displays it on the screen.

[0066] Database Update

[0067] Users input new English words, illustrations, or vocabulary information that is easily confused into their device and send it to the server. The server analyzes the received information and updates the database. This allows the latest vocabulary and illustrations to be added to the database, making it possible to continuously update the learning content.

[0068] Overall flow

[0069] In this way, users can learn English naturally while playing, through a series of steps from inputting the alphabet to displaying the pronunciation and illustrations of English words, generating conversations in an interactive format, and updating the database. It also provides an environment where parents and children can learn together while having fun.

[0070] The processing flow will be explained below.

[0071] Processing alphabet input and displaying pronunciation and illustrations

[0072] Step 1:

[0073] User: Enter an alphabet (e.g., "A") into the terminal's input interface.

[0074] Step 2:

[0075] Terminal: Takes the input alphabet and sends an HTTP POST request to the server. The request contains the input alphabet.

[0076] Step 3:

[0077] Server: Analyzes the received alphabet and queries the database to obtain the corresponding English word (e.g., "Apple") and the URL of the corresponding illustration.

[0078] example:

[0079] sql

[0080] SELECT word, image_url FROM words WHERE alphabet = 'A';

[0081] Step 4:

[0082] Server: Sends an HTTP response to the terminal containing the URL of the acquired English word and illustration.

[0083] Step 5:

[0084] Terminal: The English word returned from the server (e.g., "Apple") is passed to the speech synthesis engine and pronounced. In addition, an illustration is displayed on the screen using the illustration URL.

[0085] Handling simple conversation generation in a conversational format

[0086] Step 1:

[0087] User: Selects the interaction mode in the device UI.

[0088] Step 2:

[0089] Terminal: Sends a request to start interactive mode to the server via HTTP protocol.

[0090] Step 3:

[0091] Server: Generates a simple question (e.g., "What's your favorite color?") to begin the conversation and sends it to the terminal.

[0092] Step 4:

[0093] Terminal: The question received from the server is passed to the speech synthesis engine, which pronounces it and displays the question on the screen.

[0094] Step 5:

[0095] User: Enters the answer to the question (e.g., "Blue") into the device's input field.

[0096] Step 6:

[0097] Terminal: Gets the user's answer and sends it to the server via an HTTP POST request.

[0098] Step 7:

[0099] Server: Based on the user's answers, it generates the next question or comment (e.g., "Blue is a nice color! Do you like blue flowers or blue cars?") and sends it to the device.

[0100] Step 8:

[0101] Terminal: New questions and comments received from the server are passed to the speech synthesis engine, spoken, and the new questions and comments are displayed on the screen.

[0102] Database update process

[0103] Step 1:

[0104] User: Enter information about new English words, illustrations, similar words, and commonly confused vocabulary into a dedicated form on the device.

[0105] Step 2:

[0106] Terminal: Obtains the entered information and sends it to the server via an HTTP POST request.

[0107] Step 3:

[0108] Server: Analyzes the received information and adds it as a new entry in the database or updates existing data as needed.

[0109] example:

[0110] sql

[0111] INSERT INTO similar_words (word1, word2) VALUES ('Pear', 'Pair')

[0112] ON DUPLICATE KEY UPDATE word1='Pear', word2='Pair';

[0113]

[0114] Example 1

[0115] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0116] Conventional learning systems make it difficult for children to learn English naturally while playing, and do not provide an interactive learning environment. In addition, updating the latest vocabulary and illustration information is done manually, making efficient learning difficult.

[0117] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0118] In this invention, the server includes means for accepting input of alphabets, means for acquiring vocabulary based on the input alphabets, means for audibly outputting the acquired vocabulary, means for displaying graphics corresponding to the vocabulary, means for acquiring the vocabulary from a database on the Internet, means for generating dialogue-style questions based on the vocabulary, means for presenting the questions to a user and accepting answers from the user, means for generating a next dialogue based on the answers, and means for analyzing the vocabulary and the graphics and updating the database with the latest learning content. This makes it possible to provide an interactive learning environment in which children can efficiently learn English naturally while playing and in which the latest learning content is automatically updated.

[0119] The "means for accepting alphabetic input" is an interface that provides a function for a user to input alphabetic characters using a terminal.

[0120] The "means for acquiring vocabulary" is a function for searching and acquiring corresponding vocabulary from a database based on input alphabets.

[0121] The "means for outputting vocabulary by voice" is a function that uses a voice synthesis engine to present the acquired vocabulary to the user in voice format.

[0122] "Means for displaying graphics" is a function that displays images or illustrations corresponding to acquired vocabulary on the screen.

[0123] "Means of obtaining from databases on the Internet" refers to the function of obtaining necessary vocabulary information and graphics from remote databases via a network.

[0124] The "means for generating interactive questions" is a function for generating questions to promote interactive dialogue with the user based on the acquired vocabulary.

[0125] The "means for presenting a question to a user and receiving an answer from the user" is a function for presenting a generated question to a user by voice or text and receiving an answer from the user.

[0126] The "means for generating the next dialogue" is a function that generates the next question or comment based on the user's answer and continues the interactive dialogue.

[0127] "Means for updating the latest learning content to the database" is a function that continuously updates learning content based on user input and new information and reflects it in the database.

[0128] The present invention is a learning system that allows children to learn English naturally while playing. This system is composed of a terminal operated by the user (child), a server that manages data on English words and graphics, and various program components for outputting audio and displaying images. Specific embodiments of the present invention are described below.

[0129] Hardware and software used

[0130] Terminal: A device such as a tablet, smartphone, or computer that a user interacts with through an interface.

[0131] Server: A remote computer system containing a database that manages vocabulary and graphic data and generates dialogue.

[0132] Speech synthesis engine: Software for converting text into speech. For example, Google® Text-to-Speech (TTS) or Amazon Polly can be used.

[0133] Program processing

[0134] 1. User input of alphabet

[0135] A user inputs an alphabet into the input interface of a terminal. For example, the user inputs "A."

[0136] 2. Sending the Alphabet

[0137] The device captures the input alphabet and sends it to the server as an HTTP request.

[0138] 3. Acquire English words and illustrations

[0139] The server analyzes the received alphabet and retrieves the corresponding English word (e.g., "Apple") and the URL of its illustration from the database. The retrieved information is then sent back to the device.

[0140] 4. Speech synthesis of English words and illustration display

[0141] The device receives the English word and illustration URL returned from the server, pronounces the English word using a speech synthesis engine (e.g., Google Text-to-Speech), and displays the corresponding graphic on the screen.

[0142] 5. Conversational conversation generation

[0143] The user selects a conversation mode, and the device sends a request to start a conversation to the server. The server generates an initial question (e.g., "What's your favorite color?") and sends it to the device. The device pronounces the received question using a speech synthesis engine and displays it on the screen. When the user inputs an answer to the question (e.g., "Blue"), the device sends it to the server, and the server generates the next question or comment (e.g., "Blue is a nice color! Do you like blue flowers or blue cars?") and sends it back to the device.

[0144] 6. Updating the database

[0145] The user inputs new English words, illustrations, or confusing vocabulary information into the device and sends it to the server. The server analyzes the received information and updates the database, adding the latest vocabulary and illustration data to the database.

[0146] Specific examples

[0147] For example, when a user types "A" into a device, the device sends this to the server. The server retrieves the English word "Apple" and its illustration URL from the database and sends it back to the device. The device pronounces "Apple" using a speech synthesis engine and displays the illustration on the screen.

[0148] Also, consider the case where the user selects "Interactive mode" and the terminal requests the question "What's your favorite color?" from the server. If the user answers "Blue," the server generates the next question, "Blue is a nice color! Do you like blue flowers or blue cars?" and sends it back to the terminal. The terminal then pronounces this aloud and displays it on the screen.

[0149] Prompt Sentence Examples

[0150] The following prompts can be input to the generative AI model:

[0151] "A child uses the device to enter the letter 'A'. The server retrieves the English word 'Apple' and the URL of its illustration from the database and sends it back to the device. The device pronounces 'Apple' using its speech synthesis engine and displays the illustration on the screen. If interactive mode is selected, the device pronounces the question generated by the server using its speech synthesis engine and generates the next question based on the user's answer."

[0152] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0153] Step 1:

[0154] User input of alphabetic characters

[0155] A user inputs an alphabet using the input interface (keyboard or touch screen) of a terminal. For example, a user inputs the alphabet "A." The input is displayed as a character in a text box on the terminal. This input becomes the starting point for subsequent data processing.

[0156] Step 2:

[0157] Sending the alphabet

[0158] The terminal captures the alphabet "A" entered by the user and generates an HTTP request to send it to the server. The input data (alphabet "A") is sent to the server. The HTTP request also includes metadata such as the client ID and request timestamp.

[0159] Input: The alphabet "A" entered by the user

[0160] Output: HTTP requests containing the letter "A"

[0161] What it does: The terminal program captures the user's input and constructs a request to send this data to the server.

[0162] Step 3:

[0163] Acquire English words and illustrations

[0164] The server extracts the letter "A" from the received HTTP request, generates and executes a database query, retrieves the corresponding English word (e.g., "Apple") and the URL of its illustration from the database, and returns this to the terminal as an HTTP response. The English word and illustration URL are output as the results of the database query.

[0165] Input: HTTP request containing the letter "A"

[0166] Output: HTTP response containing the English word "Apple" and an illustration URL

[0167] Specific operation: The server queries the database to retrieve the English word "Apple" and its related illustration URLs.

[0168] Step 4:

[0169] Speech synthesis of English words and illustration display

[0170] The device receives the HTTP response returned from the server and generates the English word "Apple" aloud using a text-to-speech API or speech synthesis engine (for example, Google Text-to-Speech). It also obtains image data from the obtained illustration URL and displays it on the screen. The speech synthesis generates the audio data, and the image data is rendered on the screen.

[0171] Input: HTTP response containing the English word "Apple" and an illustration URL

[0172] Output: Voice output of the English word "Apple" and display of an illustration on the screen

[0173] Specific operation: The device synthesizes "Apple" through the voice speaker and displays an illustration on the screen.

[0174] Step 5:

[0175] Selecting an Interactive Mode

[0176] The user selects an interaction mode on the terminal interface. The selection triggers the system to transition to the interactive question generation module. The selection information is recorded as internal data of the terminal.

[0177] Input: The user's action to select the interaction mode

[0178] Output:Internal flag for interactive mode selection

[0179] Specific action: The user taps an interaction mode button in the UI.

[0180] Step 6:

[0181] Conversation Initiating Request

[0182] The terminal requests the server to start the dialogue mode. This request triggers the server to generate the initial question. The terminal sends a dialogue start request to the server, and the initial question is generated based on this request.

[0183] Input: Request based on internal flag for interactive mode selection

[0184] Output: A request to generate the first question

[0185] Specific operation: The terminal sends a request to start a conversation to the server.

[0186] Step 7:

[0187] Question pronunciation and display

[0188] The server receives the dialogue start request and generates the initial question (e.g., "What's your favorite color?"). The generated question is sent to the device as an HTTP response. The device then outputs the received question aloud using a speech synthesis engine and displays it as text on the screen.

[0189] Input: A request to generate the initial question.

[0190] Output: HTTP response containing the initial question, "What's your favorite color?"

[0191] Specific operation: The device synthesizes the question, outputs it, and displays it on the screen.

[0192] Step 8:

[0193] Submitting user answers

[0194] The user uses the device to input an answer to the question, for example, "Blue." The device captures this answer and sends it to the server. The input data is sent to the server and triggers the generation of the next question.

[0195] Input: Answer data "Blue" entered by the user

[0196] Output: HTTP request with response data "Blue"

[0197] Specific operation: The user types the answer on the keyboard, and the device captures it and sends it to the server.

[0198] Step 9:

[0199] Generate the next conversation

[0200] The server receives the user's answer and generates the next question or comment, for example, "Blue is a nice color! Do you like blue flowers or blue cars?", and sends it to the terminal as an HTTP response.

[0201] Input: HTTP request containing user answer data "Blue"

[0202] Output: HTTP response containing the question "Blue is a nice color! Do you like blue flowers or blue cars?"

[0203] Specific operation: The server analyzes the response data and generates the next dialogue.

[0204] Step 10:

[0205] Database Update

[0206] The user inputs new English words, graphics, and confusing vocabulary information into the terminal and sends it to the server. The server analyzes the received information and updates the database. The updated database is reflected in subsequent data acquisition and question generation.

[0207] Input: New English words, graphics, and vocabulary information

[0208] Output: Updated database

[0209] What happens: The user enters new information, which the server parses to update the database.

[0210] (Application example 1)

[0211] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0212] Previous English learning systems for children were limited to displaying words and illustrations and outputting audio on fixed devices such as PCs and tablets, which meant that learning lacked flexibility and enjoyment. Furthermore, they lacked the functionality to deepen learning through interactive question and answer formats, and did not adequately provide an environment for children to learn English naturally. Furthermore, the database was not continually updated to reflect the latest information in real time.

[0213] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0214] In this invention, the server includes means for accepting input of alphabets, means for acquiring English words based on the input alphabets, means for audibly outputting the acquired English words, means for displaying illustrations corresponding to the English words, means for displaying the English words and the illustrations on the smart glasses and outputting them audibly, means for generating dialogue-style questions based on the English words, means for presenting the questions to the user and accepting answers from the user, means for generating a next dialogue based on the answers, and means for displaying the dialogue-style questions and answers on the smart glasses and outputting them audibly, thereby enabling children to learn English more naturally and enjoyably.

[0215] "Alphabet" refers to the 26 letters that make up the basic set of letters in the English language.

[0216] "Means for accepting input" refers to interface devices or software for receiving character information such as alphabets from the user.

[0217] The "means for obtaining English words" is a mechanism for finding and extracting corresponding English words based on the input alphabet.

[0218] "Means for audio output" refers to devices such as a speech synthesis engine or speaker that reproduces the acquired English words as sound.

[0219] "Means for displaying illustrations" refers to mechanisms or software for displaying images corresponding to English words on a screen.

[0220] "Smart glasses" are wearable devices that have a display for displaying information in front of the eyes and have audio output capabilities.

[0221] "Interactive question generation means" refers to algorithms or software that generate appropriate questions to promote English learning through dialogue with the user.

[0222] "Means for accepting responses from users" refers to an interface or software for recognizing and appropriately accepting responses from users to questions.

[0223] The "means for generating the next dialogue" refers to algorithms or software for generating the next appropriate question or comment based on the user's answers.

[0224] An "Internet database" is an information management system that stores data remotely and makes it accessible over a network.

[0225] MODE FOR CARRYING OUT THE INVENTION

[0226] The system of the present invention is a learning system that aims to help children learn English naturally while playing, and is implemented using smart glasses. The system allows users to easily input alphabets, obtain English words based on the input, output them as audio, and display corresponding illustrations. It also provides a deeper learning experience through interactive question and answer sessions.

[0227] System configuration

[0228] Hardware:

[0229] Smart glasses (e.g., Google Glass (registered trademark)): Uses a display and built-in speakers.

[0230] Server: Manages English words and illustrations and generates dialogue.

[0231] software:

[0232] Speech synthesis engine (e.g. Google Text-to-Speech API)

[0233] Database (e.g. MySQL (registered trademark))

[0234] API server (e.g. Flask)

[0235] Program processing

[0236] 1. Enter the alphabet:

[0237] The user (child) uses the touchpad of the smart glasses to input alphabets, which are recognized by software in the smart glasses and sent to the server.

[0238] 2. Acquire English words and illustrations:

[0239] The server retrieves the corresponding English word and illustration URL from the database based on the received alphabet, and this information is then sent back to the smart glasses.

[0240] 3. English word pronunciation and illustration display:

[0241] The smart glasses pronounce the received English words using a speech synthesis engine and simultaneously display illustrations on the screen.

[0242] 4. Interactive learning:

[0243] When the user selects the dialogue mode, the smart glasses display an initial question and read it aloud. The user responds by voice or using the touchpad, and the answer is sent to the server. The server generates a new question and sends it back to the smart glasses. By repeating this process, the user can learn English through dialogue.

[0244] Specific examples

[0245] Alphabet input and word generation:

[0246] When a user uses the smart glasses to input the letter "B," the smart glasses will display the word "Banana" on the display and pronounce it as "Banana."

[0247] Interactive Question and Answer:

[0248] The user selects the interactive mode and the smart glasses ask, "What is your favorite fruit?" If the user answers, "Apple," the smart glasses generate the next question, "Do you like red apples or green apples?", which the smart glasses display and pronounce.

[0249] Prompt Sentence Examples

[0250] I entered "B". What English word do you want to see and how do you pronounce it? The next question is "What is your favorite fruit?" If the user answers "Apple", please generate the next question.

[0251] This system allows children to receive real-time responses and feedback through the smart glasses, enabling them to effectively learn English while playing.

[0252] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0253] Step 1:

[0254] A user inputs alphabets using the touchpad of the smart glasses, which are then recognized by software within the smart glasses. The input of this process is the alphabets input by the user, and the output is the recognized alphabet data.

[0255] Step 2:

[0256] The smart glasses send the recognized alphabet data to the server, which receives the data and queries the database to find the English word and its corresponding illustration. The input of this process is the alphabet data sent from the smart glasses, and the output is the URL of the corresponding English word and illustration.

[0257] Step 3:

[0258] The server returns the URL of the English word and illustration retrieved from the database to the smart glasses. The smart glasses receive it, pronounce the English word using a speech synthesis engine, and display the illustration on the display. The input of this process is the URL of the English word and illustration returned from the server, and the output is the pronunciation of the English word and the display of the illustration by the smart glasses.

[0259] Step 4:

[0260] When the user selects an interaction mode, the smart glasses send a request to start the interaction to the server. The server generates an initial question and sends it to the smart glasses. The input of this process is the request to select the interaction mode, and the output is the generated question.

[0261] Step 5:

[0262] The smart glasses will then use a speech synthesis engine to pronounce the received question and display it on the display. The user can then respond to the question by voice or using the touchpad. The input to this process is the question sent from the server, and the output is the question displayed on the smart glasses' display and the voice output.

[0263] Step 6:

[0264] The user's answer is recognized by the smart glasses and sent to the server. The server generates a next question based on the received answer and sends it back to the smart glasses. The input of this process is the user's answer and the output is the generated next question.

[0265] Step 7:

[0266] The smart glasses receive the next question sent from the server, pronounce it again using the speech synthesis engine, and display it on the display. This process continues until the user finishes the dialogue, and all dialogue is recorded and the database is updated as necessary. The input of this process is the next question, and the output is the voice output and display based on that question.

[0267] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0268] This invention relates to a learning system that allows children to learn English naturally while playing. In particular, this system has the ability to recognize the user's emotions and adjust the learning content accordingly. In addition to inputting the alphabet, acquiring and pronouncing English words, displaying illustrations, and generating interactive questions, this system can also recognize the user's emotions to provide an optimal learning experience.

[0269] System configuration

[0270] This system consists of a terminal operated by the user (child), a server that manages data on English words and illustrations, various program components for outputting voice and displaying illustrations, and an emotion engine that recognizes the user's emotions.

[0271] Program processing

[0272] Processing alphabet input and displaying pronunciation and illustrations

[0273] 1. User input of alphabet

[0274] User: Enter an alphabet (e.g., "A") into the terminal's input interface.

[0275] 2. Sending the alphabet and acquiring English words

[0276] Terminal: Takes the input alphabet and sends an HTTP POST request to the server. The request contains the input alphabet.

[0277] Server: Analyzes the received alphabet and queries the database to obtain the corresponding English word (e.g., "Apple") and the URL of the corresponding illustration.

[0278] example:

[0279] sql

[0280] SELECT word, image_url FROM words WHERE alphabet = 'A';

[0281] 3. English word pronunciation and illustrations

[0282] Server: Sends an HTTP response to the terminal containing the URL of the acquired English word and illustration.

[0283] Terminal: The English word returned from the server (e.g., "Apple") is passed to the speech synthesis engine and pronounced. In addition, an illustration is displayed on the screen using the illustration URL.

[0284] Conversational conversation generation

[0285] 1. Selecting the interaction mode

[0286] User: Selects the interaction mode in the device UI.

[0287] 2. Request to start a conversation

[0288] Terminal: Sends a request to start a conversation to the server using the HTTP protocol.

[0289] Server: Generates the initial question and sends it to the device.

[0290] 3. Pronunciation and display of questions

[0291] Terminal: The question received from the server is passed to the speech synthesis engine, which pronounces it, and the question is displayed on the screen.

[0292] 4. Accepting User Answers

[0293] User: Enters the answer to the question into the input field on the device.

[0294] Terminal: Send this response to the server.

[0295] 5. Generating the next dialogue

[0296] Server: Generates the next question or comment based on the user's answer and sends it to the device.

[0297] Terminal: New questions and comments received from the server are passed to the speech synthesis engine, spoken, and the new questions and comments are displayed on the screen.

[0298] Use of emotion engine

[0299] 1. Emotional awareness

[0300] Device: The built-in camera and microphone are used to input the user's voice and facial expressions into the emotion engine.

[0301] Emotion engine: Analyzes the user's emotional state (e.g., happiness, sadness, excitement) and outputs this as data.

[0302] 2. Transmission and processing of emotional data

[0303] Terminal: Sends emotion data obtained from the emotion engine to the server.

[0304] Server: Analyzes emotional data and adjusts learning content and dialogue.

[0305] 3. Adjusting learning content

[0306] Server: Adjusts the content of the next dialogue and the difficulty of the learning task based on the user's emotions and sends that information to the terminal.

[0307] Terminal: Adjusts the screen display, audio tone, and output content based on information sent from the server.

[0308] Specific examples

[0309] For example, when a user inputs "A," the device sends this to the server, which retrieves the English word "Apple" and an illustration URL from the database and sends them to the device. The device then pronounces "Apple" and displays the illustration.

[0310] In dialogue mode, when the user is asked "What's your favorite color?" and answers "Blue," the server generates the next question: "Blue is a nice color! Do you like blue flowers or blue cars?" Furthermore, the emotion engine recognizes the user's emotions (e.g., excitement) from their facial expressions and voice, and the server adjusts the dialogue content and difficulty accordingly.

[0311] This allows children to learn English naturally while playing, providing an optimal learning experience that responds to their emotions, and creating an environment where parents and children can enjoy learning together.

[0312] The processing flow will be explained below.

[0313] Specific processing steps of a system that combines emotion engines

[0314] Processing alphabet input and displaying pronunciation and illustrations

[0315] Step 1:

[0316] User: Enter an alphabet (e.g., "A") into the terminal's input interface.

[0317] Step 2:

[0318] Terminal: Takes the input alphabet and sends an HTTP POST request to the server. The request contains the input alphabet.

[0319] Step 3:

[0320] Server: Analyzes the received alphabet and queries the database to obtain the corresponding English word (e.g., "Apple") and the URL of the corresponding illustration.

[0321] example:

[0322] sql

[0323] SELECT word, image_url FROM words WHERE alphabet = 'A';

[0324] Step 4:

[0325] Server: Sends an HTTP response to the terminal containing the URL of the acquired English word and illustration.

[0326] Step 5:

[0327] Terminal: The English word returned from the server (e.g., "Apple") is passed to the speech synthesis engine and pronounced. In addition, an illustration is displayed on the screen using the illustration URL.

[0328] Handling conversational dialogue generation

[0329] Step 1:

[0330] User: Selects the interaction mode in the device UI.

[0331] Step 2:

[0332] Terminal: Sends a request to start interactive mode to the server via HTTP protocol.

[0333] Step 3:

[0334] Server: Generates a simple question (e.g., "What's your favorite color?") to begin the conversation and sends it to the terminal.

[0335] Step 4:

[0336] Terminal: The question received from the server is passed to the speech synthesis engine, which pronounces it and displays the question on the screen.

[0337] Step 5:

[0338] User: Enters the answer to the question (e.g., "Blue") into the device's input field.

[0339] Step 6:

[0340] Terminal: Gets the user's answer and sends it to the server via an HTTP POST request.

[0341] Step 7:

[0342] Server: Based on the user's answers, it generates the next question or comment (e.g., "Blue is a nice color! Do you like blue flowers or blue cars?") and sends it to the device.

[0343] Step 8:

[0344] Terminal: New questions and comments received from the server are passed to the speech synthesis engine, spoken, and the new questions and comments are displayed on the screen.

[0345] Use of emotion engine

[0346] Step 1:

[0347] Device: The built-in camera and microphone are used to input the user's voice and facial expressions into the emotion engine.

[0348] Step 2:

[0349] Emotion engine: Analyzes the user's emotional state (e.g., happiness, sadness, excitement) and outputs this as data.

[0350] Step 3:

[0351] Terminal: Sends emotion data obtained from the emotion engine to the server.

[0352] Step 4:

[0353] Server: Analyzes emotional data and adjusts learning content and dialogue.

[0354] Step 5:

[0355] Server: Adjusts the content of the next dialogue and the difficulty of the learning task based on the user's emotions and sends that information to the terminal.

[0356] Step 6:

[0357] Terminal: Adjusts the screen display, audio tone, and output content based on information sent from the server.

[0358] Specific examples

[0359] Alphabetical input example:

[0360] 1. When the user types "A" into the terminal,

[0361] 2. The device sends this to the server,

[0362] 3. The server retrieves the English word "Apple" and the illustration URL from the database and returns them to the device.

[0363] 4. The device will pronounce "Apple" and display an illustration.

[0364] Interactive example:

[0365] 1. The user selects an interaction mode,

[0366] 2. The device requests the question "What's your favorite color?" from the server.

[0367] 3. If the user answers "Blue,"

[0368] 4. The server generates the next question, "Blue is a nice color! Do you like blue flowers or blue cars?" and sends it back to the terminal.

[0369] 5. The device will pronounce this aloud and display it on the screen.

[0370] Emotion engine example:

[0371] 1. The user's facial expressions and voice are input into the emotion engine via the device's camera and microphone.

[0372] 2. The emotion engine recognizes the user's emotion (e.g., excitement).

[0373] 3. Emotional data is sent to the server, which then adjusts the learning accordingly.

[0374] 4. The device displays tailored learning content to provide an optimal learning experience.

[0375] This will create a learning system that takes the user's emotions into consideration, allowing children to learn English naturally while having fun. It will also create an environment where parents can feel at ease and support their children's learning.

[0376] Example 2

[0377] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0378] Conventional English learning systems lacked the ability to adjust learning content based on the user's emotions, making it difficult to provide an effective learning experience. Furthermore, even when users engaged in interactive learning, the system often consisted of one-way question-and-answer exchanges, making it difficult for users to maintain their interest and attention in the progress of their learning. Furthermore, there were few integrated learning systems that included voice synthesis and illustration display.

[0379] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0380] In this invention, the server includes means for accepting alphabetic input, means for acquiring English word and illustration data based on the input alphabet, means for audibly outputting the acquired English words, means for displaying illustrations corresponding to the English words, means for recognizing the user's emotions, and means for adjusting the learning content based on the recognized emotions. This allows for an optimal learning experience that is tailored to the user's emotions, thereby maintaining the user's interest and attention. Furthermore, even in interactive learning, the content of the next dialogue and the difficulty of the learning tasks can be adjusted based on the user's emotions and responses, thereby achieving effective learning.

[0381] The "means for accepting alphabetic input" is an input interface that allows the terminal to recognize alphabetic input by the user and pass it on to subsequent processing.

[0382] "Means for obtaining data on English words and illustrations" refers to the process by which the server queries the database based on the alphabet sent from the terminal and obtains the URL of the corresponding English word and its illustration.

[0383] The "means for outputting English words by voice" is a function for converting the acquired English words into voice using a voice synthesis engine and allowing the user to hear it.

[0384] "Means for displaying illustrations" is a function for reading image data from the URL of the acquired illustration and displaying it on the terminal screen.

[0385] "Means for recognizing user emotions" refers to the process of using a built-in camera and microphone to analyze the user's facial expressions and voice to determine the user's emotional state (e.g., joy, sadness, excitement).

[0386] The "means for adjusting learning content based on recognized emotions" is a function that dynamically changes the learning content, difficulty level of interactive questions, tone, display content, etc. according to the user's emotional state.

[0387] The "means for generating interactive questions" is a process for creating questions to proceed in an interactive format with the user based on the acquired English vocabulary and learning content.

[0388] The "means for accepting an answer from a user" is an input interface for receiving an answer entered by a user in an interactive format and passing it on to subsequent processing.

[0389] "Means for generating the next dialogue" refers to the process of creating new questions or comments based on the user's answers, etc., and continuing the dialogue.

[0390] "Means of obtaining data from a database on the Internet" refers to a function that obtains the necessary English word and illustration data in real time from a remote database via a network.

[0391] The present invention is a learning system that can recognize a user's emotions and adjust the English learning content accordingly. This system is configured using the following hardware and software.

[0392] System configuration

[0393] Hardware

[0394] 1. Device: A device used by a user, such as a tablet or smartphone, that has a touchscreen, camera, and microphone.

[0395] 2. Server: A cloud computing platform, such as Amazon Web Services (AWS®) or Microsoft® Azure®, that handles database operations and API requests.

[0396] software

[0397] 1. Database: A database management system such as MySQL or PostgreSQL is used to store the English word and illustration data.

[0398] 2. Speech synthesis engine: Uses Google Text-to-Speech API or Amazon Polly to output English words aloud on the device.

[0399] 3. Emotion Recognition Engine: Recognizes user emotions using Microsoft Azure Cognitive Services and IBM Watson (registered trademark).

[0400] 4. HTTP protocol: Used for data communication between the terminal and the server.

[0401] Operation explanation

[0402] 1. Alphabetical input

[0403] The user uses the device's input interface to input alphabetic characters, which can come from a keyboard or touchscreen and are then captured by a program within the device.

[0404] 2. Data Acquisition and Display

[0405] The device receives the input alphabet and sends it as an HTTP POST request to the server. The server queries the database based on the received alphabet and obtains the corresponding English word (e.g., "Apple") and the URL of the illustration.

[0406] The acquired English words are output as voice using a speech synthesis engine, and illustrations are displayed on the device screen.

[0407] 3. Conversational conversation generation

[0408] When the user selects the interactive mode, the terminal sends a request to the server to start the interaction, and the server generates an initial question and sends it to the terminal.

[0409] When the user enters an answer to a question into the terminal, the terminal sends the answer to the server, which then generates the next question or comment and sends it back to the terminal, thus continuing the dialogue.

[0410] 4. Emotion recognition and learning adjustment

[0411] The device uses a camera and microphone to capture the user's facial expressions and voice, and inputs them into an emotion recognition engine.

[0412] The emotion recognition engine analyzes the user's emotional state and returns data based on this to the device, which then transmits this data to the server, which then adjusts the learning content and difficulty of the questions based on the user's emotions.

[0413] The adjusted learning content is sent to the terminal and presented to the user in an appropriate format.

[0414] Specific examples

[0415] For example, when a user inputs "A," the device sends the input to the server, which retrieves the English word "Apple" and its illustration from the database and sends it to the device. The device then outputs "Apple" aloud and displays the illustration on the screen.

[0416] In a conversational example, if a user is asked "What's your favorite color?" and answers "Blue," the server generates the next question: "Blue is a nice color! Do you like blue flowers or blue cars?" Furthermore, an emotion recognition engine detects excitement from the user's facial expressions and voice, and the server adjusts the tone and content of the questions accordingly.

[0417] Prompt Sentence Examples

[0418] Examples of prompts to input to a generative AI model include:

[0419] "When the user types the letter 'A', retrieve and display words and illustrations related to that letter."

[0420] "Conduct a dialogue between the user and the system, asking and answering questions in English."

[0421] These can be used to provide appropriate instructions to the generative AI model.

[0422] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0423] Step 1:

[0424] User input of alphabetic characters

[0425] Input: A user types an alphabet (e.g., "A") into the terminal's input interface.

[0426] How it works: The user enters alphabetic characters using the device's keyboard or touchscreen.

[0427] Output: The input alphabet is retrieved by the terminal program.

[0428] Step 2:

[0429] Sending the alphabet and getting English words

[0430] Input: Alphabet typed into the terminal.

[0431] Operation: The device includes the obtained alphabet in the payload of an HTTP POST request and sends it to the server.

[0432] The server receives the HTTP request and parses the alphabet.

[0433] The server performs a database query based on the parsed alphabet and retrieves the corresponding English word (e.g., "Apple") and the URL of the illustration.

[0434] Example database query:

[0435] sql

[0436] SELECT word, image_url FROM words WHERE alphabet = 'A';

[0437] Output: The URL of the acquired English word and illustration is sent from the server to the device as an HTTP response.

[0438] Step 3:

[0439] English word pronunciation and illustration display

[0440] Input: English word and illustration URL sent from the server.

[0441] How it works: The device passes the received English word to a speech synthesis engine (e.g., Google Text-to-Speech API) and pronounces it.

[0442] At the same time, the device loads the image data using the illustration's URL and displays it on the screen.

[0443] Output: Pronunciation of English words and illustrations.

[0444] Step 4:

[0445] Selecting an Interactive Mode

[0446] Input: User selection of an interactive mode.

[0447] How it works: The user selects an interaction mode in the device's UI, which triggers a request to start an interaction.

[0448] Output: Interactive mode selections are saved to the terminal and the program continues to the next step.

[0449] Step 5:

[0450] Interaction-starting request

[0451] Input: Selection information for interactive mode.

[0452] Operation: The terminal sends a request to start a conversation to the server using the HTTP protocol.

[0453] The server receives the request and generates an initial question, which can be retrieved from a database or generated based on logic.

[0454] Output: The initial question generated by the server is sent to the terminal.

[0455] Step 6:

[0456] Question pronunciation and display

[0457] Input: The question sent by the server.

[0458] How it works: The device passes the received question to a speech synthesis engine (e.g., Google Text-to-Speech API) and pronounces it aloud.

[0459] At the same time, the device displays the question on the screen.

[0460] Output: Questions are spoken and displayed on the screen.

[0461] Step 7:

[0462] Accepting user answers

[0463] Input: The user's answer to the question (e.g., "Blue").

[0464] Action: The user enters an answer into an input field on the terminal.

[0465] The terminal receives this response and sends it to the server.

[0466] Output: The user's answer sent to the server.

[0467] Step 8:

[0468] Generate the next conversation

[0469] Input: User answers and server interaction logic.

[0470] How it works: The server analyzes the user's answers and generates the next question or comment based on them.

[0471] The server sends the generated questions and comments to the terminal as an HTTP response.

[0472] Output: New questions and comments are sent to the device.

[0473] Step 9:

[0474] Emotion recognition

[0475] Input: User's facial expressions and voice.

[0476] Operation: The device uses the built-in camera and microphone to capture the user's facial expressions and voice, and inputs them into the emotion engine.

[0477] The emotion engine analyzes the user's emotional state and generates emotion data.

[0478] Output: Emotion data obtained from the emotion engine.

[0479] Step 10:

[0480] Sending and processing emotional data

[0481] Input: Emotion data.

[0482] Operation: The device sends emotion data to the server as an HTTP request.

[0483] The server analyzes the received emotional data, understands the user's emotional state, and adjusts the learning content and dialogue accordingly.

[0484] Output: Learning and dialogue content adjusted based on user sentiment.

[0485] Step 11:

[0486] Adjusting learning content

[0487] Input: Coordination information and emotion data.

[0488] Operation: The server adjusts the content of the next dialogue and the difficulty of the learning task based on the user's emotional state, and sends new learning content and dialogue content based on this to the terminal.

[0489] The terminal adjusts the screen display and audio tone based on the received adjustment content.

[0490] Output: The adjusted learning and interaction content is presented to the user.

[0491] (Application example 2)

[0492] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0493] Traditional learning systems only provide static learning content for children to learn English naturally while playing. As a result, it is difficult to maintain learners' interest and it is not possible to provide an optimal learning experience that suits each individual's emotional state and learning progress. There is also a lack of ways to provide an interactive learning experience in physical stores.

[0494] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0495] In this invention, the server includes a means for accepting alphabetic input, a means for acquiring English words based on the input alphabet, a means for audibly outputting the acquired English words, a means for displaying illustrations corresponding to the English words, a means for reading a QR code (registered trademark), a means for acquiring the alphabet by reading the QR code, a means for recognizing the user's emotions, and a means for dynamically adjusting the learning content based on the recognized emotions. This enables the system to analyze the user's (child's) emotional state and reactions in real time and provide an optimal learning experience. Furthermore, an interactive learning experience is also provided in a physical store, creating an environment where children can enjoy learning English while playing.

[0496] The "means for accepting alphabetic input" is an interface that allows the user to input any alphabetic character.

[0497] The "means for acquiring English words based on the input alphabet" is a system having a function for searching and acquiring English words based on the input alphabet.

[0498] The "means for outputting the acquired English words by voice" refers to a device or program that pronounces the acquired English words using voice synthesis technology.

[0499] The "means for displaying an illustration corresponding to the English word" is a system that displays an image related to the English word on a display device such as a monitor.

[0500] A "means for reading a QR code" is a device or program that uses a device such as a camera to scan a QR code and obtain its contents.

[0501] The "means for obtaining the alphabet by reading the QR code" is a system that has the function of analyzing the data obtained from the QR code and obtaining the corresponding alphabet.

[0502] "Means for recognizing user emotions" refers to technology that uses sensors such as cameras and microphones to analyze the user's facial expressions and tone of voice to determine their emotional state.

[0503] "Means for dynamically adjusting learning content based on recognized emotions" refers to a system that changes the difficulty level of learning content and question format in real time based on recognized emotional data.

[0504] An "interactive question generator" is a program that has the ability to automatically generate new questions based on user input and context.

[0505] The "means for accepting answers from the user" is an interface for receiving and saving answers entered by the user.

[0506] The "means for generating the next dialogue based on the answer" is a technique for analyzing the user's answer and generating the next question or comment in response to it.

[0507] System configuration

[0508] This invention is a learning system that allows users (children) to learn English naturally while playing. In particular, it has the function of dynamically adjusting the learning content according to the user's emotions using emotion recognition technology. This system is composed of a series of hardware and software.

[0509] Hardware and software used

[0510] The system uses the following major hardware and software:

[0511] Terminal: A device operated by the user, such as a smartphone or tablet, that scans QR codes, outputs audio, and displays illustrations.

[0512] Server: Connected to the central database, it retrieves English words and illustrations, generates dialogue content, and processes emotion recognition data.

[0513] Camera and microphone: The sensors required for user emotion recognition.

[0514] Software: Emotion recognition models using libraries such as OpenCV and Keras, a speech synthesis engine, and an API for processing HTTP requests.

[0515] Program processing

[0516] Alphabet input and English word acquisition

[0517] 1. Alphabet input: The user inputs any alphabet through the input interface of the terminal. This is the input method.

[0518] 2. Obtaining English words: The input alphabet is sent to the server via an HTTP request. The server then obtains the corresponding English word and illustration URL from the database.

[0519] 3. Voice output and illustration display: The acquired English words are pronounced using a speech synthesis engine, and illustrations are displayed on the device screen.

[0520] Scanning QR codes

[0521] 1. QR Code Reading: Using the device's camera, scan the QR code placed in the store. The data obtained from the QR code is analyzed to obtain the corresponding alphabet. The data is used as a means of obtaining the alphabet.

[0522] 2. Display of English words and illustrations: English words and illustrations are displayed in alphabetical order based on the scanned QR code, and the words are simultaneously read out loud.

[0523] Emotion recognition and dynamic adjustment of learning content

[0524] 1. Emotion recognition: The camera and microphone are used to analyze the user's facial expressions and voice in real time, and the emotion is determined using an emotion recognition model (Keras model). This is an emotion recognition method.

[0525] 2. Adjustment of learning content: The recognized emotion data is sent to the server, which then dynamically adjusts the learning content and the difficulty of the dialogue based on the data. This is a means of dynamically adjusting the learning content.

[0526] Interactive question generation

[0527] 1. Question generation: The server generates interactive questions based on English words, presents the questions to the user, and accepts answers from the user.

[0528] 2. Next dialogue generation: After receiving the user's answer, the server uses a generative AI model to generate the next question and dynamically adjust the content of the dialogue based on the user's emotions.

[0529] Specific examples

[0530] For example, when a child enters the letter "A" or scans a QR code, the device displays the English word "Apple" and an illustration of an apple, and pronounces "Apple." Next, if the user's emotion is recognized as "joy," the server provides dialogue content and learning tasks tailored to the emotion. For example, the next question generated might be, "Blue is a nice color! Do you like blue flowers or blue cars?"

[0531] Prompt Sentence Examples

[0532] "Generate the following English learning content for children: The user's answer is 'Blue'. What should the next question be?"

[0533] This allows the learning system to combine user emotions with dynamic emotion recognition to provide an optimal environment for learning English while having fun.

[0534] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0535] Step 1:

[0536] The terminal accepts alphabetic input from the user. The input alphabetic data is temporarily stored in the terminal. The input data is used for subsequent processing.

[0537] Step 2:

[0538] The device sends an HTTP POST request to the server based on the input alphabet. The request includes the input alphabet. The server receives the request and retrieves the URL of the English word and corresponding illustration stored in a database. This is done through a database query. The server sends the retrieved English word and illustration URL to the device as an HTTP response.

[0539] Step 3:

[0540] The device receives the response from the server, passes the retrieved English words to the speech synthesis engine to pronounce them, and displays illustrations on the screen using the illustration URL. Specifically, the speech synthesis engine converts the input English words from text to speech data. Meanwhile, the illustration display retrieves an image from the URL and renders it on the screen.

[0541] Step 4:

[0542] A user holds a device and uses the camera to read a QR code placed in a physical store. The device analyzes the data obtained from the QR code and extracts the corresponding alphabet. This alphabet is used to obtain English words and illustrations according to the procedure described above.

[0543] Step 5:

[0544] The device uses a built-in camera and microphone to collect the user's facial expressions and voice, and recognizes emotions in real time. Emotion recognition is performed using a machine learning model with the Keras library. The collected data is preprocessed and input into the model to output emotional data.

[0545] Step 6:

[0546] The device sends the recognized emotion data to the server, which analyzes the emotion data and dynamically adjusts the learning content and dialogue questions to best suit the user. Specifically, it adjusts the difficulty level of the learning content and selects dialogue content templates.

[0547] Step 7:

[0548] The server generates the next dialogue-style question based on the English words. The next question is generated using a generative AI model based on the user's response. The question content and expression are adjusted based on emotional data. The generated question is sent to the device as an HTTP response, and the device pronounces it aloud and displays it on the screen.

[0549] Step 8:

[0550] The user enters answers to questions posed by the server. The device accepts the answers and sends them to the server as an HTTP POST request. The server analyzes the user's answers and generates the next dialogue based on them. This process is repeated, and learning progresses.

[0551] Examples:

[0552] For example, when a child enters the letter "A" or scans a QR code, the device displays the English word "Apple" and an illustration of an apple, and pronounces "Apple." Next, if the user's emotion is recognized as "joy," the server provides dialogue content and learning tasks tailored to the emotion. For example, the next question generated might be, "Blue is a nice color! Do you like blue flowers or blue cars?"

[0553] Example prompt sentence:

[0554] "Generate the following English learning content for children: The user's answer is 'Blue'. What should the next question be?"

[0555] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0556] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0557] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0558] [Second embodiment]

[0559] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0560] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0561] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0562] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0563] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0564] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0565] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0566] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0567] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0568] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0569] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0570] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0571] This invention relates to a learning system that allows children to learn English naturally while playing. This system accepts alphabet input, acquires English words based on the input, outputs them aloud, and displays corresponding illustrations. It can also generate simple dialogues and generate the next dialogue based on the user's answers.

[0572] System configuration

[0573] This system consists of a terminal operated by the user (child), a server that manages data on English words and illustrations, and various program components for outputting voice and displaying illustrations.

[0574] Program processing

[0575] 1. User input of alphabet

[0576] User: Enters alphabets into the input interface of the terminal. For example, the user enters "A."

[0577] 2. Sending the alphabet and acquiring English words

[0578] Terminal: The input alphabet is acquired and sent to the server. The server then acquires the corresponding English word (e.g., "Apple") from the database based on the received alphabet.

[0579] Server: Returns the acquired English word and the URL of the illustration corresponding to that word to the device.

[0580] 3. English word pronunciation and illustrations

[0581] Terminal: Receives English words sent from the server, pronounces them using a speech synthesis engine, and displays the corresponding illustrations on the screen using the corresponding illustration URL.

[0582] Specific examples

[0583] For example, when a user types "A" into a device, the device sends this to the server. The server retrieves the English word "Apple" and an illustration URL from the database and sends them back to the device. The device pronounces "Apple" using a speech synthesis engine and displays the illustration on the screen.

[0584] Conversational conversation generation

[0585] 1. Selecting the interaction mode

[0586] User: Select the interaction mode.

[0587] 2. Request to start a conversation

[0588] Terminal: Sends a request to start a conversation to the server.

[0589] Server: Generates the initial question and sends it to the device.

[0590] 3. Pronunciation and display of questions

[0591] Device: The received question is pronounced by a speech synthesis engine and displayed on the screen. For example, the question "What's your favorite color?" is displayed and pronounced aloud.

[0592] 4. Accepting User Answers

[0593] User: Enters an answer to the question. For example, enters "Blue."

[0594] Terminal: Send this response to the server.

[0595] 5. Generating the next dialogue

[0596] Server: Generates the next question or comment based on the user's answer and sends it to the device.

[0597] Specific examples

[0598] For example, if a user selects "Interactive mode," the device requests the server to ask the question "What's your favorite color?" If the user answers "Blue," the server generates the next question, "Blue is a nice color! Do you like blue flowers or blue cars?" and sends it back to the device. The device then pronounces this aloud and displays it on the screen.

[0599] Database Update

[0600] Users input new English words, illustrations, or vocabulary information that is easily confused into their device and send it to the server. The server analyzes the received information and updates the database. This allows the latest vocabulary and illustrations to be added to the database, making it possible to continuously update the learning content.

[0601] Overall flow

[0602] In this way, users can learn English naturally while playing, through a series of steps from inputting the alphabet to displaying the pronunciation and illustrations of English words, generating conversations in an interactive format, and updating the database. It also provides an environment where parents and children can learn together while having fun.

[0603] The processing flow will be explained below.

[0604] Processing alphabet input and displaying pronunciation and illustrations

[0605] Step 1:

[0606] User: Enter an alphabet (e.g., "A") into the terminal's input interface.

[0607] Step 2:

[0608] Terminal: Takes the input alphabet and sends an HTTP POST request to the server. The request contains the input alphabet.

[0609] Step 3:

[0610] Server: Analyzes the received alphabet and queries the database to obtain the corresponding English word (e.g., "Apple") and the URL of the corresponding illustration.

[0611] example:

[0612] sql

[0613] SELECT word, image_url FROM words WHERE alphabet = 'A';

[0614] Step 4:

[0615] Server: Sends an HTTP response to the terminal containing the URL of the acquired English word and illustration.

[0616] Step 5:

[0617] Terminal: The English word returned from the server (e.g., "Apple") is passed to the speech synthesis engine and pronounced. In addition, an illustration is displayed on the screen using the illustration URL.

[0618] Handling simple conversation generation in a conversational format

[0619] Step 1:

[0620] User: Selects the interaction mode in the device UI.

[0621] Step 2:

[0622] Terminal: Sends a request to start interactive mode to the server via HTTP protocol.

[0623] Step 3:

[0624] Server: Generates a simple question (e.g., "What's your favorite color?") to begin the conversation and sends it to the terminal.

[0625] Step 4:

[0626] Terminal: The question received from the server is passed to the speech synthesis engine, which pronounces it and displays the question on the screen.

[0627] Step 5:

[0628] User: Enters the answer to the question (e.g., "Blue") into the device's input field.

[0629] Step 6:

[0630] Terminal: Gets the user's answer and sends it to the server via an HTTP POST request.

[0631] Step 7:

[0632] Server: Based on the user's answers, it generates the next question or comment (e.g., "Blue is a nice color! Do you like blue flowers or blue cars?") and sends it to the device.

[0633] Step 8:

[0634] Terminal: New questions and comments received from the server are passed to the speech synthesis engine, spoken, and the new questions and comments are displayed on the screen.

[0635] Database update process

[0636] Step 1:

[0637] User: Enter information about new English words, illustrations, similar words, and commonly confused vocabulary into a dedicated form on the device.

[0638] Step 2:

[0639] Terminal: Obtains the entered information and sends it to the server via an HTTP POST request.

[0640] Step 3:

[0641] Server: Analyzes the received information and adds it as a new entry in the database or updates existing data as needed.

[0642] example:

[0643] sql

[0644] INSERT INTO similar_words (word1, word2) VALUES ('Pear', 'Pair')

[0645] ON DUPLICATE KEY UPDATE word1='Pear', word2='Pair';

[0646]

[0647] Example 1

[0648] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0649] Conventional learning systems make it difficult for children to learn English naturally while playing, and do not provide an interactive learning environment. In addition, updating the latest vocabulary and illustration information is done manually, making efficient learning difficult.

[0650] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0651] In this invention, the server includes means for accepting input of alphabets, means for acquiring vocabulary based on the input alphabets, means for audibly outputting the acquired vocabulary, means for displaying graphics corresponding to the vocabulary, means for acquiring the vocabulary from a database on the Internet, means for generating dialogue-style questions based on the vocabulary, means for presenting the questions to a user and accepting answers from the user, means for generating a next dialogue based on the answers, and means for analyzing the vocabulary and the graphics and updating the database with the latest learning content. This makes it possible to provide an interactive learning environment in which children can efficiently learn English naturally while playing and in which the latest learning content is automatically updated.

[0652] The "means for accepting alphabetic input" is an interface that provides a function for a user to input alphabetic characters using a terminal.

[0653] The "means for acquiring vocabulary" is a function for searching and acquiring corresponding vocabulary from a database based on input alphabets.

[0654] The "means for outputting vocabulary by voice" is a function that uses a voice synthesis engine to present the acquired vocabulary to the user in voice format.

[0655] "Means for displaying graphics" is a function that displays images or illustrations corresponding to acquired vocabulary on the screen.

[0656] "Means of obtaining from databases on the Internet" refers to the function of obtaining necessary vocabulary information and graphics from remote databases via a network.

[0657] The "means for generating interactive questions" is a function for generating questions to promote interactive dialogue with the user based on the acquired vocabulary.

[0658] The "means for presenting a question to a user and receiving an answer from the user" is a function for presenting a generated question to a user by voice or text and receiving an answer from the user.

[0659] The "means for generating the next dialogue" is a function that generates the next question or comment based on the user's answer and continues the interactive dialogue.

[0660] "Means for updating the latest learning content to the database" is a function that continuously updates learning content based on user input and new information and reflects it in the database.

[0661] The present invention is a learning system that allows children to learn English naturally while playing. This system is composed of a terminal operated by the user (child), a server that manages data on English words and graphics, and various program components for outputting audio and displaying images. Specific embodiments of the present invention are described below.

[0662] Hardware and software used

[0663] Terminal: A device such as a tablet, smartphone, or computer that a user interacts with through an interface.

[0664] Server: A remote computer system containing a database that manages vocabulary and graphic data and generates dialogue.

[0665] Speech synthesis engine: Software for converting text to speech, such as Google Text-to-Speech (TTS) or Amazon Polly.

[0666] Program processing

[0667] 1. User input of alphabet

[0668] A user inputs an alphabet into the input interface of a terminal. For example, the user inputs "A."

[0669] 2. Sending the Alphabet

[0670] The device captures the input alphabet and sends it to the server as an HTTP request.

[0671] 3. Acquire English words and illustrations

[0672] The server analyzes the received alphabet and retrieves the corresponding English word (e.g., "Apple") and the URL of its illustration from the database. The retrieved information is then sent back to the device.

[0673] 4. Speech synthesis of English words and illustration display

[0674] The device receives the English word and illustration URL returned from the server, pronounces the English word using a speech synthesis engine (e.g., Google Text-to-Speech), and displays the corresponding graphic on the screen.

[0675] 5. Conversational conversation generation

[0676] The user selects a conversation mode, and the device sends a request to start a conversation to the server. The server generates an initial question (e.g., "What's your favorite color?") and sends it to the device. The device pronounces the received question using a speech synthesis engine and displays it on the screen. When the user inputs an answer to the question (e.g., "Blue"), the device sends it to the server, and the server generates the next question or comment (e.g., "Blue is a nice color! Do you like blue flowers or blue cars?") and sends it back to the device.

[0677] 6. Updating the database

[0678] The user inputs new English words, illustrations, or confusing vocabulary information into the device and sends it to the server. The server analyzes the received information and updates the database, adding the latest vocabulary and illustration data to the database.

[0679] Specific examples

[0680] For example, when a user types "A" into a device, the device sends this to the server. The server retrieves the English word "Apple" and its illustration URL from the database and sends it back to the device. The device pronounces "Apple" using a speech synthesis engine and displays the illustration on the screen.

[0681] Also, consider the case where the user selects "Interactive mode" and the terminal requests the question "What's your favorite color?" from the server. If the user answers "Blue," the server generates the next question, "Blue is a nice color! Do you like blue flowers or blue cars?" and sends it back to the terminal. The terminal then pronounces this aloud and displays it on the screen.

[0682] Prompt Sentence Examples

[0683] The following prompts can be input to the generative AI model:

[0684] "A child uses the device to enter the letter 'A'. The server retrieves the English word 'Apple' and the URL of its illustration from the database and sends it back to the device. The device pronounces 'Apple' using its speech synthesis engine and displays the illustration on the screen. If interactive mode is selected, the device pronounces the question generated by the server using its speech synthesis engine and generates the next question based on the user's answer."

[0685] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0686] Step 1:

[0687] User input of alphabetic characters

[0688] A user inputs an alphabet using the input interface (keyboard or touch screen) of a terminal. For example, a user inputs the alphabet "A." The input is displayed as a character in a text box on the terminal. This input becomes the starting point for subsequent data processing.

[0689] Step 2:

[0690] Sending the alphabet

[0691] The terminal captures the alphabet "A" entered by the user and generates an HTTP request to send it to the server. The input data (alphabet "A") is sent to the server. The HTTP request also includes metadata such as the client ID and request timestamp.

[0692] Input: The alphabet "A" entered by the user

[0693] Output: HTTP requests containing the letter "A"

[0694] What it does: The terminal program captures the user's input and constructs a request to send this data to the server.

[0695] Step 3:

[0696] Acquire English words and illustrations

[0697] The server extracts the letter "A" from the received HTTP request, generates and executes a database query, retrieves the corresponding English word (e.g., "Apple") and the URL of its illustration from the database, and returns this to the terminal as an HTTP response. The English word and illustration URL are output as the results of the database query.

[0698] Input: HTTP request containing the letter "A"

[0699] Output: HTTP response containing the English word "Apple" and an illustration URL

[0700] Specific operation: The server queries the database to retrieve the English word "Apple" and its related illustration URLs.

[0701] Step 4:

[0702] Speech synthesis of English words and illustration display

[0703] The device receives the HTTP response returned from the server and generates the English word "Apple" aloud using a text-to-speech API or speech synthesis engine (for example, Google Text-to-Speech). It also obtains image data from the obtained illustration URL and displays it on the screen. The speech synthesis generates the audio data, and the image data is rendered on the screen.

[0704] Input: HTTP response containing the English word "Apple" and an illustration URL

[0705] Output: Voice output of the English word "Apple" and display of an illustration on the screen

[0706] Specific operation: The device synthesizes "Apple" through the voice speaker and displays an illustration on the screen.

[0707] Step 5:

[0708] Selecting an Interactive Mode

[0709] The user selects an interaction mode on the terminal interface. The selection triggers the system to transition to the interactive question generation module. The selection information is recorded as internal data of the terminal.

[0710] Input: The user's action to select the interaction mode

[0711] Output:Internal flag for interactive mode selection

[0712] Specific action: The user taps an interaction mode button in the UI.

[0713] Step 6:

[0714] Conversation Initiating Request

[0715] The terminal requests the server to start the dialogue mode. This request triggers the server to generate the initial question. The terminal sends a dialogue start request to the server, and the initial question is generated based on this request.

[0716] Input: Request based on internal flag for interactive mode selection

[0717] Output: A request to generate the first question

[0718] Specific operation: The terminal sends a request to start a conversation to the server.

[0719] Step 7:

[0720] Question pronunciation and display

[0721] The server receives the dialogue start request and generates the initial question (e.g., "What's your favorite color?"). The generated question is sent to the device as an HTTP response. The device then outputs the received question aloud using a speech synthesis engine and displays it as text on the screen.

[0722] Input: A request to generate the initial question.

[0723] Output: HTTP response containing the initial question, "What's your favorite color?"

[0724] Specific operation: The device synthesizes the question, outputs it, and displays it on the screen.

[0725] Step 8:

[0726] Submitting user answers

[0727] The user uses the device to input an answer to the question, for example, "Blue." The device captures this answer and sends it to the server. The input data is sent to the server and triggers the generation of the next question.

[0728] Input: Answer data "Blue" entered by the user

[0729] Output: HTTP request with response data "Blue"

[0730] Specific operation: The user types the answer on the keyboard, and the device captures it and sends it to the server.

[0731] Step 9:

[0732] Generate the next conversation

[0733] The server receives the user's answer and generates the next question or comment, for example, "Blue is a nice color! Do you like blue flowers or blue cars?", and sends it to the terminal as an HTTP response.

[0734] Input: HTTP request containing user answer data "Blue"

[0735] Output: HTTP response containing the question "Blue is a nice color! Do you like blue flowers or blue cars?"

[0736] Specific operation: The server analyzes the response data and generates the next dialogue.

[0737] Step 10:

[0738] Database Update

[0739] The user inputs new English words, graphics, and confusing vocabulary information into the terminal and sends it to the server. The server analyzes the received information and updates the database. The updated database is reflected in subsequent data acquisition and question generation.

[0740] Input: New English words, graphics, and vocabulary information

[0741] Output: Updated database

[0742] What happens: The user enters new information, which the server parses to update the database.

[0743] (Application example 1)

[0744] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0745] Previous English learning systems for children were limited to displaying words and illustrations and outputting audio on fixed devices such as PCs and tablets, which meant that learning lacked flexibility and enjoyment. Furthermore, they lacked the functionality to deepen learning through interactive question and answer formats, and did not adequately provide an environment for children to learn English naturally. Furthermore, the database was not continually updated to reflect the latest information in real time.

[0746] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0747] In this invention, the server includes means for accepting input of alphabets, means for acquiring English words based on the input alphabets, means for audibly outputting the acquired English words, means for displaying illustrations corresponding to the English words, means for displaying the English words and the illustrations on the smart glasses and outputting them audibly, means for generating dialogue-style questions based on the English words, means for presenting the questions to the user and accepting answers from the user, means for generating a next dialogue based on the answers, and means for displaying the dialogue-style questions and answers on the smart glasses and outputting them audibly, thereby enabling children to learn English more naturally and enjoyably.

[0748] "Alphabet" refers to the 26 letters that make up the basic set of letters in the English language.

[0749] "Means for accepting input" refers to interface devices or software for receiving character information such as alphabets from the user.

[0750] The "means for obtaining English words" is a mechanism for finding and extracting corresponding English words based on the input alphabet.

[0751] "Means for audio output" refers to devices such as a speech synthesis engine or speaker that reproduces the acquired English words as sound.

[0752] "Means for displaying illustrations" refers to mechanisms or software for displaying images corresponding to English words on a screen.

[0753] "Smart glasses" are wearable devices that have a display for displaying information in front of the eyes and have audio output capabilities.

[0754] "Interactive question generation means" refers to algorithms or software that generate appropriate questions to promote English learning through dialogue with the user.

[0755] "Means for accepting responses from users" refers to an interface or software for recognizing and appropriately accepting responses from users to questions.

[0756] The "means for generating the next dialogue" refers to algorithms or software for generating the next appropriate question or comment based on the user's answers.

[0757] An "Internet database" is an information management system that stores data remotely and makes it accessible over a network.

[0758] MODE FOR CARRYING OUT THE INVENTION

[0759] The system of the present invention is a learning system that aims to help children learn English naturally while playing, and is implemented using smart glasses. The system allows users to easily input alphabets, obtain English words based on the input, output them as audio, and display corresponding illustrations. It also provides a deeper learning experience through interactive question and answer sessions.

[0760] System configuration

[0761] Hardware:

[0762] Smart glasses (e.g. Google Glass): Uses a display and built-in speakers.

[0763] Server: Manages English words and illustrations and generates dialogue.

[0764] software:

[0765] Speech synthesis engine (e.g. Google Text-to-Speech API)

[0766] Database (e.g. MySQL)

[0767] API server (e.g. Flask)

[0768] Program processing

[0769] 1. Enter the alphabet:

[0770] The user (child) uses the touchpad of the smart glasses to input alphabets, which are recognized by software in the smart glasses and sent to the server.

[0771] 2. Acquire English words and illustrations:

[0772] The server retrieves the corresponding English word and illustration URL from the database based on the received alphabet, and this information is then sent back to the smart glasses.

[0773] 3. English word pronunciation and illustration display:

[0774] The smart glasses pronounce the received English words using a speech synthesis engine and simultaneously display illustrations on the screen.

[0775] 4. Interactive learning:

[0776] When the user selects the dialogue mode, the smart glasses display an initial question and read it aloud. The user responds by voice or using the touchpad, and the answer is sent to the server. The server generates a new question and sends it back to the smart glasses. By repeating this process, the user can learn English through dialogue.

[0777] Specific examples

[0778] Alphabet input and word generation:

[0779] When a user uses the smart glasses to input the letter "B," the smart glasses will display the word "Banana" on the display and pronounce it as "Banana."

[0780] Interactive Question and Answer:

[0781] The user selects the interactive mode and the smart glasses ask, "What is your favorite fruit?" If the user answers, "Apple," the smart glasses generate the next question, "Do you like red apples or green apples?", which the smart glasses display and pronounce.

[0782] Prompt Sentence Examples

[0783] I entered "B". What English word do you want to see and how do you pronounce it? The next question is "What is your favorite fruit?" If the user answers "Apple", please generate the next question.

[0784] This system allows children to receive real-time responses and feedback through the smart glasses, enabling them to effectively learn English while playing.

[0785] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0786] Step 1:

[0787] A user inputs alphabets using the touchpad of the smart glasses, which are then recognized by software within the smart glasses. The input of this process is the alphabets input by the user, and the output is the recognized alphabet data.

[0788] Step 2:

[0789] The smart glasses send the recognized alphabet data to the server, which receives the data and queries the database to find the English word and its corresponding illustration. The input of this process is the alphabet data sent from the smart glasses, and the output is the URL of the corresponding English word and illustration.

[0790] Step 3:

[0791] The server returns the URL of the English word and illustration retrieved from the database to the smart glasses. The smart glasses receive it, pronounce the English word using a speech synthesis engine, and display the illustration on the display. The input of this process is the URL of the English word and illustration returned from the server, and the output is the pronunciation of the English word and the display of the illustration by the smart glasses.

[0792] Step 4:

[0793] When the user selects an interaction mode, the smart glasses send a request to start the interaction to the server. The server generates an initial question and sends it to the smart glasses. The input of this process is the request to select the interaction mode, and the output is the generated question.

[0794] Step 5:

[0795] The smart glasses will then use a speech synthesis engine to pronounce the received question and display it on the display. The user can then respond to the question by voice or using the touchpad. The input to this process is the question sent from the server, and the output is the question displayed on the smart glasses' display and the voice output.

[0796] Step 6:

[0797] The user's answer is recognized by the smart glasses and sent to the server. The server generates a next question based on the received answer and sends it back to the smart glasses. The input of this process is the user's answer and the output is the generated next question.

[0798] Step 7:

[0799] The smart glasses receive the next question sent from the server, pronounce it again using the speech synthesis engine, and display it on the display. This process continues until the user finishes the dialogue, and all dialogue is recorded and the database is updated as necessary. The input of this process is the next question, and the output is the voice output and display based on that question.

[0800] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0801] This invention relates to a learning system that allows children to learn English naturally while playing. In particular, this system has the ability to recognize the user's emotions and adjust the learning content accordingly. In addition to inputting the alphabet, acquiring and pronouncing English words, displaying illustrations, and generating interactive questions, this system can also recognize the user's emotions to provide an optimal learning experience.

[0802] System configuration

[0803] This system consists of a terminal operated by the user (child), a server that manages data on English words and illustrations, various program components for outputting voice and displaying illustrations, and an emotion engine that recognizes the user's emotions.

[0804] Program processing

[0805] Processing alphabet input and displaying pronunciation and illustrations

[0806] 1. User input of alphabet

[0807] User: Enter an alphabet (e.g., "A") into the terminal's input interface.

[0808] 2. Sending the alphabet and acquiring English words

[0809] Terminal: Takes the input alphabet and sends an HTTP POST request to the server. The request contains the input alphabet.

[0810] Server: Analyzes the received alphabet and queries the database to obtain the corresponding English word (e.g., "Apple") and the URL of the corresponding illustration.

[0811] example:

[0812] sql

[0813] SELECT word, image_url FROM words WHERE alphabet = 'A';

[0814] 3. English word pronunciation and illustrations

[0815] Server: Sends an HTTP response to the terminal containing the URL of the acquired English word and illustration.

[0816] Terminal: The English word returned from the server (e.g., "Apple") is passed to the speech synthesis engine and pronounced. In addition, an illustration is displayed on the screen using the illustration URL.

[0817] Conversational conversation generation

[0818] 1. Selecting the interaction mode

[0819] User: Selects the interaction mode in the device UI.

[0820] 2. Request to start a conversation

[0821] Terminal: Sends a request to start a conversation to the server using the HTTP protocol.

[0822] Server: Generates the initial question and sends it to the device.

[0823] 3. Pronunciation and display of questions

[0824] Terminal: The question received from the server is passed to the speech synthesis engine, which pronounces it, and the question is displayed on the screen.

[0825] 4. Accepting User Answers

[0826] User: Enters the answer to the question into the input field on the device.

[0827] Terminal: Send this response to the server.

[0828] 5. Generating the next dialogue

[0829] Server: Generates the next question or comment based on the user's answer and sends it to the device.

[0830] Terminal: New questions and comments received from the server are passed to the speech synthesis engine, spoken, and the new questions and comments are displayed on the screen.

[0831] Use of emotion engine

[0832] 1. Emotional awareness

[0833] Device: The built-in camera and microphone are used to input the user's voice and facial expressions into the emotion engine.

[0834] Emotion engine: Analyzes the user's emotional state (e.g., happiness, sadness, excitement) and outputs this as data.

[0835] 2. Transmission and processing of emotional data

[0836] Terminal: Sends emotion data obtained from the emotion engine to the server.

[0837] Server: Analyzes emotional data and adjusts learning content and dialogue.

[0838] 3. Adjusting learning content

[0839] Server: Adjusts the content of the next dialogue and the difficulty of the learning task based on the user's emotions and sends that information to the terminal.

[0840] Terminal: Adjusts the screen display, audio tone, and output content based on information sent from the server.

[0841] Specific examples

[0842] For example, when a user inputs "A," the device sends this to the server, which retrieves the English word "Apple" and an illustration URL from the database and sends them to the device. The device then pronounces "Apple" and displays the illustration.

[0843] In dialogue mode, when the user is asked "What's your favorite color?" and answers "Blue," the server generates the next question: "Blue is a nice color! Do you like blue flowers or blue cars?" Furthermore, the emotion engine recognizes the user's emotions (e.g., excitement) from their facial expressions and voice, and the server adjusts the dialogue content and difficulty accordingly.

[0844] This allows children to learn English naturally while playing, providing an optimal learning experience that responds to their emotions, and creating an environment where parents and children can enjoy learning together.

[0845] The processing flow will be explained below.

[0846] Specific processing steps of a system that combines emotion engines

[0847] Processing alphabet input and displaying pronunciation and illustrations

[0848] Step 1:

[0849] User: Enter an alphabet (e.g., "A") into the terminal's input interface.

[0850] Step 2:

[0851] Terminal: Takes the input alphabet and sends an HTTP POST request to the server. The request contains the input alphabet.

[0852] Step 3:

[0853] Server: Analyzes the received alphabet and queries the database to obtain the corresponding English word (e.g., "Apple") and the URL of the corresponding illustration.

[0854] example:

[0855] sql

[0856] SELECT word, image_url FROM words WHERE alphabet = 'A';

[0857] Step 4:

[0858] Server: Sends an HTTP response to the terminal containing the URL of the acquired English word and illustration.

[0859] Step 5:

[0860] Terminal: The English word returned from the server (e.g., "Apple") is passed to the speech synthesis engine and pronounced. In addition, an illustration is displayed on the screen using the illustration URL.

[0861] Handling conversational dialogue generation

[0862] Step 1:

[0863] User: Selects the interaction mode in the device UI.

[0864] Step 2:

[0865] Terminal: Sends a request to start interactive mode to the server via HTTP protocol.

[0866] Step 3:

[0867] Server: Generates a simple question (e.g., "What's your favorite color?") to begin the conversation and sends it to the terminal.

[0868] Step 4:

[0869] Terminal: The question received from the server is passed to the speech synthesis engine, which pronounces it and displays the question on the screen.

[0870] Step 5:

[0871] User: Enters the answer to the question (e.g., "Blue") into the device's input field.

[0872] Step 6:

[0873] Terminal: Gets the user's answer and sends it to the server via an HTTP POST request.

[0874] Step 7:

[0875] Server: Based on the user's answers, it generates the next question or comment (e.g., "Blue is a nice color! Do you like blue flowers or blue cars?") and sends it to the device.

[0876] Step 8:

[0877] Terminal: New questions and comments received from the server are passed to the speech synthesis engine, spoken, and the new questions and comments are displayed on the screen.

[0878] Use of emotion engine

[0879] Step 1:

[0880] Device: The built-in camera and microphone are used to input the user's voice and facial expressions into the emotion engine.

[0881] Step 2:

[0882] Emotion engine: Analyzes the user's emotional state (e.g., happiness, sadness, excitement) and outputs this as data.

[0883] Step 3:

[0884] Terminal: Sends emotion data obtained from the emotion engine to the server.

[0885] Step 4:

[0886] Server: Analyzes emotional data and adjusts learning content and dialogue.

[0887] Step 5:

[0888] Server: Adjusts the content of the next dialogue and the difficulty of the learning task based on the user's emotions and sends that information to the terminal.

[0889] Step 6:

[0890] Terminal: Adjusts the screen display, audio tone, and output content based on information sent from the server.

[0891] Specific examples

[0892] Alphabetical input example:

[0893] 1. When the user types "A" into the terminal,

[0894] 2. The device sends this to the server,

[0895] 3. The server retrieves the English word "Apple" and the illustration URL from the database and returns them to the device.

[0896] 4. The device will pronounce "Apple" and display an illustration.

[0897] Interactive example:

[0898] 1. The user selects an interaction mode,

[0899] 2. The device requests the question "What's your favorite color?" from the server.

[0900] 3. If the user answers "Blue,"

[0901] 4. The server generates the next question, "Blue is a nice color! Do you like blue flowers or blue cars?" and sends it back to the terminal.

[0902] 5. The device will pronounce this aloud and display it on the screen.

[0903] Emotion engine example:

[0904] 1. The user's facial expressions and voice are input into the emotion engine via the device's camera and microphone.

[0905] 2. The emotion engine recognizes the user's emotion (e.g., excitement).

[0906] 3. Emotional data is sent to the server, which then adjusts the learning accordingly.

[0907] 4. The device displays tailored learning content to provide an optimal learning experience.

[0908] This will create a learning system that takes the user's emotions into consideration, allowing children to learn English naturally while having fun. It will also create an environment where parents can feel at ease and support their children's learning.

[0909] Example 2

[0910] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0911] Conventional English learning systems lacked the ability to adjust learning content based on the user's emotions, making it difficult to provide an effective learning experience. Furthermore, even when users engaged in interactive learning, the system often consisted of one-way question-and-answer exchanges, making it difficult for users to maintain their interest and attention in the progress of their learning. Furthermore, there were few integrated learning systems that included voice synthesis and illustration display.

[0912] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0913] In this invention, the server includes means for accepting alphabetic input, means for acquiring English word and illustration data based on the input alphabet, means for audibly outputting the acquired English words, means for displaying illustrations corresponding to the English words, means for recognizing the user's emotions, and means for adjusting the learning content based on the recognized emotions. This allows for an optimal learning experience that is tailored to the user's emotions, thereby maintaining the user's interest and attention. Furthermore, even in interactive learning, the content of the next dialogue and the difficulty of the learning tasks can be adjusted based on the user's emotions and responses, thereby achieving effective learning.

[0914] The "means for accepting alphabetic input" is an input interface that allows the terminal to recognize alphabetic input by the user and pass it on to subsequent processing.

[0915] "Means for obtaining data on English words and illustrations" refers to the process by which the server queries the database based on the alphabet sent from the terminal and obtains the URL of the corresponding English word and its illustration.

[0916] The "means for outputting English words by voice" is a function for converting the acquired English words into voice using a voice synthesis engine and allowing the user to hear it.

[0917] "Means for displaying illustrations" is a function for reading image data from the URL of the acquired illustration and displaying it on the terminal screen.

[0918] "Means for recognizing user emotions" refers to the process of using a built-in camera and microphone to analyze the user's facial expressions and voice to determine the user's emotional state (e.g., joy, sadness, excitement).

[0919] The "means for adjusting learning content based on recognized emotions" is a function that dynamically changes the learning content, difficulty level of interactive questions, tone, display content, etc. according to the user's emotional state.

[0920] The "means for generating interactive questions" is a process for creating questions to proceed in an interactive format with the user based on the acquired English vocabulary and learning content.

[0921] The "means for accepting an answer from a user" is an input interface for receiving an answer entered by a user in an interactive format and passing it on to subsequent processing.

[0922] "Means for generating the next dialogue" refers to the process of creating new questions or comments based on the user's answers, etc., and continuing the dialogue.

[0923] "Means of obtaining data from a database on the Internet" refers to a function that obtains the necessary English word and illustration data in real time from a remote database via a network.

[0924] The present invention is a learning system that can recognize a user's emotions and adjust the English learning content accordingly. This system is configured using the following hardware and software.

[0925] System configuration

[0926] Hardware

[0927] 1. Device: A device used by a user, such as a tablet or smartphone, that has a touchscreen, camera, and microphone.

[0928] 2. Server: A cloud computing platform, such as Amazon Web Services (AWS) or Microsoft Azure, that can handle database operations and API requests.

[0929] software

[0930] 1. Database: A database management system such as MySQL or PostgreSQL is used to store the English word and illustration data.

[0931] 2. Speech synthesis engine: Uses Google Text-to-Speech API or Amazon Polly to output English words aloud on the device.

[0932] 3. Emotion recognition engine: Recognizes user emotions using Microsoft Azure Cognitive Services and IBM Watson.

[0933] 4. HTTP protocol: Used for data communication between the terminal and the server.

[0934] Operation explanation

[0935] 1. Alphabetical input

[0936] The user uses the device's input interface to input alphabetic characters, which can come from a keyboard or touchscreen and are then captured by a program within the device.

[0937] 2. Data Acquisition and Display

[0938] The device receives the input alphabet and sends it as an HTTP POST request to the server. The server queries the database based on the received alphabet and obtains the corresponding English word (e.g., "Apple") and the URL of the illustration.

[0939] The acquired English words are output as voice using a speech synthesis engine, and illustrations are displayed on the device screen.

[0940] 3. Conversational conversation generation

[0941] When the user selects the interactive mode, the terminal sends a request to the server to start the interaction, and the server generates an initial question and sends it to the terminal.

[0942] When the user enters an answer to a question into the terminal, the terminal sends the answer to the server, which then generates the next question or comment and sends it back to the terminal, thus continuing the dialogue.

[0943] 4. Emotion recognition and learning adjustment

[0944] The device uses a camera and microphone to capture the user's facial expressions and voice, and inputs them into an emotion recognition engine.

[0945] The emotion recognition engine analyzes the user's emotional state and returns data based on this to the device, which then transmits this data to the server, which then adjusts the learning content and difficulty of the questions based on the user's emotions.

[0946] The adjusted learning content is sent to the terminal and presented to the user in an appropriate format.

[0947] Specific examples

[0948] For example, when a user inputs "A," the device sends the input to the server, which retrieves the English word "Apple" and its illustration from the database and sends it to the device. The device then outputs "Apple" aloud and displays the illustration on the screen.

[0949] In a conversational example, if a user is asked "What's your favorite color?" and answers "Blue," the server generates the next question: "Blue is a nice color! Do you like blue flowers or blue cars?" Furthermore, an emotion recognition engine detects excitement from the user's facial expressions and voice, and the server adjusts the tone and content of the questions accordingly.

[0950] Prompt Sentence Examples

[0951] Examples of prompts to input to a generative AI model include:

[0952] "When the user types the letter 'A', retrieve and display words and illustrations related to that letter."

[0953] "Conduct a dialogue between the user and the system, asking and answering questions in English."

[0954] These can be used to provide appropriate instructions to the generative AI model.

[0955] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0956] Step 1:

[0957] User input of alphabetic characters

[0958] Input: A user types an alphabet (e.g., "A") into the terminal's input interface.

[0959] How it works: The user enters alphabetic characters using the device's keyboard or touchscreen.

[0960] Output: The input alphabet is retrieved by the terminal program.

[0961] Step 2:

[0962] Sending the alphabet and getting English words

[0963] Input: Alphabet typed into the terminal.

[0964] Operation: The device includes the obtained alphabet in the payload of an HTTP POST request and sends it to the server.

[0965] The server receives the HTTP request and parses the alphabet.

[0966] The server performs a database query based on the parsed alphabet and retrieves the corresponding English word (e.g., "Apple") and the URL of the illustration.

[0967] Example database query:

[0968] sql

[0969] SELECT word, image_url FROM words WHERE alphabet = 'A';

[0970] Output: The URL of the acquired English word and illustration is sent from the server to the device as an HTTP response.

[0971] Step 3:

[0972] English word pronunciation and illustration display

[0973] Input: English word and illustration URL sent from the server.

[0974] How it works: The device passes the received English word to a speech synthesis engine (e.g., Google Text-to-Speech API) and pronounces it.

[0975] At the same time, the device loads the image data using the illustration's URL and displays it on the screen.

[0976] Output: Pronunciation of English words and illustrations.

[0977] Step 4:

[0978] Selecting an Interactive Mode

[0979] Input: User selection of an interactive mode.

[0980] How it works: The user selects an interaction mode in the device's UI, which triggers a request to start an interaction.

[0981] Output: Interactive mode selections are saved to the terminal and the program continues to the next step.

[0982] Step 5:

[0983] Interaction-starting request

[0984] Input: Selection information for interactive mode.

[0985] Operation: The terminal sends a request to start a conversation to the server using the HTTP protocol.

[0986] The server receives the request and generates an initial question, which can be retrieved from a database or generated based on logic.

[0987] Output: The initial question generated by the server is sent to the terminal.

[0988] Step 6:

[0989] Question pronunciation and display

[0990] Input: The question sent by the server.

[0991] How it works: The device passes the received question to a speech synthesis engine (e.g., Google Text-to-Speech API) and pronounces it aloud.

[0992] At the same time, the device displays the question on the screen.

[0993] Output: Questions are spoken and displayed on the screen.

[0994] Step 7:

[0995] Accepting user answers

[0996] Input: The user's answer to the question (e.g., "Blue").

[0997] Action: The user enters an answer into an input field on the terminal.

[0998] The terminal receives this response and sends it to the server.

[0999] Output: The user's answer sent to the server.

[1000] Step 8:

[1001] Generate the next conversation

[1002] Input: User answers and server interaction logic.

[1003] How it works: The server analyzes the user's answers and generates the next question or comment based on them.

[1004] The server sends the generated questions and comments to the terminal as an HTTP response.

[1005] Output: New questions and comments are sent to the device.

[1006] Step 9:

[1007] Emotion recognition

[1008] Input: User's facial expressions and voice.

[1009] Operation: The device uses the built-in camera and microphone to capture the user's facial expressions and voice, and inputs them into the emotion engine.

[1010] The emotion engine analyzes the user's emotional state and generates emotion data.

[1011] Output: Emotion data obtained from the emotion engine.

[1012] Step 10:

[1013] Sending and processing emotional data

[1014] Input: Emotion data.

[1015] Operation: The device sends emotion data to the server as an HTTP request.

[1016] The server analyzes the received emotional data, understands the user's emotional state, and adjusts the learning content and dialogue accordingly.

[1017] Output: Learning and dialogue content adjusted based on user sentiment.

[1018] Step 11:

[1019] Adjusting learning content

[1020] Input: Coordination information and emotion data.

[1021] Operation: The server adjusts the content of the next dialogue and the difficulty of the learning task based on the user's emotional state, and sends new learning content and dialogue content based on this to the terminal.

[1022] The terminal adjusts the screen display and audio tone based on the received adjustment content.

[1023] Output: The adjusted learning and interaction content is presented to the user.

[1024] (Application example 2)

[1025] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1026] Traditional learning systems only provide static learning content for children to learn English naturally while playing. As a result, it is difficult to maintain learners' interest and it is not possible to provide an optimal learning experience that suits each individual's emotional state and learning progress. There is also a lack of ways to provide an interactive learning experience in physical stores.

[1027] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1028] In this invention, the server includes a means for accepting alphabet input, a means for acquiring English words based on the input alphabet, a means for audibly outputting the acquired English words, a means for displaying illustrations corresponding to the English words, a means for reading QR codes, a means for acquiring alphabets by reading the QR codes, a means for recognizing a user's emotions, and a means for dynamically adjusting learning content based on the recognized emotions. This enables real-time analysis of the user's (child's) emotional state and reactions, providing an optimal learning experience. Furthermore, an interactive learning experience is also provided in a physical store, creating an environment where children can enjoy learning English while playing.

[1029] The "means for accepting alphabetic input" is an interface that allows the user to input any alphabetic character.

[1030] The "means for acquiring English words based on the input alphabet" is a system having a function for searching and acquiring English words based on the input alphabet.

[1031] The "means for outputting the acquired English words by voice" refers to a device or program that pronounces the acquired English words using voice synthesis technology.

[1032] The "means for displaying an illustration corresponding to the English word" is a system that displays an image related to the English word on a display device such as a monitor.

[1033] A "means for reading a QR code" is a device or program that uses a device such as a camera to scan a QR code and obtain its contents.

[1034] The "means for obtaining the alphabet by reading the QR code" is a system that has the function of analyzing the data obtained from the QR code and obtaining the corresponding alphabet.

[1035] "Means for recognizing user emotions" refers to technology that uses sensors such as cameras and microphones to analyze the user's facial expressions and tone of voice to determine their emotional state.

[1036] "Means for dynamically adjusting learning content based on recognized emotions" refers to a system that changes the difficulty level of learning content and question format in real time based on recognized emotional data.

[1037] An "interactive question generator" is a program that has the ability to automatically generate new questions based on user input and context.

[1038] The "means for accepting answers from the user" is an interface for receiving and saving answers entered by the user.

[1039] The "means for generating the next dialogue based on the answer" is a technique for analyzing the user's answer and generating the next question or comment in response to it.

[1040] System configuration

[1041] This invention is a learning system that allows users (children) to learn English naturally while playing. In particular, it has the function of dynamically adjusting the learning content according to the user's emotions using emotion recognition technology. This system is composed of a series of hardware and software.

[1042] Hardware and software used

[1043] The system uses the following major hardware and software:

[1044] Terminal: A device operated by the user, such as a smartphone or tablet, that scans QR codes, outputs audio, and displays illustrations.

[1045] Server: Connected to the central database, it retrieves English words and illustrations, generates dialogue content, and processes emotion recognition data.

[1046] Camera and microphone: The sensors required for user emotion recognition.

[1047] Software: Emotion recognition models using libraries such as OpenCV and Keras, a speech synthesis engine, and an API for processing HTTP requests.

[1048] Program processing

[1049] Alphabet input and English word acquisition

[1050] 1. Alphabet input: The user inputs any alphabet through the input interface of the terminal. This is the input method.

[1051] 2. Obtaining English words: The input alphabet is sent to the server via an HTTP request. The server then obtains the corresponding English word and illustration URL from the database.

[1052] 3. Voice output and illustration display: The acquired English words are pronounced using a speech synthesis engine, and illustrations are displayed on the device screen.

[1053] Scanning QR codes

[1054] 1. QR Code Reading: Using the device's camera, scan the QR code placed in the store. The data obtained from the QR code is analyzed to obtain the corresponding alphabet. The data is used as a means of obtaining the alphabet.

[1055] 2. Display of English words and illustrations: English words and illustrations are displayed in alphabetical order based on the scanned QR code, and the words are simultaneously read out loud.

[1056] Emotion recognition and dynamic adjustment of learning content

[1057] 1. Emotion recognition: The camera and microphone are used to analyze the user's facial expressions and voice in real time, and the emotion is determined using an emotion recognition model (Keras model). This is an emotion recognition method.

[1058] 2. Adjustment of learning content: The recognized emotion data is sent to the server, which then dynamically adjusts the learning content and the difficulty of the dialogue based on the data. This is a means of dynamically adjusting the learning content.

[1059] Interactive question generation

[1060] 1. Question generation: The server generates interactive questions based on English words, presents the questions to the user, and accepts answers from the user.

[1061] 2. Next dialogue generation: After receiving the user's answer, the server uses a generative AI model to generate the next question and dynamically adjust the content of the dialogue based on the user's emotions.

[1062] Specific examples

[1063] For example, when a child enters the letter "A" or scans a QR code, the device displays the English word "Apple" and an illustration of an apple, and pronounces "Apple." Next, if the user's emotion is recognized as "joy," the server provides dialogue content and learning tasks tailored to the emotion. For example, the next question generated might be, "Blue is a nice color! Do you like blue flowers or blue cars?"

[1064] Prompt Sentence Examples

[1065] "Generate the following English learning content for children: The user's answer is 'Blue'. What should the next question be?"

[1066] This allows the learning system to combine user emotions with dynamic emotion recognition to provide an optimal environment for learning English while having fun.

[1067] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1068] Step 1:

[1069] The terminal accepts alphabetic input from the user. The input alphabetic data is temporarily stored in the terminal. The input data is used for subsequent processing.

[1070] Step 2:

[1071] The device sends an HTTP POST request to the server based on the input alphabet. The request includes the input alphabet. The server receives the request and retrieves the URL of the English word and corresponding illustration stored in a database. This is done through a database query. The server sends the retrieved English word and illustration URL to the device as an HTTP response.

[1072] Step 3:

[1073] The device receives the response from the server, passes the retrieved English words to the speech synthesis engine to pronounce them, and displays illustrations on the screen using the illustration URL. Specifically, the speech synthesis engine converts the input English words from text to speech data. Meanwhile, the illustration display retrieves an image from the URL and renders it on the screen.

[1074] Step 4:

[1075] A user holds a device and uses the camera to read a QR code placed in a physical store. The device analyzes the data obtained from the QR code and extracts the corresponding alphabet. This alphabet is used to obtain English words and illustrations according to the procedure described above.

[1076] Step 5:

[1077] The device uses a built-in camera and microphone to collect the user's facial expressions and voice, and recognizes emotions in real time. Emotion recognition is performed using a machine learning model with the Keras library. The collected data is preprocessed and input into the model to output emotional data.

[1078] Step 6:

[1079] The device sends the recognized emotion data to the server, which analyzes the emotion data and dynamically adjusts the learning content and dialogue questions to best suit the user. Specifically, it adjusts the difficulty level of the learning content and selects dialogue content templates.

[1080] Step 7:

[1081] The server generates the next dialogue-style question based on the English words. The next question is generated using a generative AI model based on the user's response. The question content and expression are adjusted based on emotional data. The generated question is sent to the device as an HTTP response, and the device pronounces it aloud and displays it on the screen.

[1082] Step 8:

[1083] The user enters answers to questions posed by the server. The device accepts the answers and sends them to the server as an HTTP POST request. The server analyzes the user's answers and generates the next dialogue based on them. This process is repeated, and learning progresses.

[1084] Examples:

[1085] For example, when a child enters the letter "A" or scans a QR code, the device displays the English word "Apple" and an illustration of an apple, and pronounces "Apple." Next, if the user's emotion is recognized as "joy," the server provides dialogue content and learning tasks tailored to the emotion. For example, the next question generated might be, "Blue is a nice color! Do you like blue flowers or blue cars?"

[1086] Example prompt sentence:

[1087] "Generate the following English learning content for children: The user's answer is 'Blue'. What should the next question be?"

[1088] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1089] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1090] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1091] [Third embodiment]

[1092] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1093] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1094] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1095] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1096] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1097] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1098] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1099] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1100] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1101] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1102] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1103] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1104] This invention relates to a learning system that allows children to learn English naturally while playing. This system accepts alphabet input, acquires English words based on the input, outputs them aloud, and displays corresponding illustrations. It can also generate simple dialogues and generate the next dialogue based on the user's answers.

[1105] System configuration

[1106] This system consists of a terminal operated by the user (child), a server that manages data on English words and illustrations, and various program components for outputting voice and displaying illustrations.

[1107] Program processing

[1108] 1. User input of alphabet

[1109] User: Enters alphabets into the input interface of the terminal. For example, the user enters "A."

[1110] 2. Sending the alphabet and acquiring English words

[1111] Terminal: The input alphabet is acquired and sent to the server. The server then acquires the corresponding English word (e.g., "Apple") from the database based on the received alphabet.

[1112] Server: Returns the acquired English word and the URL of the illustration corresponding to that word to the device.

[1113] 3. English word pronunciation and illustrations

[1114] Terminal: Receives English words sent from the server, pronounces them using a speech synthesis engine, and displays the corresponding illustrations on the screen using the corresponding illustration URL.

[1115] Specific examples

[1116] For example, when a user types "A" into a device, the device sends this to the server. The server retrieves the English word "Apple" and an illustration URL from the database and sends them back to the device. The device pronounces "Apple" using a speech synthesis engine and displays the illustration on the screen.

[1117] Conversational conversation generation

[1118] 1. Selecting the interaction mode

[1119] User: Select the interaction mode.

[1120] 2. Request to start a conversation

[1121] Terminal: Sends a request to start a conversation to the server.

[1122] Server: Generates the initial question and sends it to the device.

[1123] 3. Pronunciation and display of questions

[1124] Device: The received question is pronounced by a speech synthesis engine and displayed on the screen. For example, the question "What's your favorite color?" is displayed and pronounced aloud.

[1125] 4. Accepting User Answers

[1126] User: Enters an answer to the question. For example, enters "Blue."

[1127] Terminal: Send this response to the server.

[1128] 5. Generating the next dialogue

[1129] Server: Generates the next question or comment based on the user's answer and sends it to the device.

[1130] Specific examples

[1131] For example, if a user selects "Interactive mode," the device requests the server to ask the question "What's your favorite color?" If the user answers "Blue," the server generates the next question, "Blue is a nice color! Do you like blue flowers or blue cars?" and sends it back to the device. The device then pronounces this aloud and displays it on the screen.

[1132] Database Update

[1133] Users input new English words, illustrations, or vocabulary information that is easily confused into their device and send it to the server. The server analyzes the received information and updates the database. This allows the latest vocabulary and illustrations to be added to the database, making it possible to continuously update the learning content.

[1134] Overall flow

[1135] In this way, users can learn English naturally while playing, through a series of steps from inputting the alphabet to displaying the pronunciation and illustrations of English words, generating conversations in an interactive format, and updating the database. It also provides an environment where parents and children can learn together while having fun.

[1136] The processing flow will be explained below.

[1137] Processing alphabet input and displaying pronunciation and illustrations

[1138] Step 1:

[1139] User: Enter an alphabet (e.g., "A") into the terminal's input interface.

[1140] Step 2:

[1141] Terminal: Takes the input alphabet and sends an HTTP POST request to the server. The request contains the input alphabet.

[1142] Step 3:

[1143] Server: Analyzes the received alphabet and queries the database to obtain the corresponding English word (e.g., "Apple") and the URL of the corresponding illustration.

[1144] example:

[1145] sql

[1146] SELECT word, image_url FROM words WHERE alphabet = 'A';

[1147] Step 4:

[1148] Server: Sends an HTTP response to the terminal containing the URL of the acquired English word and illustration.

[1149] Step 5:

[1150] Terminal: The English word returned from the server (e.g., "Apple") is passed to the speech synthesis engine and pronounced. In addition, an illustration is displayed on the screen using the illustration URL.

[1151] Handling simple conversation generation in a conversational format

[1152] Step 1:

[1153] User: Selects the interaction mode in the device UI.

[1154] Step 2:

[1155] Terminal: Sends a request to start interactive mode to the server via HTTP protocol.

[1156] Step 3:

[1157] Server: Generates a simple question (e.g., "What's your favorite color?") to begin the conversation and sends it to the terminal.

[1158] Step 4:

[1159] Terminal: The question received from the server is passed to the speech synthesis engine, which pronounces it and displays the question on the screen.

[1160] Step 5:

[1161] User: Enters the answer to the question (e.g., "Blue") into the device's input field.

[1162] Step 6:

[1163] Terminal: Gets the user's answer and sends it to the server via an HTTP POST request.

[1164] Step 7:

[1165] Server: Based on the user's answers, it generates the next question or comment (e.g., "Blue is a nice color! Do you like blue flowers or blue cars?") and sends it to the device.

[1166] Step 8:

[1167] Terminal: New questions and comments received from the server are passed to the speech synthesis engine, spoken, and the new questions and comments are displayed on the screen.

[1168] Database update process

[1169] Step 1:

[1170] User: Enter information about new English words, illustrations, similar words, and commonly confused vocabulary into a dedicated form on the device.

[1171] Step 2:

[1172] Terminal: Obtains the entered information and sends it to the server via an HTTP POST request.

[1173] Step 3:

[1174] Server: Analyzes the received information and adds it as a new entry in the database or updates existing data as needed.

[1175] example:

[1176] sql

[1177] INSERT INTO similar_words (word1, word2) VALUES ('Pear', 'Pair')

[1178] ON DUPLICATE KEY UPDATE word1='Pear', word2='Pair';

[1179]

[1180] Example 1

[1181] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1182] Conventional learning systems make it difficult for children to learn English naturally while playing, and do not provide an interactive learning environment. In addition, updating the latest vocabulary and illustration information is done manually, making efficient learning difficult.

[1183] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1184] In this invention, the server includes means for accepting input of alphabets, means for acquiring vocabulary based on the input alphabets, means for audibly outputting the acquired vocabulary, means for displaying graphics corresponding to the vocabulary, means for acquiring the vocabulary from a database on the Internet, means for generating dialogue-style questions based on the vocabulary, means for presenting the questions to a user and accepting answers from the user, means for generating a next dialogue based on the answers, and means for analyzing the vocabulary and the graphics and updating the database with the latest learning content. This makes it possible to provide an interactive learning environment in which children can efficiently learn English naturally while playing and in which the latest learning content is automatically updated.

[1185] The "means for accepting alphabetic input" is an interface that provides a function for a user to input alphabetic characters using a terminal.

[1186] The "means for acquiring vocabulary" is a function for searching and acquiring corresponding vocabulary from a database based on input alphabets.

[1187] The "means for outputting vocabulary by voice" is a function that uses a voice synthesis engine to present the acquired vocabulary to the user in voice format.

[1188] "Means for displaying graphics" is a function that displays images or illustrations corresponding to acquired vocabulary on the screen.

[1189] "Means of obtaining from databases on the Internet" refers to the function of obtaining necessary vocabulary information and graphics from remote databases via a network.

[1190] The "means for generating interactive questions" is a function for generating questions to promote interactive dialogue with the user based on the acquired vocabulary.

[1191] The "means for presenting a question to a user and receiving an answer from the user" is a function for presenting a generated question to a user by voice or text and receiving an answer from the user.

[1192] The "means for generating the next dialogue" is a function that generates the next question or comment based on the user's answer and continues the interactive dialogue.

[1193] "Means for updating the latest learning content to the database" is a function that continuously updates learning content based on user input and new information and reflects it in the database.

[1194] The present invention is a learning system that allows children to learn English naturally while playing. This system is composed of a terminal operated by the user (child), a server that manages data on English words and graphics, and various program components for outputting audio and displaying images. Specific embodiments of the present invention are described below.

[1195] Hardware and software used

[1196] Terminal: A device such as a tablet, smartphone, or computer that a user interacts with through an interface.

[1197] Server: A remote computer system containing a database that manages vocabulary and graphic data and generates dialogue.

[1198] Speech synthesis engine: Software for converting text to speech, such as Google Text-to-Speech (TTS) or Amazon Polly.

[1199] Program processing

[1200] 1. User input of alphabet

[1201] A user inputs an alphabet into the input interface of a terminal. For example, the user inputs "A."

[1202] 2. Sending the Alphabet

[1203] The device captures the input alphabet and sends it to the server as an HTTP request.

[1204] 3. Acquire English words and illustrations

[1205] The server analyzes the received alphabet and retrieves the corresponding English word (e.g., "Apple") and the URL of its illustration from the database. The retrieved information is then sent back to the device.

[1206] 4. Speech synthesis of English words and illustration display

[1207] The device receives the English word and illustration URL returned from the server, pronounces the English word using a speech synthesis engine (e.g., Google Text-to-Speech), and displays the corresponding graphic on the screen.

[1208] 5. Conversational conversation generation

[1209] The user selects a conversation mode, and the device sends a request to start a conversation to the server. The server generates an initial question (e.g., "What's your favorite color?") and sends it to the device. The device pronounces the received question using a speech synthesis engine and displays it on the screen. When the user inputs an answer to the question (e.g., "Blue"), the device sends it to the server, and the server generates the next question or comment (e.g., "Blue is a nice color! Do you like blue flowers or blue cars?") and sends it back to the device.

[1210] 6. Updating the database

[1211] The user inputs new English words, illustrations, or confusing vocabulary information into the device and sends it to the server. The server analyzes the received information and updates the database, adding the latest vocabulary and illustration data to the database.

[1212] Specific examples

[1213] For example, when a user types "A" into a device, the device sends this to the server. The server retrieves the English word "Apple" and its illustration URL from the database and sends it back to the device. The device pronounces "Apple" using a speech synthesis engine and displays the illustration on the screen.

[1214] Also, consider the case where the user selects "Interactive mode" and the terminal requests the question "What's your favorite color?" from the server. If the user answers "Blue," the server generates the next question, "Blue is a nice color! Do you like blue flowers or blue cars?" and sends it back to the terminal. The terminal then pronounces this aloud and displays it on the screen.

[1215] Prompt Sentence Examples

[1216] The following prompts can be input to the generative AI model:

[1217] "A child uses the device to enter the letter 'A'. The server retrieves the English word 'Apple' and the URL of its illustration from the database and sends it back to the device. The device pronounces 'Apple' using its speech synthesis engine and displays the illustration on the screen. If interactive mode is selected, the device pronounces the question generated by the server using its speech synthesis engine and generates the next question based on the user's answer."

[1218] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1219] Step 1:

[1220] User input of alphabetic characters

[1221] A user inputs an alphabet using the input interface (keyboard or touch screen) of a terminal. For example, a user inputs the alphabet "A." The input is displayed as a character in a text box on the terminal. This input becomes the starting point for subsequent data processing.

[1222] Step 2:

[1223] Sending the alphabet

[1224] The terminal captures the alphabet "A" entered by the user and generates an HTTP request to send it to the server. The input data (alphabet "A") is sent to the server. The HTTP request also includes metadata such as the client ID and request timestamp.

[1225] Input: The alphabet "A" entered by the user

[1226] Output: HTTP requests containing the letter "A"

[1227] What it does: The terminal program captures the user's input and constructs a request to send this data to the server.

[1228] Step 3:

[1229] Acquire English words and illustrations

[1230] The server extracts the letter "A" from the received HTTP request, generates and executes a database query, retrieves the corresponding English word (e.g., "Apple") and the URL of its illustration from the database, and returns this to the terminal as an HTTP response. The English word and illustration URL are output as the results of the database query.

[1231] Input: HTTP request containing the letter "A"

[1232] Output: HTTP response containing the English word "Apple" and an illustration URL

[1233] Specific operation: The server queries the database to retrieve the English word "Apple" and its related illustration URLs.

[1234] Step 4:

[1235] Speech synthesis of English words and illustration display

[1236] The device receives the HTTP response returned from the server and generates the English word "Apple" aloud using a text-to-speech API or speech synthesis engine (for example, Google Text-to-Speech). It also obtains image data from the obtained illustration URL and displays it on the screen. The speech synthesis generates the audio data, and the image data is rendered on the screen.

[1237] Input: HTTP response containing the English word "Apple" and an illustration URL

[1238] Output: Voice output of the English word "Apple" and display of an illustration on the screen

[1239] Specific operation: The device synthesizes "Apple" through the voice speaker and displays an illustration on the screen.

[1240] Step 5:

[1241] Selecting an Interactive Mode

[1242] The user selects an interaction mode on the terminal interface. The selection triggers the system to transition to the interactive question generation module. The selection information is recorded as internal data of the terminal.

[1243] Input: The user's action to select the interaction mode

[1244] Output:Internal flag for interactive mode selection

[1245] Specific action: The user taps an interaction mode button in the UI.

[1246] Step 6:

[1247] Conversation Initiating Request

[1248] The terminal requests the server to start the dialogue mode. This request triggers the server to generate the initial question. The terminal sends a dialogue start request to the server, and the initial question is generated based on this request.

[1249] Input: Request based on internal flag for interactive mode selection

[1250] Output: A request to generate the first question

[1251] Specific operation: The terminal sends a request to start a conversation to the server.

[1252] Step 7:

[1253] Question pronunciation and display

[1254] The server receives the dialogue start request and generates the initial question (e.g., "What's your favorite color?"). The generated question is sent to the device as an HTTP response. The device then outputs the received question aloud using a speech synthesis engine and displays it as text on the screen.

[1255] Input: A request to generate the initial question.

[1256] Output: HTTP response containing the initial question, "What's your favorite color?"

[1257] Specific operation: The device synthesizes the question, outputs it, and displays it on the screen.

[1258] Step 8:

[1259] Submitting user answers

[1260] The user uses the device to input an answer to the question, for example, "Blue." The device captures this answer and sends it to the server. The input data is sent to the server and triggers the generation of the next question.

[1261] Input: Answer data "Blue" entered by the user

[1262] Output: HTTP request with response data "Blue"

[1263] Specific operation: The user types the answer on the keyboard, and the device captures it and sends it to the server.

[1264] Step 9:

[1265] Generate the next conversation

[1266] The server receives the user's answer and generates the next question or comment, for example, "Blue is a nice color! Do you like blue flowers or blue cars?", and sends it to the terminal as an HTTP response.

[1267] Input: HTTP request containing user answer data "Blue"

[1268] Output: HTTP response containing the question "Blue is a nice color! Do you like blue flowers or blue cars?"

[1269] Specific operation: The server analyzes the response data and generates the next dialogue.

[1270] Step 10:

[1271] Database Update

[1272] The user inputs new English words, graphics, and confusing vocabulary information into the terminal and sends it to the server. The server analyzes the received information and updates the database. The updated database is reflected in subsequent data acquisition and question generation.

[1273] Input: New English words, graphics, and vocabulary information

[1274] Output: Updated database

[1275] What happens: The user enters new information, which the server parses to update the database.

[1276] (Application example 1)

[1277] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1278] Previous English learning systems for children were limited to displaying words and illustrations and outputting audio on fixed devices such as PCs and tablets, which meant that learning lacked flexibility and enjoyment. Furthermore, they lacked the functionality to deepen learning through interactive question and answer formats, and did not adequately provide an environment for children to learn English naturally. Furthermore, the database was not continually updated to reflect the latest information in real time.

[1279] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1280] In this invention, the server includes means for accepting input of alphabets, means for acquiring English words based on the input alphabets, means for audibly outputting the acquired English words, means for displaying illustrations corresponding to the English words, means for displaying the English words and the illustrations on the smart glasses and outputting them audibly, means for generating dialogue-style questions based on the English words, means for presenting the questions to the user and accepting answers from the user, means for generating a next dialogue based on the answers, and means for displaying the dialogue-style questions and answers on the smart glasses and outputting them audibly, thereby enabling children to learn English more naturally and enjoyably.

[1281] "Alphabet" refers to the 26 letters that make up the basic set of letters in the English language.

[1282] "Means for accepting input" refers to interface devices or software for receiving character information such as alphabets from the user.

[1283] The "means for obtaining English words" is a mechanism for finding and extracting corresponding English words based on the input alphabet.

[1284] "Means for audio output" refers to devices such as a speech synthesis engine or speaker that reproduces the acquired English words as sound.

[1285] "Means for displaying illustrations" refers to mechanisms or software for displaying images corresponding to English words on a screen.

[1286] "Smart glasses" are wearable devices that have a display for displaying information in front of the eyes and have audio output capabilities.

[1287] "Interactive question generation means" refers to algorithms or software that generate appropriate questions to promote English learning through dialogue with the user.

[1288] "Means for accepting responses from users" refers to an interface or software for recognizing and appropriately accepting responses from users to questions.

[1289] The "means for generating the next dialogue" refers to algorithms or software for generating the next appropriate question or comment based on the user's answers.

[1290] An "Internet database" is an information management system that stores data remotely and makes it accessible over a network.

[1291] MODE FOR CARRYING OUT THE INVENTION

[1292] The system of the present invention is a learning system that aims to help children learn English naturally while playing, and is implemented using smart glasses. The system allows users to easily input alphabets, obtain English words based on the input, output them as audio, and display corresponding illustrations. It also provides a deeper learning experience through interactive question and answer sessions.

[1293] System configuration

[1294] Hardware:

[1295] Smart glasses (e.g. Google Glass): Uses a display and built-in speakers.

[1296] Server: Manages English words and illustrations and generates dialogue.

[1297] software:

[1298] Speech synthesis engine (e.g. Google Text-to-Speech API)

[1299] Database (e.g. MySQL)

[1300] API server (e.g. Flask)

[1301] Program processing

[1302] 1. Enter the alphabet:

[1303] The user (child) uses the touchpad of the smart glasses to input alphabets, which are recognized by software in the smart glasses and sent to the server.

[1304] 2. Acquire English words and illustrations:

[1305] The server retrieves the corresponding English word and illustration URL from the database based on the received alphabet, and this information is then sent back to the smart glasses.

[1306] 3. English word pronunciation and illustration display:

[1307] The smart glasses pronounce the received English words using a speech synthesis engine and simultaneously display illustrations on the screen.

[1308] 4. Interactive learning:

[1309] When the user selects the dialogue mode, the smart glasses display an initial question and read it aloud. The user responds by voice or using the touchpad, and the answer is sent to the server. The server generates a new question and sends it back to the smart glasses. By repeating this process, the user can learn English through dialogue.

[1310] Specific examples

[1311] Alphabet input and word generation:

[1312] When a user uses the smart glasses to input the letter "B," the smart glasses will display the word "Banana" on the display and pronounce it as "Banana."

[1313] Interactive Question and Answer:

[1314] The user selects the interactive mode and the smart glasses ask, "What is your favorite fruit?" If the user answers, "Apple," the smart glasses generate the next question, "Do you like red apples or green apples?", which the smart glasses display and pronounce.

[1315] Prompt Sentence Examples

[1316] I entered "B". What English word do you want to see and how do you pronounce it? The next question is "What is your favorite fruit?" If the user answers "Apple", please generate the next question.

[1317] This system allows children to receive real-time responses and feedback through the smart glasses, enabling them to effectively learn English while playing.

[1318] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1319] Step 1:

[1320] A user inputs alphabets using the touchpad of the smart glasses, which are then recognized by software within the smart glasses. The input of this process is the alphabets input by the user, and the output is the recognized alphabet data.

[1321] Step 2:

[1322] The smart glasses send the recognized alphabet data to the server, which receives the data and queries the database to find the English word and its corresponding illustration. The input of this process is the alphabet data sent from the smart glasses, and the output is the URL of the corresponding English word and illustration.

[1323] Step 3:

[1324] The server returns the URL of the English word and illustration retrieved from the database to the smart glasses. The smart glasses receive it, pronounce the English word using a speech synthesis engine, and display the illustration on the display. The input of this process is the URL of the English word and illustration returned from the server, and the output is the pronunciation of the English word and the display of the illustration by the smart glasses.

[1325] Step 4:

[1326] When the user selects an interaction mode, the smart glasses send a request to start the interaction to the server. The server generates an initial question and sends it to the smart glasses. The input of this process is the request to select the interaction mode, and the output is the generated question.

[1327] Step 5:

[1328] The smart glasses will then use a speech synthesis engine to pronounce the received question and display it on the display. The user can then respond to the question by voice or using the touchpad. The input to this process is the question sent from the server, and the output is the question displayed on the smart glasses' display and the voice output.

[1329] Step 6:

[1330] The user's answer is recognized by the smart glasses and sent to the server. The server generates a next question based on the received answer and sends it back to the smart glasses. The input of this process is the user's answer and the output is the generated next question.

[1331] Step 7:

[1332] The smart glasses receive the next question sent from the server, pronounce it again using the speech synthesis engine, and display it on the display. This process continues until the user finishes the dialogue, and all dialogue is recorded and the database is updated as necessary. The input of this process is the next question, and the output is the voice output and display based on that question.

[1333] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1334] This invention relates to a learning system that allows children to learn English naturally while playing. In particular, this system has the ability to recognize the user's emotions and adjust the learning content accordingly. In addition to inputting the alphabet, acquiring and pronouncing English words, displaying illustrations, and generating interactive questions, this system can also recognize the user's emotions to provide an optimal learning experience.

[1335] System configuration

[1336] This system consists of a terminal operated by the user (child), a server that manages data on English words and illustrations, various program components for outputting voice and displaying illustrations, and an emotion engine that recognizes the user's emotions.

[1337] Program processing

[1338] Processing alphabet input and displaying pronunciation and illustrations

[1339] 1. User input of alphabet

[1340] User: Enter an alphabet (e.g., "A") into the terminal's input interface.

[1341] 2. Sending the alphabet and acquiring English words

[1342] Terminal: Takes the input alphabet and sends an HTTP POST request to the server. The request contains the input alphabet.

[1343] Server: Analyzes the received alphabet and queries the database to obtain the corresponding English word (e.g., "Apple") and the URL of the corresponding illustration.

[1344] example:

[1345] sql

[1346] SELECT word, image_url FROM words WHERE alphabet = 'A';

[1347] 3. English word pronunciation and illustrations

[1348] Server: Sends an HTTP response to the terminal containing the URL of the acquired English word and illustration.

[1349] Terminal: The English word returned from the server (e.g., "Apple") is passed to the speech synthesis engine and pronounced. In addition, an illustration is displayed on the screen using the illustration URL.

[1350] Conversational conversation generation

[1351] 1. Selecting the interaction mode

[1352] User: Selects the interaction mode in the device UI.

[1353] 2. Request to start a conversation

[1354] Terminal: Sends a request to start a conversation to the server using the HTTP protocol.

[1355] Server: Generates the initial question and sends it to the device.

[1356] 3. Pronunciation and display of questions

[1357] Terminal: The question received from the server is passed to the speech synthesis engine, which pronounces it, and the question is displayed on the screen.

[1358] 4. Accepting User Answers

[1359] User: Enters the answer to the question into the input field on the device.

[1360] Terminal: Send this response to the server.

[1361] 5. Generating the next dialogue

[1362] Server: Generates the next question or comment based on the user's answer and sends it to the device.

[1363] Terminal: New questions and comments received from the server are passed to the speech synthesis engine, spoken, and the new questions and comments are displayed on the screen.

[1364] Use of emotion engine

[1365] 1. Emotional awareness

[1366] Device: The built-in camera and microphone are used to input the user's voice and facial expressions into the emotion engine.

[1367] Emotion engine: Analyzes the user's emotional state (e.g., happiness, sadness, excitement) and outputs this as data.

[1368] 2. Transmission and processing of emotional data

[1369] Terminal: Sends emotion data obtained from the emotion engine to the server.

[1370] Server: Analyzes emotional data and adjusts learning content and dialogue.

[1371] 3. Adjusting learning content

[1372] Server: Adjusts the content of the next dialogue and the difficulty of the learning task based on the user's emotions and sends that information to the terminal.

[1373] Terminal: Adjusts the screen display, audio tone, and output content based on information sent from the server.

[1374] Specific examples

[1375] For example, when a user inputs "A," the device sends this to the server, which retrieves the English word "Apple" and an illustration URL from the database and sends them to the device. The device then pronounces "Apple" and displays the illustration.

[1376] In dialogue mode, when the user is asked "What's your favorite color?" and answers "Blue," the server generates the next question: "Blue is a nice color! Do you like blue flowers or blue cars?" Furthermore, the emotion engine recognizes the user's emotions (e.g., excitement) from their facial expressions and voice, and the server adjusts the dialogue content and difficulty accordingly.

[1377] This allows children to learn English naturally while playing, providing an optimal learning experience that responds to their emotions, and creating an environment where parents and children can enjoy learning together.

[1378] The processing flow will be explained below.

[1379] Specific processing steps of a system that combines emotion engines

[1380] Processing alphabet input and displaying pronunciation and illustrations

[1381] Step 1:

[1382] User: Enter an alphabet (e.g., "A") into the terminal's input interface.

[1383] Step 2:

[1384] Terminal: Takes the input alphabet and sends an HTTP POST request to the server. The request contains the input alphabet.

[1385] Step 3:

[1386] Server: Analyzes the received alphabet and queries the database to obtain the corresponding English word (e.g., "Apple") and the URL of the corresponding illustration.

[1387] example:

[1388] sql

[1389] SELECT word, image_url FROM words WHERE alphabet = 'A';

[1390] Step 4:

[1391] Server: Sends an HTTP response to the terminal containing the URL of the acquired English word and illustration.

[1392] Step 5:

[1393] Terminal: The English word returned from the server (e.g., "Apple") is passed to the speech synthesis engine and pronounced. In addition, an illustration is displayed on the screen using the illustration URL.

[1394] Handling conversational dialogue generation

[1395] Step 1:

[1396] User: Selects the interaction mode in the device UI.

[1397] Step 2:

[1398] Terminal: Sends a request to start interactive mode to the server via HTTP protocol.

[1399] Step 3:

[1400] Server: Generates a simple question (e.g., "What's your favorite color?") to begin the conversation and sends it to the terminal.

[1401] Step 4:

[1402] Terminal: The question received from the server is passed to the speech synthesis engine, which pronounces it and displays the question on the screen.

[1403] Step 5:

[1404] User: Enters the answer to the question (e.g., "Blue") into the device's input field.

[1405] Step 6:

[1406] Terminal: Gets the user's answer and sends it to the server via an HTTP POST request.

[1407] Step 7:

[1408] Server: Based on the user's answers, it generates the next question or comment (e.g., "Blue is a nice color! Do you like blue flowers or blue cars?") and sends it to the device.

[1409] Step 8:

[1410] Terminal: New questions and comments received from the server are passed to the speech synthesis engine, spoken, and the new questions and comments are displayed on the screen.

[1411] Use of emotion engine

[1412] Step 1:

[1413] Device: The built-in camera and microphone are used to input the user's voice and facial expressions into the emotion engine.

[1414] Step 2:

[1415] Emotion engine: Analyzes the user's emotional state (e.g., happiness, sadness, excitement) and outputs this as data.

[1416] Step 3:

[1417] Terminal: Sends emotion data obtained from the emotion engine to the server.

[1418] Step 4:

[1419] Server: Analyzes emotional data and adjusts learning content and dialogue.

[1420] Step 5:

[1421] Server: Adjusts the content of the next dialogue and the difficulty of the learning task based on the user's emotions and sends that information to the terminal.

[1422] Step 6:

[1423] Terminal: Adjusts the screen display, audio tone, and output content based on information sent from the server.

[1424] Specific examples

[1425] Alphabetical input example:

[1426] 1. When the user types "A" into the terminal,

[1427] 2. The device sends this to the server,

[1428] 3. The server retrieves the English word "Apple" and the illustration URL from the database and returns them to the device.

[1429] 4. The device will pronounce "Apple" and display an illustration.

[1430] Interactive example:

[1431] 1. The user selects an interaction mode,

[1432] 2. The device requests the question "What's your favorite color?" from the server.

[1433] 3. If the user answers "Blue,"

[1434] 4. The server generates the next question, "Blue is a nice color! Do you like blue flowers or blue cars?" and sends it back to the terminal.

[1435] 5. The device will pronounce this aloud and display it on the screen.

[1436] Emotion engine example:

[1437] 1. The user's facial expressions and voice are input into the emotion engine via the device's camera and microphone.

[1438] 2. The emotion engine recognizes the user's emotion (e.g., excitement).

[1439] 3. Emotional data is sent to the server, which then adjusts the learning accordingly.

[1440] 4. The device displays tailored learning content to provide an optimal learning experience.

[1441] This will create a learning system that takes the user's emotions into consideration, allowing children to learn English naturally while having fun. It will also create an environment where parents can feel at ease and support their children's learning.

[1442] Example 2

[1443] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1444] Conventional English learning systems lacked the ability to adjust learning content based on the user's emotions, making it difficult to provide an effective learning experience. Furthermore, even when users engaged in interactive learning, the system often consisted of one-way question-and-answer exchanges, making it difficult for users to maintain their interest and attention in the progress of their learning. Furthermore, there were few integrated learning systems that included voice synthesis and illustration display.

[1445] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1446] In this invention, the server includes means for accepting alphabetic input, means for acquiring English word and illustration data based on the input alphabet, means for audibly outputting the acquired English words, means for displaying illustrations corresponding to the English words, means for recognizing the user's emotions, and means for adjusting the learning content based on the recognized emotions. This allows for an optimal learning experience that is tailored to the user's emotions, thereby maintaining the user's interest and attention. Furthermore, even in interactive learning, the content of the next dialogue and the difficulty of the learning tasks can be adjusted based on the user's emotions and responses, thereby achieving effective learning.

[1447] The "means for accepting alphabetic input" is an input interface that allows the terminal to recognize alphabetic input by the user and pass it on to subsequent processing.

[1448] "Means for obtaining data on English words and illustrations" refers to the process by which the server queries the database based on the alphabet sent from the terminal and obtains the URL of the corresponding English word and its illustration.

[1449] The "means for outputting English words by voice" is a function for converting the acquired English words into voice using a voice synthesis engine and allowing the user to hear it.

[1450] "Means for displaying illustrations" is a function for reading image data from the URL of the acquired illustration and displaying it on the terminal screen.

[1451] "Means for recognizing user emotions" refers to the process of using a built-in camera and microphone to analyze the user's facial expressions and voice to determine the user's emotional state (e.g., joy, sadness, excitement).

[1452] The "means for adjusting learning content based on recognized emotions" is a function that dynamically changes the learning content, difficulty level of interactive questions, tone, display content, etc. according to the user's emotional state.

[1453] The "means for generating interactive questions" is a process for creating questions to proceed in an interactive format with the user based on the acquired English vocabulary and learning content.

[1454] The "means for accepting an answer from a user" is an input interface for receiving an answer entered by a user in an interactive format and passing it on to subsequent processing.

[1455] "Means for generating the next dialogue" refers to the process of creating new questions or comments based on the user's answers, etc., and continuing the dialogue.

[1456] "Means of obtaining data from a database on the Internet" refers to a function that obtains the necessary English word and illustration data in real time from a remote database via a network.

[1457] The present invention is a learning system that can recognize a user's emotions and adjust the English learning content accordingly. This system is configured using the following hardware and software.

[1458] System configuration

[1459] Hardware

[1460] 1. Device: A device used by a user, such as a tablet or smartphone, that has a touchscreen, camera, and microphone.

[1461] 2. Server: A cloud computing platform, such as Amazon Web Services (AWS) or Microsoft Azure, that can handle database operations and API requests.

[1462] software

[1463] 1. Database: A database management system such as MySQL or PostgreSQL is used to store the English word and illustration data.

[1464] 2. Speech synthesis engine: Uses Google Text-to-Speech API or Amazon Polly to output English words aloud on the device.

[1465] 3. Emotion recognition engine: Recognizes user emotions using Microsoft Azure Cognitive Services and IBM Watson.

[1466] 4. HTTP protocol: Used for data communication between the terminal and the server.

[1467] Operation explanation

[1468] 1. Alphabetical input

[1469] The user uses the device's input interface to input alphabetic characters, which can come from a keyboard or touchscreen and are then captured by a program within the device.

[1470] 2. Data Acquisition and Display

[1471] The device receives the input alphabet and sends it as an HTTP POST request to the server. The server queries the database based on the received alphabet and obtains the corresponding English word (e.g., "Apple") and the URL of the illustration.

[1472] The acquired English words are output as voice using a speech synthesis engine, and illustrations are displayed on the device screen.

[1473] 3. Conversational conversation generation

[1474] When the user selects the interactive mode, the terminal sends a request to the server to start the interaction, and the server generates an initial question and sends it to the terminal.

[1475] When the user enters an answer to a question into the terminal, the terminal sends the answer to the server, which then generates the next question or comment and sends it back to the terminal, thus continuing the dialogue.

[1476] 4. Emotion recognition and learning adjustment

[1477] The device uses a camera and microphone to capture the user's facial expressions and voice, and inputs them into an emotion recognition engine.

[1478] The emotion recognition engine analyzes the user's emotional state and returns data based on this to the device, which then transmits this data to the server, which then adjusts the learning content and difficulty of the questions based on the user's emotions.

[1479] The adjusted learning content is sent to the terminal and presented to the user in an appropriate format.

[1480] Specific examples

[1481] For example, when a user inputs "A," the device sends the input to the server, which retrieves the English word "Apple" and its illustration from the database and sends it to the device. The device then outputs "Apple" aloud and displays the illustration on the screen.

[1482] In a conversational example, if a user is asked "What's your favorite color?" and answers "Blue," the server generates the next question: "Blue is a nice color! Do you like blue flowers or blue cars?" Furthermore, an emotion recognition engine detects excitement from the user's facial expressions and voice, and the server adjusts the tone and content of the questions accordingly.

[1483] Prompt Sentence Examples

[1484] Examples of prompts to input to a generative AI model include:

[1485] "When the user types the letter 'A', retrieve and display words and illustrations related to that letter."

[1486] "Conduct a dialogue between the user and the system, asking and answering questions in English."

[1487] These can be used to provide appropriate instructions to the generative AI model.

[1488] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1489] Step 1:

[1490] User input of alphabetic characters

[1491] Input: A user types an alphabet (e.g., "A") into the terminal's input interface.

[1492] How it works: The user enters alphabetic characters using the device's keyboard or touchscreen.

[1493] Output: The input alphabet is retrieved by the terminal program.

[1494] Step 2:

[1495] Sending the alphabet and getting English words

[1496] Input: Alphabet typed into the terminal.

[1497] Operation: The device includes the obtained alphabet in the payload of an HTTP POST request and sends it to the server.

[1498] The server receives the HTTP request and parses the alphabet.

[1499] The server performs a database query based on the parsed alphabet and retrieves the corresponding English word (e.g., "Apple") and the URL of the illustration.

[1500] Example database query:

[1501] sql

[1502] SELECT word, image_url FROM words WHERE alphabet = 'A';

[1503] Output: The URL of the acquired English word and illustration is sent from the server to the device as an HTTP response.

[1504] Step 3:

[1505] English word pronunciation and illustration display

[1506] Input: English word and illustration URL sent from the server.

[1507] How it works: The device passes the received English word to a speech synthesis engine (e.g., Google Text-to-Speech API) and pronounces it.

[1508] At the same time, the device loads the image data using the illustration's URL and displays it on the screen.

[1509] Output: Pronunciation of English words and illustrations.

[1510] Step 4:

[1511] Selecting an Interactive Mode

[1512] Input: User selection of an interactive mode.

[1513] How it works: The user selects an interaction mode in the device's UI, which triggers a request to start an interaction.

[1514] Output: Interactive mode selections are saved to the terminal and the program continues to the next step.

[1515] Step 5:

[1516] Interaction-starting request

[1517] Input: Selection information for interactive mode.

[1518] Operation: The terminal sends a request to start a conversation to the server using the HTTP protocol.

[1519] The server receives the request and generates an initial question, which can be retrieved from a database or generated based on logic.

[1520] Output: The initial question generated by the server is sent to the terminal.

[1521] Step 6:

[1522] Question pronunciation and display

[1523] Input: The question sent by the server.

[1524] How it works: The device passes the received question to a speech synthesis engine (e.g., Google Text-to-Speech API) and pronounces it aloud.

[1525] At the same time, the device displays the question on the screen.

[1526] Output: Questions are spoken and displayed on the screen.

[1527] Step 7:

[1528] Accepting user answers

[1529] Input: The user's answer to the question (e.g., "Blue").

[1530] Action: The user enters an answer into an input field on the terminal.

[1531] The terminal receives this response and sends it to the server.

[1532] Output: The user's answer sent to the server.

[1533] Step 8:

[1534] Generate the next conversation

[1535] Input: User answers and server interaction logic.

[1536] How it works: The server analyzes the user's answers and generates the next question or comment based on them.

[1537] The server sends the generated questions and comments to the terminal as an HTTP response.

[1538] Output: New questions and comments are sent to the device.

[1539] Step 9:

[1540] Emotion recognition

[1541] Input: User's facial expressions and voice.

[1542] Operation: The device uses the built-in camera and microphone to capture the user's facial expressions and voice, and inputs them into the emotion engine.

[1543] The emotion engine analyzes the user's emotional state and generates emotion data.

[1544] Output: Emotion data obtained from the emotion engine.

[1545] Step 10:

[1546] Sending and processing emotional data

[1547] Input: Emotion data.

[1548] Operation: The device sends emotion data to the server as an HTTP request.

[1549] The server analyzes the received emotional data, understands the user's emotional state, and adjusts the learning content and dialogue accordingly.

[1550] Output: Learning and dialogue content adjusted based on user sentiment.

[1551] Step 11:

[1552] Adjusting learning content

[1553] Input: Coordination information and emotion data.

[1554] Operation: The server adjusts the content of the next dialogue and the difficulty of the learning task based on the user's emotional state, and sends new learning content and dialogue content based on this to the terminal.

[1555] The terminal adjusts the screen display and audio tone based on the received adjustment content.

[1556] Output: The adjusted learning and interaction content is presented to the user.

[1557] (Application example 2)

[1558] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1559] Traditional learning systems only provide static learning content for children to learn English naturally while playing. As a result, it is difficult to maintain learners' interest and it is not possible to provide an optimal learning experience that suits each individual's emotional state and learning progress. There is also a lack of ways to provide an interactive learning experience in physical stores.

[1560] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1561] In this invention, the server includes a means for accepting alphabet input, a means for acquiring English words based on the input alphabet, a means for audibly outputting the acquired English words, a means for displaying illustrations corresponding to the English words, a means for reading QR codes, a means for acquiring alphabets by reading the QR codes, a means for recognizing a user's emotions, and a means for dynamically adjusting learning content based on the recognized emotions. This enables real-time analysis of the user's (child's) emotional state and reactions, providing an optimal learning experience. Furthermore, an interactive learning experience is also provided in a physical store, creating an environment where children can enjoy learning English while playing.

[1562] The "means for accepting alphabetic input" is an interface that allows the user to input any alphabetic character.

[1563] The "means for acquiring English words based on the input alphabet" is a system having a function for searching and acquiring English words based on the input alphabet.

[1564] The "means for outputting the acquired English words by voice" refers to a device or program that pronounces the acquired English words using voice synthesis technology.

[1565] The "means for displaying an illustration corresponding to the English word" is a system that displays an image related to the English word on a display device such as a monitor.

[1566] A "means for reading a QR code" is a device or program that uses a device such as a camera to scan a QR code and obtain its contents.

[1567] The "means for obtaining the alphabet by reading the QR code" is a system that has the function of analyzing the data obtained from the QR code and obtaining the corresponding alphabet.

[1568] "Means for recognizing user emotions" refers to technology that uses sensors such as cameras and microphones to analyze the user's facial expressions and tone of voice to determine their emotional state.

[1569] "Means for dynamically adjusting learning content based on recognized emotions" refers to a system that changes the difficulty level of learning content and question format in real time based on recognized emotional data.

[1570] An "interactive question generator" is a program that has the ability to automatically generate new questions based on user input and context.

[1571] The "means for accepting answers from the user" is an interface for receiving and saving answers entered by the user.

[1572] The "means for generating the next dialogue based on the answer" is a technique for analyzing the user's answer and generating the next question or comment in response to it.

[1573] System configuration

[1574] This invention is a learning system that allows users (children) to learn English naturally while playing. In particular, it has the function of dynamically adjusting the learning content according to the user's emotions using emotion recognition technology. This system is composed of a series of hardware and software.

[1575] Hardware and software used

[1576] The system uses the following major hardware and software:

[1577] Terminal: A device operated by the user, such as a smartphone or tablet, that scans QR codes, outputs audio, and displays illustrations.

[1578] Server: Connected to the central database, it retrieves English words and illustrations, generates dialogue content, and processes emotion recognition data.

[1579] Camera and microphone: The sensors required for user emotion recognition.

[1580] Software: Emotion recognition models using libraries such as OpenCV and Keras, a speech synthesis engine, and an API for processing HTTP requests.

[1581] Program processing

[1582] Alphabet input and English word acquisition

[1583] 1. Alphabet input: The user inputs any alphabet through the input interface of the terminal. This is the input method.

[1584] 2. Obtaining English words: The input alphabet is sent to the server via an HTTP request. The server then obtains the corresponding English word and illustration URL from the database.

[1585] 3. Voice output and illustration display: The acquired English words are pronounced using a speech synthesis engine, and illustrations are displayed on the device screen.

[1586] Scanning QR codes

[1587] 1. QR Code Reading: Using the device's camera, scan the QR code placed in the store. The data obtained from the QR code is analyzed to obtain the corresponding alphabet. The data is used as a means of obtaining the alphabet.

[1588] 2. Display of English words and illustrations: English words and illustrations are displayed in alphabetical order based on the scanned QR code, and the words are simultaneously read out loud.

[1589] Emotion recognition and dynamic adjustment of learning content

[1590] 1. Emotion recognition: The camera and microphone are used to analyze the user's facial expressions and voice in real time, and the emotion is determined using an emotion recognition model (Keras model). This is an emotion recognition method.

[1591] 2. Adjustment of learning content: The recognized emotion data is sent to the server, which then dynamically adjusts the learning content and the difficulty of the dialogue based on the data. This is a means of dynamically adjusting the learning content.

[1592] Interactive question generation

[1593] 1. Question generation: The server generates interactive questions based on English words, presents the questions to the user, and accepts answers from the user.

[1594] 2. Next dialogue generation: After receiving the user's answer, the server uses a generative AI model to generate the next question and dynamically adjust the content of the dialogue based on the user's emotions.

[1595] Specific examples

[1596] For example, when a child enters the letter "A" or scans a QR code, the device displays the English word "Apple" and an illustration of an apple, and pronounces "Apple." Next, if the user's emotion is recognized as "joy," the server provides dialogue content and learning tasks tailored to the emotion. For example, the next question generated might be, "Blue is a nice color! Do you like blue flowers or blue cars?"

[1597] Prompt Sentence Examples

[1598] "Generate the following English learning content for children: The user's answer is 'Blue'. What should the next question be?"

[1599] This allows the learning system to combine user emotions with dynamic emotion recognition to provide an optimal environment for learning English while having fun.

[1600] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1601] Step 1:

[1602] The terminal accepts alphabetic input from the user. The input alphabetic data is temporarily stored in the terminal. The input data is used for subsequent processing.

[1603] Step 2:

[1604] The device sends an HTTP POST request to the server based on the input alphabet. The request includes the input alphabet. The server receives the request and retrieves the URL of the English word and corresponding illustration stored in a database. This is done through a database query. The server sends the retrieved English word and illustration URL to the device as an HTTP response.

[1605] Step 3:

[1606] The device receives the response from the server, passes the retrieved English words to the speech synthesis engine to pronounce them, and displays illustrations on the screen using the illustration URL. Specifically, the speech synthesis engine converts the input English words from text to speech data. Meanwhile, the illustration display retrieves an image from the URL and renders it on the screen.

[1607] Step 4:

[1608] A user holds a device and uses the camera to read a QR code placed in a physical store. The device analyzes the data obtained from the QR code and extracts the corresponding alphabet. This alphabet is used to obtain English words and illustrations according to the procedure described above.

[1609] Step 5:

[1610] The device uses a built-in camera and microphone to collect the user's facial expressions and voice, and recognizes emotions in real time. Emotion recognition is performed using a machine learning model with the Keras library. The collected data is preprocessed and input into the model to output emotional data.

[1611] Step 6:

[1612] The device sends the recognized emotion data to the server, which analyzes the emotion data and dynamically adjusts the learning content and dialogue questions to best suit the user. Specifically, it adjusts the difficulty level of the learning content and selects dialogue content templates.

[1613] Step 7:

[1614] The server generates the next dialogue-style question based on the English words. The next question is generated using a generative AI model based on the user's response. The question content and expression are adjusted based on emotional data. The generated question is sent to the device as an HTTP response, and the device pronounces it aloud and displays it on the screen.

[1615] Step 8:

[1616] The user enters answers to questions posed by the server. The device accepts the answers and sends them to the server as an HTTP POST request. The server analyzes the user's answers and generates the next dialogue based on them. This process is repeated, and learning progresses.

[1617] Examples:

[1618] For example, when a child enters the letter "A" or scans a QR code, the device displays the English word "Apple" and an illustration of an apple, and pronounces "Apple." Next, if the user's emotion is recognized as "joy," the server provides dialogue content and learning tasks tailored to the emotion. For example, the next question generated might be, "Blue is a nice color! Do you like blue flowers or blue cars?"

[1619] Example prompt sentence:

[1620] "Generate the following English learning content for children: The user's answer is 'Blue'. What should the next question be?"

[1621] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1622] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1623] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1624] [Fourth embodiment]

[1625] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1626] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1627] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1628] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1629] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1630] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1631] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1632] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1633] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1634] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1635] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1636] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1637] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1638] This invention relates to a learning system that allows children to learn English naturally while playing. This system accepts alphabet input, acquires English words based on the input, outputs them aloud, and displays corresponding illustrations. It can also generate simple dialogues and generate the next dialogue based on the user's answers.

[1639] System configuration

[1640] This system consists of a terminal operated by the user (child), a server that manages data on English words and illustrations, and various program components for outputting voice and displaying illustrations.

[1641] Program processing

[1642] 1. User input of alphabet

[1643] User: Enters alphabets into the input interface of the terminal. For example, the user enters "A."

[1644] 2. Sending the alphabet and acquiring English words

[1645] Terminal: The input alphabet is acquired and sent to the server. The server then acquires the corresponding English word (e.g., "Apple") from the database based on the received alphabet.

[1646] Server: Returns the acquired English word and the URL of the illustration corresponding to that word to the device.

[1647] 3. English word pronunciation and illustrations

[1648] Terminal: Receives English words sent from the server, pronounces them using a speech synthesis engine, and displays the corresponding illustrations on the screen using the corresponding illustration URL.

[1649] Specific examples

[1650] For example, when a user types "A" into a device, the device sends this to the server. The server retrieves the English word "Apple" and an illustration URL from the database and sends them back to the device. The device pronounces "Apple" using a speech synthesis engine and displays the illustration on the screen.

[1651] Conversational conversation generation

[1652] 1. Selecting the interaction mode

[1653] User: Select the interaction mode.

[1654] 2. Request to start a conversation

[1655] Terminal: Sends a request to start a conversation to the server.

[1656] Server: Generates the initial question and sends it to the device.

[1657] 3. Pronunciation and display of questions

[1658] Device: The received question is pronounced by a speech synthesis engine and displayed on the screen. For example, the question "What's your favorite color?" is displayed and pronounced aloud.

[1659] 4. Accepting User Answers

[1660] User: Enters an answer to the question. For example, enters "Blue."

[1661] Terminal: Send this response to the server.

[1662] 5. Generating the next dialogue

[1663] Server: Generates the next question or comment based on the user's answer and sends it to the device.

[1664] Specific examples

[1665] For example, if a user selects "Interactive mode," the device requests the server to ask the question "What's your favorite color?" If the user answers "Blue," the server generates the next question, "Blue is a nice color! Do you like blue flowers or blue cars?" and sends it back to the device. The device then pronounces this aloud and displays it on the screen.

[1666] Database Update

[1667] Users input new English words, illustrations, or vocabulary information that is easily confused into their device and send it to the server. The server analyzes the received information and updates the database. This allows the latest vocabulary and illustrations to be added to the database, making it possible to continuously update the learning content.

[1668] Overall flow

[1669] In this way, users can learn English naturally while playing, through a series of steps from inputting the alphabet to displaying the pronunciation and illustrations of English words, generating conversations in an interactive format, and updating the database. It also provides an environment where parents and children can learn together while having fun.

[1670] The processing flow will be explained below.

[1671] Processing alphabet input and displaying pronunciation and illustrations

[1672] Step 1:

[1673] User: Enter an alphabet (e.g., "A") into the terminal's input interface.

[1674] Step 2:

[1675] Terminal: Takes the input alphabet and sends an HTTP POST request to the server. The request contains the input alphabet.

[1676] Step 3:

[1677] Server: Analyzes the received alphabet and queries the database to obtain the corresponding English word (e.g., "Apple") and the URL of the corresponding illustration.

[1678] example:

[1679] sql

[1680] SELECT word, image_url FROM words WHERE alphabet = 'A';

[1681] Step 4:

[1682] Server: Sends an HTTP response to the terminal containing the URL of the acquired English word and illustration.

[1683] Step 5:

[1684] Terminal: The English word returned from the server (e.g., "Apple") is passed to the speech synthesis engine and pronounced. In addition, an illustration is displayed on the screen using the illustration URL.

[1685] Handling simple conversation generation in a conversational format

[1686] Step 1:

[1687] User: Selects the interaction mode in the device UI.

[1688] Step 2:

[1689] Terminal: Sends a request to start interactive mode to the server via HTTP protocol.

[1690] Step 3:

[1691] Server: Generates a simple question (e.g., "What's your favorite color?") to begin the conversation and sends it to the terminal.

[1692] Step 4:

[1693] Terminal: The question received from the server is passed to the speech synthesis engine, which pronounces it and displays the question on the screen.

[1694] Step 5:

[1695] User: Enters the answer to the question (e.g., "Blue") into the device's input field.

[1696] Step 6:

[1697] Terminal: Gets the user's answer and sends it to the server via an HTTP POST request.

[1698] Step 7:

[1699] Server: Based on the user's answers, it generates the next question or comment (e.g., "Blue is a nice color! Do you like blue flowers or blue cars?") and sends it to the device.

[1700] Step 8:

[1701] Terminal: New questions and comments received from the server are passed to the speech synthesis engine, spoken, and the new questions and comments are displayed on the screen.

[1702] Database update process

[1703] Step 1:

[1704] User: Enter information about new English words, illustrations, similar words, and commonly confused vocabulary into a dedicated form on the device.

[1705] Step 2:

[1706] Terminal: Obtains the entered information and sends it to the server via an HTTP POST request.

[1707] Step 3:

[1708] Server: Analyzes the received information and adds it as a new entry in the database or updates existing data as needed.

[1709] example:

[1710] sql

[1711] INSERT INTO similar_words (word1, word2) VALUES ('Pear', 'Pair')

[1712] ON DUPLICATE KEY UPDATE word1='Pear', word2='Pair';

[1713]

[1714] Example 1

[1715] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1716] Conventional learning systems make it difficult for children to learn English naturally while playing, and do not provide an interactive learning environment. In addition, updating the latest vocabulary and illustration information is done manually, making efficient learning difficult.

[1717] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1718] In this invention, the server includes means for accepting input of alphabets, means for acquiring vocabulary based on the input alphabets, means for audibly outputting the acquired vocabulary, means for displaying graphics corresponding to the vocabulary, means for acquiring the vocabulary from a database on the Internet, means for generating dialogue-style questions based on the vocabulary, means for presenting the questions to a user and accepting answers from the user, means for generating a next dialogue based on the answers, and means for analyzing the vocabulary and the graphics and updating the database with the latest learning content. This makes it possible to provide an interactive learning environment in which children can efficiently learn English naturally while playing and in which the latest learning content is automatically updated.

[1719] The "means for accepting alphabetic input" is an interface that provides a function for a user to input alphabetic characters using a terminal.

[1720] The "means for acquiring vocabulary" is a function for searching and acquiring corresponding vocabulary from a database based on input alphabets.

[1721] The "means for outputting vocabulary by voice" is a function that uses a voice synthesis engine to present the acquired vocabulary to the user in voice format.

[1722] "Means for displaying graphics" is a function that displays images or illustrations corresponding to acquired vocabulary on the screen.

[1723] "Means of obtaining from databases on the Internet" refers to the function of obtaining necessary vocabulary information and graphics from remote databases via a network.

[1724] The "means for generating interactive questions" is a function for generating questions to promote interactive dialogue with the user based on the acquired vocabulary.

[1725] The "means for presenting a question to a user and receiving an answer from the user" is a function for presenting a generated question to a user by voice or text and receiving an answer from the user.

[1726] The "means for generating the next dialogue" is a function that generates the next question or comment based on the user's answer and continues the interactive dialogue.

[1727] "Means for updating the latest learning content to the database" is a function that continuously updates learning content based on user input and new information and reflects it in the database.

[1728] The present invention is a learning system that allows children to learn English naturally while playing. This system is composed of a terminal operated by the user (child), a server that manages data on English words and graphics, and various program components for outputting audio and displaying images. Specific embodiments of the present invention are described below.

[1729] Hardware and software used

[1730] Terminal: A device such as a tablet, smartphone, or computer that a user interacts with through an interface.

[1731] Server: A remote computer system containing a database that manages vocabulary and graphic data and generates dialogue.

[1732] Speech synthesis engine: Software for converting text to speech, such as Google Text-to-Speech (TTS) or Amazon Polly.

[1733] Program processing

[1734] 1. User input of alphabet

[1735] A user inputs an alphabet into the input interface of a terminal. For example, the user inputs "A."

[1736] 2. Sending the Alphabet

[1737] The device captures the input alphabet and sends it to the server as an HTTP request.

[1738] 3. Acquire English words and illustrations

[1739] The server analyzes the received alphabet and retrieves the corresponding English word (e.g., "Apple") and the URL of its illustration from the database. The retrieved information is then sent back to the device.

[1740] 4. Speech synthesis of English words and illustration display

[1741] The device receives the English word and illustration URL returned from the server, pronounces the English word using a speech synthesis engine (e.g., Google Text-to-Speech), and displays the corresponding graphic on the screen.

[1742] 5. Conversational conversation generation

[1743] The user selects a conversation mode, and the device sends a request to start a conversation to the server. The server generates an initial question (e.g., "What's your favorite color?") and sends it to the device. The device pronounces the received question using a speech synthesis engine and displays it on the screen. When the user inputs an answer to the question (e.g., "Blue"), the device sends it to the server, and the server generates the next question or comment (e.g., "Blue is a nice color! Do you like blue flowers or blue cars?") and sends it back to the device.

[1744] 6. Updating the database

[1745] The user inputs new English words, illustrations, or confusing vocabulary information into the device and sends it to the server. The server analyzes the received information and updates the database, adding the latest vocabulary and illustration data to the database.

[1746] Specific examples

[1747] For example, when a user types "A" into a device, the device sends this to the server. The server retrieves the English word "Apple" and its illustration URL from the database and sends it back to the device. The device pronounces "Apple" using a speech synthesis engine and displays the illustration on the screen.

[1748] Also, consider the case where the user selects "Interactive mode" and the terminal requests the question "What's your favorite color?" from the server. If the user answers "Blue," the server generates the next question, "Blue is a nice color! Do you like blue flowers or blue cars?" and sends it back to the terminal. The terminal then pronounces this aloud and displays it on the screen.

[1749] Prompt Sentence Examples

[1750] The following prompts can be input to the generative AI model:

[1751] "A child uses the device to enter the letter 'A'. The server retrieves the English word 'Apple' and the URL of its illustration from the database and sends it back to the device. The device pronounces 'Apple' using its speech synthesis engine and displays the illustration on the screen. If interactive mode is selected, the device pronounces the question generated by the server using its speech synthesis engine and generates the next question based on the user's answer."

[1752] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1753] Step 1:

[1754] User input of alphabetic characters

[1755] A user inputs an alphabet using the input interface (keyboard or touch screen) of a terminal. For example, a user inputs the alphabet "A." The input is displayed as a character in a text box on the terminal. This input becomes the starting point for subsequent data processing.

[1756] Step 2:

[1757] Sending the alphabet

[1758] The terminal captures the alphabet "A" entered by the user and generates an HTTP request to send it to the server. The input data (alphabet "A") is sent to the server. The HTTP request also includes metadata such as the client ID and request timestamp.

[1759] Input: The alphabet "A" entered by the user

[1760] Output: HTTP requests containing the letter "A"

[1761] What it does: The terminal program captures the user's input and constructs a request to send this data to the server.

[1762] Step 3:

[1763] Acquire English words and illustrations

[1764] The server extracts the letter "A" from the received HTTP request, generates and executes a database query, retrieves the corresponding English word (e.g., "Apple") and the URL of its illustration from the database, and returns this to the terminal as an HTTP response. The English word and illustration URL are output as the results of the database query.

[1765] Input: HTTP request containing the letter "A"

[1766] Output: HTTP response containing the English word "Apple" and an illustration URL

[1767] Specific operation: The server queries the database to retrieve the English word "Apple" and its related illustration URLs.

[1768] Step 4:

[1769] Speech synthesis of English words and illustration display

[1770] The device receives the HTTP response returned from the server and generates the English word "Apple" aloud using a text-to-speech API or speech synthesis engine (for example, Google Text-to-Speech). It also obtains image data from the obtained illustration URL and displays it on the screen. The speech synthesis generates the audio data, and the image data is rendered on the screen.

[1771] Input: HTTP response containing the English word "Apple" and an illustration URL

[1772] Output: Voice output of the English word "Apple" and display of an illustration on the screen

[1773] Specific operation: The device synthesizes "Apple" through the voice speaker and displays an illustration on the screen.

[1774] Step 5:

[1775] Selecting an Interactive Mode

[1776] The user selects an interaction mode on the terminal interface. The selection triggers the system to transition to the interactive question generation module. The selection information is recorded as internal data of the terminal.

[1777] Input: The user's action to select the interaction mode

[1778] Output:Internal flag for interactive mode selection

[1779] Specific action: The user taps an interaction mode button in the UI.

[1780] Step 6:

[1781] Conversation Initiating Request

[1782] The terminal requests the server to start the dialogue mode. This request triggers the server to generate the initial question. The terminal sends a dialogue start request to the server, and the initial question is generated based on this request.

[1783] Input: Request based on internal flag for interactive mode selection

[1784] Output: A request to generate the first question

[1785] Specific operation: The terminal sends a request to start a conversation to the server.

[1786] Step 7:

[1787] Question pronunciation and display

[1788] The server receives the dialogue start request and generates the initial question (e.g., "What's your favorite color?"). The generated question is sent to the device as an HTTP response. The device then outputs the received question aloud using a speech synthesis engine and displays it as text on the screen.

[1789] Input: A request to generate the initial question.

[1790] Output: HTTP response containing the initial question, "What's your favorite color?"

[1791] Specific operation: The device synthesizes the question, outputs it, and displays it on the screen.

[1792] Step 8:

[1793] Submitting user answers

[1794] The user uses the device to input an answer to the question, for example, "Blue." The device captures this answer and sends it to the server. The input data is sent to the server and triggers the generation of the next question.

[1795] Input: Answer data "Blue" entered by the user

[1796] Output: HTTP request with response data "Blue"

[1797] Specific operation: The user types the answer on the keyboard, and the device captures it and sends it to the server.

[1798] Step 9:

[1799] Generate the next conversation

[1800] The server receives the user's answer and generates the next question or comment, for example, "Blue is a nice color! Do you like blue flowers or blue cars?", and sends it to the terminal as an HTTP response.

[1801] Input: HTTP request containing user answer data "Blue"

[1802] Output: HTTP response containing the question "Blue is a nice color! Do you like blue flowers or blue cars?"

[1803] Specific operation: The server analyzes the response data and generates the next dialogue.

[1804] Step 10:

[1805] Database Update

[1806] The user inputs new English words, graphics, and confusing vocabulary information into the terminal and sends it to the server. The server analyzes the received information and updates the database. The updated database is reflected in subsequent data acquisition and question generation.

[1807] Input: New English words, graphics, and vocabulary information

[1808] Output: Updated database

[1809] What happens: The user enters new information, which the server parses to update the database.

[1810] (Application example 1)

[1811] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1812] Previous English learning systems for children were limited to displaying words and illustrations and outputting audio on fixed devices such as PCs and tablets, which meant that learning lacked flexibility and enjoyment. Furthermore, they lacked the functionality to deepen learning through interactive question and answer formats, and did not adequately provide an environment for children to learn English naturally. Furthermore, the database was not continually updated to reflect the latest information in real time.

[1813] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1814] In this invention, the server includes means for accepting input of alphabets, means for acquiring English words based on the input alphabets, means for audibly outputting the acquired English words, means for displaying illustrations corresponding to the English words, means for displaying the English words and the illustrations on the smart glasses and outputting them audibly, means for generating dialogue-style questions based on the English words, means for presenting the questions to the user and accepting answers from the user, means for generating a next dialogue based on the answers, and means for displaying the dialogue-style questions and answers on the smart glasses and outputting them audibly, thereby enabling children to learn English more naturally and enjoyably.

[1815] "Alphabet" refers to the 26 letters that make up the basic set of letters in the English language.

[1816] "Means for accepting input" refers to interface devices or software for receiving character information such as alphabets from the user.

[1817] The "means for obtaining English words" is a mechanism for finding and extracting corresponding English words based on the input alphabet.

[1818] "Means for audio output" refers to devices such as a speech synthesis engine or speaker that reproduces the acquired English words as sound.

[1819] "Means for displaying illustrations" refers to mechanisms or software for displaying images corresponding to English words on a screen.

[1820] "Smart glasses" are wearable devices that have a display for displaying information in front of the eyes and have audio output capabilities.

[1821] "Interactive question generation means" refers to algorithms or software that generate appropriate questions to promote English learning through dialogue with the user.

[1822] "Means for accepting responses from users" refers to an interface or software for recognizing and appropriately accepting responses from users to questions.

[1823] The "means for generating the next dialogue" refers to algorithms or software for generating the next appropriate question or comment based on the user's answers.

[1824] An "Internet database" is an information management system that stores data remotely and makes it accessible over a network.

[1825] MODE FOR CARRYING OUT THE INVENTION

[1826] The system of the present invention is a learning system that aims to help children learn English naturally while playing, and is implemented using smart glasses. The system allows users to easily input alphabets, obtain English words based on the input, output them as audio, and display corresponding illustrations. It also provides a deeper learning experience through interactive question and answer sessions.

[1827] System configuration

[1828] Hardware:

[1829] Smart glasses (e.g. Google Glass): Uses a display and built-in speakers.

[1830] Server: Manages English words and illustrations and generates dialogue.

[1831] software:

[1832] Speech synthesis engine (e.g. Google Text-to-Speech API)

[1833] Database (e.g. MySQL)

[1834] API server (e.g. Flask)

[1835] Program processing

[1836] 1. Enter the alphabet:

[1837] The user (child) uses the touchpad of the smart glasses to input alphabets, which are recognized by software in the smart glasses and sent to the server.

[1838] 2. Acquire English words and illustrations:

[1839] The server retrieves the corresponding English word and illustration URL from the database based on the received alphabet, and this information is then sent back to the smart glasses.

[1840] 3. English word pronunciation and illustration display:

[1841] The smart glasses pronounce the received English words using a speech synthesis engine and simultaneously display illustrations on the screen.

[1842] 4. Interactive learning:

[1843] When the user selects the dialogue mode, the smart glasses display an initial question and read it aloud. The user responds by voice or using the touchpad, and the answer is sent to the server. The server generates a new question and sends it back to the smart glasses. By repeating this process, the user can learn English through dialogue.

[1844] Specific examples

[1845] Alphabet input and word generation:

[1846] When a user uses the smart glasses to input the letter "B," the smart glasses will display the word "Banana" on the display and pronounce it as "Banana."

[1847] Interactive Question and Answer:

[1848] The user selects the interactive mode and the smart glasses ask, "What is your favorite fruit?" If the user answers, "Apple," the smart glasses generate the next question, "Do you like red apples or green apples?", which the smart glasses display and pronounce.

[1849] Prompt Sentence Examples

[1850] I entered "B". What English word do you want to see and how do you pronounce it? The next question is "What is your favorite fruit?" If the user answers "Apple", please generate the next question.

[1851] This system allows children to receive real-time responses and feedback through the smart glasses, enabling them to effectively learn English while playing.

[1852] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1853] Step 1:

[1854] A user inputs alphabets using the touchpad of the smart glasses, which are then recognized by software within the smart glasses. The input of this process is the alphabets input by the user, and the output is the recognized alphabet data.

[1855] Step 2:

[1856] The smart glasses send the recognized alphabet data to the server, which receives the data and queries the database to find the English word and its corresponding illustration. The input of this process is the alphabet data sent from the smart glasses, and the output is the URL of the corresponding English word and illustration.

[1857] Step 3:

[1858] The server returns the URL of the English word and illustration retrieved from the database to the smart glasses. The smart glasses receive it, pronounce the English word using a speech synthesis engine, and display the illustration on the display. The input of this process is the URL of the English word and illustration returned from the server, and the output is the pronunciation of the English word and the display of the illustration by the smart glasses.

[1859] Step 4:

[1860] When the user selects an interaction mode, the smart glasses send a request to start the interaction to the server. The server generates an initial question and sends it to the smart glasses. The input of this process is the request to select the interaction mode, and the output is the generated question.

[1861] Step 5:

[1862] The smart glasses will then use a speech synthesis engine to pronounce the received question and display it on the display. The user can then respond to the question by voice or using the touchpad. The input to this process is the question sent from the server, and the output is the question displayed on the smart glasses' display and the voice output.

[1863] Step 6:

[1864] The user's answer is recognized by the smart glasses and sent to the server. The server generates a next question based on the received answer and sends it back to the smart glasses. The input of this process is the user's answer and the output is the generated next question.

[1865] Step 7:

[1866] The smart glasses receive the next question sent from the server, pronounce it again using the speech synthesis engine, and display it on the display. This process continues until the user finishes the dialogue, and all dialogue is recorded and the database is updated as necessary. The input of this process is the next question, and the output is the voice output and display based on that question.

[1867] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1868] This invention relates to a learning system that allows children to learn English naturally while playing. In particular, this system has the ability to recognize the user's emotions and adjust the learning content accordingly. In addition to inputting the alphabet, acquiring and pronouncing English words, displaying illustrations, and generating interactive questions, this system can also recognize the user's emotions to provide an optimal learning experience.

[1869] System configuration

[1870] This system consists of a terminal operated by the user (child), a server that manages data on English words and illustrations, various program components for outputting voice and displaying illustrations, and an emotion engine that recognizes the user's emotions.

[1871] Program processing

[1872] Processing alphabet input and displaying pronunciation and illustrations

[1873] 1. User input of alphabet

[1874] User: Enter an alphabet (e.g., "A") into the terminal's input interface.

[1875] 2. Sending the alphabet and acquiring English words

[1876] Terminal: Takes the input alphabet and sends an HTTP POST request to the server. The request contains the input alphabet.

[1877] Server: Analyzes the received alphabet and queries the database to obtain the corresponding English word (e.g., "Apple") and the URL of the corresponding illustration.

[1878] example:

[1879] sql

[1880] SELECT word, image_url FROM words WHERE alphabet = 'A';

[1881] 3. English word pronunciation and illustrations

[1882] Server: Sends an HTTP response to the terminal containing the URL of the acquired English word and illustration.

[1883] Terminal: The English word returned from the server (e.g., "Apple") is passed to the speech synthesis engine and pronounced. In addition, an illustration is displayed on the screen using the illustration URL.

[1884] Conversational conversation generation

[1885] 1. Selecting the interaction mode

[1886] User: Selects the interaction mode in the device UI.

[1887] 2. Request to start a conversation

[1888] Terminal: Sends a request to start a conversation to the server using the HTTP protocol.

[1889] Server: Generates the initial question and sends it to the device.

[1890] 3. Pronunciation and display of questions

[1891] Terminal: The question received from the server is passed to the speech synthesis engine, which pronounces it, and the question is displayed on the screen.

[1892] 4. Accepting User Answers

[1893] User: Enters the answer to the question into the input field on the device.

[1894] Terminal: Send this response to the server.

[1895] 5. Generating the next dialogue

[1896] Server: Generates the next question or comment based on the user's answer and sends it to the device.

[1897] Terminal: New questions and comments received from the server are passed to the speech synthesis engine, spoken, and the new questions and comments are displayed on the screen.

[1898] Use of emotion engine

[1899] 1. Emotional awareness

[1900] Device: The built-in camera and microphone are used to input the user's voice and facial expressions into the emotion engine.

[1901] Emotion engine: Analyzes the user's emotional state (e.g., happiness, sadness, excitement) and outputs this as data.

[1902] 2. Transmission and processing of emotional data

[1903] Terminal: Sends emotion data obtained from the emotion engine to the server.

[1904] Server: Analyzes emotional data and adjusts learning content and dialogue.

[1905] 3. Adjusting learning content

[1906] Server: Adjusts the content of the next dialogue and the difficulty of the learning task based on the user's emotions and sends that information to the terminal.

[1907] Terminal: Adjusts the screen display, audio tone, and output content based on information sent from the server.

[1908] Specific examples

[1909] For example, when a user inputs "A," the device sends this to the server, which retrieves the English word "Apple" and an illustration URL from the database and sends them to the device. The device then pronounces "Apple" and displays the illustration.

[1910] In dialogue mode, when the user is asked "What's your favorite color?" and answers "Blue," the server generates the next question: "Blue is a nice color! Do you like blue flowers or blue cars?" Furthermore, the emotion engine recognizes the user's emotions (e.g., excitement) from their facial expressions and voice, and the server adjusts the dialogue content and difficulty accordingly.

[1911] This allows children to learn English naturally while playing, providing an optimal learning experience that responds to their emotions, and creating an environment where parents and children can enjoy learning together.

[1912] The processing flow will be explained below.

[1913] Specific processing steps of a system that combines emotion engines

[1914] Processing alphabet input and displaying pronunciation and illustrations

[1915] Step 1:

[1916] User: Enter an alphabet (e.g., "A") into the terminal's input interface.

[1917] Step 2:

[1918] Terminal: Takes the input alphabet and sends an HTTP POST request to the server. The request contains the input alphabet.

[1919] Step 3:

[1920] Server: Analyzes the received alphabet and queries the database to obtain the corresponding English word (e.g., "Apple") and the URL of the corresponding illustration.

[1921] example:

[1922] sql

[1923] SELECT word, image_url FROM words WHERE alphabet = 'A';

[1924] Step 4:

[1925] Server: Sends an HTTP response to the terminal containing the URL of the acquired English word and illustration.

[1926] Step 5:

[1927] Terminal: The English word returned from the server (e.g., "Apple") is passed to the speech synthesis engine and pronounced. In addition, an illustration is displayed on the screen using the illustration URL.

[1928] Handling conversational dialogue generation

[1929] Step 1:

[1930] User: Selects the interaction mode in the device UI.

[1931] Step 2:

[1932] Terminal: Sends a request to start interactive mode to the server via HTTP protocol.

[1933] Step 3:

[1934] Server: Generates a simple question (e.g., "What's your favorite color?") to begin the conversation and sends it to the terminal.

[1935] Step 4:

[1936] Terminal: The question received from the server is passed to the speech synthesis engine, which pronounces it and displays the question on the screen.

[1937] Step 5:

[1938] User: Enters the answer to the question (e.g., "Blue") into the device's input field.

[1939] Step 6:

[1940] Terminal: Gets the user's answer and sends it to the server via an HTTP POST request.

[1941] Step 7:

[1942] Server: Based on the user's answers, it generates the next question or comment (e.g., "Blue is a nice color! Do you like blue flowers or blue cars?") and sends it to the device.

[1943] Step 8:

[1944] Terminal: New questions and comments received from the server are passed to the speech synthesis engine, spoken, and the new questions and comments are displayed on the screen.

[1945] Use of emotion engine

[1946] Step 1:

[1947] Device: The built-in camera and microphone are used to input the user's voice and facial expressions into the emotion engine.

[1948] Step 2:

[1949] Emotion engine: Analyzes the user's emotional state (e.g., happiness, sadness, excitement) and outputs this as data.

[1950] Step 3:

[1951] Terminal: Sends emotion data obtained from the emotion engine to the server.

[1952] Step 4:

[1953] Server: Analyzes emotional data and adjusts learning content and dialogue.

[1954] Step 5:

[1955] Server: Adjusts the content of the next dialogue and the difficulty of the learning task based on the user's emotions and sends that information to the terminal.

[1956] Step 6:

[1957] Terminal: Adjusts the screen display, audio tone, and output content based on information sent from the server.

[1958] Specific examples

[1959] Alphabetical input example:

[1960] 1. When the user types "A" into the terminal,

[1961] 2. The device sends this to the server,

[1962] 3. The server retrieves the English word "Apple" and the illustration URL from the database and returns them to the device.

[1963] 4. The device will pronounce "Apple" and display an illustration.

[1964] Interactive example:

[1965] 1. The user selects an interaction mode,

[1966] 2. The device requests the question "What's your favorite color?" from the server.

[1967] 3. If the user answers "Blue,"

[1968] 4. The server generates the next question, "Blue is a nice color! Do you like blue flowers or blue cars?" and sends it back to the terminal.

[1969] 5. The device will pronounce this aloud and display it on the screen.

[1970] Emotion engine example:

[1971] 1. The user's facial expressions and voice are input into the emotion engine via the device's camera and microphone.

[1972] 2. The emotion engine recognizes the user's emotion (e.g., excitement).

[1973] 3. Emotional data is sent to the server, which then adjusts the learning accordingly.

[1974] 4. The device displays tailored learning content to provide an optimal learning experience.

[1975] This will create a learning system that takes the user's emotions into consideration, allowing children to learn English naturally while having fun. It will also create an environment where parents can feel at ease and support their children's learning.

[1976] Example 2

[1977] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1978] Conventional English learning systems lacked the ability to adjust learning content based on the user's emotions, making it difficult to provide an effective learning experience. Furthermore, even when users engaged in interactive learning, the system often consisted of one-way question-and-answer exchanges, making it difficult for users to maintain their interest and attention in the progress of their learning. Furthermore, there were few integrated learning systems that included voice synthesis and illustration display.

[1979] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1980] In this invention, the server includes means for accepting alphabetic input, means for acquiring English word and illustration data based on the input alphabet, means for audibly outputting the acquired English words, means for displaying illustrations corresponding to the English words, means for recognizing the user's emotions, and means for adjusting the learning content based on the recognized emotions. This allows for an optimal learning experience that is tailored to the user's emotions, thereby maintaining the user's interest and attention. Furthermore, even in interactive learning, the content of the next dialogue and the difficulty of the learning tasks can be adjusted based on the user's emotions and responses, thereby achieving effective learning.

[1981] The "means for accepting alphabetic input" is an input interface that allows the terminal to recognize alphabetic input by the user and pass it on to subsequent processing.

[1982] "Means for obtaining data on English words and illustrations" refers to the process by which the server queries the database based on the alphabet sent from the terminal and obtains the URL of the corresponding English word and its illustration.

[1983] The "means for outputting English words by voice" is a function for converting the acquired English words into voice using a voice synthesis engine and allowing the user to hear it.

[1984] "Means for displaying illustrations" is a function for reading image data from the URL of the acquired illustration and displaying it on the terminal screen.

[1985] "Means for recognizing user emotions" refers to the process of using a built-in camera and microphone to analyze the user's facial expressions and voice to determine the user's emotional state (e.g., joy, sadness, excitement).

[1986] The "means for adjusting learning content based on recognized emotions" is a function that dynamically changes the learning content, difficulty level of interactive questions, tone, display content, etc. according to the user's emotional state.

[1987] The "means for generating interactive questions" is a process for creating questions to proceed in an interactive format with the user based on the acquired English vocabulary and learning content.

[1988] The "means for accepting an answer from a user" is an input interface for receiving an answer entered by a user in an interactive format and passing it on to subsequent processing.

[1989] "Means for generating the next dialogue" refers to the process of creating new questions or comments based on the user's answers, etc., and continuing the dialogue.

[1990] "Means of obtaining data from a database on the Internet" refers to a function that obtains the necessary English word and illustration data in real time from a remote database via a network.

[1991] The present invention is a learning system that can recognize a user's emotions and adjust the English learning content accordingly. This system is configured using the following hardware and software.

[1992] System configuration

[1993] Hardware

[1994] 1. Device: A device used by a user, such as a tablet or smartphone, that has a touchscreen, camera, and microphone.

[1995] 2. Server: A cloud computing platform, such as Amazon Web Services (AWS) or Microsoft Azure, that can handle database operations and API requests.

[1996] software

[1997] 1. Database: A database management system such as MySQL or PostgreSQL is used to store the English word and illustration data.

[1998] 2. Speech synthesis engine: Uses Google Text-to-Speech API or Amazon Polly to output English words aloud on the device.

[1999] 3. Emotion recognition engine: Recognizes user emotions using Microsoft Azure Cognitive Services and IBM Watson.

[2000] 4. HTTP protocol: Used for data communication between the terminal and the server.

[2001] Operation explanation

[2002] 1. Alphabetical input

[2003] The user uses the device's input interface to input alphabetic characters, which can come from a keyboard or touchscreen and are then captured by a program within the device.

[2004] 2. Data Acquisition and Display

[2005] The device receives the input alphabet and sends it as an HTTP POST request to the server. The server queries the database based on the received alphabet and obtains the corresponding English word (e.g., "Apple") and the URL of the illustration.

[2006] The acquired English words are output as voice using a speech synthesis engine, and illustrations are displayed on the device screen.

[2007] 3. Conversational conversation generation

[2008] When the user selects the interactive mode, the terminal sends a request to the server to start the interaction, and the server generates an initial question and sends it to the terminal.

[2009] When the user enters an answer to a question into the terminal, the terminal sends the answer to the server, which then generates the next question or comment and sends it back to the terminal, thus continuing the dialogue.

[2010] 4. Emotion recognition and learning adjustment

[2011] The device uses a camera and microphone to capture the user's facial expressions and voice, and inputs them into an emotion recognition engine.

[2012] The emotion recognition engine analyzes the user's emotional state and returns data based on this to the device, which then transmits this data to the server, which then adjusts the learning content and difficulty of the questions based on the user's emotions.

[2013] The adjusted learning content is sent to the terminal and presented to the user in an appropriate format.

[2014] Specific examples

[2015] For example, when a user inputs "A," the device sends the input to the server, which retrieves the English word "Apple" and its illustration from the database and sends it to the device. The device then outputs "Apple" aloud and displays the illustration on the screen.

[2016] In a conversational example, if a user is asked "What's your favorite color?" and answers "Blue," the server generates the next question: "Blue is a nice color! Do you like blue flowers or blue cars?" Furthermore, an emotion recognition engine detects excitement from the user's facial expressions and voice, and the server adjusts the tone and content of the questions accordingly.

[2017] Prompt Sentence Examples

[2018] Examples of prompts to input to a generative AI model include:

[2019] "When the user types the letter 'A', retrieve and display words and illustrations related to that letter."

[2020] "Conduct a dialogue between the user and the system, asking and answering questions in English."

[2021] These can be used to provide appropriate instructions to the generative AI model.

[2022] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2023] Step 1:

[2024] User input of alphabetic characters

[2025] Input: A user types an alphabet (e.g., "A") into the terminal's input interface.

[2026] How it works: The user enters alphabetic characters using the device's keyboard or touchscreen.

[2027] Output: The input alphabet is retrieved by the terminal program.

[2028] Step 2:

[2029] Sending the alphabet and getting English words

[2030] Input: Alphabet typed into the terminal.

[2031] Operation: The device includes the obtained alphabet in the payload of an HTTP POST request and sends it to the server.

[2032] The server receives the HTTP request and parses the alphabet.

[2033] The server performs a database query based on the parsed alphabet and retrieves the corresponding English word (e.g., "Apple") and the URL of the illustration.

[2034] Example database query:

[2035] sql

[2036] SELECT word, image_url FROM words WHERE alphabet = 'A';

[2037] Output: The URL of the acquired English word and illustration is sent from the server to the device as an HTTP response.

[2038] Step 3:

[2039] English word pronunciation and illustration display

[2040] Input: English word and illustration URL sent from the server.

[2041] How it works: The device passes the received English word to a speech synthesis engine (e.g., Google Text-to-Speech API) and pronounces it.

[2042] At the same time, the device loads the image data using the illustration's URL and displays it on the screen.

[2043] Output: Pronunciation of English words and illustrations.

[2044] Step 4:

[2045] Selecting an Interactive Mode

[2046] Input: User selection of an interactive mode.

[2047] How it works: The user selects an interaction mode in the device's UI, which triggers a request to start an interaction.

[2048] Output: Interactive mode selections are saved to the terminal and the program continues to the next step.

[2049] Step 5:

[2050] Interaction-starting request

[2051] Input: Selection information for interactive mode.

[2052] Operation: The terminal sends a request to start a conversation to the server using the HTTP protocol.

[2053] The server receives the request and generates an initial question, which can be retrieved from a database or generated based on logic.

[2054] Output: The initial question generated by the server is sent to the terminal.

[2055] Step 6:

[2056] Question pronunciation and display

[2057] Input: The question sent by the server.

[2058] How it works: The device passes the received question to a speech synthesis engine (e.g., Google Text-to-Speech API) and pronounces it aloud.

[2059] At the same time, the device displays the question on the screen.

[2060] Output: Questions are spoken and displayed on the screen.

[2061] Step 7:

[2062] Accepting user answers

[2063] Input: The user's answer to the question (e.g., "Blue").

[2064] Action: The user enters an answer into an input field on the terminal.

[2065] The terminal receives this response and sends it to the server.

[2066] Output: The user's answer sent to the server.

[2067] Step 8:

[2068] Generate the next conversation

[2069] Input: User answers and server interaction logic.

[2070] How it works: The server analyzes the user's answers and generates the next question or comment based on them.

[2071] The server sends the generated questions and comments to the terminal as an HTTP response.

[2072] Output: New questions and comments are sent to the device.

[2073] Step 9:

[2074] Emotion recognition

[2075] Input: User's facial expressions and voice.

[2076] Operation: The device uses the built-in camera and microphone to capture the user's facial expressions and voice, and inputs them into the emotion engine.

[2077] The emotion engine analyzes the user's emotional state and generates emotion data.

[2078] Output: Emotion data obtained from the emotion engine.

[2079] Step 10:

[2080] Sending and processing emotional data

[2081] Input: Emotion data.

[2082] Operation: The device sends emotion data to the server as an HTTP request.

[2083] The server analyzes the received emotional data, understands the user's emotional state, and adjusts the learning content and dialogue accordingly.

[2084] Output: Learning and dialogue content adjusted based on user sentiment.

[2085] Step 11:

[2086] Adjusting learning content

[2087] Input: Coordination information and emotion data.

[2088] Operation: The server adjusts the content of the next dialogue and the difficulty of the learning task based on the user's emotional state, and sends new learning content and dialogue content based on this to the terminal.

[2089] The terminal adjusts the screen display and audio tone based on the received adjustment content.

[2090] Output: The adjusted learning and interaction content is presented to the user.

[2091] (Application example 2)

[2092] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2093] Traditional learning systems only provide static learning content for children to learn English naturally while playing. As a result, it is difficult to maintain learners' interest and it is not possible to provide an optimal learning experience that suits each individual's emotional state and learning progress. There is also a lack of ways to provide an interactive learning experience in physical stores.

[2094] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2095] In this invention, the server includes a means for accepting alphabet input, a means for acquiring English words based on the input alphabet, a means for audibly outputting the acquired English words, a means for displaying illustrations corresponding to the English words, a means for reading QR codes, a means for acquiring alphabets by reading the QR codes, a means for recognizing a user's emotions, and a means for dynamically adjusting learning content based on the recognized emotions. This enables real-time analysis of the user's (child's) emotional state and reactions, providing an optimal learning experience. Furthermore, an interactive learning experience is also provided in a physical store, creating an environment where children can enjoy learning English while playing.

[2096] The "means for accepting alphabetic input" is an interface that allows the user to input any alphabetic character.

[2097] The "means for acquiring English words based on the input alphabet" is a system having a function for searching and acquiring English words based on the input alphabet.

[2098] The "means for outputting the acquired English words by voice" refers to a device or program that pronounces the acquired English words using voice synthesis technology.

[2099] The "means for displaying an illustration corresponding to the English word" is a system that displays an image related to the English word on a display device such as a monitor.

[2100] A "means for reading a QR code" is a device or program that uses a device such as a camera to scan a QR code and obtain its contents.

[2101] The "means for obtaining the alphabet by reading the QR code" is a system that has the function of analyzing the data obtained from the QR code and obtaining the corresponding alphabet.

[2102] "Means for recognizing user emotions" refers to technology that uses sensors such as cameras and microphones to analyze the user's facial expressions and tone of voice to determine their emotional state.

[2103] "Means for dynamically adjusting learning content based on recognized emotions" refers to a system that changes the difficulty level of learning content and question format in real time based on recognized emotional data.

[2104] An "interactive question generator" is a program that has the ability to automatically generate new questions based on user input and context.

[2105] The "means for accepting answers from the user" is an interface for receiving and saving answers entered by the user.

[2106] The "means for generating the next dialogue based on the answer" is a technique for analyzing the user's answer and generating the next question or comment in response to it.

[2107] System configuration

[2108] This invention is a learning system that allows users (children) to learn English naturally while playing. In particular, it has the function of dynamically adjusting the learning content according to the user's emotions using emotion recognition technology. This system is composed of a series of hardware and software.

[2109] Hardware and software used

[2110] The system uses the following major hardware and software:

[2111] Terminal: A device operated by the user, such as a smartphone or tablet, that scans QR codes, outputs audio, and displays illustrations.

[2112] Server: Connected to the central database, it retrieves English words and illustrations, generates dialogue content, and processes emotion recognition data.

[2113] Camera and microphone: The sensors required for user emotion recognition.

[2114] Software: Emotion recognition models using libraries such as OpenCV and Keras, a speech synthesis engine, and an API for processing HTTP requests.

[2115] Program processing

[2116] Alphabet input and English word acquisition

[2117] 1. Alphabet input: The user inputs any alphabet through the input interface of the terminal. This is the input method.

[2118] 2. Obtaining English words: The input alphabet is sent to the server via an HTTP request. The server then obtains the corresponding English word and illustration URL from the database.

[2119] 3. Voice output and illustration display: The acquired English words are pronounced using a speech synthesis engine, and illustrations are displayed on the device screen.

[2120] Scanning QR codes

[2121] 1. QR Code Reading: Using the device's camera, scan the QR code placed in the store. The data obtained from the QR code is analyzed to obtain the corresponding alphabet. The data is used as a means of obtaining the alphabet.

[2122] 2. Display of English words and illustrations: English words and illustrations are displayed in alphabetical order based on the scanned QR code, and the words are simultaneously read out loud.

[2123] Emotion recognition and dynamic adjustment of learning content

[2124] 1. Emotion recognition: The camera and microphone are used to analyze the user's facial expressions and voice in real time, and the emotion is determined using an emotion recognition model (Keras model). This is an emotion recognition method.

[2125] 2. Adjustment of learning content: The recognized emotion data is sent to the server, which then dynamically adjusts the learning content and the difficulty of the dialogue based on the data. This is a means of dynamically adjusting the learning content.

[2126] Interactive question generation

[2127] 1. Question generation: The server generates interactive questions based on English words, presents the questions to the user, and accepts answers from the user.

[2128] 2. Next dialogue generation: After receiving the user's answer, the server uses a generative AI model to generate the next question and dynamically adjust the content of the dialogue based on the user's emotions.

[2129] Specific examples

[2130] For example, when a child enters the letter "A" or scans a QR code, the device displays the English word "Apple" and an illustration of an apple, and pronounces "Apple." Next, if the user's emotion is recognized as "joy," the server provides dialogue content and learning tasks tailored to the emotion. For example, the next question generated might be, "Blue is a nice color! Do you like blue flowers or blue cars?"

[2131] Prompt Sentence Examples

[2132] "Generate the following English learning content for children: The user's answer is 'Blue'. What should the next question be?"

[2133] This allows the learning system to combine user emotions with dynamic emotion recognition to provide an optimal environment for learning English while having fun.

[2134] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2135] Step 1:

[2136] The terminal accepts alphabetic input from the user. The input alphabetic data is temporarily stored in the terminal. The input data is used for subsequent processing.

[2137] Step 2:

[2138] The device sends an HTTP POST request to the server based on the input alphabet. The request includes the input alphabet. The server receives the request and retrieves the URL of the English word and corresponding illustration stored in a database. This is done through a database query. The server sends the retrieved English word and illustration URL to the device as an HTTP response.

[2139] Step 3:

[2140] The device receives the response from the server, passes the retrieved English words to the speech synthesis engine to pronounce them, and displays illustrations on the screen using the illustration URL. Specifically, the speech synthesis engine converts the input English words from text to speech data. Meanwhile, the illustration display retrieves an image from the URL and renders it on the screen.

[2141] Step 4:

[2142] A user holds a device and uses the camera to read a QR code placed in a physical store. The device analyzes the data obtained from the QR code and extracts the corresponding alphabet. This alphabet is used to obtain English words and illustrations according to the procedure described above.

[2143] Step 5:

[2144] The device uses a built-in camera and microphone to collect the user's facial expressions and voice, and recognizes emotions in real time. Emotion recognition is performed using a machine learning model with the Keras library. The collected data is preprocessed and input into the model to output emotional data.

[2145] Step 6:

[2146] The device sends the recognized emotion data to the server, which analyzes the emotion data and dynamically adjusts the learning content and dialogue questions to best suit the user. Specifically, it adjusts the difficulty level of the learning content and selects dialogue content templates.

[2147] Step 7:

[2148] The server generates the next dialogue-style question based on the English words. The next question is generated using a generative AI model based on the user's response. The question content and expression are adjusted based on emotional data. The generated question is sent to the device as an HTTP response, and the device pronounces it aloud and displays it on the screen.

[2149] Step 8:

[2150] The user enters answers to questions posed by the server. The device accepts the answers and sends them to the server as an HTTP POST request. The server analyzes the user's answers and generates the next dialogue based on them. This process is repeated, and learning progresses.

[2151] Examples:

[2152] For example, when a child enters the letter "A" or scans a QR code, the device displays the English word "Apple" and an illustration of an apple, and pronounces "Apple." Next, if the user's emotion is recognized as "joy," the server provides dialogue content and learning tasks tailored to the emotion. For example, the next question generated might be, "Blue is a nice color! Do you like blue flowers or blue cars?"

[2153] Example prompt sentence:

[2154] "Generate the following English learning content for children: The user's answer is 'Blue'. What should the next question be?"

[2155] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2156] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2157] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2158] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2159] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2160] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2161] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2162] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2163] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2164] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2165] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2166] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2167] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2168] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2169] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2170] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2171] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2172] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources...

Claims

1. means for accepting alphabetic input; means for acquiring English words based on the input alphabet; a means for outputting the acquired English words by voice; means for displaying an illustration corresponding to the English word; A learning system including.

2. means for generating interactive questions based on the English words; means for presenting the question to a user and accepting an answer from the user; means for generating a next dialogue based on the answer; The learning system of claim 1 further comprising:

3. 2. The learning system according to claim 1, further comprising means for acquiring said English words and said illustrations from a database on the Internet.

Images