System

Abstract

A system is provided.SOLUTION: The system includes a means for registering and storing user information, a means for authenticating the registered information and managing the learning progress of the user, a means for selecting and providing proper learning contents on the basis of the progress of the user, a means for analyzing voice and text in real time and generating feedback, and a means for displaying or transmitting the generated feedback to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, English language learning has become mandatory for elementary school students from the early grades, and many elementary school students are expected to acquire basic English communication skills. However, conventional teaching materials and learning methods make it difficult for children to learn English effectively while having fun, and they lack practice in pronunciation and conversation in particular. Furthermore, it is difficult to provide individualized support tailored to each child's progress and proficiency level, resulting in a decrease in learning effectiveness. The present invention aims to solve these problems by providing a system that allows children to learn English in a fun and efficient way through conversations with virtual friends. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for registering and saving user information, a means for authenticating the registered information, and a means for managing the user's learning progress. The system also includes a means for selecting and providing appropriate learning content based on the user's progress, a means for analyzing speech and text in real time to generate feedback, and a means for displaying or communicating the generated feedback to the user. This system provides a unique and interactive learning experience, particularly for enhancing pronunciation and conversation practice. This allows children to learn English effectively while having fun.

[0006] "User information" refers to the personal information and learning attributes of users of the system (such as name, age, grade, and email address).

[0007] "Storage means" refers to a system or process for storing user information and learning progress data in a storage device such as a database.

[0008] "Authentication" refers to the means of verifying that the information entered by a user when accessing a system matches the registered information and confirming that the user is a legitimate user.

[0009] "Study progress" refers to data and indicators that indicate the level of progress and understanding a user has achieved in their English studies.

[0010] "Means of management" refers to a system or method for tracking a user's learning progress, recording it as data, and reflecting it in the next learning.

[0011] "Learning Content" refers to learning materials and learning activities (e.g., English conversation simulations, vocabulary quizzes, etc.) provided to users to help them learn English.

[0012] "Means of selection and provision" refers to algorithms and systems that select the most appropriate learning content based on the user's learning situation and attributes and present it to the user.

[0013] "Speech and text analysis" refers to techniques and methods for processing user-entered speech and text data and evaluating its content and quality.

[0014] "Means for generating feedback" refers to a system that creates and provides appropriate evaluations and advice to users based on the analysis results.

[0015] "Means for displaying or communicating" refers to the interface or technology used to visually or audibly communicate the generated feedback to the user. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The system of the present invention allows users to learn English efficiently and effectively by registering and providing appropriate learning content based on their progress. Below, we will explain in natural language how the program of this system works.

[0038] User Registration and Login

[0039] User Registration:

[0040] The user launches the app for the first time and clicks the new registration button.

[0041] The terminal prompts the user to enter information such as name, email address, password, and grade.

[0042] When the user enters the information and clicks the send button, the terminal sends this information to the server.

[0043] The server stores the received information in a database and notifies the terminal that registration is complete.

[0044] The terminal displays a message to the user indicating that registration is complete.

[0045] Login:

[0046] The user enters their email address and password on the login screen and clicks the login button.

[0047] The terminal transmits the input information to the server.

[0048] The server checks the information against its database and authenticates it.

[0049] If the authentication is successful, the server obtains the user's learning progress data and sends it to the terminal.

[0050] The terminal displays a successful login message and displays the user's learning dashboard.

[0051] Start learning English

[0052] Select your learning content:

[0053] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[0054] The terminal sends this request to the server.

[0055] The server selects appropriate content based on the user's progress data and transmits it to the terminal.

[0056] The terminal displays the content and allows the user to begin learning.

[0057] Learning implementation:

[0058] The user progresses through the displayed scenarios and quizzes.

[0059] The device transmits the user's input (voice or text) to the server in real time.

[0060] The server analyzes the input data and generates appropriate feedback.

[0061] Send feedback to the device.

[0062] The terminal displays this feedback to the user, who then continues learning based on it.

[0063] Dialogue with AI

[0064] Start the conversation:

[0065] The user selects the option to start a conversation with "AI Tomo."

[0066] The terminal notifies the server of this.

[0067] The server selects an appropriate interaction scenario and sends an initial message to the terminal.

[0068] The terminal will display or audibly convey this message to the user.

[0069] Continuing the dialogue:

[0070] The user responds verbally to the initial message.

[0071] The terminal transmits this audio to the server.

[0072] The server performs speech analysis and generates an appropriate response.

[0073] Send the response to the device.

[0074] The terminal communicates the response to the user and continues the dialogue.

[0075] Management of learning data

[0076] Save the training results:

[0077] When the user finishes learning, the terminal transmits the learning results to the server.

[0078] The server stores the results in a database and uses them the next time the learning is performed.

[0079] View your learning history:

[0080] When a user wants to check the learning history, the terminal sends a request to the server.

[0081] The server acquires the history data and transmits it to the terminal.

[0082] The terminal displays this to the user.

[0083] Provision of additional services (programming education)

[0084] New service information:

[0085] The server generates a guide for programming education for users who meet certain criteria and transmits it to the terminal.

[0086] The terminal displays this information to the user as a notification or pop-up.

[0087] Register for a new service:

[0088] If the user wishes to register, the terminal prompts the user to enter the necessary information and transmits it to the server.

[0089] The server stores the registration information and sends a registration completion message to the terminal.

[0090] The terminal notifies the user that the new service is ready.

[0091] In this way, the system of the present invention provides an interactive learning experience tailored to each user's individual progress, and can support English learning efficiently and effectively. By continuously managing users' learning data and providing additional services such as programming education as needed, the system provides comprehensive educational support.

[0092] The processing flow will be explained below.

[0093] User Registration and Login

[0094] User Registration:

[0095] Step 1:

[0096] The user launches the app for the first time and clicks the Sign Up button.

[0097] Step 2:

[0098] The terminal displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[0099] Step 3:

[0100] The user enters the required information and clicks the submit button.

[0101] Step 4:

[0102] The terminal transmits the input information to the server.

[0103] Step 5:

[0104] The server stores the received user information in a database.

[0105] Step 6:

[0106] The server sends a registration completion response to the terminal.

[0107] Step 7:

[0108] The terminal displays a message to the user indicating that registration is complete.

[0109] Login:

[0110] Step 1:

[0111] The user enters their email address and password and clicks the Login button.

[0112] Step 2:

[0113] The terminal transmits the input information to the server.

[0114] Step 3:

[0115] The server authenticates the user by checking the registration information in the database.

[0116] Step 4:

[0117] If the authentication is successful, the server acquires the user's learning progress data and transmits it to the terminal.

[0118] Step 5:

[0119] The terminal will display the user's learning dashboard along with a successful login message.

[0120] Start learning English

[0121] Select your learning content:

[0122] Step 1:

[0123] The user selects learning content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[0124] Step 2:

[0125] The terminal sends a request for the selected learning content to the server.

[0126] Step 3:

[0127] The server selects appropriate study content based on the user's progress data and transmits it to the terminal.

[0128] Step 4:

[0129] The terminal displays the selected study content and allows the user to begin studying.

[0130] Learning implementation:

[0131] Step 1:

[0132] The user proceeds through the displayed English conversation scenarios and vocabulary quizzes.

[0133] Step 2:

[0134] The device transmits the user's input (voice or text) to the server in real time.

[0135] Step 3:

[0136] The server analyzes the received data using voice recognition and natural language processing.

[0137] Step 4:

[0138] The server generates appropriate feedback from the analysis results and sends it to the terminal.

[0139] Step 5:

[0140] The terminal displays or conveys the feedback from the server to the user.

[0141] Dialogue with AI

[0142] Start the conversation:

[0143] Step 1:

[0144] The user selects the option to start a conversation with "AI Tomo."

[0145] Step 2:

[0146] The terminal sends a request to start a conversation to the server.

[0147] Step 3:

[0148] The server selects an appropriate dialogue scenario and sends an initial message (e.g., Hello! How are you today?) to the terminal.

[0149] Step 4:

[0150] The terminal displays or speaks an initial message to the user.

[0151] Continuing the dialogue:

[0152] Step 1:

[0153] The user responds verbally to the initial message.

[0154] Step 2:

[0155] The terminal transmits the user's voice to the server.

[0156] Step 3:

[0157] The server uses voice recognition technology to analyze and evaluate the user's responses.

[0158] Step 4:

[0159] The server generates and sends appropriate feedback and next interaction messages to the terminal.

[0160] Step 5:

[0161] The terminal displays or speaks feedback and the next interaction message to the user.

[0162] Management of learning data

[0163] Save the training results:

[0164] Step 1:

[0165] The user finishes learning.

[0166] Step 2:

[0167] The terminal transmits the learning result to the server.

[0168] Step 3:

[0169] The server stores the received learning results in a database.

[0170] View your learning history:

[0171] Step 1:

[0172] The user sends a request to check the learning history.

[0173] Step 2:

[0174] The terminal sends a request for the learning history to the server.

[0175] Step 3:

[0176] The server acquires the user's learning history from the database and transmits it to the terminal.

[0177] Step 4:

[0178] The terminal displays the learning history to the user.

[0179] Provision of additional services (programming education)

[0180] New service information:

[0181] Step 1:

[0182] The server generates information data for programming education services for users whose English conversation studies meet a certain standard.

[0183] Step 2:

[0184] The terminal receives the guidance data from the server and displays it to the user as a notification or a pop-up.

[0185] Register for a new service:

[0186] Step 1:

[0187] A user becomes interested in a programming education service and wishes to register.

[0188] Step 2:

[0189] The terminal prompts the user to enter the necessary information and transmits it to the server.

[0190] Step 3:

[0191] The server stores the entered registration information in a database and sends a notice to the terminal informing the user that the new service can be used.

[0192] Step 4:

[0193] The terminal displays a guide to the user on how to start using the new service, and allows the user to start learning the programming education service.

[0194] This allows users to efficiently learn English and also receive programming education.

[0195] Example 1

[0196] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0197] In modern society, there is a high demand for English language learning, but efficient and effective learning requires personalized learning support and advanced feedback functions. Furthermore, to encourage continued learning, users need interactive learning experiences and additional educational services. Conventional systems often lack the functionality to provide appropriate learning content based on individual users' progress, analyze and provide feedback in real time, and effectively introduce new educational services. This results in problems that limit the efficiency and effectiveness of users' learning.

[0198] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0199] In this invention, the server includes means for registering and saving user information, means for authenticating the registered information and managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for displaying or communicating the generated feedback to the user, means for the user to start and continue a dialogue with the "AI," means for saving the user's learning results and using them the next time they study, and means for generating information about new services and notifying the user. This enables the provision of accurate content and feedback based on the user's individual learning progress, and also enables effective information about and registration for new educational services.

[0200] "User Information" refers to personal identification information provided by a user at the time of registration, including name, email address, password, grade level, etc.

[0201] The "storage means" is a means for recording user information and learning progress data and storing them in a database or the like.

[0202] "Authentication means" refers to a process or system for verifying whether a user is legitimate based on input user information.

[0203] "Study progress" refers to the history, results, and progress of a user's learning activities.

[0204] "Learning content" refers to educational materials such as learning materials, scenarios, quizzes, and simulations that are provided for users to study.

[0205] An "analysis means" is a process or system that analyzes input voice or text data and generates appropriate feedback.

[0206] "Feedback" is a response that includes evaluation, advice, and corrections to the user's learning activities.

[0207] "Interaction means" refers to a process or system where a user continuously interacts with an artificial intelligence.

[0208] "Learning results" refers to data on the results, grades, and achievement levels obtained after a user uses learning content.

[0209] "Notification means" refers to means such as alerts, pop-ups, and emails that notify users of new services and information.

[0210] This invention is a system that provides appropriate learning content based on the user's progress, enabling efficient and effective English learning. This system is composed of means for registering and saving user information, means for managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for the user to initiate and continue a dialogue with an "AI," means for saving the learning results and using them the next time the user studies, and means for generating information about new services and notifying the user.

[0211] Configuration and Operation Procedures

[0212] User Registration and Login

[0213] User Registration:

[0214] (User) launches the app for the first time and clicks the new registration button.

[0215] (Terminal) prompts the user to enter information such as name, email address, password, and grade.

[0216] The server stores this information in a database (e.g., MySQL) and notifies the terminal of a registration completion message.

[0217] Login:

[0218] (User) enters email address and password on the login screen.

[0219] The server compares the information in the database and authenticates the user. If authentication is successful, it acquires the user's learning progress data and sends it to the device.

[0220] Select learning content and start learning English

[0221] Select your learning content:

[0222] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[0223] The server selects appropriate content based on the user's progress data and sends it to the terminal.

[0224] The terminal displays the selected content to the user.

[0225] Learning implementation:

[0226] (User) progresses through scenarios and quizzes.

[0227] The (server) analyzes the user's input (voice or text) in real time (e.g., Google Cloud Speech-to-Text) and generates appropriate feedback.

[0228] The feedback is displayed to the user through the terminal.

[0229] Dialogue with AI

[0230] Initiating and continuing a dialogue:

[0231] (User) selects "Dialogue with AI" and begins the dialogue.

[0232] The server selects an appropriate dialogue scenario, generates an initial message, and sends it to the terminal.

[0233] The user responds to the initial message with a voice message, which the device sends to the server, which analyzes the speech (e.g., with Amazon Transcribe) and generates an appropriate response, which is then transmitted to the user via the device.

[0234] Management of learning data

[0235] Saving and displaying training results:

[0236] When the user finishes learning, the device sends the learning results to the server, which stores them in a database.

[0237] When a user wants to check their learning history, the terminal sends a request to the server, the server acquires the history data, and sends it to the terminal, which then displays it.

[0238] Providing additional services

[0239] New service information:

[0240] The server generates a guide to programming education for users who meet the criteria and sends it to the terminal.

[0241] The device will display this information to the user as a notification or pop-up.

[0242] Examples of concrete examples and prompts

[0243] Sample prompt 1: "Select an option to begin interacting with the AI."

[0244] Example prompt 2: "Please select the learning content you would like to do (e.g., English conversation simulation, vocabulary quiz)."

[0245] In this way, the system of the present invention provides an interactive learning experience that responds to the user's individual progress, and can support English learning efficiently and effectively. It also continuously manages the user's learning data and provides additional services such as programming education as needed, thereby achieving comprehensive educational support.

[0246] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0247] Step 1:

[0248] Display the initial registration screen:

[0249] When a user launches the app for the first time, the device displays a new registration screen. As input, the user launches the app. As output, the new registration screen is displayed.

[0250] Step 2:

[0251] Enter your user information:

[0252] The user enters information such as name, email address, password, and grade into the new registration screen. Specifically, the input is provided by entering information into a text field and clicking the "Submit" button. As an output, the entered information is saved on the terminal.

[0253] Step 3:

[0254] Submit your input:

[0255] When the user clicks the "Send" button, the terminal sends the entered information to the server. The input is the user's click operation. The output is data transmission from the terminal to the server.

[0256] Step 4:

[0257] Data storage and verification:

[0258] The server saves the received information in a database (e.g. MySQL). After saving is complete, it notifies the terminal of a registration completion message. The input is the user information sent from the terminal. The output is that the information is saved in the database and a completion notification is sent to the terminal.

[0259] Step 5:

[0260] Registration complete message:

[0261] The terminal displays a message to the user that registration is complete. The input is a completion notification from the server. The output is a completion message displayed on the terminal screen.

[0262] Step 6:

[0263] Display the login screen:

[0264] The device displays the login screen. Input: The user launches the app again. Output: The login screen is displayed.

[0265] Step 7:

[0266] Enter your user information:

[0267] The user enters an email address and password on the login screen. Specifically, the input is to enter an email address and password in the text field and click the "Login" button. The output is to save the entered information on the device.

[0268] Step 8:

[0269] Sending credentials:

[0270] When the user clicks the "Login" button, the terminal sends the entered information to the server. The input is the user's click operation. The output is data sent from the terminal to the server.

[0271] Step 9:

[0272] User authentication:

[0273] The server authenticates the user by comparing the information in the database with the information it receives. The input is the user information sent from the terminal. The output is the authentication result.

[0274] Step 10:

[0275] Capture and send learning progress data:

[0276] If authentication is successful, the server retrieves the user's learning progress data from the database and sends it to the terminal. The inputs are the authentication result and the database search. The output is the learning progress data sent to the terminal.

[0277] Step 11:

[0278] Successful login display:

[0279] The terminal displays a login success message to the user and displays the learning dashboard. As input, there is learning progress data from the server. As output, the terminal screen displays a login success message and the learning dashboard.

[0280] Step 12:

[0281] Display the learning menu:

[0282] The terminal displays the learning menu to the user. As an input, there is the user's login completion action. As an output, the learning menu is displayed on the terminal screen.

[0283] Step 13:

[0284] Select content:

[0285] The user selects "English Conversation Simulation" or "Vocabulary Quiz" from the learning menu. The input is the user's selection action. The output is that the selection is saved on the device.

[0286] Step 14:

[0287] Submitting a content request:

[0288] When a user selects a content, the terminal sends this request to the server. The input is the user's selection. The output is the transmission of request data from the terminal to the server.

[0289] Step 15:

[0290] Content selection and delivery:

[0291] The server selects appropriate content based on the user's progress data and sends it to the terminal. The input is the user's progress data and the request in the server. The output is the selected content that is generated and sent to the terminal.

[0292] Step 16:

[0293] Show content:

[0294] The terminal displays the selected content to the user. As input, there is the content sent from the server. As output, the content is displayed on the terminal screen.

[0295] Step 17:

[0296] Check what's displayed:

[0297] The user checks the displayed scenarios and quizzes. The input is the content displayed on the device display. The output is the user's continuing learning action.

[0298] Step 18:

[0299] Collecting user input:

[0300] The terminal transmits voice and text data input by the user to the server in real time. The input includes the user's voice and text input. The output is real-time data transmission from the terminal to the server.

[0301] Step 19:

[0302] Data analysis and feedback generation:

[0303] The server analyzes the received data (e.g., Google Cloud Speech-to-Text) and generates appropriate feedback. As input, there is the user data sent from the device. As output, there is the generated feedback generated by the server.

[0304] Step 20:

[0305] Send feedback:

[0306] The server sends the generated feedback to the terminal. The input is the generated feedback. The output is the transmission of feedback data to the terminal.

[0307] Step 21:

[0308] Show feedback:

[0309] The device displays the feedback to the user, who then continues learning based on it. As input, there is feedback data from the server. As output, the feedback is displayed on the device screen.

[0310] Step 22:

[0311] View dialogue options:

[0312] The terminal displays an option to start a "dialogue with AI." As input, there is the user's login status. As output, the dialogue options are displayed on the terminal.

[0313] Step 23:

[0314] Select a dialogue option:

[0315] The user selects this option. As input, we have the user's selection action. As output, the selection is saved on the device.

[0316] Step 24:

[0317] Submitting a request:

[0318] The terminal notifies the server of this. The input is the user's selection data. The output is the transmission of request data from the terminal to the server.

[0319] Step 25:

[0320] Create and send the initial message:

[0321] The server selects an appropriate interaction scenario, generates an initial message, and sends it to the terminal. The input is a user interaction request. The output is the generated initial message sent to the terminal.

[0322] Step 26:

[0323] Displaying messages:

[0324] The terminal displays or speaks this message to the user. The input is an initial message from the server. The output is the initial message displayed on the terminal screen or played aloud.

[0325] Step 27:

[0326] Audio data collection:

[0327] The user responds to the initial message with voice. The terminal transmits this voice to the server. The input is the user's voice response. The output is the transmission of voice data from the terminal to the server.

[0328] Step 28:

[0329] Speech analysis and response generation:

[0330] The server performs speech analysis (e.g., Amazon Transcribe) and generates an appropriate response. The input is the speech data sent from the device. The output is the generated response generated by the server.

[0331] Step 29:

[0332] Sending a response:

[0333] The server sends the generated response to the terminal. The input is the generated response data. The output is the transmission of the response data to the terminal.

[0334] Step 30:

[0335] Show Responses:

[0336] The terminal communicates the response to the user and continues the dialogue. The input is the response data from the server. The output is the response displayed on the terminal screen or played back aloud.

[0337] Step 31:

[0338] Sending a termination request:

[0339] When the user finishes learning, the device sends the learning results to the server. The input is the user's termination operation. The output is the transmission of learning result data from the device to the server.

[0340] Step 32:

[0341] Data storage:

[0342] The server stores the results in a database and uses them the next time the machine learns. The input is the learning result data sent from the device. The output is the learning result stored in the database.

[0343] Step 33:

[0344] Submitting a request:

[0345] When a user wants to check their learning history, the terminal sends a request to the server. The input is the user's history check request. The output is the transmission of request data from the terminal to the server.

[0346] Step 34:

[0347] Retrieving and sending historical data:

[0348] The server acquires the history data and sends it to the terminal. The inputs are request data and database search. The output is the history data sent to the terminal.

[0349] Step 35:

[0350] View History:

[0351] The terminal displays this to the user. As input, there is historical data from the server. As output, the historical data is displayed on the terminal screen.

[0352] Step 36:

[0353] Generate and send service announcements:

[0354] The server generates a programming education guide for users who meet the criteria and sends it to the terminal. The input is the user's learning progress data. The output is the generated service guide sent to the terminal.

[0355] Step 37:

[0356] Directions displayed:

[0357] The terminal displays this information to the user as a notification or a pop-up.,Input:,The service information sent from the server.,Output:,The service information is displayed on the screen of the,terminal.

[0358] (Application example 1)

[0359] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0360] Conventional English learning systems have limitations in their ability to meticulously manage users' learning progress and provide appropriate content. Maximizing learning effectiveness through real-time feedback and the introduction of conversational AI is also a challenge. Furthermore, there is a lack of interactive learning methods that allow users to use smart devices.

[0361] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0362] In this invention, the server includes means for registering and storing user information, means for authenticating the registered information and managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for displaying or communicating the generated feedback to the user, means for providing interactive content on the smart device based on the user's learning progress, means for initiating a dialogue with an AI and utilizing a generative AI model to generate a response based on the user's voice input, and means for displaying the generated AI response to the user. This enables efficient and effective learning according to the user's progress and maximizes the learning effect through an interactive experience.

[0363] "User information" is data for specifying and identifying individual users, and includes names, email addresses, passwords, grades, etc.

[0364] The "storage means" is a device or function for storing data, and stores user information and learning progress data in a database on a server.

[0365] "Authentication" is the process of verifying whether the entered user information matches the registered information and authenticating the user.

[0366] "Study progress" is information indicating how far the user has progressed in their studies, and includes the number of completed lessons, score, and the like.

[0367] The "content selection means" is a function that selects and provides appropriate study content based on the user's study progress.

[0368] "Real-time analysis" is the process of instantly analyzing voice and text data entered by the user and generating appropriate feedback.

[0369] "Feedback" is response data that includes evaluations and advice regarding the user's learning.

[0370] "Display means" refers to a device or function for visually or audibly conveying generated feedback and learning content to the user.

[0371] "Interactive content" refers to learning content that progresses through real-time interaction between the user and the system.

[0372] A "smart device" is a portable information terminal that has Internet connectivity and is capable of running applications.

[0373] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses and feedback in response to user input.

[0374] "Generated AI response" is response data to user input generated by a generative AI model.

[0375] The system for realizing the present invention provides users with an effective English learning experience using smart devices. A specific embodiment of the system will be described below.

[0376] Overall system configuration

[0377] 1. User registration and login function

[0378] The server provides a means to register and store user information. The user enters their name, email address, password, grade, etc. on their smart device and sends it to the server. The server stores this information in a database and notifies the device that registration is complete. The user then enters their information on the authentication screen, and the server verifies the information. If authentication is successful, login is complete.

[0379] 2. Providing learning content

[0380] The server provides a means to select appropriate learning content based on the user's progress. When the user selects an English conversation simulation or vocabulary quiz on their smart device, the request is sent to the server. The server selects appropriate content based on the progress data and sends it to the device. The user then uses this content to advance their learning.

[0381] 3. Real-time analysis and feedback

[0382] The device sends user input (voice or text) to the server in real time, which then analyzes it using a generative AI model. For example, the user selects the "Start dialogue with AI" option, and an initial message is displayed on the smart device. The server then generates an appropriate response to the user's voice input and sends it to the device.

[0383] 4. Interactive learning content

[0384] The system provides interactive content based on the user's progress. For example, users can be prompted to answer quizzes or follow-up questions during the English conversation simulation to maximize learning outcomes. It also leverages generative AI models to provide real-time feedback.

[0385] 5. Management of learning data

[0386] The server stores the user's learning results and provides a means to update the user's progress the next time they study. Users can check their learning history on their smart devices, and the server updates it in real time.

[0387] Hardware and Software

[0388] Hardware: Smartphone (iOS or Android)

[0389] Software: React Native framework, Node.js-based server, MongoDB database

[0390] By using this hardware and software, user information registration, progress management, real-time feedback, and interactive learning content can be efficiently carried out.

[0391] Specific examples and examples of generative AI model prompts

[0392] Specific examples

[0393] For users with a learning progress score of 80, the next lesson will be recommended as "Practice everyday English conversation." When the user types "Hello, how are you?" into the AI, the AI ​​responds with "I'm good, thank you. How about you?"

[0394] Example prompts to input to the generative AI model

[0395] User input: "Hello, how are you?"

[0396] Generative AI model input prompt: "Interact with the user: Provide an appropriate response to 'Hello, how are you?'"

[0397] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0398] Step 1:

[0399] User Registration

[0400] Input: The user enters information such as name, email address, password, and grade on the new registration screen on the device.

[0401] Processing: The terminal sends this input information to the server.

[0402] Output: The server stores the received user information in the database and sends a registration completion message to the terminal.

[0403] Step 2:

[0404] User Login

[0405] Input: The user enters their email address and password on the login screen and clicks the Login button.

[0406] Processing: The terminal sends the entered information to the server, which compares it with information in a database to authenticate the user.

[0407] Output: The server sends the authentication result to the device, and if authentication is successful, obtains the user's learning progress data and displays it on the device.

[0408] Step 3:

[0409] Select learning content

[0410] Input: The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu on the device.

[0411] Processing: The device sends the user's selection to the server, and the server selects appropriate content based on the user's progress data and sends it to the device.

[0412] Output: The device displays the content from the server to the user, who then begins learning.

[0413] Step 4:

[0414] Learning implementation

[0415] Input: The user responds to presented scenarios and quizzes by voice or text.

[0416] Processing: The device sends user input in real time to the server, which analyzes it using a generative AI model.

[0417] Output: The server generates appropriate feedback based on the analysis results and sends it to the device, which then displays the feedback to the user, who can then continue learning based on it.

[0418] Step 5:

[0419] Dialogue with AI

[0420] Input: The user selects the "Start a conversation with AI" option and responds verbally to the initial message.

[0421] Processing: The device sends the voice data to the server, which uses a generative AI model to analyze the voice and generate a response. Example prompt: "Interact with the user: Provide an appropriate response to 'Hello, how are you?'"

[0422] Output: The server sends the AI's generated response to the terminal, which then displays or speaks it to the user to continue the dialogue.

[0423] Step 6:

[0424] Management of learning data

[0425] Input: When the user finishes learning, the device sends the learning results to the server.

[0426] Processing: The server saves the learning results in a database and updates the progress based on them the next time the learning is performed.

[0427] Output: When the user wants to check the learning history, the server sends the history data to the terminal, which displays it to the user.

[0428] Step 7:

[0429] Programming education guide

[0430] Input: For users who meet the criteria, the server generates a guide to programming education and notifies the terminal.

[0431] Processing: If the user wishes to register, they enter the necessary information, which the terminal sends to the server.

[0432] Output: The server saves the registration information and sends a registration complete message to the device, informing the user that the device is ready for the new service.

[0433] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0434] The system of the present invention is an interactive English learning system incorporating an "emotion engine" that recognizes the user's emotions and provides appropriate learning feedback. This system allows users to have a more effective and enjoyable learning experience. Below, we will explain in natural language how the program of this system works.

[0435] User Registration and Login

[0436] User Registration:

[0437] The user launches the app for the first time and clicks the Sign Up button.

[0438] The terminal displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[0439] When the user enters the required information and clicks the send button, the terminal sends this information to the server.

[0440] The server stores the received user information in a database.

[0441] The server sends a registration completion response to the terminal.

[0442] The terminal displays a message to the user indicating that registration is complete.

[0443] Login:

[0444] The user enters their email address and password and clicks the Login button.

[0445] The terminal transmits the input information to the server.

[0446] The server authenticates the user by checking the registration information in the database.

[0447] If the authentication is successful, the server acquires the user's learning progress data and sends it to the terminal.

[0448] The terminal will display the user's learning dashboard along with a successful login message.

[0449] Start learning English

[0450] Select your learning content:

[0451] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[0452] The terminal sends a request for the selected learning content to the server.

[0453] The server selects appropriate learning content based on the user's progress data and emotional data and transmits it to the terminal.

[0454] The terminal displays the selected study content and allows the user to begin studying.

[0455] Learning implementation:

[0456] The user proceeds through the displayed English conversation scenarios and vocabulary quizzes.

[0457] The device transmits user input (voice and text) and emotion data to the server in real time.

[0458] The server analyzes the input data and emotion data and generates appropriate feedback.

[0459] Send feedback to the device.

[0460] The terminal displays or conveys feedback from the server to the user to continue learning.

[0461] Dialogue with AI

[0462] Start the conversation:

[0463] The user selects the option to start a conversation with "AI Tomo."

[0464] The terminal notifies the server of this.

[0465] The server selects an appropriate dialogue scenario and sends an initial message (e.g., Hello! How are you today?) to the terminal.

[0466] The terminal will display or audibly convey this message to the user.

[0467] Continuing the dialogue:

[0468] The user responds verbally to the initial message.

[0469] The terminal transmits the user's voice and emotion data to the server.

[0470] The server analyzes and evaluates the user's responses using speech and emotion recognition technologies.

[0471] The server generates and sends appropriate feedback and next interaction messages to the terminal.

[0472] The terminal conveys feedback and the next interaction message to the user and continues the interaction.

[0473] Management of learning data

[0474] Save the training results:

[0475] The user finishes learning.

[0476] The terminal transmits the learning results and emotion data to the server.

[0477] The server stores the received learning results and emotion data in a database.

[0478] View your learning history:

[0479] The user submits a request to check their learning history.

[0480] The terminal sends a request for the learning history to the server.

[0481] The server retrieves the user's learning history and emotion data from the database and transmits them to the terminal.

[0482] The terminal displays the learning history to the user.

[0483] Provision of additional services (programming education)

[0484] New service information:

[0485] The server generates information data for programming education services for users whose English conversation studies meet a certain standard.

[0486] The terminal receives the guidance data from the server and displays it to the user as a notification or a pop-up.

[0487] Register for a new service:

[0488] A user becomes interested in a programming education service and wishes to register.

[0489] The terminal prompts the user to enter the necessary information and transmits it to the server.

[0490] The server stores the entered registration information in a database and sends a notice to the terminal informing the user that the new service can be used.

[0491] The terminal displays a guide to the user on how to start using the new service, and allows the user to start learning the programming education service.

[0492] Specific examples

[0493] English conversation simulation

[0494] 1. The user selects an English conversation simulation.

[0495] 2. The terminal sends the user's selection information to the server.

[0496] 3. The server selects an appropriate scenario and sends an initial message to the terminal.

[0497] 4. The terminal displays or speaks an initial message to the user.

[0498] 5. The user responds verbally, and the device sends this to the server.

[0499] 6. The server analyzes the voice data and emotion data and generates the following response message:

[0500] 7. Repeat this process until the conversation is complete.

[0501] Emotion-based feedback

[0502] 1. The user takes a vocabulary quiz.

[0503] 2. The device recognizes the user's emotions and transmits them to the server.

[0504] 3. The server analyzes the user's emotional data and determines that the user is "emotionally depressed."

[0505] 4. The server generates positive feedback that reflects the user's emotions (e.g., "Great! Let's try again next time!").

[0506] 5. The device displays the feedback to the user.

[0507] The system of the present invention recognizes the user's emotions and provides learning support in response to those emotions, allowing the user to study more proactively. Individual support based on emotion recognition also contributes to improving learning effectiveness.

[0508] The processing flow will be explained below.

[0509] User Registration and Login

[0510] User Registration:

[0511] Step 1:

[0512] The user launches the app for the first time and clicks the Sign Up button.

[0513] Step 2:

[0514] The terminal displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[0515] Step 3:

[0516] The user enters the required information and clicks the submit button.

[0517] Step 4:

[0518] The terminal transmits the input information to the server.

[0519] Step 5:

[0520] The server stores the received user information in a database.

[0521] Step 6:

[0522] The server sends a registration completion response to the terminal.

[0523] Step 7:

[0524] The terminal displays a message to the user indicating that registration is complete.

[0525] Login:

[0526] Step 1:

[0527] The user enters their email address and password and clicks the Login button.

[0528] Step 2:

[0529] The terminal transmits the input information to the server.

[0530] Step 3:

[0531] The server authenticates the user by checking the registration information in the database.

[0532] Step 4:

[0533] If the authentication is successful, the server acquires the user's learning progress data and transmits it to the terminal.

[0534] Step 5:

[0535] The terminal displays a successful login message and displays the user's learning dashboard.

[0536] Start learning English

[0537] Select your learning content:

[0538] Step 1:

[0539] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[0540] Step 2:

[0541] The terminal sends a request for the selected learning content to the server.

[0542] Step 3:

[0543] The server selects appropriate learning content based on the user's progress data and emotional data and transmits it to the terminal.

[0544] Step 4:

[0545] The terminal displays the selected study content and allows the user to begin studying.

[0546] Learning implementation:

[0547] Step 1:

[0548] The user proceeds through the displayed English conversation scenarios and vocabulary quizzes.

[0549] Step 2:

[0550] The device transmits user input (voice and text) and emotion data to the server in real time.

[0551] Step 3:

[0552] The server analyzes the input data and the emotion data.

[0553] Step 4:

[0554] The server generates appropriate feedback from the analysis results and sends it to the terminal.

[0555] Step 5:

[0556] The terminal displays feedback from the server to the user, allowing them to continue learning.

[0557] Dialogue with AI

[0558] Start the conversation:

[0559] Step 1:

[0560] The user selects the option to start a conversation with "AI Tomo."

[0561] Step 2:

[0562] The terminal sends a request to start a conversation to the server.

[0563] Step 3:

[0564] The server selects an appropriate dialogue scenario and sends an initial message (e.g., Hello! How are you today?) to the terminal.

[0565] Step 4:

[0566] The terminal displays or speaks an initial message to the user.

[0567] Continuing the dialogue:

[0568] Step 1:

[0569] The user responds verbally to the initial message.

[0570] Step 2:

[0571] The terminal transmits the user's voice and emotion data to the server.

[0572] Step 3:

[0573] The server analyzes the user's responses using voice and emotion recognition techniques.

[0574] Step 4:

[0575] The server generates appropriate feedback and the next dialogue message based on the analysis results and sends them to the terminal.

[0576] Step 5:

[0577] The terminal displays or speaks feedback and the next interaction message to the user.

[0578] Management of learning data

[0579] Save the training results:

[0580] Step 1:

[0581] The user finishes learning.

[0582] Step 2:

[0583] The terminal transmits the learning results and emotion data to the server.

[0584] Step 3:

[0585] The server stores the received learning results and emotion data in a database.

[0586] View your learning history:

[0587] Step 1:

[0588] The user submits a request to check their learning history.

[0589] Step 2:

[0590] The terminal sends a request for the learning history to the server.

[0591] Step 3:

[0592] The server retrieves the user's learning history and emotion data from the database and transmits them to the terminal.

[0593] Step 4:

[0594] The terminal displays the learning history to the user.

[0595] Provision of additional services (programming education)

[0596] New service information:

[0597] Step 1:

[0598] The server generates information data for programming education services for users whose English conversation studies meet a certain standard.

[0599] Step 2:

[0600] The terminal receives the guidance data from the server and displays it to the user as a notification or a pop-up.

[0601] Register for a new service:

[0602] Step 1:

[0603] A user becomes interested in a programming education service and wishes to register.

[0604] Step 2:

[0605] The terminal prompts the user to enter the necessary information and transmits it to the server.

[0606] Step 3:

[0607] The server stores the entered registration information in a database and sends a notice to the terminal informing the user that the new service can be used.

[0608] Step 4:

[0609] The terminal displays a guide to the user on how to start using the new service, and allows the user to start learning the programming education service.

[0610] Specific examples

[0611] English conversation simulation

[0612] Step 1:

[0613] The user selects an English conversation simulation.

[0614] Step 2:

[0615] The terminal transmits the user's selection information to the server.

[0616] Step 3:

[0617] The server selects an appropriate dialogue scenario and sends an initial message to the terminal.

[0618] Step 4:

[0619] The terminal displays or speaks an initial message to the user.

[0620] Step 5:

[0621] The user responds verbally, and the terminal sends this to the server.

[0622] Step 6:

[0623] The server analyzes the voice data and emotion data and generates the following response message:

[0624] Step 7:

[0625] The server sends the following response message to the terminal:

[0626] Step 8:

[0627] The terminal displays or speaks the following response message to the user.

[0628] Step 9:

[0629] The above process is repeated until the conversation is completed.

[0630] Emotion-based feedback

[0631] Step 1:

[0632] The user takes a word quiz.

[0633] Step 2:

[0634] The device recognizes the user's emotions and transmits them to the server.

[0635] Step 3:

[0636] The server analyzes the user's emotional data and determines that the user is "emotionally depressed."

[0637] Step 4:

[0638] The server generates positive feedback that reflects the user's emotions (e.g., "Great! Let's try harder next time!").

[0639] Step 5:

[0640] The server transmits the generated feedback to the terminal.

[0641] Step 6:

[0642] The terminal displays the feedback to the user.

[0643] The system of the present invention recognizes the user's emotions and provides learning support accordingly, allowing the user to take a more proactive approach to learning. Individualized support based on emotion recognition contributes to improving learning effectiveness and provides an environment that makes it easier for users to continue learning.

[0644] Example 2

[0645] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0646] Conventional learning systems do not provide feedback that takes into account the user's emotional state, resulting in issues such as reduced learning efficiency and loss of motivation. Furthermore, they lack a means to comprehensively manage the user's progress and emotional data and reflect this in the next lesson, making it difficult to provide an effective learning experience.

[0647] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0648] In this invention, the server includes means for registering and saving user information, means for authenticating the registered information and managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for displaying or communicating the generated feedback to the user, means for acquiring the user's emotional data and generating appropriate feedback based on that data, means for saving the user's emotional data and learning results, and means for reflecting the saved emotional data and learning results in the next learning session. This enables the provision of effective feedback that takes into account the user's individual emotional state and improves the learning experience.

[0649] "User information" refers to personal information such as the user's name, email address, password, and grade.

[0650] "Storage means" refers to a database or storage device for storing user information and learning data.

[0651] "Authentication means" refers to the process of authenticating a user based on registered user information and establishing access rights.

[0652] "Learning progress" refers to the progress and achievements a user makes through a learning activity.

[0653] "Learning Content" refers to learning materials and activities provided to users.

[0654] "Analysis means" refers to technology that analyzes data such as voice and text in real time to understand meaning and emotions.

[0655] "Feedback" refers to evaluation and advice messages provided to users as they learn.

[0656] "Emotion data" refers to the user's emotional state estimated from the user's facial expression, tone of voice, input content, etc.

[0657] A "dialogue scenario" is a predefined conversation flow or pattern used in interaction with a user.

[0658] "Learning results" refer to the results or grades that a user has achieved through learning activities.

[0659] The system of the present invention is an interactive English learning system incorporating an "emotion engine" that recognizes the user's emotions and provides appropriate learning feedback, allowing users to have a more effective and enjoyable learning experience.

[0660] User Registration and Login

[0661] First, the user launches the app for the first time and clicks the new registration button. The device displays a registration screen that prompts the user to enter their name, email address, password, and grade. When the user enters the required information and clicks the submit button, the device sends this information to the server. The server saves the received user information in a database and sends a registration completion response to the device. The device displays a registration completion message to the user.

[0662] Next, the user enters their email address and password and clicks the login button, and the device sends the entered information to the server. The server performs authentication by comparing the information with the registered information in the database, and if authentication is successful, it obtains the user's learning progress data and sends it to the device. The device then displays the user's learning dashboard along with a message that login was successful.

[0663] Start learning English

[0664] Next, the user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu. The device sends a request for the selected learning content to the server, and the server selects appropriate learning content based on the user's progress data and emotion data and sends it to the device. The device displays the selected learning content and allows the user to begin learning.

[0665] The user progresses through the displayed English conversation scenarios and vocabulary quizzes, and the device transmits the user's input (voice and text) and emotional data to the server in real time. The server analyzes the input data and emotional data, generates appropriate feedback, and transmits it to the device. The device then displays or communicates the feedback from the server to the user, allowing them to continue their learning.

[0666] Dialogue with AI

[0667] The user then selects an option to start a dialogue with "AI Tomo." The device notifies the server, which then selects an appropriate dialogue scenario and sends an initial message to the device. The device then displays or speaks this message to the user. When the user responds verbally, the device sends the voice and emotional data to the server. The server then analyzes and evaluates the user's response using voice recognition and emotion recognition technologies. The server then generates appropriate feedback and the next dialogue message and sends them to the device. The device then conveys the feedback and the next dialogue message to the user, continuing the dialogue.

[0668] Management of learning data

[0669] When the user finishes learning, the device sends the learning results and emotional data to the server. The server stores the received learning results and emotional data in a database. When the user sends a request to check their learning history, the device sends a learning history request to the server. The server retrieves the user's learning history and emotional data from the database and sends them to the device. The device displays the learning history to the user.

[0670] Examples and prompts

[0671] For example, if a user selects an English conversation simulation, the device sends the user's selection information to the server, which then selects an appropriate scenario and sends an initial message to the device. The device then displays or speaks this message to the user, who then responds verbally, which the device then sends to the server. The server then analyzes the voice and emotion data and generates the next response message. This process is repeated until the dialogue is completed.

[0672] Examples of prompt sentences include "Please start the English conversation simulation" and "Please provide appropriate feedback if the user's current emotions are judged to be depressed."

[0673] This allows users to study English while having their emotions properly recognized, improving the effectiveness of their learning.

[0674] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0675] Step 1:

[0676] The user launches the app for the first time and clicks the Sign Up button.

[0677] Input: User clicks on the new registration button.

[0678] Specific behavior: The app starts and the user clicks the new registration button.

[0679] Output: The terminal displays the user registration screen.

[0680] Step 2:

[0681] The device displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[0682] Input: New Registration button click event in Step 1.

[0683] What happens: A registration form is displayed and the user can enter their information.

[0684] Output: Waiting for user input.

[0685] Step 3:

[0686] The user enters the required information and clicks the submit button.

[0687] Input: A user enters their name, email address, password, and grade and clicks the submit button.

[0688] What happens: A user fills in a form and clicks the submit button.

[0689] Output: The data including the input user information is processed on the terminal side.

[0690] Step 4:

[0691] The terminal sends this information to the server.

[0692] Input: Information data entered by the user.

[0693] Specific operation: The device sends data to the server via the Internet.

[0694] Output: The user information sent to the server.

[0695] Step 5:

[0696] The server stores the received user information in a database.

[0697] Input: User information sent from the device.

[0698] Specific operation: The server receives the information and stores it in a database.

[0699] Output: User information stored in the database.

[0700] Step 6:

[0701] The server sends a registration completion response to the terminal.

[0702] Input: Confirmation of successful saving to database.

[0703] Specific operation: The server generates a registration completion message and sends it to the terminal.

[0704] Output: A registration completion message is sent to the device.

[0705] Step 7:

[0706] The terminal displays a message to the user that registration is complete.

[0707] Input: Registration completion message sent by the server.

[0708] Specific operation: The device displays a registration completion message on the screen.

[0709] Output: A successful registration message that is displayed to the user.

[0710] Step 8:

[0711] The user enters their email address and password and clicks the Login button.

[0712] Input: User enters email address and password and clicks login button.

[0713] What happens: The user fills in the login form and clicks a button.

[0714] Output: Data containing the entered information is sent to the terminal.

[0715] Step 9:

[0716] The terminal transmits the input information to the server.

[0717] Input: The email address and password entered by the user.

[0718] Specific operation: The device sends the data to the server.

[0719] Output: The authentication information sent to the server.

[0720] Step 10:

[0721] The server authenticates the user by checking the registration information in the database.

[0722] Input: The submitted credentials.

[0723] Specific operation: The server accesses the database and checks the user information.

[0724] Output: Authentication success or failure result.

[0725] Step 11:

[0726] If the authentication is successful, the server acquires the user's learning progress data and sends it to the terminal.

[0727] Input: Authentication success result.

[0728] Specific operation: The server retrieves the user's learning progress data from the database.

[0729] Output: Learning progress data sent to the device.

[0730] Step 12:

[0731] The terminal displays the user's learning dashboard along with a successful login message.

[0732] Input: Learning progress data and authentication success message sent from the server.

[0733] Specific behavior: The device displays a login success message and displays the dashboard to the user.

[0734] Output: The dashboard screen that is displayed to the user.

[0735] Step 13:

[0736] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[0737] Input: User selection of learning content.

[0738] Specific operation: The user selects the desired content from the menu.

[0739] Output: Selected content information.

[0740] Step 14:

[0741] The terminal sends a request for the selected learning content to the server.

[0742] Input: Content information selected by the user.

[0743] Specific operation: The device sends the information to the server.

[0744] Output: The content request sent to the server.

[0745] Step 15:

[0746] The server selects appropriate learning content based on the user's progress data and emotional data and sends it to the terminal.

[0747] Input: User progress data, emotion data, content requests.

[0748] Specific operation: The server selects the most suitable content based on this data.

[0749] Output: The learning content sent to the device.

[0750] Step 16:

[0751] The terminal displays the selected study content and allows the user to begin studying.

[0752] Input: The learning content sent from the server.

[0753] Specific operation: The device displays the learning content screen and prompts the user to begin learning.

[0754] Output: The displayed learning content.

[0755] Step 17:

[0756] The user progresses through the displayed English conversation scenarios and vocabulary quizzes.

[0757] Input: User conducts English conversation scenarios and vocabulary quizzes.

[0758] What happens: The user progresses through the displayed content.

[0759] Output: User's learning progress data.

[0760] Step 18:

[0761] The device transmits user input (voice and text) and emotion data to the server in real time.

[0762] Input: User voice input, text input, and emotion data.

[0763] Specific operation: The device sends this data to the server in real time.

[0764] Output: The input data and emotion data sent to the server.

[0765] Step 19:

[0766] The server analyzes the input data and emotion data and generates appropriate feedback.

[0767] Input: User input data and emotion data.

[0768] What happens: The server analyzes this data and generates feedback.

[0769] Output: The generated feedback data.

[0770] Step 20:

[0771] The server transmits the generated feedback to the terminal.

[0772] Input: Generated feedback data.

[0773] Specific operation: The server sends feedback data to the terminal.

[0774] Output: Feedback sent to the device.

[0775] Step 21:

[0776] The terminal displays or conveys feedback from the server to the user to continue learning.

[0777] Input: Feedback data sent by the server.

[0778] Specific behavior: The device displays or audibly provides feedback to the user.

[0779] Output: Feedback display to the user.

[0780] Step 22:

[0781] The user finishes learning.

[0782] Input: The user performs the operation to end learning.

[0783] Specific operation: The user clicks the End Learning button.

[0784] Output: End of learning status.

[0785] Step 23:

[0786] The terminal transmits the learning result and the emotion data to the server.

[0787] Input: Learning result data and emotion data.

[0788] Specific operation: The device sends this data to the server.

[0789] Output: Learning result data and emotion data sent to the server.

[0790] Step 24:

[0791] The server stores the received learning results and emotion data in a database.

[0792] Input: Learning result data and emotion data.

[0793] Specific operation: The server stores this data in a database.

[0794] Output: Learning results and emotion data stored in a database.

[0795] Step 25:

[0796] The user submits a request to check their learning history.

[0797] Input: A user request to check their learning history.

[0798] Specific operation: The user clicks the learning history button.

[0799] Output: The request data is sent to the terminal.

[0800] Step 26:

[0801] The terminal sends a request for the learning history to the server.

[0802] Input: The request data sent by the user to the terminal.

[0803] Specific operation: The terminal sends the request data to the server.

[0804] Output: The learning history request sent to the server.

[0805] Step 27:

[0806] The server retrieves the user's learning history and emotional data from the database and sends them to the terminal.

[0807] Input: Learning history request data.

[0808] Specific operation: The server retrieves learning history and emotion data from the database.

[0809] Output: Learning history and emotion data sent to the device.

[0810] Step 28:

[0811] The terminal displays the learning history to the user.

[0812] Input: Learning history and emotion data sent from the server.

[0813] Specific operation: The device displays this data on the screen.

[0814] Output: The learning history screen displayed to the user.

[0815] Specific examples and examples of prompts for the generative AI model

[0816] Specific examples

[0817] When the user selects an English conversation simulation, the device sends the user's selection information to the server, which then selects an appropriate scenario and sends an initial message to the device. The device then displays or speaks this message to the user, who then responds verbally, which the device then sends to the server. The server then analyzes the voice and emotion data and generates the next response message. This process is repeated until the dialogue is complete.

[0818] Prompt example

[0819] "Please start the English conversation simulation."

[0820] "Provide appropriate feedback if the user's current mood is deemed depressed."

[0821] This allows users to study English while having their emotions properly recognized, improving the effectiveness of their learning.

[0822] (Application example 2)

[0823] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0824] Modern streaming services face the challenge of making it difficult for users to find the optimal content that matches their emotions and state. In particular, when a user's mood or emotions fluctuate, conventional systems have difficulty in recommending content that responds to these changes, resulting in a less satisfying user experience.

[0825] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing a user's emotion and generating feedback based on the emotion, means for selecting and providing content appropriate to the user's state using a recommendation algorithm, and means for displaying or communicating the generated feedback and recommended content to the user. This allows the user to easily find content that best suits their emotional state.

[0826] "User information" refers to information about system users, such as their names, email addresses, passwords, and past usage history.

[0827] "Storage means" refers to a device or software for recording and storing user information, learning progress data, and emotion data.

[0828] "Authentication means" is a mechanism for verifying the authenticity of a user by comparing the information entered by the user with registered information in a database.

[0829] A "progress management means" is a device or system for tracking and recording a user's learning or viewing progress.

[0830] "Learning content" refers to digital content such as learning materials, quizzes, and simulations that users can use for learning.

[0831] The "means of providing" refers to a mechanism for displaying or conveying appropriate learning content or recommended content to users.

[0832] "Means for analyzing voice and text in real time and generating feedback" refers to a system that instantly analyzes a user's voice input or text input and generates appropriate responses or advice based on the results.

[0833] "Means for recognizing emotions" refers to technology for identifying a user's emotions by analyzing the user's facial expressions, voice, behavior, etc.

[0834] The "means for generating emotion-based feedback" is a device or system that automatically creates and provides feedback according to the recognized emotion of the user.

[0835] A "recommendation algorithm" is a computational method for selecting optimal content based on a user's emotional state and usage history.

[0836] "Recommended content" refers to content such as movies, dramas, music, documentaries, etc. that is provided in accordance with the user's emotional state and interests.

[0837] An "interface for facilitating viewing" is a user interface designed to allow a user to comfortably study or view content.

[0838] "Emotional state" is data that indicates the user's mental and emotional state, and is inferred from facial expressions, tone of voice, content of statements, etc.

[0839] The present invention provides a system for recommending appropriate content based on a user's emotional state, thereby improving the user's viewing experience. A detailed configuration of a system for realizing this application example will be described below.

[0840] System Overview

[0841] The system mainly consists of the following components:

[0842] 1. User information registration and authentication

[0843] 2. Emotion Recognition and Analysis

[0844] 3. Content Recommendation Algorithm

[0845] 4. Generating and displaying feedback

[0846] 5. Data storage and management

[0847] Hardware and software used

[0848] The main hardware and software used in this system are as follows:

[0849] Hardware: Smartphone camera, microphone

[0850] software:

[0851] Emotion Recognition: Google Cloud Vision API or Amazon Rekognition

[0852] Database: Firebase Realtime Database or AWS DynamoDB

[0853] Backend: Node.js or Python (Serverless environment: AWS Lambda or Google Cloud Functions)

[0854] Frontend: React Native (smartphone application)

[0855] User registration and authentication

[0856] When a user launches the app for the first time, an account is created and user authentication is performed using Firebase Auth or similar. Specifically, the user enters their name, email address, and password, which are then sent to the server. The server stores this information in a database and sends a registration completion response to the device. From the next time onwards, the user can log in using their email address and password.

[0857] Emotion Recognition and Analysis

[0858] The user's emotional state is recognized in real time through the smartphone's camera and microphone. For example, the Google Cloud Vision API is used to analyze emotions from the captured user's facial expressions. This emotional data is sent to the server and used in the next step.

[0859] Example prompt sentence:

[0860] def detect_emotions(image):

[0861] client = vision.ImageAnnotatorClient()

[0862] response = client.face_detection(image=image)

[0863] faces = response.face_annotations

[0864] emotions = {

[0865] 'joy': faces[0].joy_likelihood,

[0866] 'sorrow': faces[0].sorrow_likelihood,

[0867] 'anger': faces[0].anger_likelihood,

[0868] 'surprise': faces[0].surprise_likelihood,

[0869] }

[0870] return emotions

[0871] Content recommendation algorithm

[0872] The server then uses the collected emotional data to select the most suitable content using a recommendation algorithm, which takes into account the user's past viewing history and current emotional state to select the most suitable content for the user in real time.

[0873] Generating and displaying feedback

[0874] Along with the selected content, feedback based on the user's emotional data is also generated and provided to the user. For example, if the system detects that the user is feeling stressed, it will recommend relaxing content and send positive messages such as "Relax and have fun."

[0875] Data storage and management

[0876] After viewing, the app recognizes the user's emotional state again and stores that data in a database. This allows past data to be used in the next content recommendation, providing a more personalized experience. Users can also check their viewing history and emotional log at any time within the app.

[0877] For example, if a user inputs "I want to relax" into an emotion recognition system, the system will recommend relaxing music or nature documentaries. If the user is detected as being in a "happy mood," the system will recommend comedy movies or fun short videos with positive messages.

[0878] In this way, the system can provide the most appropriate content depending on the user's emotional state, improving the viewer's experience.

[0879] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0880] Step 1:

[0881] The user launches the app for the first time and begins the process of registering.

[0882] Input: Name, Email Address, Password

[0883] Action: The user enters the required information and clicks the submit button.

[0884] Output: The device sends this information to the server.

[0885] Step 2:

[0886] The server stores the received user information in a database.

[0887] Input: Registration information entered by the user

[0888] Operation: The server saves the registration information in a database and sends a registration completion response to the terminal.

[0889] Output: The terminal displays a message to the user that registration is complete.

[0890] Step 3:

[0891] The user enters their email address and password and clicks the login button.

[0892] Input: Email address, password

[0893] Operation: The terminal sends the entered information to the server, which then verifies it by comparing it with the registered information in the database.

[0894] Output: If authentication is successful, the server will retrieve the user's learning progress data and send it to the terminal, and the terminal will display the user's dashboard along with a login success message.

[0895] Step 4:

[0896] The user's face is captured by a camera and emotion recognition is performed.

[0897] Input: User's face image

[0898] How it works: The device uses the camera to capture the user's face and performs emotion analysis using the Google Cloud Vision API.

[0899] Output: Sends emotion data (e.g., joy, sadness, anger, surprise) to the server.

[0900] Step 5:

[0901] The server uses a recommendation algorithm to select the most suitable content based on the emotional data.

[0902] Input: Emotion data, user's past viewing history

[0903] How it works: The server runs a recommendation algorithm to select the most relevant content for the user.

[0904] Output: Sends the recommended content list to the device.

[0905] Step 6:

[0906] The terminal displays the recommended content list to the user.

[0907] Input: Recommended Content List

[0908] How it works: The device displays the recommended content to the user in a list format.

[0909] Output: The user selects the content they want to watch.

[0910] Step 7:

[0911] The user views the selected content.

[0912] Input: Selected content

[0913] Actions: The user uses the device to play and watch the selected content.

[0914] Output: The device collects content viewing progress data and emotion data.

[0915] Step 8:

[0916] The user's emotional changes are captured again by camera and emotion analysis is performed.

[0917] Input: User's face image (recaptured)

[0918] How it works: The device recaptures the user's face after viewing and performs emotion analysis using the Google Cloud Vision API.

[0919] Output: Emotional data after viewing is sent to the server.

[0920] Step 9:

[0921] The server stores emotion data before and after viewing and viewing progress data in a database.

[0922] Input: Emotional data before and after viewing, viewing progress data

[0923] How it works: The server stores this data in a database and uses it in the next recommendation algorithm.

[0924] Output: Updated user data in the database

[0925] Step 10:

[0926] Users can check their viewing history and emotion log from the menu displayed within the app.

[0927] Input: User request

[0928] How it works: The device sends a request to the server, which retrieves viewing history and emotion logs from the database.

[0929] Output: The terminal displays the acquired data to the user.

[0930] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0931] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0932] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0933] [Second embodiment]

[0934] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0935] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0936] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0937] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0938] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0939] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0940] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0941] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0942] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0943] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0944] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0945] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0946] The system of the present invention allows users to learn English efficiently and effectively by registering and providing appropriate learning content based on their progress. Below, we will explain in natural language how the program of this system works.

[0947] User Registration and Login

[0948] User Registration:

[0949] The user launches the app for the first time and clicks the new registration button.

[0950] The terminal prompts the user to enter information such as name, email address, password, and grade.

[0951] When the user enters the information and clicks the send button, the terminal sends this information to the server.

[0952] The server stores the received information in a database and notifies the terminal that registration is complete.

[0953] The terminal displays a message to the user indicating that registration is complete.

[0954] Login:

[0955] The user enters their email address and password on the login screen and clicks the login button.

[0956] The terminal transmits the input information to the server.

[0957] The server checks the information against its database and authenticates it.

[0958] If the authentication is successful, the server obtains the user's learning progress data and sends it to the terminal.

[0959] The terminal displays a successful login message and displays the user's learning dashboard.

[0960] Start learning English

[0961] Select your learning content:

[0962] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[0963] The terminal sends this request to the server.

[0964] The server selects appropriate content based on the user's progress data and transmits it to the terminal.

[0965] The terminal displays the content and allows the user to begin learning.

[0966] Learning implementation:

[0967] The user progresses through the displayed scenarios and quizzes.

[0968] The device transmits the user's input (voice or text) to the server in real time.

[0969] The server analyzes the input data and generates appropriate feedback.

[0970] Send feedback to the device.

[0971] The terminal displays this feedback to the user, who then continues learning based on it.

[0972] Dialogue with AI

[0973] Start the conversation:

[0974] The user selects the option to start a conversation with "AI Tomo."

[0975] The terminal notifies the server of this.

[0976] The server selects an appropriate interaction scenario and sends an initial message to the terminal.

[0977] The terminal will display or audibly convey this message to the user.

[0978] Continuing the dialogue:

[0979] The user responds verbally to the initial message.

[0980] The terminal transmits this audio to the server.

[0981] The server performs speech analysis and generates an appropriate response.

[0982] Send the response to the device.

[0983] The terminal communicates the response to the user and continues the dialogue.

[0984] Management of learning data

[0985] Save the training results:

[0986] When the user finishes learning, the terminal transmits the learning results to the server.

[0987] The server stores the results in a database and uses them the next time the learning is performed.

[0988] View your learning history:

[0989] When a user wants to check the learning history, the terminal sends a request to the server.

[0990] The server acquires the history data and transmits it to the terminal.

[0991] The terminal displays this to the user.

[0992] Provision of additional services (programming education)

[0993] New service information:

[0994] The server generates a guide for programming education for users who meet certain criteria and transmits it to the terminal.

[0995] The terminal displays this information to the user as a notification or pop-up.

[0996] Register for a new service:

[0997] If the user wishes to register, the terminal prompts the user to enter the necessary information and transmits it to the server.

[0998] The server stores the registration information and sends a registration completion message to the terminal.

[0999] The terminal notifies the user that the new service is ready.

[1000] In this way, the system of the present invention provides an interactive learning experience tailored to each user's individual progress, and can support English learning efficiently and effectively. By continuously managing users' learning data and providing additional services such as programming education as needed, the system provides comprehensive educational support.

[1001] The processing flow will be explained below.

[1002] User Registration and Login

[1003] User Registration:

[1004] Step 1:

[1005] The user launches the app for the first time and clicks the Sign Up button.

[1006] Step 2:

[1007] The terminal displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[1008] Step 3:

[1009] The user enters the required information and clicks the submit button.

[1010] Step 4:

[1011] The terminal transmits the input information to the server.

[1012] Step 5:

[1013] The server stores the received user information in a database.

[1014] Step 6:

[1015] The server sends a registration completion response to the terminal.

[1016] Step 7:

[1017] The terminal displays a message to the user indicating that registration is complete.

[1018] Login:

[1019] Step 1:

[1020] The user enters their email address and password and clicks the Login button.

[1021] Step 2:

[1022] The terminal transmits the input information to the server.

[1023] Step 3:

[1024] The server authenticates the user by checking the registration information in the database.

[1025] Step 4:

[1026] If the authentication is successful, the server acquires the user's learning progress data and transmits it to the terminal.

[1027] Step 5:

[1028] The terminal will display the user's learning dashboard along with a successful login message.

[1029] Start learning English

[1030] Select your learning content:

[1031] Step 1:

[1032] The user selects learning content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[1033] Step 2:

[1034] The terminal sends a request for the selected learning content to the server.

[1035] Step 3:

[1036] The server selects appropriate study content based on the user's progress data and transmits it to the terminal.

[1037] Step 4:

[1038] The terminal displays the selected study content and allows the user to begin studying.

[1039] Learning implementation:

[1040] Step 1:

[1041] The user proceeds through the displayed English conversation scenarios and vocabulary quizzes.

[1042] Step 2:

[1043] The device transmits the user's input (voice or text) to the server in real time.

[1044] Step 3:

[1045] The server analyzes the received data using voice recognition and natural language processing.

[1046] Step 4:

[1047] The server generates appropriate feedback from the analysis results and sends it to the terminal.

[1048] Step 5:

[1049] The terminal displays or conveys the feedback from the server to the user.

[1050] Dialogue with AI

[1051] Start the conversation:

[1052] Step 1:

[1053] The user selects the option to start a conversation with "AI Tomo."

[1054] Step 2:

[1055] The terminal sends a request to start a conversation to the server.

[1056] Step 3:

[1057] The server selects an appropriate dialogue scenario and sends an initial message (e.g., Hello! How are you today?) to the terminal.

[1058] Step 4:

[1059] The terminal displays or speaks an initial message to the user.

[1060] Continuing the dialogue:

[1061] Step 1:

[1062] The user responds verbally to the initial message.

[1063] Step 2:

[1064] The terminal transmits the user's voice to the server.

[1065] Step 3:

[1066] The server uses voice recognition technology to analyze and evaluate the user's responses.

[1067] Step 4:

[1068] The server generates and sends appropriate feedback and next interaction messages to the terminal.

[1069] Step 5:

[1070] The terminal displays or speaks feedback and the next interaction message to the user.

[1071] Management of learning data

[1072] Save the training results:

[1073] Step 1:

[1074] The user finishes learning.

[1075] Step 2:

[1076] The terminal transmits the learning result to the server.

[1077] Step 3:

[1078] The server stores the received learning results in a database.

[1079] View your learning history:

[1080] Step 1:

[1081] The user sends a request to check the learning history.

[1082] Step 2:

[1083] The terminal sends a request for the learning history to the server.

[1084] Step 3:

[1085] The server acquires the user's learning history from the database and transmits it to the terminal.

[1086] Step 4:

[1087] The terminal displays the learning history to the user.

[1088] Provision of additional services (programming education)

[1089] New service information:

[1090] Step 1:

[1091] The server generates information data for programming education services for users whose English conversation studies meet a certain standard.

[1092] Step 2:

[1093] The terminal receives the guidance data from the server and displays it to the user as a notification or a pop-up.

[1094] Register for a new service:

[1095] Step 1:

[1096] A user becomes interested in a programming education service and wishes to register.

[1097] Step 2:

[1098] The terminal prompts the user to enter the necessary information and transmits it to the server.

[1099] Step 3:

[1100] The server stores the entered registration information in a database and sends a notice to the terminal informing the user that the new service can be used.

[1101] Step 4:

[1102] The terminal displays a guide to the user on how to start using the new service, and allows the user to start learning the programming education service.

[1103] This allows users to efficiently learn English and also receive programming education.

[1104] Example 1

[1105] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1106] In modern society, there is a high demand for English language learning, but efficient and effective learning requires personalized learning support and advanced feedback functions. Furthermore, to encourage continued learning, users need interactive learning experiences and additional educational services. Conventional systems often lack the functionality to provide appropriate learning content based on individual users' progress, analyze and provide feedback in real time, and effectively introduce new educational services. This results in problems that limit the efficiency and effectiveness of users' learning.

[1107] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1108] In this invention, the server includes means for registering and saving user information, means for authenticating the registered information and managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for displaying or communicating the generated feedback to the user, means for the user to start and continue a dialogue with the "AI," means for saving the user's learning results and using them the next time they study, and means for generating information about new services and notifying the user. This enables the provision of accurate content and feedback based on the user's individual learning progress, and also enables effective information about and registration for new educational services.

[1109] "User Information" refers to personal identification information provided by a user at the time of registration, including name, email address, password, grade level, etc.

[1110] The "storage means" is a means for recording user information and learning progress data and storing them in a database or the like.

[1111] "Authentication means" refers to a process or system for verifying whether a user is legitimate based on input user information.

[1112] "Study progress" refers to the history, results, and progress of a user's learning activities.

[1113] "Learning content" refers to educational materials such as learning materials, scenarios, quizzes, and simulations that are provided for users to study.

[1114] An "analysis means" is a process or system that analyzes input voice or text data and generates appropriate feedback.

[1115] "Feedback" is a response that includes evaluation, advice, and corrections to the user's learning activities.

[1116] "Interaction means" refers to a process or system where a user continuously interacts with an artificial intelligence.

[1117] "Learning results" refers to data on the results, grades, and achievement levels obtained after a user uses learning content.

[1118] "Notification means" refers to means such as alerts, pop-ups, and emails that notify users of new services and information.

[1119] This invention is a system that provides appropriate learning content based on the user's progress, enabling efficient and effective English learning. This system is composed of means for registering and saving user information, means for managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for the user to initiate and continue a dialogue with an "AI," means for saving the learning results and using them the next time the user studies, and means for generating information about new services and notifying the user.

[1120] Configuration and Operation Procedures

[1121] User Registration and Login

[1122] User Registration:

[1123] (User) launches the app for the first time and clicks the new registration button.

[1124] (Terminal) prompts the user to enter information such as name, email address, password, and grade.

[1125] The server stores this information in a database (e.g., MySQL) and notifies the terminal of a registration completion message.

[1126] Login:

[1127] (User) enters email address and password on the login screen.

[1128] The server compares the information in the database and authenticates the user. If authentication is successful, it acquires the user's learning progress data and sends it to the device.

[1129] Select learning content and start learning English

[1130] Select your learning content:

[1131] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[1132] The server selects appropriate content based on the user's progress data and sends it to the terminal.

[1133] The terminal displays the selected content to the user.

[1134] Learning implementation:

[1135] (User) progresses through scenarios and quizzes.

[1136] The (server) analyzes the user's input (voice or text) in real time (e.g., Google Cloud Speech-to-Text) and generates appropriate feedback.

[1137] The feedback is displayed to the user through the terminal.

[1138] Dialogue with AI

[1139] Initiating and continuing a dialogue:

[1140] (User) selects "Dialogue with AI" and begins the dialogue.

[1141] The server selects an appropriate dialogue scenario, generates an initial message, and sends it to the terminal.

[1142] The user responds to the initial message with a voice message, which the device sends to the server, which analyzes the speech (e.g., with Amazon Transcribe) and generates an appropriate response, which is then transmitted to the user via the device.

[1143] Management of learning data

[1144] Saving and displaying training results:

[1145] When the user finishes learning, the device sends the learning results to the server, which stores them in a database.

[1146] When a user wants to check their learning history, the terminal sends a request to the server, the server acquires the history data, and sends it to the terminal, which then displays it.

[1147] Providing additional services

[1148] New service information:

[1149] The server generates a guide to programming education for users who meet the criteria and sends it to the terminal.

[1150] The device will display this information to the user as a notification or pop-up.

[1151] Examples of concrete examples and prompts

[1152] Sample prompt 1: "Select an option to begin interacting with the AI."

[1153] Example prompt 2: "Please select the learning content you would like to do (e.g., English conversation simulation, vocabulary quiz)."

[1154] In this way, the system of the present invention provides an interactive learning experience that responds to the user's individual progress, and can support English learning efficiently and effectively. It also continuously manages the user's learning data and provides additional services such as programming education as needed, thereby achieving comprehensive educational support.

[1155] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1156] Step 1:

[1157] Display the initial registration screen:

[1158] When a user launches the app for the first time, the device displays a new registration screen. As input, the user launches the app. As output, the new registration screen is displayed.

[1159] Step 2:

[1160] Enter your user information:

[1161] The user enters information such as name, email address, password, and grade into the new registration screen. Specifically, the input is provided by entering information into a text field and clicking the "Submit" button. As an output, the entered information is saved on the terminal.

[1162] Step 3:

[1163] Submit your input:

[1164] When the user clicks the "Send" button, the terminal sends the entered information to the server. The input is the user's click operation. The output is data transmission from the terminal to the server.

[1165] Step 4:

[1166] Data storage and verification:

[1167] The server saves the received information in a database (e.g. MySQL). After saving is complete, it notifies the terminal of a registration completion message. The input is the user information sent from the terminal. The output is that the information is saved in the database and a completion notification is sent to the terminal.

[1168] Step 5:

[1169] Registration complete message:

[1170] The terminal displays a message to the user that registration is complete. The input is a completion notification from the server. The output is a completion message displayed on the terminal screen.

[1171] Step 6:

[1172] Display the login screen:

[1173] The device displays the login screen. Input: The user launches the app again. Output: The login screen is displayed.

[1174] Step 7:

[1175] Enter your user information:

[1176] The user enters an email address and password on the login screen. Specifically, the input is to enter an email address and password in the text field and click the "Login" button. The output is to save the entered information on the device.

[1177] Step 8:

[1178] Sending credentials:

[1179] When the user clicks the "Login" button, the terminal sends the entered information to the server. The input is the user's click operation. The output is data sent from the terminal to the server.

[1180] Step 9:

[1181] User authentication:

[1182] The server authenticates the user by comparing the information in the database with the information it receives. The input is the user information sent from the terminal. The output is the authentication result.

[1183] Step 10:

[1184] Capture and send learning progress data:

[1185] If authentication is successful, the server retrieves the user's learning progress data from the database and sends it to the terminal. The inputs are the authentication result and the database search. The output is the learning progress data sent to the terminal.

[1186] Step 11:

[1187] Successful login display:

[1188] The terminal displays a login success message to the user and displays the learning dashboard. As input, there is learning progress data from the server. As output, the terminal screen displays a login success message and the learning dashboard.

[1189] Step 12:

[1190] Display the learning menu:

[1191] The terminal displays the learning menu to the user. As an input, there is the user's login completion action. As an output, the learning menu is displayed on the terminal screen.

[1192] Step 13:

[1193] Select content:

[1194] The user selects "English Conversation Simulation" or "Vocabulary Quiz" from the learning menu. The input is the user's selection action. The output is that the selection is saved on the device.

[1195] Step 14:

[1196] Submitting a content request:

[1197] When a user selects a content, the terminal sends this request to the server. The input is the user's selection. The output is the transmission of request data from the terminal to the server.

[1198] Step 15:

[1199] Content selection and delivery:

[1200] The server selects appropriate content based on the user's progress data and sends it to the terminal. The input is the user's progress data and the request in the server. The output is the selected content that is generated and sent to the terminal.

[1201] Step 16:

[1202] Show content:

[1203] The terminal displays the selected content to the user. As input, there is the content sent from the server. As output, the content is displayed on the terminal screen.

[1204] Step 17:

[1205] Check what's displayed:

[1206] The user checks the displayed scenarios and quizzes. The input is the content displayed on the device display. The output is the user's continuing learning action.

[1207] Step 18:

[1208] Collecting user input:

[1209] The terminal transmits voice and text data input by the user to the server in real time. The input includes the user's voice and text input. The output is real-time data transmission from the terminal to the server.

[1210] Step 19:

[1211] Data analysis and feedback generation:

[1212] The server analyzes the received data (e.g., Google Cloud Speech-to-Text) and generates appropriate feedback. As input, there is the user data sent from the device. As output, there is the generated feedback generated by the server.

[1213] Step 20:

[1214] Send feedback:

[1215] The server sends the generated feedback to the terminal. The input is the generated feedback. The output is the transmission of feedback data to the terminal.

[1216] Step 21:

[1217] Show feedback:

[1218] The device displays the feedback to the user, who then continues learning based on it. As input, there is feedback data from the server. As output, the feedback is displayed on the device screen.

[1219] Step 22:

[1220] View dialogue options:

[1221] The terminal displays an option to start a "dialogue with AI." As input, there is the user's login status. As output, the dialogue options are displayed on the terminal.

[1222] Step 23:

[1223] Select a dialogue option:

[1224] The user selects this option. As input, we have the user's selection action. As output, the selection is saved on the device.

[1225] Step 24:

[1226] Submitting a request:

[1227] The terminal notifies the server of this. The input is the user's selection data. The output is the transmission of request data from the terminal to the server.

[1228] Step 25:

[1229] Create and send the initial message:

[1230] The server selects an appropriate interaction scenario, generates an initial message, and sends it to the terminal. The input is a user interaction request. The output is the generated initial message sent to the terminal.

[1231] Step 26:

[1232] Displaying messages:

[1233] The terminal displays or speaks this message to the user. The input is an initial message from the server. The output is the initial message displayed on the terminal screen or played aloud.

[1234] Step 27:

[1235] Audio data collection:

[1236] The user responds to the initial message with voice. The terminal transmits this voice to the server. The input is the user's voice response. The output is the transmission of voice data from the terminal to the server.

[1237] Step 28:

[1238] Speech analysis and response generation:

[1239] The server performs speech analysis (e.g., Amazon Transcribe) and generates an appropriate response. The input is the speech data sent from the device. The output is the generated response generated by the server.

[1240] Step 29:

[1241] Sending a response:

[1242] The server sends the generated response to the terminal. The input is the generated response data. The output is the transmission of the response data to the terminal.

[1243] Step 30:

[1244] Show Responses:

[1245] The terminal communicates the response to the user and continues the dialogue. The input is the response data from the server. The output is the response displayed on the terminal screen or played back aloud.

[1246] Step 31:

[1247] Sending a termination request:

[1248] When the user finishes learning, the device sends the learning results to the server. The input is the user's termination operation. The output is the transmission of learning result data from the device to the server.

[1249] Step 32:

[1250] Data storage:

[1251] The server stores the results in a database and uses them the next time the machine learns. The input is the learning result data sent from the device. The output is the learning result stored in the database.

[1252] Step 33:

[1253] Submitting a request:

[1254] When a user wants to check their learning history, the terminal sends a request to the server. The input is the user's history check request. The output is the transmission of request data from the terminal to the server.

[1255] Step 34:

[1256] Retrieving and sending historical data:

[1257] The server acquires the history data and sends it to the terminal. The inputs are request data and database search. The output is the history data sent to the terminal.

[1258] Step 35:

[1259] View History:

[1260] The terminal displays this to the user. As input, there is historical data from the server. As output, the historical data is displayed on the terminal screen.

[1261] Step 36:

[1262] Generate and send service announcements:

[1263] The server generates a programming education guide for users who meet the criteria and sends it to the terminal. The input is the user's learning progress data. The output is the generated service guide sent to the terminal.

[1264] Step 37:

[1265] Directions displayed:

[1266] The terminal displays this information to the user as a notification or a pop-up.,Input:,The service information sent from the server.,Output:,The service information is displayed on the screen of the,terminal.

[1267] (Application example 1)

[1268] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1269] Conventional English learning systems have limitations in their ability to meticulously manage users' learning progress and provide appropriate content. Maximizing learning effectiveness through real-time feedback and the introduction of conversational AI is also a challenge. Furthermore, there is a lack of interactive learning methods that allow users to use smart devices.

[1270] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1271] In this invention, the server includes means for registering and storing user information, means for authenticating the registered information and managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for displaying or communicating the generated feedback to the user, means for providing interactive content on the smart device based on the user's learning progress, means for initiating a dialogue with an AI and utilizing a generative AI model to generate a response based on the user's voice input, and means for displaying the generated AI response to the user. This enables efficient and effective learning according to the user's progress and maximizes the learning effect through an interactive experience.

[1272] "User information" is data for specifying and identifying individual users, and includes names, email addresses, passwords, grades, etc.

[1273] The "storage means" is a device or function for storing data, and stores user information and learning progress data in a database on a server.

[1274] "Authentication" is the process of verifying whether the entered user information matches the registered information and authenticating the user.

[1275] "Study progress" is information indicating how far the user has progressed in their studies, and includes the number of completed lessons, score, and the like.

[1276] The "content selection means" is a function that selects and provides appropriate study content based on the user's study progress.

[1277] "Real-time analysis" is the process of instantly analyzing voice and text data entered by the user and generating appropriate feedback.

[1278] "Feedback" is response data that includes evaluations and advice regarding the user's learning.

[1279] "Display means" refers to a device or function for visually or audibly conveying generated feedback and learning content to the user.

[1280] "Interactive content" refers to learning content that progresses through real-time interaction between the user and the system.

[1281] A "smart device" is a portable information terminal that has Internet connectivity and is capable of running applications.

[1282] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses and feedback in response to user input.

[1283] "Generated AI response" is response data to user input generated by a generative AI model.

[1284] The system for realizing the present invention provides users with an effective English learning experience using smart devices. A specific embodiment of the system will be described below.

[1285] Overall system configuration

[1286] 1. User registration and login function

[1287] The server provides a means to register and store user information. The user enters their name, email address, password, grade, etc. on their smart device and sends it to the server. The server stores this information in a database and notifies the device that registration is complete. The user then enters their information on the authentication screen, and the server verifies the information. If authentication is successful, login is complete.

[1288] 2. Providing learning content

[1289] The server provides a means to select appropriate learning content based on the user's progress. When the user selects an English conversation simulation or vocabulary quiz on their smart device, the request is sent to the server. The server selects appropriate content based on the progress data and sends it to the device. The user then uses this content to advance their learning.

[1290] 3. Real-time analysis and feedback

[1291] The device sends user input (voice or text) to the server in real time, which then analyzes it using a generative AI model. For example, the user selects the "Start dialogue with AI" option, and an initial message is displayed on the smart device. The server then generates an appropriate response to the user's voice input and sends it to the device.

[1292] 4. Interactive learning content

[1293] The system provides interactive content based on the user's progress. For example, users can be prompted to answer quizzes or follow-up questions during the English conversation simulation to maximize learning outcomes. It also leverages generative AI models to provide real-time feedback.

[1294] 5. Management of learning data

[1295] The server stores the user's learning results and provides a means to update the user's progress the next time they study. Users can check their learning history on their smart devices, and the server updates it in real time.

[1296] Hardware and Software

[1297] Hardware: Smartphone (iOS or Android)

[1298] Software: React Native framework, Node.js-based server, MongoDB database

[1299] By using this hardware and software, user information registration, progress management, real-time feedback, and interactive learning content can be efficiently carried out.

[1300] Specific examples and examples of generative AI model prompts

[1301] Specific examples

[1302] For users with a learning progress score of 80, the next lesson will be recommended as "Practice everyday English conversation." When the user types "Hello, how are you?" into the AI, the AI ​​responds with "I'm good, thank you. How about you?"

[1303] Example prompts to input to the generative AI model

[1304] User input: "Hello, how are you?"

[1305] Generative AI model input prompt: "Interact with the user: Provide an appropriate response to 'Hello, how are you?'"

[1306] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1307] Step 1:

[1308] User Registration

[1309] Input: The user enters information such as name, email address, password, and grade on the new registration screen on the device.

[1310] Processing: The terminal sends this input information to the server.

[1311] Output: The server stores the received user information in the database and sends a registration completion message to the terminal.

[1312] Step 2:

[1313] User Login

[1314] Input: The user enters their email address and password on the login screen and clicks the Login button.

[1315] Processing: The terminal sends the entered information to the server, which compares it with information in a database to authenticate the user.

[1316] Output: The server sends the authentication result to the device, and if authentication is successful, obtains the user's learning progress data and displays it on the device.

[1317] Step 3:

[1318] Select learning content

[1319] Input: The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu on the device.

[1320] Processing: The device sends the user's selection to the server, and the server selects appropriate content based on the user's progress data and sends it to the device.

[1321] Output: The device displays the content from the server to the user, who then begins learning.

[1322] Step 4:

[1323] Learning implementation

[1324] Input: The user responds to presented scenarios and quizzes by voice or text.

[1325] Processing: The device sends user input in real time to the server, which analyzes it using a generative AI model.

[1326] Output: The server generates appropriate feedback based on the analysis results and sends it to the device, which then displays the feedback to the user, who can then continue learning based on it.

[1327] Step 5:

[1328] Dialogue with AI

[1329] Input: The user selects the "Start a conversation with AI" option and responds verbally to the initial message.

[1330] Processing: The device sends the voice data to the server, which uses a generative AI model to analyze the voice and generate a response. Example prompt: "Interact with the user: Provide an appropriate response to 'Hello, how are you?'"

[1331] Output: The server sends the AI's generated response to the terminal, which then displays or speaks it to the user to continue the dialogue.

[1332] Step 6:

[1333] Management of learning data

[1334] Input: When the user finishes learning, the device sends the learning results to the server.

[1335] Processing: The server saves the learning results in a database and updates the progress based on them the next time the learning is performed.

[1336] Output: When the user wants to check the learning history, the server sends the history data to the terminal, which displays it to the user.

[1337] Step 7:

[1338] Programming education guide

[1339] Input: For users who meet the criteria, the server generates a guide to programming education and notifies the terminal.

[1340] Processing: If the user wishes to register, they enter the necessary information, which the terminal sends to the server.

[1341] Output: The server saves the registration information and sends a registration complete message to the device, informing the user that the device is ready for the new service.

[1342] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1343] The system of the present invention is an interactive English learning system incorporating an "emotion engine" that recognizes the user's emotions and provides appropriate learning feedback. This system allows users to have a more effective and enjoyable learning experience. Below, we will explain in natural language how the program of this system works.

[1344] User Registration and Login

[1345] User Registration:

[1346] The user launches the app for the first time and clicks the Sign Up button.

[1347] The terminal displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[1348] When the user enters the required information and clicks the send button, the terminal sends this information to the server.

[1349] The server stores the received user information in a database.

[1350] The server sends a registration completion response to the terminal.

[1351] The terminal displays a message to the user indicating that registration is complete.

[1352] Login:

[1353] The user enters their email address and password and clicks the Login button.

[1354] The terminal transmits the input information to the server.

[1355] The server authenticates the user by checking the registration information in the database.

[1356] If the authentication is successful, the server acquires the user's learning progress data and sends it to the terminal.

[1357] The terminal will display the user's learning dashboard along with a successful login message.

[1358] Start learning English

[1359] Select your learning content:

[1360] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[1361] The terminal sends a request for the selected learning content to the server.

[1362] The server selects appropriate learning content based on the user's progress data and emotional data and transmits it to the terminal.

[1363] The terminal displays the selected study content and allows the user to begin studying.

[1364] Learning implementation:

[1365] The user proceeds through the displayed English conversation scenarios and vocabulary quizzes.

[1366] The device transmits user input (voice and text) and emotion data to the server in real time.

[1367] The server analyzes the input data and emotion data and generates appropriate feedback.

[1368] Send feedback to the device.

[1369] The terminal displays or conveys feedback from the server to the user to continue learning.

[1370] Dialogue with AI

[1371] Start the conversation:

[1372] The user selects the option to start a conversation with "AI Tomo."

[1373] The terminal notifies the server of this.

[1374] The server selects an appropriate dialogue scenario and sends an initial message (e.g., Hello! How are you today?) to the terminal.

[1375] The terminal will display or audibly convey this message to the user.

[1376] Continuing the dialogue:

[1377] The user responds verbally to the initial message.

[1378] The terminal transmits the user's voice and emotion data to the server.

[1379] The server analyzes and evaluates the user's responses using speech and emotion recognition technologies.

[1380] The server generates and sends appropriate feedback and next interaction messages to the terminal.

[1381] The terminal conveys feedback and the next interaction message to the user and continues the interaction.

[1382] Management of learning data

[1383] Save the training results:

[1384] The user finishes learning.

[1385] The terminal transmits the learning results and emotion data to the server.

[1386] The server stores the received learning results and emotion data in a database.

[1387] View your learning history:

[1388] The user submits a request to check their learning history.

[1389] The terminal sends a request for the learning history to the server.

[1390] The server retrieves the user's learning history and emotion data from the database and transmits them to the terminal.

[1391] The terminal displays the learning history to the user.

[1392] Provision of additional services (programming education)

[1393] New service information:

[1394] The server generates information data for programming education services for users whose English conversation studies meet a certain standard.

[1395] The terminal receives the guidance data from the server and displays it to the user as a notification or a pop-up.

[1396] Register for a new service:

[1397] A user becomes interested in a programming education service and wishes to register.

[1398] The terminal prompts the user to enter the necessary information and transmits it to the server.

[1399] The server stores the entered registration information in a database and sends a notice to the terminal informing the user that the new service can be used.

[1400] The terminal displays a guide to the user on how to start using the new service, and allows the user to start learning the programming education service.

[1401] Specific examples

[1402] English conversation simulation

[1403] 1. The user selects an English conversation simulation.

[1404] 2. The terminal sends the user's selection information to the server.

[1405] 3. The server selects an appropriate scenario and sends an initial message to the terminal.

[1406] 4. The terminal displays or speaks an initial message to the user.

[1407] 5. The user responds verbally, and the device sends this to the server.

[1408] 6. The server analyzes the voice data and emotion data and generates the following response message:

[1409] 7. Repeat this process until the conversation is complete.

[1410] Emotion-based feedback

[1411] 1. The user takes a vocabulary quiz.

[1412] 2. The device recognizes the user's emotions and transmits them to the server.

[1413] 3. The server analyzes the user's emotional data and determines that the user is "emotionally depressed."

[1414] 4. The server generates positive feedback that reflects the user's emotions (e.g., "Great! Let's try again next time!").

[1415] 5. The device displays the feedback to the user.

[1416] The system of the present invention recognizes the user's emotions and provides learning support in response to those emotions, allowing the user to study more proactively. Individual support based on emotion recognition also contributes to improving learning effectiveness.

[1417] The processing flow will be explained below.

[1418] User Registration and Login

[1419] User Registration:

[1420] Step 1:

[1421] The user launches the app for the first time and clicks the Sign Up button.

[1422] Step 2:

[1423] The terminal displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[1424] Step 3:

[1425] The user enters the required information and clicks the submit button.

[1426] Step 4:

[1427] The terminal transmits the input information to the server.

[1428] Step 5:

[1429] The server stores the received user information in a database.

[1430] Step 6:

[1431] The server sends a registration completion response to the terminal.

[1432] Step 7:

[1433] The terminal displays a message to the user indicating that registration is complete.

[1434] Login:

[1435] Step 1:

[1436] The user enters their email address and password and clicks the Login button.

[1437] Step 2:

[1438] The terminal transmits the input information to the server.

[1439] Step 3:

[1440] The server authenticates the user by checking the registration information in the database.

[1441] Step 4:

[1442] If the authentication is successful, the server acquires the user's learning progress data and transmits it to the terminal.

[1443] Step 5:

[1444] The terminal displays a successful login message and displays the user's learning dashboard.

[1445] Start learning English

[1446] Select your learning content:

[1447] Step 1:

[1448] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[1449] Step 2:

[1450] The terminal sends a request for the selected learning content to the server.

[1451] Step 3:

[1452] The server selects appropriate learning content based on the user's progress data and emotional data and transmits it to the terminal.

[1453] Step 4:

[1454] The terminal displays the selected study content and allows the user to begin studying.

[1455] Learning implementation:

[1456] Step 1:

[1457] The user proceeds through the displayed English conversation scenarios and vocabulary quizzes.

[1458] Step 2:

[1459] The device transmits user input (voice and text) and emotion data to the server in real time.

[1460] Step 3:

[1461] The server analyzes the input data and the emotion data.

[1462] Step 4:

[1463] The server generates appropriate feedback from the analysis results and sends it to the terminal.

[1464] Step 5:

[1465] The terminal displays feedback from the server to the user, allowing them to continue learning.

[1466] Dialogue with AI

[1467] Start the conversation:

[1468] Step 1:

[1469] The user selects the option to start a conversation with "AI Tomo."

[1470] Step 2:

[1471] The terminal sends a request to start a conversation to the server.

[1472] Step 3:

[1473] The server selects an appropriate dialogue scenario and sends an initial message (e.g., Hello! How are you today?) to the terminal.

[1474] Step 4:

[1475] The terminal displays or speaks an initial message to the user.

[1476] Continuing the dialogue:

[1477] Step 1:

[1478] The user responds verbally to the initial message.

[1479] Step 2:

[1480] The terminal transmits the user's voice and emotion data to the server.

[1481] Step 3:

[1482] The server analyzes the user's responses using voice and emotion recognition techniques.

[1483] Step 4:

[1484] The server generates appropriate feedback and the next dialogue message based on the analysis results and sends them to the terminal.

[1485] Step 5:

[1486] The terminal displays or speaks feedback and the next interaction message to the user.

[1487] Management of learning data

[1488] Save the training results:

[1489] Step 1:

[1490] The user finishes learning.

[1491] Step 2:

[1492] The terminal transmits the learning results and emotion data to the server.

[1493] Step 3:

[1494] The server stores the received learning results and emotion data in a database.

[1495] View your learning history:

[1496] Step 1:

[1497] The user submits a request to check their learning history.

[1498] Step 2:

[1499] The terminal sends a request for the learning history to the server.

[1500] Step 3:

[1501] The server retrieves the user's learning history and emotion data from the database and transmits them to the terminal.

[1502] Step 4:

[1503] The terminal displays the learning history to the user.

[1504] Provision of additional services (programming education)

[1505] New service information:

[1506] Step 1:

[1507] The server generates information data for programming education services for users whose English conversation studies meet a certain standard.

[1508] Step 2:

[1509] The terminal receives the guidance data from the server and displays it to the user as a notification or a pop-up.

[1510] Register for a new service:

[1511] Step 1:

[1512] A user becomes interested in a programming education service and wishes to register.

[1513] Step 2:

[1514] The terminal prompts the user to enter the necessary information and transmits it to the server.

[1515] Step 3:

[1516] The server stores the entered registration information in a database and sends a notice to the terminal informing the user that the new service can be used.

[1517] Step 4:

[1518] The terminal displays a guide to the user on how to start using the new service, and allows the user to start learning the programming education service.

[1519] Specific examples

[1520] English conversation simulation

[1521] Step 1:

[1522] The user selects an English conversation simulation.

[1523] Step 2:

[1524] The terminal transmits the user's selection information to the server.

[1525] Step 3:

[1526] The server selects an appropriate dialogue scenario and sends an initial message to the terminal.

[1527] Step 4:

[1528] The terminal displays or speaks an initial message to the user.

[1529] Step 5:

[1530] The user responds verbally, and the terminal sends this to the server.

[1531] Step 6:

[1532] The server analyzes the voice data and emotion data and generates the following response message:

[1533] Step 7:

[1534] The server sends the following response message to the terminal:

[1535] Step 8:

[1536] The terminal displays or speaks the following response message to the user.

[1537] Step 9:

[1538] The above process is repeated until the conversation is completed.

[1539] Emotion-based feedback

[1540] Step 1:

[1541] The user takes a word quiz.

[1542] Step 2:

[1543] The device recognizes the user's emotions and transmits them to the server.

[1544] Step 3:

[1545] The server analyzes the user's emotional data and determines that the user is "emotionally depressed."

[1546] Step 4:

[1547] The server generates positive feedback that reflects the user's emotions (e.g., "Great! Let's try harder next time!").

[1548] Step 5:

[1549] The server transmits the generated feedback to the terminal.

[1550] Step 6:

[1551] The terminal displays the feedback to the user.

[1552] The system of the present invention recognizes the user's emotions and provides learning support accordingly, allowing the user to take a more proactive approach to learning. Individualized support based on emotion recognition contributes to improving learning effectiveness and provides an environment that makes it easier for users to continue learning.

[1553] Example 2

[1554] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1555] Conventional learning systems do not provide feedback that takes into account the user's emotional state, resulting in issues such as reduced learning efficiency and loss of motivation. Furthermore, they lack a means to comprehensively manage the user's progress and emotional data and reflect this in the next lesson, making it difficult to provide an effective learning experience.

[1556] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1557] In this invention, the server includes means for registering and saving user information, means for authenticating the registered information and managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for displaying or communicating the generated feedback to the user, means for acquiring the user's emotional data and generating appropriate feedback based on that data, means for saving the user's emotional data and learning results, and means for reflecting the saved emotional data and learning results in the next learning session. This enables the provision of effective feedback that takes into account the user's individual emotional state and improves the learning experience.

[1558] "User information" refers to personal information such as the user's name, email address, password, and grade.

[1559] "Storage means" refers to a database or storage device for storing user information and learning data.

[1560] "Authentication means" refers to the process of authenticating a user based on registered user information and establishing access rights.

[1561] "Learning progress" refers to the progress and achievements a user makes through a learning activity.

[1562] "Learning Content" refers to learning materials and activities provided to users.

[1563] "Analysis means" refers to technology that analyzes data such as voice and text in real time to understand meaning and emotions.

[1564] "Feedback" refers to evaluation and advice messages provided to users as they learn.

[1565] "Emotion data" refers to the user's emotional state estimated from the user's facial expression, tone of voice, input content, etc.

[1566] A "dialogue scenario" is a predefined conversation flow or pattern used in interaction with a user.

[1567] "Learning results" refer to the results or grades that a user has achieved through learning activities.

[1568] The system of the present invention is an interactive English learning system incorporating an "emotion engine" that recognizes the user's emotions and provides appropriate learning feedback, allowing users to have a more effective and enjoyable learning experience.

[1569] User Registration and Login

[1570] First, the user launches the app for the first time and clicks the new registration button. The device displays a registration screen that prompts the user to enter their name, email address, password, and grade. When the user enters the required information and clicks the submit button, the device sends this information to the server. The server saves the received user information in a database and sends a registration completion response to the device. The device displays a registration completion message to the user.

[1571] Next, the user enters their email address and password and clicks the login button, and the device sends the entered information to the server. The server performs authentication by comparing the information with the registered information in the database, and if authentication is successful, it obtains the user's learning progress data and sends it to the device. The device then displays the user's learning dashboard along with a message that login was successful.

[1572] Start learning English

[1573] Next, the user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu. The device sends a request for the selected learning content to the server, and the server selects appropriate learning content based on the user's progress data and emotion data and sends it to the device. The device displays the selected learning content and allows the user to begin learning.

[1574] The user progresses through the displayed English conversation scenarios and vocabulary quizzes, and the device transmits the user's input (voice and text) and emotional data to the server in real time. The server analyzes the input data and emotional data, generates appropriate feedback, and transmits it to the device. The device then displays or communicates the feedback from the server to the user, allowing them to continue their learning.

[1575] Dialogue with AI

[1576] The user then selects an option to start a dialogue with "AI Tomo." The device notifies the server, which then selects an appropriate dialogue scenario and sends an initial message to the device. The device then displays or speaks this message to the user. When the user responds verbally, the device sends the voice and emotional data to the server. The server then analyzes and evaluates the user's response using voice recognition and emotion recognition technologies. The server then generates appropriate feedback and the next dialogue message and sends them to the device. The device then conveys the feedback and the next dialogue message to the user, continuing the dialogue.

[1577] Management of learning data

[1578] When the user finishes learning, the device sends the learning results and emotional data to the server. The server stores the received learning results and emotional data in a database. When the user sends a request to check their learning history, the device sends a learning history request to the server. The server retrieves the user's learning history and emotional data from the database and sends them to the device. The device displays the learning history to the user.

[1579] Examples and prompts

[1580] For example, if a user selects an English conversation simulation, the device sends the user's selection information to the server, which then selects an appropriate scenario and sends an initial message to the device. The device then displays or speaks this message to the user, who then responds verbally, which the device then sends to the server. The server then analyzes the voice and emotion data and generates the next response message. This process is repeated until the dialogue is completed.

[1581] Examples of prompt sentences include "Please start the English conversation simulation" and "Please provide appropriate feedback if the user's current emotions are judged to be depressed."

[1582] This allows users to study English while having their emotions properly recognized, improving the effectiveness of their learning.

[1583] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1584] Step 1:

[1585] The user launches the app for the first time and clicks the Sign Up button.

[1586] Input: User clicks on the new registration button.

[1587] Specific behavior: The app starts and the user clicks the new registration button.

[1588] Output: The terminal displays the user registration screen.

[1589] Step 2:

[1590] The device displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[1591] Input: New Registration button click event in Step 1.

[1592] What happens: A registration form is displayed and the user can enter their information.

[1593] Output: Waiting for user input.

[1594] Step 3:

[1595] The user enters the required information and clicks the submit button.

[1596] Input: A user enters their name, email address, password, and grade and clicks the submit button.

[1597] What happens: A user fills in a form and clicks the submit button.

[1598] Output: The data including the input user information is processed on the terminal side.

[1599] Step 4:

[1600] The terminal sends this information to the server.

[1601] Input: Information data entered by the user.

[1602] Specific operation: The device sends data to the server via the Internet.

[1603] Output: The user information sent to the server.

[1604] Step 5:

[1605] The server stores the received user information in a database.

[1606] Input: User information sent from the device.

[1607] Specific operation: The server receives the information and stores it in a database.

[1608] Output: User information stored in the database.

[1609] Step 6:

[1610] The server sends a registration completion response to the terminal.

[1611] Input: Confirmation of successful saving to database.

[1612] Specific operation: The server generates a registration completion message and sends it to the terminal.

[1613] Output: A registration completion message is sent to the device.

[1614] Step 7:

[1615] The terminal displays a message to the user that registration is complete.

[1616] Input: Registration completion message sent by the server.

[1617] Specific operation: The device displays a registration completion message on the screen.

[1618] Output: A successful registration message that is displayed to the user.

[1619] Step 8:

[1620] The user enters their email address and password and clicks the Login button.

[1621] Input: User enters email address and password and clicks login button.

[1622] What happens: The user fills in the login form and clicks a button.

[1623] Output: Data containing the entered information is sent to the terminal.

[1624] Step 9:

[1625] The terminal transmits the input information to the server.

[1626] Input: The email address and password entered by the user.

[1627] Specific operation: The device sends the data to the server.

[1628] Output: The authentication information sent to the server.

[1629] Step 10:

[1630] The server authenticates the user by checking the registration information in the database.

[1631] Input: The submitted credentials.

[1632] Specific operation: The server accesses the database and checks the user information.

[1633] Output: Authentication success or failure result.

[1634] Step 11:

[1635] If the authentication is successful, the server acquires the user's learning progress data and sends it to the terminal.

[1636] Input: Authentication success result.

[1637] Specific operation: The server retrieves the user's learning progress data from the database.

[1638] Output: Learning progress data sent to the device.

[1639] Step 12:

[1640] The terminal displays the user's learning dashboard along with a successful login message.

[1641] Input: Learning progress data and authentication success message sent from the server.

[1642] Specific behavior: The device displays a login success message and displays the dashboard to the user.

[1643] Output: The dashboard screen that is displayed to the user.

[1644] Step 13:

[1645] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[1646] Input: User selection of learning content.

[1647] Specific operation: The user selects the desired content from the menu.

[1648] Output: Selected content information.

[1649] Step 14:

[1650] The terminal sends a request for the selected learning content to the server.

[1651] Input: Content information selected by the user.

[1652] Specific operation: The device sends the information to the server.

[1653] Output: The content request sent to the server.

[1654] Step 15:

[1655] The server selects appropriate learning content based on the user's progress data and emotional data and sends it to the terminal.

[1656] Input: User progress data, emotion data, content requests.

[1657] Specific operation: The server selects the most suitable content based on this data.

[1658] Output: The learning content sent to the device.

[1659] Step 16:

[1660] The terminal displays the selected study content and allows the user to begin studying.

[1661] Input: The learning content sent from the server.

[1662] Specific operation: The device displays the learning content screen and prompts the user to begin learning.

[1663] Output: The displayed learning content.

[1664] Step 17:

[1665] The user progresses through the displayed English conversation scenarios and vocabulary quizzes.

[1666] Input: User conducts English conversation scenarios and vocabulary quizzes.

[1667] What happens: The user progresses through the displayed content.

[1668] Output: User's learning progress data.

[1669] Step 18:

[1670] The device transmits user input (voice and text) and emotion data to the server in real time.

[1671] Input: User voice input, text input, and emotion data.

[1672] Specific operation: The device sends this data to the server in real time.

[1673] Output: The input data and emotion data sent to the server.

[1674] Step 19:

[1675] The server analyzes the input data and emotion data and generates appropriate feedback.

[1676] Input: User input data and emotion data.

[1677] What happens: The server analyzes this data and generates feedback.

[1678] Output: The generated feedback data.

[1679] Step 20:

[1680] The server transmits the generated feedback to the terminal.

[1681] Input: Generated feedback data.

[1682] Specific operation: The server sends feedback data to the terminal.

[1683] Output: Feedback sent to the device.

[1684] Step 21:

[1685] The terminal displays or conveys feedback from the server to the user to continue learning.

[1686] Input: Feedback data sent by the server.

[1687] Specific behavior: The device displays or audibly provides feedback to the user.

[1688] Output: Feedback display to the user.

[1689] Step 22:

[1690] The user finishes learning.

[1691] Input: The user performs the operation to end learning.

[1692] Specific operation: The user clicks the End Learning button.

[1693] Output: End of learning status.

[1694] Step 23:

[1695] The terminal transmits the learning result and the emotion data to the server.

[1696] Input: Learning result data and emotion data.

[1697] Specific operation: The device sends this data to the server.

[1698] Output: Learning result data and emotion data sent to the server.

[1699] Step 24:

[1700] The server stores the received learning results and emotion data in a database.

[1701] Input: Learning result data and emotion data.

[1702] Specific operation: The server stores this data in a database.

[1703] Output: Learning results and emotion data stored in a database.

[1704] Step 25:

[1705] The user submits a request to check their learning history.

[1706] Input: A user request to check their learning history.

[1707] Specific operation: The user clicks the learning history button.

[1708] Output: The request data is sent to the terminal.

[1709] Step 26:

[1710] The terminal sends a request for the learning history to the server.

[1711] Input: The request data sent by the user to the terminal.

[1712] Specific operation: The terminal sends the request data to the server.

[1713] Output: The learning history request sent to the server.

[1714] Step 27:

[1715] The server retrieves the user's learning history and emotional data from the database and sends them to the terminal.

[1716] Input: Learning history request data.

[1717] Specific operation: The server retrieves learning history and emotion data from the database.

[1718] Output: Learning history and emotion data sent to the device.

[1719] Step 28:

[1720] The terminal displays the learning history to the user.

[1721] Input: Learning history and emotion data sent from the server.

[1722] Specific operation: The device displays this data on the screen.

[1723] Output: The learning history screen displayed to the user.

[1724] Specific examples and examples of prompts for the generative AI model

[1725] Specific examples

[1726] When the user selects an English conversation simulation, the device sends the user's selection information to the server, which then selects an appropriate scenario and sends an initial message to the device. The device then displays or speaks this message to the user, who then responds verbally, which the device then sends to the server. The server then analyzes the voice and emotion data and generates the next response message. This process is repeated until the dialogue is complete.

[1727] Prompt example

[1728] "Please start the English conversation simulation."

[1729] "Provide appropriate feedback if the user's current mood is deemed depressed."

[1730] This allows users to study English while having their emotions properly recognized, improving the effectiveness of their learning.

[1731] (Application example 2)

[1732] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1733] Modern streaming services face the challenge of making it difficult for users to find the optimal content that matches their emotions and state. In particular, when a user's mood or emotions fluctuate, conventional systems have difficulty in recommending content that responds to these changes, resulting in a less satisfying user experience.

[1734] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing a user's emotion and generating feedback based on the emotion, means for selecting and providing content appropriate to the user's state using a recommendation algorithm, and means for displaying or communicating the generated feedback and recommended content to the user. This allows the user to easily find content that best suits their emotional state.

[1735] "User information" refers to information about system users, such as their names, email addresses, passwords, and past usage history.

[1736] "Storage means" refers to a device or software for recording and storing user information, learning progress data, and emotion data.

[1737] "Authentication means" is a mechanism for verifying the authenticity of a user by comparing the information entered by the user with registered information in a database.

[1738] A "progress management means" is a device or system for tracking and recording a user's learning or viewing progress.

[1739] "Learning content" refers to digital content such as learning materials, quizzes, and simulations that users can use for learning.

[1740] The "means of providing" refers to a mechanism for displaying or conveying appropriate learning content or recommended content to users.

[1741] "Means for analyzing voice and text in real time and generating feedback" refers to a system that instantly analyzes a user's voice input or text input and generates appropriate responses or advice based on the results.

[1742] "Means for recognizing emotions" refers to technology for identifying a user's emotions by analyzing the user's facial expressions, voice, behavior, etc.

[1743] The "means for generating emotion-based feedback" is a device or system that automatically creates and provides feedback according to the recognized emotion of the user.

[1744] A "recommendation algorithm" is a computational method for selecting optimal content based on a user's emotional state and usage history.

[1745] "Recommended content" refers to content such as movies, dramas, music, documentaries, etc. that is provided in accordance with the user's emotional state and interests.

[1746] An "interface for facilitating viewing" is a user interface designed to allow a user to comfortably study or view content.

[1747] "Emotional state" is data that indicates the user's mental and emotional state, and is inferred from facial expressions, tone of voice, content of statements, etc.

[1748] The present invention provides a system for recommending appropriate content based on a user's emotional state, thereby improving the user's viewing experience. A detailed configuration of a system for realizing this application example will be described below.

[1749] System Overview

[1750] The system mainly consists of the following components:

[1751] 1. User information registration and authentication

[1752] 2. Emotion Recognition and Analysis

[1753] 3. Content Recommendation Algorithm

[1754] 4. Generating and displaying feedback

[1755] 5. Data storage and management

[1756] Hardware and software used

[1757] The main hardware and software used in this system are as follows:

[1758] Hardware: Smartphone camera, microphone

[1759] software:

[1760] Emotion Recognition: Google Cloud Vision API or Amazon Rekognition

[1761] Database: Firebase Realtime Database or AWS DynamoDB

[1762] Backend: Node.js or Python (Serverless environment: AWS Lambda or Google Cloud Functions)

[1763] Frontend: React Native (smartphone application)

[1764] User registration and authentication

[1765] When a user launches the app for the first time, an account is created and user authentication is performed using Firebase Auth or similar. Specifically, the user enters their name, email address, and password, which are then sent to the server. The server stores this information in a database and sends a registration completion response to the device. From the next time onwards, the user can log in using their email address and password.

[1766] Emotion Recognition and Analysis

[1767] The user's emotional state is recognized in real time through the smartphone's camera and microphone. For example, the Google Cloud Vision API is used to analyze emotions from the captured user's facial expressions. This emotional data is sent to the server and used in the next step.

[1768] Example prompt sentence:

[1769] def detect_emotions(image):

[1770] client = vision.ImageAnnotatorClient()

[1771] response = client.face_detection(image=image)

[1772] faces = response.face_annotations

[1773] emotions = {

[1774] 'joy': faces[0].joy_likelihood,

[1775] 'sorrow': faces[0].sorrow_likelihood,

[1776] 'anger': faces[0].anger_likelihood,

[1777] 'surprise': faces[0].surprise_likelihood,

[1778] }

[1779] return emotions

[1780] Content recommendation algorithm

[1781] The server then uses the collected emotional data to select the most suitable content using a recommendation algorithm, which takes into account the user's past viewing history and current emotional state to select the most suitable content for the user in real time.

[1782] Generating and displaying feedback

[1783] Along with the selected content, feedback based on the user's emotional data is also generated and provided to the user. For example, if the system detects that the user is feeling stressed, it will recommend relaxing content and send positive messages such as "Relax and have fun."

[1784] Data storage and management

[1785] After viewing, the app recognizes the user's emotional state again and stores that data in a database. This allows past data to be used in the next content recommendation, providing a more personalized experience. Users can also check their viewing history and emotional log at any time within the app.

[1786] For example, if a user inputs "I want to relax" into an emotion recognition system, the system will recommend relaxing music or nature documentaries. If the user is detected as being in a "happy mood," the system will recommend comedy movies or fun short videos with positive messages.

[1787] In this way, the system can provide the most appropriate content depending on the user's emotional state, improving the viewer's experience.

[1788] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1789] Step 1:

[1790] The user launches the app for the first time and begins the process of registering.

[1791] Input: Name, Email Address, Password

[1792] Action: The user enters the required information and clicks the submit button.

[1793] Output: The device sends this information to the server.

[1794] Step 2:

[1795] The server stores the received user information in a database.

[1796] Input: Registration information entered by the user

[1797] Operation: The server saves the registration information in a database and sends a registration completion response to the terminal.

[1798] Output: The terminal displays a message to the user that registration is complete.

[1799] Step 3:

[1800] The user enters their email address and password and clicks the login button.

[1801] Input: Email address, password

[1802] Operation: The terminal sends the entered information to the server, which then verifies it by comparing it with the registered information in the database.

[1803] Output: If authentication is successful, the server will retrieve the user's learning progress data and send it to the terminal, and the terminal will display the user's dashboard along with a login success message.

[1804] Step 4:

[1805] The user's face is captured by a camera and emotion recognition is performed.

[1806] Input: User's face image

[1807] How it works: The device uses the camera to capture the user's face and performs emotion analysis using the Google Cloud Vision API.

[1808] Output: Sends emotion data (e.g., joy, sadness, anger, surprise) to the server.

[1809] Step 5:

[1810] The server uses a recommendation algorithm to select the most suitable content based on the emotional data.

[1811] Input: Emotion data, user's past viewing history

[1812] How it works: The server runs a recommendation algorithm to select the most relevant content for the user.

[1813] Output: Sends the recommended content list to the device.

[1814] Step 6:

[1815] The terminal displays the recommended content list to the user.

[1816] Input: Recommended Content List

[1817] How it works: The device displays the recommended content to the user in a list format.

[1818] Output: The user selects the content they want to watch.

[1819] Step 7:

[1820] The user views the selected content.

[1821] Input: Selected content

[1822] Actions: The user uses the device to play and watch the selected content.

[1823] Output: The device collects content viewing progress data and emotion data.

[1824] Step 8:

[1825] The user's emotional changes are captured again by camera and emotion analysis is performed.

[1826] Input: User's face image (recaptured)

[1827] How it works: The device recaptures the user's face after viewing and performs emotion analysis using the Google Cloud Vision API.

[1828] Output: Emotional data after viewing is sent to the server.

[1829] Step 9:

[1830] The server stores emotion data before and after viewing and viewing progress data in a database.

[1831] Input: Emotional data before and after viewing, viewing progress data

[1832] How it works: The server stores this data in a database and uses it in the next recommendation algorithm.

[1833] Output: Updated user data in the database

[1834] Step 10:

[1835] Users can check their viewing history and emotion log from the menu displayed within the app.

[1836] Input: User request

[1837] How it works: The device sends a request to the server, which retrieves viewing history and emotion logs from the database.

[1838] Output: The terminal displays the acquired data to the user.

[1839] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1840] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1841] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1842] [Third embodiment]

[1843] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1844] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1845] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1846] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1847] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1848] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1849] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1850] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1851] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1852] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1853] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1854] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1855] The system of the present invention allows users to learn English efficiently and effectively by registering and providing appropriate learning content based on their progress. Below, we will explain in natural language how the program of this system works.

[1856] User Registration and Login

[1857] User Registration:

[1858] The user launches the app for the first time and clicks the new registration button.

[1859] The terminal prompts the user to enter information such as name, email address, password, and grade.

[1860] When the user enters the information and clicks the send button, the terminal sends this information to the server.

[1861] The server stores the received information in a database and notifies the terminal that registration is complete.

[1862] The terminal displays a message to the user indicating that registration is complete.

[1863] Login:

[1864] The user enters their email address and password on the login screen and clicks the login button.

[1865] The terminal transmits the input information to the server.

[1866] The server checks the information against its database and authenticates it.

[1867] If the authentication is successful, the server obtains the user's learning progress data and sends it to the terminal.

[1868] The terminal displays a successful login message and displays the user's learning dashboard.

[1869] Start learning English

[1870] Select your learning content:

[1871] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[1872] The terminal sends this request to the server.

[1873] The server selects appropriate content based on the user's progress data and transmits it to the terminal.

[1874] The terminal displays the content and allows the user to begin learning.

[1875] Learning implementation:

[1876] The user progresses through the displayed scenarios and quizzes.

[1877] The device transmits the user's input (voice or text) to the server in real time.

[1878] The server analyzes the input data and generates appropriate feedback.

[1879] Send feedback to the device.

[1880] The terminal displays this feedback to the user, who then continues learning based on it.

[1881] Dialogue with AI

[1882] Start the conversation:

[1883] The user selects the option to start a conversation with "AI Tomo."

[1884] The terminal notifies the server of this.

[1885] The server selects an appropriate interaction scenario and sends an initial message to the terminal.

[1886] The terminal will display or audibly convey this message to the user.

[1887] Continuing the dialogue:

[1888] The user responds verbally to the initial message.

[1889] The terminal transmits this audio to the server.

[1890] The server performs speech analysis and generates an appropriate response.

[1891] Send the response to the device.

[1892] The terminal communicates the response to the user and continues the dialogue.

[1893] Management of learning data

[1894] Save the training results:

[1895] When the user finishes learning, the terminal transmits the learning results to the server.

[1896] The server stores the results in a database and uses them the next time the learning is performed.

[1897] View your learning history:

[1898] When a user wants to check the learning history, the terminal sends a request to the server.

[1899] The server acquires the history data and transmits it to the terminal.

[1900] The terminal displays this to the user.

[1901] Provision of additional services (programming education)

[1902] New service information:

[1903] The server generates a guide for programming education for users who meet certain criteria and transmits it to the terminal.

[1904] The terminal displays this information to the user as a notification or pop-up.

[1905] Register for a new service:

[1906] If the user wishes to register, the terminal prompts the user to enter the necessary information and transmits it to the server.

[1907] The server stores the registration information and sends a registration completion message to the terminal.

[1908] The terminal notifies the user that the new service is ready.

[1909] In this way, the system of the present invention provides an interactive learning experience tailored to each user's individual progress, and can support English learning efficiently and effectively. By continuously managing users' learning data and providing additional services such as programming education as needed, the system provides comprehensive educational support.

[1910] The processing flow will be explained below.

[1911] User Registration and Login

[1912] User Registration:

[1913] Step 1:

[1914] The user launches the app for the first time and clicks the Sign Up button.

[1915] Step 2:

[1916] The terminal displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[1917] Step 3:

[1918] The user enters the required information and clicks the submit button.

[1919] Step 4:

[1920] The terminal transmits the input information to the server.

[1921] Step 5:

[1922] The server stores the received user information in a database.

[1923] Step 6:

[1924] The server sends a registration completion response to the terminal.

[1925] Step 7:

[1926] The terminal displays a message to the user indicating that registration is complete.

[1927] Login:

[1928] Step 1:

[1929] The user enters their email address and password and clicks the Login button.

[1930] Step 2:

[1931] The terminal transmits the input information to the server.

[1932] Step 3:

[1933] The server authenticates the user by checking the registration information in the database.

[1934] Step 4:

[1935] If the authentication is successful, the server acquires the user's learning progress data and transmits it to the terminal.

[1936] Step 5:

[1937] The terminal will display the user's learning dashboard along with a successful login message.

[1938] Start learning English

[1939] Select your learning content:

[1940] Step 1:

[1941] The user selects learning content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[1942] Step 2:

[1943] The terminal sends a request for the selected learning content to the server.

[1944] Step 3:

[1945] The server selects appropriate study content based on the user's progress data and transmits it to the terminal.

[1946] Step 4:

[1947] The terminal displays the selected study content and allows the user to begin studying.

[1948] Learning implementation:

[1949] Step 1:

[1950] The user proceeds through the displayed English conversation scenarios and vocabulary quizzes.

[1951] Step 2:

[1952] The device transmits the user's input (voice or text) to the server in real time.

[1953] Step 3:

[1954] The server analyzes the received data using voice recognition and natural language processing.

[1955] Step 4:

[1956] The server generates appropriate feedback from the analysis results and sends it to the terminal.

[1957] Step 5:

[1958] The terminal displays or conveys the feedback from the server to the user.

[1959] Dialogue with AI

[1960] Start the conversation:

[1961] Step 1:

[1962] The user selects the option to start a conversation with "AI Tomo."

[1963] Step 2:

[1964] The terminal sends a request to start a conversation to the server.

[1965] Step 3:

[1966] The server selects an appropriate dialogue scenario and sends an initial message (e.g., Hello! How are you today?) to the terminal.

[1967] Step 4:

[1968] The terminal displays or speaks an initial message to the user.

[1969] Continuing the dialogue:

[1970] Step 1:

[1971] The user responds verbally to the initial message.

[1972] Step 2:

[1973] The terminal transmits the user's voice to the server.

[1974] Step 3:

[1975] The server uses voice recognition technology to analyze and evaluate the user's responses.

[1976] Step 4:

[1977] The server generates and sends appropriate feedback and next interaction messages to the terminal.

[1978] Step 5:

[1979] The terminal displays or speaks feedback and the next interaction message to the user.

[1980] Management of learning data

[1981] Save the training results:

[1982] Step 1:

[1983] The user finishes learning.

[1984] Step 2:

[1985] The terminal transmits the learning result to the server.

[1986] Step 3:

[1987] The server stores the received learning results in a database.

[1988] View your learning history:

[1989] Step 1:

[1990] The user sends a request to check the learning history.

[1991] Step 2:

[1992] The terminal sends a request for the learning history to the server.

[1993] Step 3:

[1994] The server acquires the user's learning history from the database and transmits it to the terminal.

[1995] Step 4:

[1996] The terminal displays the learning history to the user.

[1997] Provision of additional services (programming education)

[1998] New service information:

[1999] Step 1:

[2000] The server generates information data for programming education services for users whose English conversation studies meet a certain standard.

[2001] Step 2:

[2002] The terminal receives the guidance data from the server and displays it to the user as a notification or a pop-up.

[2003] Register for a new service:

[2004] Step 1:

[2005] A user becomes interested in a programming education service and wishes to register.

[2006] Step 2:

[2007] The terminal prompts the user to enter the necessary information and transmits it to the server.

[2008] Step 3:

[2009] The server stores the entered registration information in a database and sends a notice to the terminal informing the user that the new service can be used.

[2010] Step 4:

[2011] The terminal displays a guide to the user on how to start using the new service, and allows the user to start learning the programming education service.

[2012] This allows users to efficiently learn English and also receive programming education.

[2013] Example 1

[2014] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2015] In modern society, there is a high demand for English language learning, but efficient and effective learning requires personalized learning support and advanced feedback functions. Furthermore, to encourage continued learning, users need interactive learning experiences and additional educational services. Conventional systems often lack the functionality to provide appropriate learning content based on individual users' progress, analyze and provide feedback in real time, and effectively introduce new educational services. This results in problems that limit the efficiency and effectiveness of users' learning.

[2016] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2017] In this invention, the server includes means for registering and saving user information, means for authenticating the registered information and managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for displaying or communicating the generated feedback to the user, means for the user to start and continue a dialogue with the "AI," means for saving the user's learning results and using them the next time they study, and means for generating information about new services and notifying the user. This enables the provision of accurate content and feedback based on the user's individual learning progress, and also enables effective information about and registration for new educational services.

[2018] "User Information" refers to personal identification information provided by a user at the time of registration, including name, email address, password, grade level, etc.

[2019] The "storage means" is a means for recording user information and learning progress data and storing them in a database or the like.

[2020] "Authentication means" refers to a process or system for verifying whether a user is legitimate based on input user information.

[2021] "Study progress" refers to the history, results, and progress of a user's learning activities.

[2022] "Learning content" refers to educational materials such as learning materials, scenarios, quizzes, and simulations that are provided for users to study.

[2023] An "analysis means" is a process or system that analyzes input voice or text data and generates appropriate feedback.

[2024] "Feedback" is a response that includes evaluation, advice, and corrections to the user's learning activities.

[2025] "Interaction means" refers to a process or system where a user continuously interacts with an artificial intelligence.

[2026] "Learning results" refers to data on the results, grades, and achievement levels obtained after a user uses learning content.

[2027] "Notification means" refers to means such as alerts, pop-ups, and emails that notify users of new services and information.

[2028] This invention is a system that provides appropriate learning content based on the user's progress, enabling efficient and effective English learning. This system is composed of means for registering and saving user information, means for managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for the user to initiate and continue a dialogue with an "AI," means for saving the learning results and using them the next time the user studies, and means for generating information about new services and notifying the user.

[2029] Configuration and Operation Procedures

[2030] User Registration and Login

[2031] User Registration:

[2032] (User) launches the app for the first time and clicks the new registration button.

[2033] (Terminal) prompts the user to enter information such as name, email address, password, and grade.

[2034] The server stores this information in a database (e.g., MySQL) and notifies the terminal of a registration completion message.

[2035] Login:

[2036] (User) enters email address and password on the login screen.

[2037] The server compares the information in the database and authenticates the user. If authentication is successful, it acquires the user's learning progress data and sends it to the device.

[2038] Select learning content and start learning English

[2039] Select your learning content:

[2040] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[2041] The server selects appropriate content based on the user's progress data and sends it to the terminal.

[2042] The terminal displays the selected content to the user.

[2043] Learning implementation:

[2044] (User) progresses through scenarios and quizzes.

[2045] The (server) analyzes the user's input (voice or text) in real time (e.g., Google Cloud Speech-to-Text) and generates appropriate feedback.

[2046] The feedback is displayed to the user through the terminal.

[2047] Dialogue with AI

[2048] Initiating and continuing a dialogue:

[2049] (User) selects "Dialogue with AI" and begins the dialogue.

[2050] The server selects an appropriate dialogue scenario, generates an initial message, and sends it to the terminal.

[2051] The user responds to the initial message with a voice message, which the device sends to the server, which analyzes the speech (e.g., with Amazon Transcribe) and generates an appropriate response, which is then transmitted to the user via the device.

[2052] Management of learning data

[2053] Saving and displaying training results:

[2054] When the user finishes learning, the device sends the learning results to the server, which stores them in a database.

[2055] When a user wants to check their learning history, the terminal sends a request to the server, the server acquires the history data, and sends it to the terminal, which then displays it.

[2056] Providing additional services

[2057] New service information:

[2058] The server generates a guide to programming education for users who meet the criteria and sends it to the terminal.

[2059] The device will display this information to the user as a notification or pop-up.

[2060] Examples of concrete examples and prompts

[2061] Sample prompt 1: "Select an option to begin interacting with the AI."

[2062] Example prompt 2: "Please select the learning content you would like to do (e.g., English conversation simulation, vocabulary quiz)."

[2063] In this way, the system of the present invention provides an interactive learning experience that responds to the user's individual progress, and can support English learning efficiently and effectively. It also continuously manages the user's learning data and provides additional services such as programming education as needed, thereby achieving comprehensive educational support.

[2064] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2065] Step 1:

[2066] Display the initial registration screen:

[2067] When a user launches the app for the first time, the device displays a new registration screen. As input, the user launches the app. As output, the new registration screen is displayed.

[2068] Step 2:

[2069] Enter your user information:

[2070] The user enters information such as name, email address, password, and grade into the new registration screen. Specifically, the input is provided by entering information into a text field and clicking the "Submit" button. As an output, the entered information is saved on the terminal.

[2071] Step 3:

[2072] Submit your input:

[2073] When the user clicks the "Send" button, the terminal sends the entered information to the server. The input is the user's click operation. The output is data transmission from the terminal to the server.

[2074] Step 4:

[2075] Data storage and verification:

[2076] The server saves the received information in a database (e.g. MySQL). After saving is complete, it notifies the terminal of a registration completion message. The input is the user information sent from the terminal. The output is that the information is saved in the database and a completion notification is sent to the terminal.

[2077] Step 5:

[2078] Registration complete message:

[2079] The terminal displays a message to the user that registration is complete. The input is a completion notification from the server. The output is a completion message displayed on the terminal screen.

[2080] Step 6:

[2081] Display the login screen:

[2082] The device displays the login screen. Input: The user launches the app again. Output: The login screen is displayed.

[2083] Step 7:

[2084] Enter your user information:

[2085] The user enters an email address and password on the login screen. Specifically, the input is to enter an email address and password in the text field and click the "Login" button. The output is to save the entered information on the device.

[2086] Step 8:

[2087] Sending credentials:

[2088] When the user clicks the "Login" button, the terminal sends the entered information to the server. The input is the user's click operation. The output is data sent from the terminal to the server.

[2089] Step 9:

[2090] User authentication:

[2091] The server authenticates the user by comparing the information in the database with the information it receives. The input is the user information sent from the terminal. The output is the authentication result.

[2092] Step 10:

[2093] Capture and send learning progress data:

[2094] If authentication is successful, the server retrieves the user's learning progress data from the database and sends it to the terminal. The inputs are the authentication result and the database search. The output is the learning progress data sent to the terminal.

[2095] Step 11:

[2096] Successful login display:

[2097] The terminal displays a login success message to the user and displays the learning dashboard. As input, there is learning progress data from the server. As output, the terminal screen displays a login success message and the learning dashboard.

[2098] Step 12:

[2099] Display the learning menu:

[2100] The terminal displays the learning menu to the user. As an input, there is the user's login completion action. As an output, the learning menu is displayed on the terminal screen.

[2101] Step 13:

[2102] Select content:

[2103] The user selects "English Conversation Simulation" or "Vocabulary Quiz" from the learning menu. The input is the user's selection action. The output is that the selection is saved on the device.

[2104] Step 14:

[2105] Submitting a content request:

[2106] When a user selects a content, the terminal sends this request to the server. The input is the user's selection. The output is the transmission of request data from the terminal to the server.

[2107] Step 15:

[2108] Content selection and delivery:

[2109] The server selects appropriate content based on the user's progress data and sends it to the terminal. The input is the user's progress data and the request in the server. The output is the selected content that is generated and sent to the terminal.

[2110] Step 16:

[2111] Show content:

[2112] The terminal displays the selected content to the user. As input, there is the content sent from the server. As output, the content is displayed on the terminal screen.

[2113] Step 17:

[2114] Check what's displayed:

[2115] The user checks the displayed scenarios and quizzes. The input is the content displayed on the device display. The output is the user's continuing learning action.

[2116] Step 18:

[2117] Collecting user input:

[2118] The terminal transmits voice and text data input by the user to the server in real time. The input includes the user's voice and text input. The output is real-time data transmission from the terminal to the server.

[2119] Step 19:

[2120] Data analysis and feedback generation:

[2121] The server analyzes the received data (e.g., Google Cloud Speech-to-Text) and generates appropriate feedback. As input, there is the user data sent from the device. As output, there is the generated feedback generated by the server.

[2122] Step 20:

[2123] Send feedback:

[2124] The server sends the generated feedback to the terminal. The input is the generated feedback. The output is the transmission of feedback data to the terminal.

[2125] Step 21:

[2126] Show feedback:

[2127] The device displays the feedback to the user, who then continues learning based on it. As input, there is feedback data from the server. As output, the feedback is displayed on the device screen.

[2128] Step 22:

[2129] View dialogue options:

[2130] The terminal displays an option to start a "dialogue with AI." As input, there is the user's login status. As output, the dialogue options are displayed on the terminal.

[2131] Step 23:

[2132] Select a dialogue option:

[2133] The user selects this option. As input, we have the user's selection action. As output, the selection is saved on the device.

[2134] Step 24:

[2135] Submitting a request:

[2136] The terminal notifies the server of this. The input is the user's selection data. The output is the transmission of request data from the terminal to the server.

[2137] Step 25:

[2138] Create and send the initial message:

[2139] The server selects an appropriate interaction scenario, generates an initial message, and sends it to the terminal. The input is a user interaction request. The output is the generated initial message sent to the terminal.

[2140] Step 26:

[2141] Displaying messages:

[2142] The terminal displays or speaks this message to the user. The input is an initial message from the server. The output is the initial message displayed on the terminal screen or played aloud.

[2143] Step 27:

[2144] Audio data collection:

[2145] The user responds to the initial message with voice. The terminal transmits this voice to the server. The input is the user's voice response. The output is the transmission of voice data from the terminal to the server.

[2146] Step 28:

[2147] Speech analysis and response generation:

[2148] The server performs speech analysis (e.g., Amazon Transcribe) and generates an appropriate response. The input is the speech data sent from the device. The output is the generated response generated by the server.

[2149] Step 29:

[2150] Sending a response:

[2151] The server sends the generated response to the terminal. The input is the generated response data. The output is the transmission of the response data to the terminal.

[2152] Step 30:

[2153] Show Responses:

[2154] The terminal communicates the response to the user and continues the dialogue. The input is the response data from the server. The output is the response displayed on the terminal screen or played back aloud.

[2155] Step 31:

[2156] Sending a termination request:

[2157] When the user finishes learning, the device sends the learning results to the server. The input is the user's termination operation. The output is the transmission of learning result data from the device to the server.

[2158] Step 32:

[2159] Data storage:

[2160] The server stores the results in a database and uses them the next time the machine learns. The input is the learning result data sent from the device. The output is the learning result stored in the database.

[2161] Step 33:

[2162] Submitting a request:

[2163] When a user wants to check their learning history, the terminal sends a request to the server. The input is the user's history check request. The output is the transmission of request data from the terminal to the server.

[2164] Step 34:

[2165] Retrieving and sending historical data:

[2166] The server acquires the history data and sends it to the terminal. The inputs are request data and database search. The output is the history data sent to the terminal.

[2167] Step 35:

[2168] View History:

[2169] The terminal displays this to the user. As input, there is historical data from the server. As output, the historical data is displayed on the terminal screen.

[2170] Step 36:

[2171] Generate and send service announcements:

[2172] The server generates a programming education guide for users who meet the criteria and sends it to the terminal. The input is the user's learning progress data. The output is the generated service guide sent to the terminal.

[2173] Step 37:

[2174] Directions displayed:

[2175] The terminal displays this information to the user as a notification or a pop-up.,Input:,The service information sent from the server.,Output:,The service information is displayed on the screen of the,terminal.

[2176] (Application example 1)

[2177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2178] Conventional English learning systems have limitations in their ability to meticulously manage users' learning progress and provide appropriate content. Maximizing learning effectiveness through real-time feedback and the introduction of conversational AI is also a challenge. Furthermore, there is a lack of interactive learning methods that allow users to use smart devices.

[2179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2180] In this invention, the server includes means for registering and storing user information, means for authenticating the registered information and managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for displaying or communicating the generated feedback to the user, means for providing interactive content on the smart device based on the user's learning progress, means for initiating a dialogue with an AI and utilizing a generative AI model to generate a response based on the user's voice input, and means for displaying the generated AI response to the user. This enables efficient and effective learning according to the user's progress and maximizes the learning effect through an interactive experience.

[2181] "User information" is data for specifying and identifying individual users, and includes names, email addresses, passwords, grades, etc.

[2182] The "storage means" is a device or function for storing data, and stores user information and learning progress data in a database on a server.

[2183] "Authentication" is the process of verifying whether the entered user information matches the registered information and authenticating the user.

[2184] "Study progress" is information indicating how far the user has progressed in their studies, and includes the number of completed lessons, score, and the like.

[2185] The "content selection means" is a function that selects and provides appropriate study content based on the user's study progress.

[2186] "Real-time analysis" is the process of instantly analyzing voice and text data entered by the user and generating appropriate feedback.

[2187] "Feedback" is response data that includes evaluations and advice regarding the user's learning.

[2188] "Display means" refers to a device or function for visually or audibly conveying generated feedback and learning content to the user.

[2189] "Interactive content" refers to learning content that progresses through real-time interaction between the user and the system.

[2190] A "smart device" is a portable information terminal that has Internet connectivity and is capable of running applications.

[2191] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses and feedback in response to user input.

[2192] "Generated AI response" is response data to user input generated by a generative AI model.

[2193] The system for realizing the present invention provides users with an effective English learning experience using smart devices. A specific embodiment of the system will be described below.

[2194] Overall system configuration

[2195] 1. User registration and login function

[2196] The server provides a means to register and store user information. The user enters their name, email address, password, grade, etc. on their smart device and sends it to the server. The server stores this information in a database and notifies the device that registration is complete. The user then enters their information on the authentication screen, and the server verifies the information. If authentication is successful, login is complete.

[2197] 2. Providing learning content

[2198] The server provides a means to select appropriate learning content based on the user's progress. When the user selects an English conversation simulation or vocabulary quiz on their smart device, the request is sent to the server. The server selects appropriate content based on the progress data and sends it to the device. The user then uses this content to advance their learning.

[2199] 3. Real-time analysis and feedback

[2200] The device sends user input (voice or text) to the server in real time, which then analyzes it using a generative AI model. For example, the user selects the "Start dialogue with AI" option, and an initial message is displayed on the smart device. The server then generates an appropriate response to the user's voice input and sends it to the device.

[2201] 4. Interactive learning content

[2202] The system provides interactive content based on the user's progress. For example, users can be prompted to answer quizzes or follow-up questions during the English conversation simulation to maximize learning outcomes. It also leverages generative AI models to provide real-time feedback.

[2203] 5. Management of learning data

[2204] The server stores the user's learning results and provides a means to update the user's progress the next time they study. Users can check their learning history on their smart devices, and the server updates it in real time.

[2205] Hardware and Software

[2206] Hardware: Smartphone (iOS or Android)

[2207] Software: React Native framework, Node.js-based server, MongoDB database

[2208] By using this hardware and software, user information registration, progress management, real-time feedback, and interactive learning content can be efficiently carried out.

[2209] Specific examples and examples of generative AI model prompts

[2210] Specific examples

[2211] For users with a learning progress score of 80, the next lesson will be recommended as "Practice everyday English conversation." When the user types "Hello, how are you?" into the AI, the AI ​​responds with "I'm good, thank you. How about you?"

[2212] Example prompts to input to the generative AI model

[2213] User input: "Hello, how are you?"

[2214] Generative AI model input prompt: "Interact with the user: Provide an appropriate response to 'Hello, how are you?'"

[2215] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2216] Step 1:

[2217] User Registration

[2218] Input: The user enters information such as name, email address, password, and grade on the new registration screen on the device.

[2219] Processing: The terminal sends this input information to the server.

[2220] Output: The server stores the received user information in the database and sends a registration completion message to the terminal.

[2221] Step 2:

[2222] User Login

[2223] Input: The user enters their email address and password on the login screen and clicks the Login button.

[2224] Processing: The terminal sends the entered information to the server, which compares it with information in a database to authenticate the user.

[2225] Output: The server sends the authentication result to the device, and if authentication is successful, obtains the user's learning progress data and displays it on the device.

[2226] Step 3:

[2227] Select learning content

[2228] Input: The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu on the device.

[2229] Processing: The device sends the user's selection to the server, and the server selects appropriate content based on the user's progress data and sends it to the device.

[2230] Output: The device displays the content from the server to the user, who then begins learning.

[2231] Step 4:

[2232] Learning implementation

[2233] Input: The user responds to presented scenarios and quizzes by voice or text.

[2234] Processing: The device sends user input in real time to the server, which analyzes it using a generative AI model.

[2235] Output: The server generates appropriate feedback based on the analysis results and sends it to the device, which then displays the feedback to the user, who can then continue learning based on it.

[2236] Step 5:

[2237] Dialogue with AI

[2238] Input: The user selects the "Start a conversation with AI" option and responds verbally to the initial message.

[2239] Processing: The device sends the voice data to the server, which uses a generative AI model to analyze the voice and generate a response. Example prompt: "Interact with the user: Provide an appropriate response to 'Hello, how are you?'"

[2240] Output: The server sends the AI's generated response to the terminal, which then displays or speaks it to the user to continue the dialogue.

[2241] Step 6:

[2242] Management of learning data

[2243] Input: When the user finishes learning, the device sends the learning results to the server.

[2244] Processing: The server saves the learning results in a database and updates the progress based on them the next time the learning is performed.

[2245] Output: When the user wants to check the learning history, the server sends the history data to the terminal, which displays it to the user.

[2246] Step 7:

[2247] Programming education guide

[2248] Input: For users who meet the criteria, the server generates a guide to programming education and notifies the terminal.

[2249] Processing: If the user wishes to register, they enter the necessary information, which the terminal sends to the server.

[2250] Output: The server saves the registration information and sends a registration complete message to the device, informing the user that the device is ready for the new service.

[2251] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2252] The system of the present invention is an interactive English learning system incorporating an "emotion engine" that recognizes the user's emotions and provides appropriate learning feedback. This system allows users to have a more effective and enjoyable learning experience. Below, we will explain in natural language how the program of this system works.

[2253] User Registration and Login

[2254] User Registration:

[2255] The user launches the app for the first time and clicks the Sign Up button.

[2256] The terminal displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[2257] When the user enters the required information and clicks the send button, the terminal sends this information to the server.

[2258] The server stores the received user information in a database.

[2259] The server sends a registration completion response to the terminal.

[2260] The terminal displays a message to the user indicating that registration is complete.

[2261] Login:

[2262] The user enters their email address and password and clicks the Login button.

[2263] The terminal transmits the input information to the server.

[2264] The server authenticates the user by checking the registration information in the database.

[2265] If the authentication is successful, the server acquires the user's learning progress data and sends it to the terminal.

[2266] The terminal will display the user's learning dashboard along with a successful login message.

[2267] Start learning English

[2268] Select your learning content:

[2269] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[2270] The terminal sends a request for the selected learning content to the server.

[2271] The server selects appropriate learning content based on the user's progress data and emotional data and transmits it to the terminal.

[2272] The terminal displays the selected study content and allows the user to begin studying.

[2273] Learning implementation:

[2274] The user proceeds through the displayed English conversation scenarios and vocabulary quizzes.

[2275] The device transmits user input (voice and text) and emotion data to the server in real time.

[2276] The server analyzes the input data and emotion data and generates appropriate feedback.

[2277] Send feedback to the device.

[2278] The terminal displays or conveys feedback from the server to the user to continue learning.

[2279] Dialogue with AI

[2280] Start the conversation:

[2281] The user selects the option to start a conversation with "AI Tomo."

[2282] The terminal notifies the server of this.

[2283] The server selects an appropriate dialogue scenario and sends an initial message (e.g., Hello! How are you today?) to the terminal.

[2284] The terminal will display or audibly convey this message to the user.

[2285] Continuing the dialogue:

[2286] The user responds verbally to the initial message.

[2287] The terminal transmits the user's voice and emotion data to the server.

[2288] The server analyzes and evaluates the user's responses using speech and emotion recognition technologies.

[2289] The server generates and sends appropriate feedback and next interaction messages to the terminal.

[2290] The terminal conveys feedback and the next interaction message to the user and continues the interaction.

[2291] Management of learning data

[2292] Save the training results:

[2293] The user finishes learning.

[2294] The terminal transmits the learning results and emotion data to the server.

[2295] The server stores the received learning results and emotion data in a database.

[2296] View your learning history:

[2297] The user submits a request to check their learning history.

[2298] The terminal sends a request for the learning history to the server.

[2299] The server retrieves the user's learning history and emotion data from the database and transmits them to the terminal.

[2300] The terminal displays the learning history to the user.

[2301] Provision of additional services (programming education)

[2302] New service information:

[2303] The server generates information data for programming education services for users whose English conversation studies meet a certain standard.

[2304] The terminal receives the guidance data from the server and displays it to the user as a notification or a pop-up.

[2305] Register for a new service:

[2306] A user becomes interested in a programming education service and wishes to register.

[2307] The terminal prompts the user to enter the necessary information and transmits it to the server.

[2308] The server stores the entered registration information in a database and sends a notice to the terminal informing the user that the new service can be used.

[2309] The terminal displays a guide to the user on how to start using the new service, and allows the user to start learning the programming education service.

[2310] Specific examples

[2311] English conversation simulation

[2312] 1. The user selects an English conversation simulation.

[2313] 2. The terminal sends the user's selection information to the server.

[2314] 3. The server selects an appropriate scenario and sends an initial message to the terminal.

[2315] 4. The terminal displays or speaks an initial message to the user.

[2316] 5. The user responds verbally, and the device sends this to the server.

[2317] 6. The server analyzes the voice data and emotion data and generates the following response message:

[2318] 7. Repeat this process until the conversation is complete.

[2319] Emotion-based feedback

[2320] 1. The user takes a vocabulary quiz.

[2321] 2. The device recognizes the user's emotions and transmits them to the server.

[2322] 3. The server analyzes the user's emotional data and determines that the user is "emotionally depressed."

[2323] 4. The server generates positive feedback that reflects the user's emotions (e.g., "Great! Let's try again next time!").

[2324] 5. The device displays the feedback to the user.

[2325] The system of the present invention recognizes the user's emotions and provides learning support in response to those emotions, allowing the user to study more proactively. Individual support based on emotion recognition also contributes to improving learning effectiveness.

[2326] The processing flow will be explained below.

[2327] User Registration and Login

[2328] User Registration:

[2329] Step 1:

[2330] The user launches the app for the first time and clicks the Sign Up button.

[2331] Step 2:

[2332] The terminal displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[2333] Step 3:

[2334] The user enters the required information and clicks the submit button.

[2335] Step 4:

[2336] The terminal transmits the input information to the server.

[2337] Step 5:

[2338] The server stores the received user information in a database.

[2339] Step 6:

[2340] The server sends a registration completion response to the terminal.

[2341] Step 7:

[2342] The terminal displays a message to the user indicating that registration is complete.

[2343] Login:

[2344] Step 1:

[2345] The user enters their email address and password and clicks the Login button.

[2346] Step 2:

[2347] The terminal transmits the input information to the server.

[2348] Step 3:

[2349] The server authenticates the user by checking the registration information in the database.

[2350] Step 4:

[2351] If the authentication is successful, the server acquires the user's learning progress data and transmits it to the terminal.

[2352] Step 5:

[2353] The terminal displays a successful login message and displays the user's learning dashboard.

[2354] Start learning English

[2355] Select your learning content:

[2356] Step 1:

[2357] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[2358] Step 2:

[2359] The terminal sends a request for the selected learning content to the server.

[2360] Step 3:

[2361] The server selects appropriate learning content based on the user's progress data and emotional data and transmits it to the terminal.

[2362] Step 4:

[2363] The terminal displays the selected study content and allows the user to begin studying.

[2364] Learning implementation:

[2365] Step 1:

[2366] The user proceeds through the displayed English conversation scenarios and vocabulary quizzes.

[2367] Step 2:

[2368] The device transmits user input (voice and text) and emotion data to the server in real time.

[2369] Step 3:

[2370] The server analyzes the input data and the emotion data.

[2371] Step 4:

[2372] The server generates appropriate feedback from the analysis results and sends it to the terminal.

[2373] Step 5:

[2374] The terminal displays feedback from the server to the user, allowing them to continue learning.

[2375] Dialogue with AI

[2376] Start the conversation:

[2377] Step 1:

[2378] The user selects the option to start a conversation with "AI Tomo."

[2379] Step 2:

[2380] The terminal sends a request to start a conversation to the server.

[2381] Step 3:

[2382] The server selects an appropriate dialogue scenario and sends an initial message (e.g., Hello! How are you today?) to the terminal.

[2383] Step 4:

[2384] The terminal displays or speaks an initial message to the user.

[2385] Continuing the dialogue:

[2386] Step 1:

[2387] The user responds verbally to the initial message.

[2388] Step 2:

[2389] The terminal transmits the user's voice and emotion data to the server.

[2390] Step 3:

[2391] The server analyzes the user's responses using voice and emotion recognition techniques.

[2392] Step 4:

[2393] The server generates appropriate feedback and the next dialogue message based on the analysis results and sends them to the terminal.

[2394] Step 5:

[2395] The terminal displays or speaks feedback and the next interaction message to the user.

[2396] Management of learning data

[2397] Save the training results:

[2398] Step 1:

[2399] The user finishes learning.

[2400] Step 2:

[2401] The terminal transmits the learning results and emotion data to the server.

[2402] Step 3:

[2403] The server stores the received learning results and emotion data in a database.

[2404] View your learning history:

[2405] Step 1:

[2406] The user submits a request to check their learning history.

[2407] Step 2:

[2408] The terminal sends a request for the learning history to the server.

[2409] Step 3:

[2410] The server retrieves the user's learning history and emotion data from the database and transmits them to the terminal.

[2411] Step 4:

[2412] The terminal displays the learning history to the user.

[2413] Provision of additional services (programming education)

[2414] New service information:

[2415] Step 1:

[2416] The server generates information data for programming education services for users whose English conversation studies meet a certain standard.

[2417] Step 2:

[2418] The terminal receives the guidance data from the server and displays it to the user as a notification or a pop-up.

[2419] Register for a new service:

[2420] Step 1:

[2421] A user becomes interested in a programming education service and wishes to register.

[2422] Step 2:

[2423] The terminal prompts the user to enter the necessary information and transmits it to the server.

[2424] Step 3:

[2425] The server stores the entered registration information in a database and sends a notice to the terminal informing the user that the new service can be used.

[2426] Step 4:

[2427] The terminal displays a guide to the user on how to start using the new service, and allows the user to start learning the programming education service.

[2428] Specific examples

[2429] English conversation simulation

[2430] Step 1:

[2431] The user selects an English conversation simulation.

[2432] Step 2:

[2433] The terminal transmits the user's selection information to the server.

[2434] Step 3:

[2435] The server selects an appropriate dialogue scenario and sends an initial message to the terminal.

[2436] Step 4:

[2437] The terminal displays or speaks an initial message to the user.

[2438] Step 5:

[2439] The user responds verbally, and the terminal sends this to the server.

[2440] Step 6:

[2441] The server analyzes the voice data and emotion data and generates the following response message:

[2442] Step 7:

[2443] The server sends the following response message to the terminal:

[2444] Step 8:

[2445] The terminal displays or speaks the following response message to the user.

[2446] Step 9:

[2447] The above process is repeated until the conversation is completed.

[2448] Emotion-based feedback

[2449] Step 1:

[2450] The user takes a word quiz.

[2451] Step 2:

[2452] The device recognizes the user's emotions and transmits them to the server.

[2453] Step 3:

[2454] The server analyzes the user's emotional data and determines that the user is "emotionally depressed."

[2455] Step 4:

[2456] The server generates positive feedback that reflects the user's emotions (e.g., "Great! Let's try harder next time!").

[2457] Step 5:

[2458] The server transmits the generated feedback to the terminal.

[2459] Step 6:

[2460] The terminal displays the feedback to the user.

[2461] The system of the present invention recognizes the user's emotions and provides learning support accordingly, allowing the user to take a more proactive approach to learning. Individualized support based on emotion recognition contributes to improving learning effectiveness and provides an environment that makes it easier for users to continue learning.

[2462] Example 2

[2463] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2464] Conventional learning systems do not provide feedback that takes into account the user's emotional state, resulting in issues such as reduced learning efficiency and loss of motivation. Furthermore, they lack a means to comprehensively manage the user's progress and emotional data and reflect this in the next lesson, making it difficult to provide an effective learning experience.

[2465] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2466] In this invention, the server includes means for registering and saving user information, means for authenticating the registered information and managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for displaying or communicating the generated feedback to the user, means for acquiring the user's emotional data and generating appropriate feedback based on that data, means for saving the user's emotional data and learning results, and means for reflecting the saved emotional data and learning results in the next learning session. This enables the provision of effective feedback that takes into account the user's individual emotional state and improves the learning experience.

[2467] "User information" refers to personal information such as the user's name, email address, password, and grade.

[2468] "Storage means" refers to a database or storage device for storing user information and learning data.

[2469] "Authentication means" refers to the process of authenticating a user based on registered user information and establishing access rights.

[2470] "Learning progress" refers to the progress and achievements a user makes through a learning activity.

[2471] "Learning Content" refers to learning materials and activities provided to users.

[2472] "Analysis means" refers to technology that analyzes data such as voice and text in real time to understand meaning and emotions.

[2473] "Feedback" refers to evaluation and advice messages provided to users as they learn.

[2474] "Emotion data" refers to the user's emotional state estimated from the user's facial expression, tone of voice, input content, etc.

[2475] A "dialogue scenario" is a predefined conversation flow or pattern used in interaction with a user.

[2476] "Learning results" refer to the results or grades that a user has achieved through learning activities.

[2477] The system of the present invention is an interactive English learning system incorporating an "emotion engine" that recognizes the user's emotions and provides appropriate learning feedback, allowing users to have a more effective and enjoyable learning experience.

[2478] User Registration and Login

[2479] First, the user launches the app for the first time and clicks the new registration button. The device displays a registration screen that prompts the user to enter their name, email address, password, and grade. When the user enters the required information and clicks the submit button, the device sends this information to the server. The server saves the received user information in a database and sends a registration completion response to the device. The device displays a registration completion message to the user.

[2480] Next, the user enters their email address and password and clicks the login button, and the device sends the entered information to the server. The server performs authentication by comparing the information with the registered information in the database, and if authentication is successful, it obtains the user's learning progress data and sends it to the device. The device then displays the user's learning dashboard along with a message that login was successful.

[2481] Start learning English

[2482] Next, the user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu. The device sends a request for the selected learning content to the server, and the server selects appropriate learning content based on the user's progress data and emotion data and sends it to the device. The device displays the selected learning content and allows the user to begin learning.

[2483] The user progresses through the displayed English conversation scenarios and vocabulary quizzes, and the device transmits the user's input (voice and text) and emotional data to the server in real time. The server analyzes the input data and emotional data, generates appropriate feedback, and transmits it to the device. The device then displays or communicates the feedback from the server to the user, allowing them to continue their learning.

[2484] Dialogue with AI

[2485] The user then selects an option to start a dialogue with "AI Tomo." The device notifies the server, which then selects an appropriate dialogue scenario and sends an initial message to the device. The device then displays or speaks this message to the user. When the user responds verbally, the device sends the voice and emotional data to the server. The server then analyzes and evaluates the user's response using voice recognition and emotion recognition technologies. The server then generates appropriate feedback and the next dialogue message and sends them to the device. The device then conveys the feedback and the next dialogue message to the user, continuing the dialogue.

[2486] Management of learning data

[2487] When the user finishes learning, the device sends the learning results and emotional data to the server. The server stores the received learning results and emotional data in a database. When the user sends a request to check their learning history, the device sends a learning history request to the server. The server retrieves the user's learning history and emotional data from the database and sends them to the device. The device displays the learning history to the user.

[2488] Examples and prompts

[2489] For example, if a user selects an English conversation simulation, the device sends the user's selection information to the server, which then selects an appropriate scenario and sends an initial message to the device. The device then displays or speaks this message to the user, who then responds verbally, which the device then sends to the server. The server then analyzes the voice and emotion data and generates the next response message. This process is repeated until the dialogue is completed.

[2490] Examples of prompt sentences include "Please start the English conversation simulation" and "Please provide appropriate feedback if the user's current emotions are judged to be depressed."

[2491] This allows users to study English while having their emotions properly recognized, improving the effectiveness of their learning.

[2492] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2493] Step 1:

[2494] The user launches the app for the first time and clicks the Sign Up button.

[2495] Input: User clicks on the new registration button.

[2496] Specific behavior: The app starts and the user clicks the new registration button.

[2497] Output: The terminal displays the user registration screen.

[2498] Step 2:

[2499] The device displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[2500] Input: New Registration button click event in Step 1.

[2501] What happens: A registration form is displayed and the user can enter their information.

[2502] Output: Waiting for user input.

[2503] Step 3:

[2504] The user enters the required information and clicks the submit button.

[2505] Input: A user enters their name, email address, password, and grade and clicks the submit button.

[2506] What happens: A user fills in a form and clicks the submit button.

[2507] Output: The data including the input user information is processed on the terminal side.

[2508] Step 4:

[2509] The terminal sends this information to the server.

[2510] Input: Information data entered by the user.

[2511] Specific operation: The device sends data to the server via the Internet.

[2512] Output: The user information sent to the server.

[2513] Step 5:

[2514] The server stores the received user information in a database.

[2515] Input: User information sent from the device.

[2516] Specific operation: The server receives the information and stores it in a database.

[2517] Output: User information stored in the database.

[2518] Step 6:

[2519] The server sends a registration completion response to the terminal.

[2520] Input: Confirmation of successful saving to database.

[2521] Specific operation: The server generates a registration completion message and sends it to the terminal.

[2522] Output: A registration completion message is sent to the device.

[2523] Step 7:

[2524] The terminal displays a message to the user that registration is complete.

[2525] Input: Registration completion message sent by the server.

[2526] Specific operation: The device displays a registration completion message on the screen.

[2527] Output: A successful registration message that is displayed to the user.

[2528] Step 8:

[2529] The user enters their email address and password and clicks the Login button.

[2530] Input: User enters email address and password and clicks login button.

[2531] What happens: The user fills in the login form and clicks a button.

[2532] Output: Data containing the entered information is sent to the terminal.

[2533] Step 9:

[2534] The terminal transmits the input information to the server.

[2535] Input: The email address and password entered by the user.

[2536] Specific operation: The device sends the data to the server.

[2537] Output: The authentication information sent to the server.

[2538] Step 10:

[2539] The server authenticates the user by checking the registration information in the database.

[2540] Input: The submitted credentials.

[2541] Specific operation: The server accesses the database and checks the user information.

[2542] Output: Authentication success or failure result.

[2543] Step 11:

[2544] If the authentication is successful, the server acquires the user's learning progress data and sends it to the terminal.

[2545] Input: Authentication success result.

[2546] Specific operation: The server retrieves the user's learning progress data from the database.

[2547] Output: Learning progress data sent to the device.

[2548] Step 12:

[2549] The terminal displays the user's learning dashboard along with a successful login message.

[2550] Input: Learning progress data and authentication success message sent from the server.

[2551] Specific behavior: The device displays a login success message and displays the dashboard to the user.

[2552] Output: The dashboard screen that is displayed to the user.

[2553] Step 13:

[2554] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[2555] Input: User selection of learning content.

[2556] Specific operation: The user selects the desired content from the menu.

[2557] Output: Selected content information.

[2558] Step 14:

[2559] The terminal sends a request for the selected learning content to the server.

[2560] Input: Content information selected by the user.

[2561] Specific operation: The device sends the information to the server.

[2562] Output: The content request sent to the server.

[2563] Step 15:

[2564] The server selects appropriate learning content based on the user's progress data and emotional data and sends it to the terminal.

[2565] Input: User progress data, emotion data, content requests.

[2566] Specific operation: The server selects the most suitable content based on this data.

[2567] Output: The learning content sent to the device.

[2568] Step 16:

[2569] The terminal displays the selected study content and allows the user to begin studying.

[2570] Input: The learning content sent from the server.

[2571] Specific operation: The device displays the learning content screen and prompts the user to begin learning.

[2572] Output: The displayed learning content.

[2573] Step 17:

[2574] The user progresses through the displayed English conversation scenarios and vocabulary quizzes.

[2575] Input: User conducts English conversation scenarios and vocabulary quizzes.

[2576] What happens: The user progresses through the displayed content.

[2577] Output: User's learning progress data.

[2578] Step 18:

[2579] The device transmits user input (voice and text) and emotion data to the server in real time.

[2580] Input: User voice input, text input, and emotion data.

[2581] Specific operation: The device sends this data to the server in real time.

[2582] Output: The input data and emotion data sent to the server.

[2583] Step 19:

[2584] The server analyzes the input data and emotion data and generates appropriate feedback.

[2585] Input: User input data and emotion data.

[2586] What happens: The server analyzes this data and generates feedback.

[2587] Output: The generated feedback data.

[2588] Step 20:

[2589] The server transmits the generated feedback to the terminal.

[2590] Input: Generated feedback data.

[2591] Specific operation: The server sends feedback data to the terminal.

[2592] Output: Feedback sent to the device.

[2593] Step 21:

[2594] The terminal displays or conveys feedback from the server to the user to continue learning.

[2595] Input: Feedback data sent by the server.

[2596] Specific behavior: The device displays or audibly provides feedback to the user.

[2597] Output: Feedback display to the user.

[2598] Step 22:

[2599] The user finishes learning.

[2600] Input: The user performs the operation to end learning.

[2601] Specific operation: The user clicks the End Learning button.

[2602] Output: End of learning status.

[2603] Step 23:

[2604] The terminal transmits the learning result and the emotion data to the server.

[2605] Input: Learning result data and emotion data.

[2606] Specific operation: The device sends this data to the server.

[2607] Output: Learning result data and emotion data sent to the server.

[2608] Step 24:

[2609] The server stores the received learning results and emotion data in a database.

[2610] Input: Learning result data and emotion data.

[2611] Specific operation: The server stores this data in a database.

[2612] Output: Learning results and emotion data stored in a database.

[2613] Step 25:

[2614] The user submits a request to check their learning history.

[2615] Input: A user request to check their learning history.

[2616] Specific operation: The user clicks the learning history button.

[2617] Output: The request data is sent to the terminal.

[2618] Step 26:

[2619] The terminal sends a request for the learning history to the server.

[2620] Input: The request data sent by the user to the terminal.

[2621] Specific operation: The terminal sends the request data to the server.

[2622] Output: The learning history request sent to the server.

[2623] Step 27:

[2624] The server retrieves the user's learning history and emotional data from the database and sends them to the terminal.

[2625] Input: Learning history request data.

[2626] Specific operation: The server retrieves learning history and emotion data from the database.

[2627] Output: Learning history and emotion data sent to the device.

[2628] Step 28:

[2629] The terminal displays the learning history to the user.

[2630] Input: Learning history and emotion data sent from the server.

[2631] Specific operation: The device displays this data on the screen.

[2632] Output: The learning history screen displayed to the user.

[2633] Specific examples and examples of prompts for the generative AI model

[2634] Specific examples

[2635] When the user selects an English conversation simulation, the device sends the user's selection information to the server, which then selects an appropriate scenario and sends an initial message to the device. The device then displays or speaks this message to the user, who then responds verbally, which the device then sends to the server. The server then analyzes the voice and emotion data and generates the next response message. This process is repeated until the dialogue is complete.

[2636] Prompt example

[2637] "Please start the English conversation simulation."

[2638] "Provide appropriate feedback if the user's current mood is deemed depressed."

[2639] This allows users to study English while having their emotions properly recognized, improving the effectiveness of their learning.

[2640] (Application example 2)

[2641] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2642] Modern streaming services face the challenge of making it difficult for users to find the optimal content that matches their emotions and state. In particular, when a user's mood or emotions fluctuate, conventional systems have difficulty in recommending content that responds to these changes, resulting in a less satisfying user experience.

[2643] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing a user's emotion and generating feedback based on the emotion, means for selecting and providing content appropriate to the user's state using a recommendation algorithm, and means for displaying or communicating the generated feedback and recommended content to the user. This allows the user to easily find content that best suits their emotional state.

[2644] "User information" refers to information about system users, such as their names, email addresses, passwords, and past usage history.

[2645] "Storage means" refers to a device or software for recording and storing user information, learning progress data, and emotion data.

[2646] "Authentication means" is a mechanism for verifying the authenticity of a user by comparing the information entered by the user with registered information in a database.

[2647] A "progress management means" is a device or system for tracking and recording a user's learning or viewing progress.

[2648] "Learning content" refers to digital content such as learning materials, quizzes, and simulations that users can use for learning.

[2649] The "means of providing" refers to a mechanism for displaying or conveying appropriate learning content or recommended content to users.

[2650] "Means for analyzing voice and text in real time and generating feedback" refers to a system that instantly analyzes a user's voice input or text input and generates appropriate responses or advice based on the results.

[2651] "Means for recognizing emotions" refers to technology for identifying a user's emotions by analyzing the user's facial expressions, voice, behavior, etc.

[2652] The "means for generating emotion-based feedback" is a device or system that automatically creates and provides feedback according to the recognized emotion of the user.

[2653] A "recommendation algorithm" is a computational method for selecting optimal content based on a user's emotional state and usage history.

[2654] "Recommended content" refers to content such as movies, dramas, music, documentaries, etc. that is provided in accordance with the user's emotional state and interests.

[2655] An "interface for facilitating viewing" is a user interface designed to allow a user to comfortably study or view content.

[2656] "Emotional state" is data that indicates the user's mental and emotional state, and is inferred from facial expressions, tone of voice, content of statements, etc.

[2657] The present invention provides a system for recommending appropriate content based on a user's emotional state, thereby improving the user's viewing experience. A detailed configuration of a system for realizing this application example will be described below.

[2658] System Overview

[2659] The system mainly consists of the following components:

[2660] 1. User information registration and authentication

[2661] 2. Emotion Recognition and Analysis

[2662] 3. Content Recommendation Algorithm

[2663] 4. Generating and displaying feedback

[2664] 5. Data storage and management

[2665] Hardware and software used

[2666] The main hardware and software used in this system are as follows:

[2667] Hardware: Smartphone camera, microphone

[2668] software:

[2669] Emotion Recognition: Google Cloud Vision API or Amazon Rekognition

[2670] Database: Firebase Realtime Database or AWS DynamoDB

[2671] Backend: Node.js or Python (Serverless environment: AWS Lambda or Google Cloud Functions)

[2672] Frontend: React Native (smartphone application)

[2673] User registration and authentication

[2674] When a user launches the app for the first time, an account is created and user authentication is performed using Firebase Auth or similar. Specifically, the user enters their name, email address, and password, which are then sent to the server. The server stores this information in a database and sends a registration completion response to the device. From the next time onwards, the user can log in using their email address and password.

[2675] Emotion Recognition and Analysis

[2676] The user's emotional state is recognized in real time through the smartphone's camera and microphone. For example, the Google Cloud Vision API is used to analyze emotions from the captured user's facial expressions. This emotional data is sent to the server and used in the next step.

[2677] Example prompt sentence:

[2678] def detect_emotions(image):

[2679] client = vision.ImageAnnotatorClient()

[2680] response = client.face_detection(image=image)

[2681] faces = response.face_annotations

[2682] emotions = {

[2683] 'joy': faces[0].joy_likelihood,

[2684] 'sorrow': faces[0].sorrow_likelihood,

[2685] 'anger': faces[0].anger_likelihood,

[2686] 'surprise': faces[0].surprise_likelihood,

[2687] }

[2688] return emotions

[2689] Content recommendation algorithm

[2690] The server then uses the collected emotional data to select the most suitable content using a recommendation algorithm, which takes into account the user's past viewing history and current emotional state to select the most suitable content for the user in real time.

[2691] Generating and displaying feedback

[2692] Along with the selected content, feedback based on the user's emotional data is also generated and provided to the user. For example, if the system detects that the user is feeling stressed, it will recommend relaxing content and send positive messages such as "Relax and have fun."

[2693] Data storage and management

[2694] After viewing, the app recognizes the user's emotional state again and stores that data in a database. This allows past data to be used in the next content recommendation, providing a more personalized experience. Users can also check their viewing history and emotional log at any time within the app.

[2695] For example, if a user inputs "I want to relax" into an emotion recognition system, the system will recommend relaxing music or nature documentaries. If the user is detected as being in a "happy mood," the system will recommend comedy movies or fun short videos with positive messages.

[2696] In this way, the system can provide the most appropriate content depending on the user's emotional state, improving the viewer's experience.

[2697] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2698] Step 1:

[2699] The user launches the app for the first time and begins the process of registering.

[2700] Input: Name, Email Address, Password

[2701] Action: The user enters the required information and clicks the submit button.

[2702] Output: The device sends this information to the server.

[2703] Step 2:

[2704] The server stores the received user information in a database.

[2705] Input: Registration information entered by the user

[2706] Operation: The server saves the registration information in a database and sends a registration completion response to the terminal.

[2707] Output: The terminal displays a message to the user that registration is complete.

[2708] Step 3:

[2709] The user enters their email address and password and clicks the login button.

[2710] Input: Email address, password

[2711] Operation: The terminal sends the entered information to the server, which then verifies it by comparing it with the registered information in the database.

[2712] Output: If authentication is successful, the server will retrieve the user's learning progress data and send it to the terminal, and the terminal will display the user's dashboard along with a login success message.

[2713] Step 4:

[2714] The user's face is captured by a camera and emotion recognition is performed.

[2715] Input: User's face image

[2716] How it works: The device uses the camera to capture the user's face and performs emotion analysis using the Google Cloud Vision API.

[2717] Output: Sends emotion data (e.g., joy, sadness, anger, surprise) to the server.

[2718] Step 5:

[2719] The server uses a recommendation algorithm to select the most suitable content based on the emotional data.

[2720] Input: Emotion data, user's past viewing history

[2721] How it works: The server runs a recommendation algorithm to select the most relevant content for the user.

[2722] Output: Sends the recommended content list to the device.

[2723] Step 6:

[2724] The terminal displays the recommended content list to the user.

[2725] Input: Recommended Content List

[2726] How it works: The device displays the recommended content to the user in a list format.

[2727] Output: The user selects the content they want to watch.

[2728] Step 7:

[2729] The user views the selected content.

[2730] Input: Selected content

[2731] Actions: The user uses the device to play and watch the selected content.

[2732] Output: The device collects content viewing progress data and emotion data.

[2733] Step 8:

[2734] The user's emotional changes are captured again by camera and emotion analysis is performed.

[2735] Input: User's face image (recaptured)

[2736] How it works: The device recaptures the user's face after viewing and performs emotion analysis using the Google Cloud Vision API.

[2737] Output: Emotional data after viewing is sent to the server.

[2738] Step 9:

[2739] The server stores emotion data before and after viewing and viewing progress data in a database.

[2740] Input: Emotional data before and after viewing, viewing progress data

[2741] How it works: The server stores this data in a database and uses it in the next recommendation algorithm.

[2742] Output: Updated user data in the database

[2743] Step 10:

[2744] Users can check their viewing history and emotion log from the menu displayed within the app.

[2745] Input: User request

[2746] How it works: The device sends a request to the server, which retrieves viewing history and emotion logs from the database.

[2747] Output: The terminal displays the acquired data to the user.

[2748] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2749] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2750] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2751] [Fourth embodiment]

[2752] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2753] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2754] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2755] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2756] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2757] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2758] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2759] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2760] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2761] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2762] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2763] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2764] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2765] The system of the present invention allows users to learn English efficiently and effectively by registering and providing appropriate learning content based on their progress. Below, we will explain in natural language how the program of this system works.

[2766] User Registration and Login

[2767] User Registration:

[2768] The user launches the app for the first time and clicks the new registration button.

[2769] The terminal prompts the user to enter information such as name, email address, password, and grade.

[2770] When the user enters the information and clicks the send button, the terminal sends this information to the server.

[2771] The server stores the received information in a database and notifies the terminal that registration is complete.

[2772] The terminal displays a message to the user indicating that registration is complete.

[2773] Login:

[2774] The user enters their email address and password on the login screen and clicks the login button.

[2775] The terminal transmits the input information to the server.

[2776] The server checks the information against its database and authenticates it.

[2777] If the authentication is successful, the server obtains the user's learning progress data and sends it to the terminal.

[2778] The terminal displays a successful login message and displays the user's learning dashboard.

[2779] Start learning English

[2780] Select your learning content:

[2781] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[2782] The terminal sends this request to the server.

[2783] The server selects appropriate content based on the user's progress data and transmits it to the terminal.

[2784] The terminal displays the content and allows the user to begin learning.

[2785] Learning implementation:

[2786] The user progresses through the displayed scenarios and quizzes.

[2787] The device transmits the user's input (voice or text) to the server in real time.

[2788] The server analyzes the input data and generates appropriate feedback.

[2789] Send feedback to the device.

[2790] The terminal displays this feedback to the user, who then continues learning based on it.

[2791] Dialogue with AI

[2792] Start the conversation:

[2793] The user selects the option to start a conversation with "AI Tomo."

[2794] The terminal notifies the server of this.

[2795] The server selects an appropriate interaction scenario and sends an initial message to the terminal.

[2796] The terminal will display or audibly convey this message to the user.

[2797] Continuing the dialogue:

[2798] The user responds verbally to the initial message.

[2799] The terminal transmits this audio to the server.

[2800] The server performs speech analysis and generates an appropriate response.

[2801] Send the response to the device.

[2802] The terminal communicates the response to the user and continues the dialogue.

[2803] Management of learning data

[2804] Save the training results:

[2805] When the user finishes learning, the terminal transmits the learning results to the server.

[2806] The server stores the results in a database and uses them the next time the learning is performed.

[2807] View your learning history:

[2808] When a user wants to check the learning history, the terminal sends a request to the server.

[2809] The server acquires the history data and transmits it to the terminal.

[2810] The terminal displays this to the user.

[2811] Provision of additional services (programming education)

[2812] New service information:

[2813] The server generates a guide for programming education for users who meet certain criteria and transmits it to the terminal.

[2814] The terminal displays this information to the user as a notification or pop-up.

[2815] Register for a new service:

[2816] If the user wishes to register, the terminal prompts the user to enter the necessary information and transmits it to the server.

[2817] The server stores the registration information and sends a registration completion message to the terminal.

[2818] The terminal notifies the user that the new service is ready.

[2819] In this way, the system of the present invention provides an interactive learning experience tailored to each user's individual progress, and can support English learning efficiently and effectively. By continuously managing users' learning data and providing additional services such as programming education as needed, the system provides comprehensive educational support.

[2820] The processing flow will be explained below.

[2821] User Registration and Login

[2822] User Registration:

[2823] Step 1:

[2824] The user launches the app for the first time and clicks the Sign Up button.

[2825] Step 2:

[2826] The terminal displays a registration screen that prompts the user to enter their name, email address, password, and grade.

[2827] Step 3:

[2828] The user enters the required information and clicks the submit button.

[2829] Step 4:

[2830] The terminal transmits the input information to the server.

[2831] Step 5:

[2832] The server stores the received user information in a database.

[2833] Step 6:

[2834] The server sends a registration completion response to the terminal.

[2835] Step 7:

[2836] The terminal displays a message to the user indicating that registration is complete.

[2837] Login:

[2838] Step 1:

[2839] The user enters their email address and password and clicks the Login button.

[2840] Step 2:

[2841] The terminal transmits the input information to the server.

[2842] Step 3:

[2843] The server authenticates the user by checking the registration information in the database.

[2844] Step 4:

[2845] If the authentication is successful, the server acquires the user's learning progress data and transmits it to the terminal.

[2846] Step 5:

[2847] The terminal will display the user's learning dashboard along with a successful login message.

[2848] Start learning English

[2849] Select your learning content:

[2850] Step 1:

[2851] The user selects learning content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[2852] Step 2:

[2853] The terminal sends a request for the selected learning content to the server.

[2854] Step 3:

[2855] The server selects appropriate study content based on the user's progress data and transmits it to the terminal.

[2856] Step 4:

[2857] The terminal displays the selected study content and allows the user to begin studying.

[2858] Learning implementation:

[2859] Step 1:

[2860] The user proceeds through the displayed English conversation scenarios and vocabulary quizzes.

[2861] Step 2:

[2862] The device transmits the user's input (voice or text) to the server in real time.

[2863] Step 3:

[2864] The server analyzes the received data using voice recognition and natural language processing.

[2865] Step 4:

[2866] The server generates appropriate feedback from the analysis results and sends it to the terminal.

[2867] Step 5:

[2868] The terminal displays or conveys the feedback from the server to the user.

[2869] Dialogue with AI

[2870] Start the conversation:

[2871] Step 1:

[2872] The user selects the option to start a conversation with "AI Tomo."

[2873] Step 2:

[2874] The terminal sends a request to start a conversation to the server.

[2875] Step 3:

[2876] The server selects an appropriate dialogue scenario and sends an initial message (e.g., Hello! How are you today?) to the terminal.

[2877] Step 4:

[2878] The terminal displays or speaks an initial message to the user.

[2879] Continuing the dialogue:

[2880] Step 1:

[2881] The user responds verbally to the initial message.

[2882] Step 2:

[2883] The terminal transmits the user's voice to the server.

[2884] Step 3:

[2885] The server uses voice recognition technology to analyze and evaluate the user's responses.

[2886] Step 4:

[2887] The server generates and sends appropriate feedback and next interaction messages to the terminal.

[2888] Step 5:

[2889] The terminal displays or speaks feedback and the next interaction message to the user.

[2890] Management of learning data

[2891] Save the training results:

[2892] Step 1:

[2893] The user finishes learning.

[2894] Step 2:

[2895] The terminal transmits the learning result to the server.

[2896] Step 3:

[2897] The server stores the received learning results in a database.

[2898] View your learning history:

[2899] Step 1:

[2900] The user sends a request to check the learning history.

[2901] Step 2:

[2902] The terminal sends a request for the learning history to the server.

[2903] Step 3:

[2904] The server acquires the user's learning history from the database and transmits it to the terminal.

[2905] Step 4:

[2906] The terminal displays the learning history to the user.

[2907] Provision of additional services (programming education)

[2908] New service information:

[2909] Step 1:

[2910] The server generates information data for programming education services for users whose English conversation studies meet a certain standard.

[2911] Step 2:

[2912] The terminal receives the guidance data from the server and displays it to the user as a notification or a pop-up.

[2913] Register for a new service:

[2914] Step 1:

[2915] A user becomes interested in a programming education service and wishes to register.

[2916] Step 2:

[2917] The terminal prompts the user to enter the necessary information and transmits it to the server.

[2918] Step 3:

[2919] The server stores the entered registration information in a database and sends a notice to the terminal informing the user that the new service can be used.

[2920] Step 4:

[2921] The terminal displays a guide to the user on how to start using the new service, and allows the user to start learning the programming education service.

[2922] This allows users to efficiently learn English and also receive programming education.

[2923] Example 1

[2924] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2925] In modern society, there is a high demand for English language learning, but efficient and effective learning requires personalized learning support and advanced feedback functions. Furthermore, to encourage continued learning, users need interactive learning experiences and additional educational services. Conventional systems often lack the functionality to provide appropriate learning content based on individual users' progress, analyze and provide feedback in real time, and effectively introduce new educational services. This results in problems that limit the efficiency and effectiveness of users' learning.

[2926] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2927] In this invention, the server includes means for registering and saving user information, means for authenticating the registered information and managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for displaying or communicating the generated feedback to the user, means for the user to start and continue a dialogue with the "AI," means for saving the user's learning results and using them the next time they study, and means for generating information about new services and notifying the user. This enables the provision of accurate content and feedback based on the user's individual learning progress, and also enables effective information about and registration for new educational services.

[2928] "User Information" refers to personal identification information provided by a user at the time of registration, including name, email address, password, grade level, etc.

[2929] The "storage means" is a means for recording user information and learning progress data and storing them in a database or the like.

[2930] "Authentication means" refers to a process or system for verifying whether a user is legitimate based on input user information.

[2931] "Study progress" refers to the history, results, and progress of a user's learning activities.

[2932] "Learning content" refers to educational materials such as learning materials, scenarios, quizzes, and simulations that are provided for users to study.

[2933] An "analysis means" is a process or system that analyzes input voice or text data and generates appropriate feedback.

[2934] "Feedback" is a response that includes evaluation, advice, and corrections to the user's learning activities.

[2935] "Interaction means" refers to a process or system where a user continuously interacts with an artificial intelligence.

[2936] "Learning results" refers to data on the results, grades, and achievement levels obtained after a user uses learning content.

[2937] "Notification means" refers to means such as alerts, pop-ups, and emails that notify users of new services and information.

[2938] This invention is a system that provides appropriate learning content based on the user's progress, enabling efficient and effective English learning. This system is composed of means for registering and saving user information, means for managing the user's learning progress, means for selecting and providing appropriate learning content based on the user's progress, means for analyzing voice and text in real time and generating feedback, means for the user to initiate and continue a dialogue with an "AI," means for saving the learning results and using them the next time the user studies, and means for generating information about new services and notifying the user.

[2939] Configuration and Operation Procedures

[2940] User Registration and Login

[2941] User Registration:

[2942] (User) launches the app for the first time and clicks the new registration button.

[2943] (Terminal) prompts the user to enter information such as name, email address, password, and grade.

[2944] The server stores this information in a database (e.g., MySQL) and notifies the terminal of a registration completion message.

[2945] Login:

[2946] (User) enters email address and password on the login screen.

[2947] The server compares the information in the database and authenticates the user. If authentication is successful, it acquires the user's learning progress data and sends it to the device.

[2948] Select learning content and start learning English

[2949] Select your learning content:

[2950] The user selects content such as "English conversation simulation" or "vocabulary quiz" from the learning menu.

[2951] The server selects appropriate content based on the user's progress data and sends it to the terminal.

[2952] The terminal displays the selected content to the user.

[2953] Learning implementation:

[2954] (User) progresses through scenarios and quizzes.

[2955] The (server) analyzes the user's input (voice or text) in real time (e.g., Google Cloud Speech-to-Text) and generates ap...

Claims

1. A means for registering and storing user information; a means for authenticating the registered information and managing the user's learning progress; a means for selecting and providing appropriate learning content based on the user's progress; A means of analyzing voice and text in real time and generating feedback; The system includes means for displaying or communicating said generated feedback to a user.

2. The system of claim 1 , further comprising: an interface for enabling a user to advance their learning by utilizing the appropriate learning content.

3. 2. The system according to claim 1, wherein the user's learning results are saved and the progress status is updated based on the saved results the next time the user studies.

Images