System

Abstract

To provide a system which enables a learner to enjoy and effectively learn English conversation.SOLUTION: The system includes a means for providing an interface for inputting user enrollment information, a means for generating a synthetic voice using a voice sample of an actor or character selected by the user, a means for preparing English conversation content for the user and creating a conversation scenario, a means for displaying a 3D model in a virtual reality context, a means for converting user utterances into text using voice recognition technology, a means for generating a natural dialogue based on user utterances, a means for evaluating dialogue results and providing feedback to the user, and a means for providing a continuous learning scenario and performing subscription management.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional English conversation learning methods have the problem of making it difficult for learners to maintain their motivation. In particular, with self-study materials and online courses, learners tend to get bored easily because they have limited conversation partners. It is also difficult to get real-time feedback when practicing English conversation. Given these issues, there is a need for a method that allows learners to acquire English conversation skills in an enjoyable and effective way. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for providing an interface for entering user registration information, a means for generating synthetic speech using voice samples of actors or characters selected by the user, a means for preparing English conversation content for the user and creating a conversation scenario, a means for displaying a 3D model in a virtual reality environment, a means for converting the user's utterances into text using speech recognition technology, a means for generating natural dialogue based on the user's utterances, a means for evaluating the dialogue results and providing feedback to the user, and a means for providing continuous learning scenarios and managing subscriptions. This system allows users to learn English while enjoying conversations with their favorite actors or characters, and the real-time feedback enables effective learning. Furthermore, the use of a virtual reality environment allows users to practice in a manner that is close to real-life situations, making it easier to maintain learner motivation.

[0006] An "interface for entering user registration information" is a means of providing a screen or form for a user to enter necessary personal information and preferences.

[0007] "Synthetic voice" is voice data that is artificially generated based on voice samples of an actor or character selected by the user.

[0008] "Means for preparing English conversation content and creating conversation scenarios" refers to technology that sets up specific English conversation lessons and situations within the system and generates scenarios based on the user's learning progress and level.

[0009] "Means for displaying 3D models in a virtual reality environment" refers to technology that allows a user to use VR goggles or a compatible device to display a three-dimensional model of a selected actor or character in a virtual reality space.

[0010] "Speech recognition technology" is a technology that converts words spoken by a user from digital voice data into text data.

[0011] "Means for generating natural dialogue" refers to a technique for creating conversation content that allows a selected actor or character to respond naturally to a user's utterances.

[0012] "Means for evaluating dialogue results and providing feedback to the user" is a function that analyzes the user's English conversation skills and provides the user with advice and areas for improvement based on the evaluation results of pronunciation, grammar, etc.

[0013] The "means for providing continuous learning scenarios and managing subscriptions" refers to a system that provides new lesson plans to continuously support users' learning and manages periodic billing processes. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention relates to a VR system that allows a user to receive English conversation coaching from a favorite actor or character. Specific embodiments for carrying out the present invention will be described below.

[0036] User Registration and Login

[0037] 1. The user installs and launches the application.

[0038] 2. The device displays an interface for the user to enter their name, email address, and favorite actor or character.

[0039] 3. The user enters the necessary information and presses the send button, and the device sends the information to the server.

[0040] 4. The server saves the user's registration information in the database. The registered email address and password are used for subsequent logins.

[0041] Preparation for synthetic voice and conversational AI

[0042] 5. Once the user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user.

[0043] 6. The server uses conversational AI to prepare an appropriate English conversation scenario for the user, which is customized based on the user's learning history and current progress.

[0044] English conversation coaching in a VR environment

[0045] 7. The device will load a virtual reality environment and display a 3D model of the selected actor or character.

[0046] 8. Users can enter a virtual environment using VR goggles and controllers and begin English conversations with actors and characters using synthesized voices.

[0047] 9. The device uses voice recognition technology to convert the user's speech into text and transmits it to the server in real time.

[0048] 10. The server analyzes the received text, uses conversational AI to generate natural dialogue, and sends the next actor or character response to the device.

[0049] 11. This allows users to practice English conversation in situations that are close to real life situations.

[0050] Feedback and continuous learning

[0051] 12. After the lesson is completed, the device sends the dialogue data to the server.

[0052] 13. The server uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates the results as feedback.

[0053] 14. The server sends the generated feedback to the device, which displays it to the user, allowing the user to see how their English conversation skills have improved.

[0054] 15. When the next learning session begins, the server generates a new conversation scenario and sends it to the device.

[0055] Subscription Management

[0056] 16. The server periodically checks the subscription status and automatically charges the monthly fee.

[0057] 17. When a subscription is about to expire, the server sends reminder notifications to users and takes them through the renewal process if necessary.

[0058] Example: When a user uses the system for the first time, he or she first enters the name "John" and email address on the registration screen and selects "Actor X" as the preferred actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares a scenario for ordering at a restaurant. John enters the VR environment and begins a conversation with Actor X, saying, "Hello, John! How can I help you today?" After the conversation ends, John receives feedback on his pronunciation and grammar and checks his or her skills.

[0059] In this way, users can continue to learn English conversation in a fun and enjoyable way.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] The user installs and launches the application.

[0063] Step 2:

[0064] The device displays an interface for the user to select their name, email address, and favorite actor or character.

[0065] Step 3:

[0066] The user enters the necessary information and presses the send button.

[0067] Step 4:

[0068] The terminal transmits the user's input information to the server.

[0069] Step 5:

[0070] The server stores the received information in a database and returns a message indicating that the information was successfully stored to the terminal.

[0071] Step 6:

[0072] The terminal will display a successful save message to the user.

[0073] Step 7:

[0074] The user enters their email address and password on the login screen and clicks the login button.

[0075] Step 8:

[0076] The terminal transmits the entered login information to the server.

[0077] Step 9:

[0078] The server retrieves user information from the database and performs authentication.

[0079] Step 10:

[0080] If the authentication is successful, the server returns a message indicating that the authentication is successful to the terminal.

[0081] Step 11:

[0082] The device will notify the user that the login was successful and begin the process of loading the VR environment.

[0083] Step 12:

[0084] The server uses synthetic voice AI to generate voice data based on voice samples of the actor or character selected by the user.

[0085] Step 13:

[0086] The server uses conversational AI to prepare English conversation content and scenarios based on the user's learning level and history.

[0087] Step 14:

[0088] The server transmits the generated voice data and conversation scenario to the terminal.

[0089] Step 15:

[0090] The device loads 3D models into the VR environment and displays the model of the selected actor or character.

[0091] Step 16:

[0092] The user puts on VR goggles and controllers and enters the virtual environment.

[0093] Step 17:

[0094] The device plays a synthesized voice, with a character saying, "Hello, [user]! How are you today?"

[0095] Step 18:

[0096] The user presses a button on the controller to reply, "I'm good, thank you! How are you?"

[0097] Step 19:

[0098] The device converts the user's speech into text using voice recognition technology and sends the text to the server.

[0099] Step 20:

[0100] The server uses conversational AI to generate appropriate responses to the user's comments and sends those responses to the device.

[0101] Step 21:

[0102] The device then plays the received response as a synthesized voice of the character and continues the conversation.

[0103] Step 22:

[0104] This process repeats until the user ends the conversation.

[0105] Step 23:

[0106] After the conversation is completed, the terminal transmits the dialogue data to the server.

[0107] Step 24:

[0108] The server uses conversational AI to evaluate the dialogue data and generate feedback on items such as pronunciation, grammar, and fluency.

[0109] Step 25:

[0110] The server transmits the generated feedback to the terminal.

[0111] Step 26:

[0112] The terminal displays feedback to the user, allowing them to confirm the learning results.

[0113] Step 27:

[0114] The next time you study, the server will generate a new English conversation scenario and send it to your device.

[0115] Step 28:

[0116] The server periodically checks the subscription status and bills the monthly fee.

[0117] Step 29:

[0118] When the subscription is about to expire, the server will send a renewal reminder to the device.

[0119] Step 30:

[0120] The device will display an update notification to the user and prompt them to take the necessary steps.

[0121] Example 1

[0122] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0123] Conventional English conversation learning systems often lack the realism required for users to practice English conversation in real situations. They also lack regular feedback and sufficient evaluation of pronunciation and grammar, making it difficult for users to continuously improve their skills. Furthermore, subscription management is often done manually, placing a heavy burden on users.

[0124] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0125] In this invention, the server includes means for providing an interface for inputting user registration information, means for generating synthetic speech using a voice sample of a character selected by the user, means for preparing English conversation content for the user and creating a conversation scenario, means for displaying a three-dimensional model in a virtual reality environment, means for converting the user's utterances into text using voice recognition technology, means for generating natural dialogue based on the user's utterances, means for evaluating the dialogue results and providing feedback to the user, means for providing continuous learning scenarios and managing subscriptions, and means for periodically checking the subscription status and automatically charging fees. This allows users to practice English conversation in realistic situations and improve their skills by receiving continuous feedback and evaluation, while also reducing the burden of subscription management.

[0126] The "interface for entering user registration information" is a graphical input screen that allows a user to enter their name, email address, and other required information.

[0127] The "means for generating synthetic voice" is a mechanism for converting text data into voice data based on a voice sample of a character selected by the user.

[0128] "Means for creating conversation scenarios" refers to a method for automatically generating appropriate English conversation scenes and situations based on the user's learning history and progress.

[0129] "Means for displaying a three-dimensional model in a virtual reality environment" refers to a method of displaying a character selected by the user in a three-dimensional virtual space using VR goggles or a display.

[0130] "Means for converting user speech into text using speech recognition technology" refers to technology that analyzes the user's speech in real time and converts it into corresponding text data.

[0131] "Means for generating natural dialogue" refers to a method for using conversational AI to generate natural-flowing dialogue based on user input.

[0132] "Means for evaluating the dialogue results and providing feedback to the user" refers to a mechanism that analyzes the content of the dialogue with the user, evaluates pronunciation, grammar, fluency, etc., and notifies the user of the results.

[0133] "Means for providing continuous learning scenarios" refers to a method for continuously providing new learning content based on the user's progress and feedback results.

[0134] A "means for managing subscriptions" is a mechanism for checking the user's subscription status and periodically billing the service fee.

[0135] An "automatic billing mechanism" is a system that automatically processes payment of the subscription fee during the subscription period.

[0136] MODE FOR CARRYING OUT THE INVENTION

[0137] The present invention relates to a VR system that allows a user to enjoy learning English conversation while receiving coaching from a character selected by the user. Specific embodiments for carrying out the present invention will be described below.

[0138] User Registration and Login

[0139] First, the user installs and launches the application. At this stage, the device displays an interface for the user to enter their name, email address, and favorite character. After the user enters the required information and presses the submit button, the device sends the information to the server. The server stores the user's registration information in a database, and the registered email address and password are used for subsequent logins.

[0140] Preparation for synthetic voice and conversational AI

[0141] When a user logs in, the server generates synthetic speech based on the voice samples of the character selected by the user. Specifically, a speech synthesis engine (e.g., Google Cloud Text-to-Speech or Amazon Polly) is used. The server also uses a conversational AI model (e.g., OpenAI's GPT-3) to prepare an English conversation scenario appropriate for the user. This scenario is customized based on the user's learning history and current progress.

[0142] English conversation coaching in a VR environment

[0143] Next, the device loads a virtual reality environment (VR environment) and displays a three-dimensional model of the selected character. VR goggles are mainly used, such as Oculus Rift or HTC Vive. The user enters the virtual environment using the VR goggles and controllers and can begin English conversation with the character using synthesized voice. The device converts the user's speech into text using voice recognition technology (e.g., Google Speech-to-Text) and sends it to the server in real time. The server analyzes the received text, generates natural dialogue using conversational AI, and sends the character's next response to the device. This allows the user to practice English conversation in a manner that is close to real-life situations.

[0144] Feedback and continuous learning

[0145] After the lesson is completed, the device sends the dialogue data to the server. The server uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results. The generated feedback is displayed to the user via the device, allowing the user to see areas where their English conversation skills have improved. When the next lesson begins, the server generates a new dialogue scenario and sends it to the device.

[0146] Subscription Management

[0147] The server periodically checks the subscription status and automatically bills the monthly fee. When the subscription expiration date approaches, the server sends a reminder to the user and takes care of the renewal procedure if necessary.

[0148] Specific examples

[0149] For example, when a user uses the system for the first time, they enter their name "John" and email address on the registration screen and select "Character X" as their favorite character. After logging in, the server generates a synthesized voice for "Character X" and prepares a scenario for ordering at a restaurant. Once John enters the VR environment, he begins a conversation with Character X, saying, "Hello, John! How can I help you today?" After the conversation ends, John receives feedback on his pronunciation and grammar and checks his skills.

[0150] Prompt Sentence Examples

[0151] "Please explain the details of the VR system that allows users to receive English conversation coaching with their favorite characters."

[0152] Thus, the present invention provides a system that allows users to practice English conversation in realistic situations and improve their skills through continuous feedback and evaluation.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Step 1:

[0155] The user installs and launches the application. After installation, by opening the application, the device displays an interface for user registration.

[0156] Input: The user installs and launches the app.

[0157] Output: The user registration interface is displayed.

[0158] Step 2:

[0159] The terminal provides the user with a form to enter their name, email address, and favorite character. The user enters the required information into the form and presses the submit button.

[0160] Input: A user fills out a form with their name, email address, and favorite character.

[0161] Output: The entered information is sent to the server.

[0162] Step 3:

[0163] The terminal sends the user information to the server, which stores the information in a database.

[0164] Input: User information sent from the device.

[0165] Output: User information stored on the server.

[0166] Step 4:

[0167] When a user logs in, the server generates a synthesized voice based on a voice sample of the character selected by the user.

[0168] Input: User login information and selected character.

[0169] Output: Synthetic speech data is generated.

[0170] Step 5:

[0171] The server uses a conversational AI model to prepare a suitable English conversation scenario for the user, which is customized based on the user's learning history and current progress.

[0172] Input: User learning history and progress data.

[0173] Output: A customized English conversation scenario.

[0174] Step 6:

[0175] The device loads a virtual reality environment and displays a three-dimensional model of the selected character. The user enters the virtual environment using VR goggles and controllers.

[0176] Input: English conversation scenario and character 3D model data.

[0177] Output: The character displayed in a VR environment.

[0178] Step 7:

[0179] The device converts the user's speech into text using voice recognition technology and sends it to the server in real time. Google Speech-to-Text is used for voice recognition.

[0180] Input: User speech.

[0181] Output: Textualized speech data.

[0182] Step 8:

[0183] The server analyzes the textual data and uses conversational AI to generate natural dialogue, which is then sent to the device.

[0184] Input: Text data of user utterances.

[0185] Output: Natural dialogue.

[0186] Step 9:

[0187] The device displays the generated dialogue, and the user continues the conversation, allowing the user to practice English conversation in a realistic situation.

[0188] Input: The generated dialogue.

[0189] Output: The displayed dialogue.

[0190] Step 10:

[0191] The device sends the conversation data to a server, which uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results.

[0192] Input: Interaction data.

[0193] Output: Feedback data.

[0194] Step 11:

[0195] The server sends the generated feedback to the terminal, which displays it to the user, allowing the user to see how their English conversation skills have improved.

[0196] Input: Feedback data.

[0197] Output: The feedback displayed to the user.

[0198] Step 12:

[0199] At the start of the next learning session, the server generates a new conversation scenario and sends it to the terminal.

[0200] Input: User learning history and feedback.

[0201] Output: A new conversation scenario.

[0202] Step 13:

[0203] The server periodically checks the subscription status and automatically bills users, sending reminders when their subscription is about to expire.

[0204] Input: Subscription information.

[0205] Output: Automatic billing and reminder notifications.

[0206] (Application example 1)

[0207] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0208] In conventional English conversation learning systems, users have limited opportunities to practice English conversation in situations close to real conversations. Furthermore, there are no systems that can provide users with an engaging experience of receiving English conversation coaching from their favorite actors or characters, making it difficult to maintain motivation to continue learning.

[0209] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0210] In this invention, the server includes a means for generating synthetic speech based on voice samples of actors or characters selected by the user, a means for providing an application to be installed on a smartphone to execute the generated conversation scenario, and a means for generating new conversation scenarios based on the user's learning progress via the server. This allows users to practice English conversation in a manner that is close to actual conversation situations through real-time dialogue with their favorite actors or characters. Furthermore, the provision of new scenarios based on the user's learning progress allows the user to enjoy constantly fresh learning content, contributing to maintaining motivation for continuous learning.

[0211] 1. An "interface for entering user registration information" is a screen or form that provides a user with the ability to enter information such as their name, email address, and favorite actors or characters.

[0212] 2. "Means for generating synthetic speech" refers to a program or algorithm that generates speech data that enables natural dialogue based on voice samples of an actor or character selected by the user.

[0213] 3. "Means for creating conversation scenarios" refers to a system for designing and preparing appropriate English conversation content in advance based on the user's learning progress and preferences.

[0214] 4. "Means for displaying 3D models in a virtual reality environment" means a system for displaying 3D character or actor models based on a user's selection using a VR headset or display.

[0215] 5. "Means of converting user speech into text using speech recognition technology" refers to technology or programs that analyze what a user says in real time and convert it into text data.

[0216] 6. "Means for generating natural dialogue" refers to a conversational AI system that generates appropriate responses based on the user's textual utterances and provides them as natural conversation.

[0217] 7. "Means for providing an application to be installed on a smartphone" means a method or service for a user to download and install the software necessary to use the English conversation coaching system on a smartphone.

[0218] 8. "Means for generating new conversation scenarios based on the user's learning progress via a server" refers to a server-based system that analyzes the user's learning data and progress to generate and provide new English conversation scenarios.

[0219] 9. "Means for evaluating a user's skills based on feedback from a server and presenting the results in the form of a widget" refers to a method for evaluating a user's pronunciation, grammar, and fluency using feedback information sent from a server, and displaying the evaluation results as a widget on a smartphone.

[0220] 10. "Means for evaluating dialogue results and providing feedback to the user" refers to a mechanism that analyzes the content of the dialogue conducted by the user, evaluates it based on the accuracy of pronunciation and grammar, and returns the results to the user.

[0221] 11. "Means for providing continuous learning scenarios and managing subscriptions" means a system for operating a subscription service for which a recurring fee is charged, by periodically providing new learning scenarios to users and managing and supervising their usage.

[0222] The following hardware and software are required to implement this invention. The hardware required is a smartphone connected to the internet, and VR goggles and controllers may be used as options. The software includes Python, PyQt5, requests, speech_recognition, and pyttsx3.

[0223] When a user installs and launches the application, an interface for entering user registration information is displayed. In this interface, the user enters their name, email address, and information about their favorite actors and characters. This information is sent to the server. The server creates a user profile based on the received user information and stores the registration information in a database.

[0224] When a user logs in, the server generates a synthesized voice based on the voice samples of the actor or character selected by the user. This synthesized voice is generated as voice data using the pyttsx3 library. The server also uses conversational AI to prepare an English conversation scenario suited to the user based on the user's learning history and progress. This allows the user to practice English conversation based on the scenario.

[0225] Next, users use a smartphone application to enter the virtual reality environment. The application works in conjunction with VR goggles and controllers to load the virtual environment and display a 3D model of the selected actor or character. Within the VR environment, users can begin English conversations with the actor or character using synthesized voice.

[0226] The user's speech is converted into text using voice recognition technology and sent to the server in real time. The server analyzes the received text data and generates natural dialogue using conversational AI. As a result, the next response of the actor or character is sent to the device, allowing the user to continue the dialogue in real time.

[0227] Once the conversation is over, the server uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results. This feedback is sent to the device and displayed to the user in the form of a widget, allowing the user to see where their skills have improved.

[0228] As a concrete example, when a user uses the system for the first time, they enter their name "John" and email address on the registration screen and select "Actor X" as their favorite actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares a scenario for ordering at a restaurant. John enters the VR environment and begins a conversation with "Actor X": "Hello, John! How can I help you today?" After the conversation ends, John receives feedback on his pronunciation and grammar and checks his skills.

[0229] Additionally, by inputting the following prompt sentences into the generative AI model, it is possible to generate an appropriate conversation scenario:

[0230] The user has selected "Actor X" as their favorite actor. Please generate natural dialogue to support the user in English conversation in a situation where they are ordering at a restaurant.

[0231] In this way, users can improve their English conversation skills while experiencing conversations with virtual actors and characters.

[0232] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0233] Step 1:

[0234] The user installs and launches the application. The device displays an interface for entering user registration information. The user enters their name, email address, and information about their favorite actor or character, and presses the submit button. The input data is sent to the server. The server receives the user information and stores it in a database.

[0235] Step 2:

[0236] The user logs in. The server verifies the user information and authenticates the login. If authentication is successful, a synthesized voice is generated based on the voice samples of the actor or character selected by the user. The server generates the voice data using the pyttsx3 library and sends the result to the device.

[0237] Step 3:

[0238] The server uses conversational AI to prepare an English conversation scenario based on the user's learning history and progress. A generative AI model is then used to generate a conversation scenario based on a specified prompt. The prompt, "The user has selected 'Actor X' as their favorite actor. Please generate a natural dialogue to support the user in English conversation in a situation where they are ordering at a restaurant," is input, and the AI ​​outputs a response scenario.

[0239] Step 4:

[0240] The device loads a virtual reality environment and displays a 3D model of the selected actor or character. The user enters the virtual environment using VR goggles and controllers. The device plays synthesized voice data from the server and begins a conversation in English with the user. The user's speech is input through a microphone.

[0241] Step 5:

[0242] The device uses voice recognition technology to convert the user's speech into text in real time. The text data is then sent to a server. The server then analyzes the received text data and uses conversational AI to generate natural dialogue. The server then generates the next actor or character's response and sends the resulting voice data to the device.

[0243] Step 6:

[0244] The user continues to have real-time conversations with actors and characters using synthesized speech. The device plays back the voice data sent from the server, providing the user with a realistic conversation experience. Every time the user speaks, speech recognition and text conversion are repeated.

[0245] Step 7:

[0246] Once the conversation is complete, the device sends the conversation data to the server, which uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results. The generated feedback is sent to the device and presented to the user in the form of a widget.

[0247] Step 8:

[0248] When the next learning session begins, the server generates a new conversation scenario based on the user's learning progress. It then uses the generative AI model again to generate a conversation scenario based on the new prompt. The server then sends the generated conversation scenario to the device, providing a continuous learning scenario.

[0249] Step 9:

[0250] The server periodically checks the subscription status and automatically bills the monthly fee. When the subscription expiration date approaches, the server sends a reminder to the user and takes care of the renewal procedure if necessary.

[0251] These steps allow users to continuously improve their English skills by experiencing English conversations with virtual actors and characters. Furthermore, by generating conversation scenarios using a generative AI model, users can enjoy constantly new learning content.

[0252] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0253] The present invention combines an emotion engine with a VR system that allows users to receive English conversation coaching from their favorite actors or characters. Specific embodiments for carrying out the present invention will be described below.

[0254] User Registration and Login

[0255] 1. The user installs and launches the application.

[0256] 2. The device displays an interface for the user to enter their name, email address, and select their favorite actor or character.

[0257] 3. The user enters the necessary information and presses the send button, and the device sends the information to the server.

[0258] 4. The server saves the user's registration information in the database. The registered email address and password are used for subsequent logins.

[0259] Preparation of synthetic voice, conversational AI and emotion recognition engine

[0260] 5. Once the user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user.

[0261] 6. The server uses conversational AI to prepare an appropriate English conversation scenario for the user, which is customized based on the user's learning history and current progress.

[0262] 7. The server also provides an emotion engine that can recognize emotions from the user's words and facial expressions.

[0263] English conversation coaching in a VR environment

[0264] 8. The device will load a virtual reality environment and display a 3D model of the selected actor or character.

[0265] 9. Users can enter a virtual environment using VR goggles and controllers and begin English conversations with characters using synthesized voices.

[0266] 10. The device uses voice recognition technology to convert the user's speech into text and transmits it to the server in real time.

[0267] 11. The server analyzes the received text, uses conversational AI to generate natural dialogue, and sends the character's next response to the device.

[0268] 12. The emotion engine analyzes the user's voice and facial expression data to assess their emotional state in real time, adjusting the tone and content of the conversation accordingly.

[0269] 13. The device plays back the conversational responses as the character's synthesized voice and continues the dialogue with the user.

[0270] Feedback and continuous learning

[0271] 14. After the lesson is completed, the device sends the dialogue data to the server.

[0272] 15. The server uses conversational AI to evaluate the dialogue data and generate feedback on items including pronunciation, grammar, fluency, and emotion recognition.

[0273] 16. The server sends the generated feedback to the device, which displays it to the user. This feedback includes comments about the learning progress and emotional fluctuations.

[0274] 17. When the next learning session begins, the server generates a new English conversation scenario and sends it to the device.

[0275] Subscription Management

[0276] 18. The server periodically checks the subscription status and automatically charges the monthly fee.

[0277] 19. When a subscription is about to expire, the server sends reminder notifications to users and takes them through the renewal process if necessary.

[0278] Example: When a user uses the system for the first time, they first enter their name "John" and email address on the registration screen and select "Actor X" as their favorite actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares the emotion engine. A scenario of ordering at a restaurant is prepared, and John enters the VR environment and starts a conversation with Actor X, saying, "Hello, John! How are you today?" When John replies, "I'm good, thank you! How are you?", the emotion engine analyzes John's emotional state from his voice and facial expressions and adjusts the character's response. After the conversation ends, John receives feedback on his pronunciation, grammar, and emotional fluctuations to check his skills.

[0279] In this way, users can continue to learn English conversation in a fun and enjoyable way while receiving coaching in an approach that responds to emotional fluctuations.

[0280] The processing flow will be explained below.

[0281] Step 1:

[0282] The user installs and launches the application.

[0283] Step 2:

[0284] The device displays an interface for the user to select their name, email address, and favorite actor or character.

[0285] Step 3:

[0286] The user enters the necessary information and presses the send button.

[0287] Step 4:

[0288] The terminal transmits the user's input information to the server.

[0289] Step 5:

[0290] The server stores the received information in a database and returns a message indicating that the information was successfully stored to the terminal.

[0291] Step 6:

[0292] The terminal will display a successful save message to the user.

[0293] Step 7:

[0294] The user enters their email address and password on the login screen and clicks the login button.

[0295] Step 8:

[0296] The terminal transmits the entered login information to the server.

[0297] Step 9:

[0298] The server retrieves user information from the database and performs authentication.

[0299] Step 10:

[0300] If the authentication is successful, the server returns a message indicating that the authentication is successful to the terminal.

[0301] Step 11:

[0302] The device will notify the user that the login was successful and begin the process of loading the VR environment.

[0303] Step 12:

[0304] The server uses synthetic voice AI to generate voice data based on voice samples of the actor or character selected by the user.

[0305] Step 13:

[0306] The server uses conversational AI to prepare English conversation content and scenarios based on the user's learning level and history.

[0307] Step 14:

[0308] The server also has an emotion engine and is configured to recognize emotions in real time from the user's comments and facial expressions.

[0309] Step 15:

[0310] The server transmits the generated voice data, the conversation scenario, and the emotion engine settings to the terminal.

[0311] Step 16:

[0312] The device loads 3D models into the VR environment and displays the model of the selected actor or character.

[0313] Step 17:

[0314] The user puts on VR goggles and controllers and enters the virtual environment.

[0315] Step 18:

[0316] The device plays a synthesized voice, with a character saying, "Hello, [user]! How are you today?"

[0317] Step 19:

[0318] The user presses a button on the controller to reply, "I'm good, thank you! How are you?"

[0319] Step 20:

[0320] The device converts the user's speech into text using voice recognition technology and sends the text to the server.

[0321] Step 21:

[0322] The server uses conversational AI to generate appropriate responses to user comments.

[0323] Step 22:

[0324] The emotion engine analyzes the user's voice and facial expression data to assess their emotional state in real time.

[0325] Step 23:

[0326] The server adjusts the character's next response based on the emotional data evaluated by the emotion engine and sends that data to the terminal.

[0327] Step 24:

[0328] The device then plays the received response as a synthesized voice of the character and continues the conversation.

[0329] Step 25:

[0330] This process repeats until the user ends the conversation.

[0331] Step 26:

[0332] After the conversation is completed, the terminal transmits the dialogue data to the server.

[0333] Step 27:

[0334] The server uses conversational AI to evaluate the dialogue data and generate feedback based on pronunciation, grammar, fluency, and emotion recognition.

[0335] Step 28:

[0336] The server transmits the generated feedback to the terminal.

[0337] Step 29:

[0338] The terminal displays feedback to the user, allowing them to confirm the learning results.

[0339] Step 30:

[0340] The next time you study, the server will generate a new English conversation scenario and send it to your device.

[0341] Step 31:

[0342] The server periodically checks the subscription status and bills the monthly fee.

[0343] Step 32:

[0344] When the subscription is about to expire, the server will send a renewal reminder to the device.

[0345] Step 33:

[0346] The device will display an update notification to the user and prompt them to take the necessary steps.

[0347] Example 2

[0348] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0349] Conventional English conversation learning systems use fixed scenarios and general speech synthesis technology without taking into account the learner's emotional state, which limits the effectiveness of learning. Furthermore, it is difficult to provide scenarios tailored to individual learners, making it difficult to maintain continuous learning motivation. Furthermore, since pronunciation and grammar evaluation is not performed in real time, feedback tends to be delayed, making it difficult to improve practical conversational skills.

[0350] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0351] In this invention, the server includes: means for providing an interface for inputting user registration information; means for generating synthetic speech using voice samples of actors or characters selected by the user; means for preparing English conversation content for the user and creating a conversation scenario; means for displaying a 3D model in a virtual reality environment; means for converting user utterances into text using speech recognition technology; means for generating natural dialogue based on the user utterances; means for evaluating the dialogue results and providing feedback to the user; means for providing continuous learning scenarios and managing subscriptions; means for recognizing emotions from user utterances and facial expressions using an emotion recognition engine; and means for providing customized English conversation scenarios using a generative AI model. This enables more effective English conversation learning by providing scenarios and feedback in real time that take the learner's emotional state into account.

[0352] "User registration information" refers to information such as name, email address, password, favorite actor or character, etc., that a user enters to use an application.

[0353] An "interface" refers to the screen or operating means through which a user inputs information into a system or performs operations.

[0354] "Voice Sample" means the recorded or digitized voice of a particular actor or character.

[0355] "Synthetic voice" is an artificial voice that imitates a human voice and is created based on voice samples.

[0356] "English conversation content" refers to conversation themes, phrases, and scenarios provided for users to study.

[0357] A "conversation scenario" is a series of dialogues that assume a specific situation, and is the content of the conversation that takes place between the user and a virtual character.

[0358] "Virtual reality environment" means a visual and operational environment for displaying 3D models and for allowing a user to experience them in virtual reality.

[0359] A "3D model" is a character or object that is represented in three dimensions using computer graphics.

[0360] "Speech recognition technology" refers to technology that recognizes what a user says and converts the voice data into text data.

[0361] "Natural dialogue" refers to the exchange of fluent, natural sentences, like conversations between people.

[0362] "Feedback" refers to evaluations, advice, and suggestions for improvement given to users in response to their learning and operation results.

[0363] A "continuous learning scenario" is a learning plan or scenario that is provided step by step to enable the user to continue learning.

[0364] "Subscription management" is a system for managing users' contract status and payment fees for subscription-based services.

[0365] An "emotion recognition engine" is a system that analyzes and recognizes a user's emotional state from their speech and facial expressions.

[0366] A "generative AI model" is an artificial intelligence model that generates new data (in this case, English conversation scenarios) based on input data.

[0367] The present invention combines an emotion engine with a VR system that allows users to receive English conversation coaching from their favorite actors or characters. Specific embodiments for carrying out the present invention will be described below.

[0368] User Registration and Login

[0369] When a user installs and launches the application, the device displays an interface for the user to select their name, email address, and favorite actor or character. After the user enters the required information and presses the submit button, the device sends the information to the server. This data is stored in a database by the server. The registered email address and password are used for subsequent logins.

[0370] Preparation of synthetic voice, conversational AI and emotion recognition engine

[0371] When a user logs in, the server generates synthetic speech based on voice samples of the actor or character selected by the user. To achieve this, it uses OpenAI's Text-to-Speech technology. The server also uses conversational AI (GPT-3) to prepare an English conversation scenario suited to the user. This scenario is customized based on the user's learning history and current progress. The server also uses Affectiva's emotion recognition engine to recognize emotions from the user's speech and facial expressions.

[0372] English conversation coaching in a VR environment

[0373] The device loads a virtual reality environment and displays a 3D model of the selected actor or character. This virtual environment is built using Unity. Users enter the virtual environment using VR goggles and controllers and can begin English conversations with the character using synthetic voice. The device uses Google Speech-to-Text to convert the user's speech into text and send it to the server in real time. The server analyzes the received text, generates natural-sounding dialogue using GPT-3, and sends the character's next response to the device. The emotion engine analyzes the user's voice and facial expression data and evaluates their emotional state in real time. Based on this, the tone and content of the conversation are adjusted, and the device plays back the character's replies as synthetic voice, continuing the dialogue with the user.

[0374] Feedback and continuous learning

[0375] After completing a lesson, the device sends the dialogue data to the server. The server uses conversational AI to evaluate the dialogue data and generate feedback on items including pronunciation, grammar, fluency, and emotion recognition. This feedback is sent to the device and displayed to the user. The feedback also includes comments on learning progress and emotional fluctuations. When starting the next lesson, the server generates a new English conversation scenario and sends it to the device.

[0376] Subscription Management

[0377] The server periodically checks the subscription status and automatically bills the monthly fee. When the subscription expiration date approaches, the server sends a reminder to the user and takes care of the renewal procedure if necessary.

[0378] Examples:

[0379] When a user first uses the system, they first enter their name ("John") and email address on the registration screen and select "Actor X" as their preferred actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares the emotion engine. A scenario of ordering at a restaurant is prepared, and John, entering the VR environment, begins a conversation with Actor X: "Hello, John! How are you today?" When John replies, "I'm good, thank you! How are you?", the emotion engine analyzes John's emotional state from his voice and facial expressions and adjusts the character's response accordingly. After the conversation ends, John receives feedback on his pronunciation, grammar, and emotional fluctuations to check his skills. In this way, users can continuously and enjoyably learn English conversation while receiving coaching tailored to their emotional fluctuations.

[0380] Example prompt for a generative AI model:

[0381] "Generate an English conversation scenario in which a user orders from actor X at a restaurant. Please include the flow of the conversation when the user has positive emotions."

[0382] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0383] Step 1:

[0384] The user installs and launches the application.

[0385] How it works: A user downloads an app from the app store on their smartphone, tablet, or PC, installs it, and then launches it. They then tap the icon of the installed app to launch it.

[0386] Step 2:

[0387] The device displays an interface for the user to select their name, email address, and favorite actor or character.

[0388] Input: None

[0389] Output: User input form

[0390] Operation details: The app screen displays a form for entering name, email address, password, and favorite actor or character.

[0391] Step 3:

[0392] The user enters the necessary information and presses the send button, and the terminal sends the information to the server.

[0393] Input: Name, email address, password, information about your favorite actor or character

[0394] Output: User-entered data is sent to the server

[0395] How it works: After the user enters their information, they tap the "Submit" button, which sends the data to the server via an HTTP request.

[0396] Step 4:

[0397] The server stores the user's registration information in a database.

[0398] Input: User-entered data

[0399] Output: User information stored in the database

[0400] How it works: A server-side script (e.g. Python, Node.js) receives HTTP requests and stores information in a database (e.g. MySQL, MongoDB).

[0401] Step 5:

[0402] Once a user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user.

[0403] Input: Information about the actor or character selected by the user

[0404] Output: Synthesized speech data

[0405] How it works: Using OpenAI's Text-to-Speech technology, a generation process is run based on a sample voice of a selected actor, and a synthesized voice file is temporarily saved.

[0406] Step 6:

[0407] The server uses conversational AI to prepare English conversation scenarios suitable for the user.

[0408] Input: User's learning history, current progress

[0409] Output: Customized English conversation scenario

[0410] How it works: Uses GPT-3 to generate prompts and create customized English conversation scenarios.

[0411] Step 7:

[0412] The server provides an emotion engine that can recognize emotions from the user's words and facial expressions.

[0413] Input: User's voice data, facial expression data

[0414] Output: Emotion recognition result

[0415] How it works: Affectiva's emotion recognition engine is used to analyze the user's voice and video data in real time.

[0416] Step 8:

[0417] The device loads a virtual reality environment and displays a 3D model of the selected actor or character.

[0418] Input: None

[0419] Output: Display of VR environment and 3D model

[0420] How it works: Uses Unity to load VR environments and 3D models and display them through a VR headset.

[0421] Step 9:

[0422] Users enter the virtual environment using VR goggles and controllers.

[0423] Input: None

[0424] Output: Immersive experience in a virtual environment

[0425] Operation details: Put on the VR goggles, pick up the controller and start operating. Positioning and field of view adjustments will be performed, and a tutorial on how to use the device will be displayed.

[0426] Step 10:

[0427] The device uses voice recognition technology to convert the user's speech into text and transmits it to the server in real time.

[0428] Input: User utterance

[0429] Output: Textualized speech data

[0430] Operation details: Uses Google Speech-to-Text to convert the user's speech into text and send it to the server via an HTTP request.

[0431] Step 11:

[0432] The server analyzes the received text, uses conversational AI to generate natural dialogue, and sends the character's next response to the device.

[0433] Input: Textual speech data

[0434] Output: Character reaction data

[0435] How it works: It uses GPT-3 to generate an appropriate response from the received text and sends it to the device.

[0436] Step 12:

[0437] The emotion engine analyzes the user's voice and facial expression data to assess their emotional state in real time.

[0438] Input: User's voice data, facial expression data

[0439] Output: Emotional state assessment result

[0440] How it works: Using Affectiva's engine, audio and video data is analyzed in real time to assess emotional state.

[0441] Step 13:

[0442] The terminal plays back the conversation's replies as the character's synthesized voice, continuing the dialogue with the user.

[0443] Input: Character reaction data

[0444] Output: Synthesized voice of the character

[0445] How it works: Plays a synthesized voice file and synchronizes the character's lip-syncing and facial expressions to create a real-time dialogue.

[0446] Step 14:

[0447] After the lesson is completed, the terminal transmits the dialogue data to the server.

[0448] Input: Interaction data

[0449] Output: Interaction data sent to the server

[0450] Operation details: Sends locally stored interaction data to the server.

[0451] Step 15:

[0452] The server uses conversational AI to evaluate the dialogue data and generate feedback on items including pronunciation, grammar, fluency, and emotion recognition.

[0453] Input: Interaction data

[0454] Output: Feedback data

[0455] How it works: Analyzes data based on various evaluation criteria and generates feedback reports.

[0456] Step 16:

[0457] The server sends the generated feedback to the terminal, which displays it to the user.

[0458] Input: Feedback data

[0459] Output: Feedback displayed on the terminal

[0460] How it works: Receives feedback data and displays it to the user in a visually friendly interface.

[0461] Step 17:

[0462] When the next learning session begins, the server generates a new English conversation scenario and sends it to the terminal.

[0463] Input: User learning history and progress data

[0464] Output: New English conversation scenario data

[0465] Operation details: Based on the user's past learning history, a new prompt is created and sent to the device as the next scenario.

[0466] Step 18:

[0467] The server periodically checks the subscription status and automatically charges the monthly fee.

[0468] Input: Subscription data

[0469] Output: Billing data

[0470] What it does: Queries user subscription data and manages expiration dates and billing information.

[0471] Step 19:

[0472] The server sends reminder notifications to users when their subscriptions are about to expire, and takes them through the renewal process if necessary.

[0473] Input: Subscription expiration data

[0474] Output: Reminder notification

[0475] What it does: Reminds users via email or in-app notification to renew their subscription.

[0476] (Application example 2)

[0477] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0478] Conventional English conversation learning systems do not provide dialogue that takes into account the user's emotions, limiting the learning effect. Furthermore, it has been difficult for content distribution services to provide an interactive experience for the content being viewed. The present invention aims to solve these problems, enhance the effectiveness of learning through dialogue that reflects the user's emotions, and provide a new interactive experience for content distribution services.

[0479] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for providing an interface for inputting user registration information, means for generating synthetic speech using voice samples of actors or characters selected by the user, means for preparing English conversation content for the user and creating a conversation scenario, means for displaying a 3D model in a virtual reality environment, means for converting user utterances into text using voice recognition technology, means for generating natural dialogue based on the user utterances, means for analyzing the user's emotions and dynamically changing the character's responses based on the results, means for providing a function that enables interactive conversations with drama and movie characters in a content distribution service, means for evaluating the dialogue results and providing feedback to the user, and means for providing continuous learning scenarios and managing subscriptions. This allows users to learn English conversation interactively through emotion recognition and enjoy dialogue with characters while watching content.

[0480] "User registration information" refers to information such as name, email address, password, and preferred actors and characters that a user enters to use the system.

[0481] "Interface" refers to a screen or input means for exchanging information between a user and a system.

[0482] "Synthetic voice" is an artificial voice generated by a computer based on real voice samples.

[0483] A "conversation scenario" is a pre-set script or content used in a dialogue with a user.

[0484] A "virtual reality environment" is a computer-generated environment used to create a user's experience in a virtual three-dimensional space.

[0485] A "3D model" is a digital representation of an object or character displayed in three dimensions using computer graphics.

[0486] "Speech recognition technology" is a technology for analyzing human speech and converting it into text data.

[0487] "Natural dialogue" is a form of dialogue that allows for fluent and flexible communication, like conversation between humans.

[0488] "Means for analyzing emotions" refers to technology that analyzes the user's voice and facial expressions to recognize their emotional state.

[0489] An "interactive conversation" is a two-way dialogue between a user and a system, in which the content of the dialogue changes depending on the user's input and the situation.

[0490] "Means for evaluating dialogue results" refers to technology for analyzing the content of a conversation and evaluating its quality, the user's reaction, learning status, etc.

[0491] A "continuous learning scenario" is a series of learning contents or scenarios that are provided in stages according to the user's learning progress and ability.

[0492] "Means for managing subscriptions" refers to technology for managing users' usage contracts and payment status, and for carrying out procedures for periodic billing and contract renewals.

[0493] The system for implementing this invention includes an interface for entering user registration information, synthetic speech, conversation scenario creation, a virtual reality environment, speech recognition technology, natural dialogue generation, emotion recognition, interactive conversation realization, dialogue result evaluation, continuous learning scenarios, and subscription management. By integrating these means, the system provides users with a high-level interactive English conversation learning and entertainment experience.

[0494] First, the server provides an interface for receiving user registration information. Through this interface, users enter their name, email address, and favorite actors or characters. This information is sent to the server and stored in a database.

[0495] The server then generates a synthetic voice using voice samples of the actor or character selected by the user. This synthetic voice is used during dialogue with the user. The server also prepares English conversation content and customizes the conversation scenario based on the user's learning history and progress.

[0496] The virtual reality environment is displayed on the device using VR goggles and a controller. 3D model characters are placed in this VR environment, and the user interacts with them in this virtual space. Voice recognition technology converts the user's speech into text and sends it to the server. The server then uses conversational AI to generate natural dialogue based on the text.

[0497] Furthermore, emotion recognition technology analyzes the user's voice and facial expressions to recognize their emotional state at any given moment, allowing the server to dynamically change the character's responses according to the user's emotions. For example, if the user is confused, the character will respond more kindly.

[0498] There is also a means for evaluating the dialogue results: the server analyzes the content of the conversation and provides feedback to the user on pronunciation, grammar, emotional fluctuations, etc. This allows the user to check their learning progress and identify areas for improvement.

[0499] Finally, continuous learning scenarios are provided and subscription management is also performed: new scenarios are provided to users on a regular basis, and monthly fees and contract renewals are automatically handled by the server.

[0500] As a specific example,

[0501] When user "John" uses the system for the first time, he first enters his name and email address and selects "Actor A" as his favorite actor. After logging in, the server generates a synthesized voice for "Actor A" and prepares the emotion engine. John puts on VR goggles and participates in a virtual restaurant ordering scenario. When he speaks, "Hello, how are you today?", the character responds, "I'm good, thank you! How are you?" based on emotion recognition technology. After the conversation ends, John receives feedback and checks his learning progress.

[0502] Example prompt sentence:

[0503] In a scenario where a user says "Hello, Character Y! How was your day?", Character Y should generate a natural response after analyzing the user's emotions.

[0504] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0505] Step 1:

[0506] A user installs and launches the application. The user enters their name, email address, and favorite actor or character into the interface. The device sends this information to the server, which stores it in a database.

[0507] Input: User's name, email address, favorite actor or character

[0508] Output: Registration information sent to the server

[0509] Step 2:

[0510] Once a user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user, and this generated voice is used to interact with the user.

[0511] Input: User login information (email address, password), selected actor and character information

[0512] Output: Synthesized voice data

[0513] Step 3:

[0514] The server creates customized English conversation scenarios based on the user's learning history and progress, including conversation topics and scenes.

[0515] Input: User learning history, progress data

[0516] Output: Customized conversation scenario

[0517] Step 4:

[0518] The device loads a virtual reality environment and displays a 3D model of the selected actor or character, and the user enters the virtual environment using VR goggles and controllers.

[0519] Input: English conversation scenario, 3D model data

[0520] Output: Virtual reality environment

[0521] Step 5:

[0522] When a user speaks in VR, the device uses voice recognition technology to convert the speech into text and send it to the server in real time.

[0523] Input: Audio data of user speech

[0524] Output: Textual speech data

[0525] Step 6:

[0526] The server uses conversational AI to generate natural dialogue based on the received text data, and the generated dialogue becomes the character's response.

[0527] Input: User speech data converted to text

[0528] Output: Text data of the generated dialogue

[0529] Step 7:

[0530] The emotion engine analyzes the user's voice and facial expression data to assess their emotional state, and the server then adjusts the character's responses accordingly—for example, if the user is confused, the character's responses will be more helpful.

[0531] Input: User's voice data, facial expression data

[0532] Output: Evaluated emotional state, adjusted character response data

[0533] Step 8:

[0534] The terminal plays back the conversation's replies as the character's synthesized voice, continuing the dialogue with the user.

[0535] Input: Adjusted character response data, synthesized voice data

[0536] Output: Character's response in synthesized voice

[0537] Step 9:

[0538] After the lesson, the device sends the conversation data to the server, which uses conversational AI to evaluate the conversation data and generate feedback on pronunciation, grammar, fluency, emotion recognition, and more.

[0539] Input: Interaction data

[0540] Output: Rating and feedback data

[0541] Step 10:

[0542] The server sends the generated feedback to the device, which displays it to the user, including comments about learning progress and emotional fluctuations.

[0543] Input: Rating and feedback data

[0544] Output: Feedback that is displayed to the user

[0545] Step 11:

[0546] When the next learning session begins, the server generates a new English conversation scenario and sends it to the terminal.

[0547] Input: User's latest learning history and progress data

[0548] Output: New English conversation scenario

[0549] Step 12:

[0550] The server periodically checks the subscription status and automatically charges the monthly fee. It also sends reminder notifications to users when their subscription is about to expire and guides them through the renewal process.

[0551] Input: Subscription information

[0552] Output: Billing data, reminder notifications

[0553] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0554] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0555] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0556] [Second embodiment]

[0557] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0558] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0559] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0560] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0561] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0562] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0563] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0564] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0565] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0566] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0567] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0568] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0569] The present invention relates to a VR system that allows a user to receive English conversation coaching from a favorite actor or character. Specific embodiments for carrying out the present invention will be described below.

[0570] User Registration and Login

[0571] 1. The user installs and launches the application.

[0572] 2. The device displays an interface for the user to enter their name, email address, and favorite actor or character.

[0573] 3. The user enters the necessary information and presses the send button, and the device sends the information to the server.

[0574] 4. The server saves the user's registration information in the database. The registered email address and password are used for subsequent logins.

[0575] Preparation for synthetic voice and conversational AI

[0576] 5. Once the user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user.

[0577] 6. The server uses conversational AI to prepare an appropriate English conversation scenario for the user, which is customized based on the user's learning history and current progress.

[0578] English conversation coaching in a VR environment

[0579] 7. The device will load a virtual reality environment and display a 3D model of the selected actor or character.

[0580] 8. Users can enter a virtual environment using VR goggles and controllers and begin English conversations with actors and characters using synthesized voices.

[0581] 9. The device uses voice recognition technology to convert the user's speech into text and transmits it to the server in real time.

[0582] 10. The server analyzes the received text, uses conversational AI to generate natural dialogue, and sends the next actor or character response to the device.

[0583] 11. This allows users to practice English conversation in situations that are close to real life situations.

[0584] Feedback and continuous learning

[0585] 12. After the lesson is completed, the device sends the dialogue data to the server.

[0586] 13. The server uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates the results as feedback.

[0587] 14. The server sends the generated feedback to the device, which displays it to the user, allowing the user to see how their English conversation skills have improved.

[0588] 15. When the next learning session begins, the server generates a new conversation scenario and sends it to the device.

[0589] Subscription Management

[0590] 16. The server periodically checks the subscription status and automatically charges the monthly fee.

[0591] 17. When a subscription is about to expire, the server sends reminder notifications to users and takes them through the renewal process if necessary.

[0592] Example: When a user uses the system for the first time, he or she first enters the name "John" and email address on the registration screen and selects "Actor X" as the preferred actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares a scenario for ordering at a restaurant. John enters the VR environment and begins a conversation with Actor X, saying, "Hello, John! How can I help you today?" After the conversation ends, John receives feedback on his pronunciation and grammar and checks his or her skills.

[0593] In this way, users can continue to learn English conversation in a fun and enjoyable way.

[0594] The processing flow will be explained below.

[0595] Step 1:

[0596] The user installs and launches the application.

[0597] Step 2:

[0598] The device displays an interface for the user to select their name, email address, and favorite actor or character.

[0599] Step 3:

[0600] The user enters the necessary information and presses the send button.

[0601] Step 4:

[0602] The terminal transmits the user's input information to the server.

[0603] Step 5:

[0604] The server stores the received information in a database and returns a message indicating that the information was successfully stored to the terminal.

[0605] Step 6:

[0606] The terminal will display a successful save message to the user.

[0607] Step 7:

[0608] The user enters their email address and password on the login screen and clicks the login button.

[0609] Step 8:

[0610] The terminal transmits the entered login information to the server.

[0611] Step 9:

[0612] The server retrieves user information from the database and performs authentication.

[0613] Step 10:

[0614] If the authentication is successful, the server returns a message indicating that the authentication is successful to the terminal.

[0615] Step 11:

[0616] The device will notify the user that the login was successful and begin the process of loading the VR environment.

[0617] Step 12:

[0618] The server uses synthetic voice AI to generate voice data based on voice samples of the actor or character selected by the user.

[0619] Step 13:

[0620] The server uses conversational AI to prepare English conversation content and scenarios based on the user's learning level and history.

[0621] Step 14:

[0622] The server transmits the generated voice data and conversation scenario to the terminal.

[0623] Step 15:

[0624] The device loads 3D models into the VR environment and displays the model of the selected actor or character.

[0625] Step 16:

[0626] The user puts on VR goggles and controllers and enters the virtual environment.

[0627] Step 17:

[0628] The device plays a synthesized voice, with a character saying, "Hello, [user]! How are you today?"

[0629] Step 18:

[0630] The user presses a button on the controller to reply, "I'm good, thank you! How are you?"

[0631] Step 19:

[0632] The device converts the user's speech into text using voice recognition technology and sends the text to the server.

[0633] Step 20:

[0634] The server uses conversational AI to generate appropriate responses to the user's comments and sends those responses to the device.

[0635] Step 21:

[0636] The device then plays the received response as a synthesized voice of the character and continues the conversation.

[0637] Step 22:

[0638] This process repeats until the user ends the conversation.

[0639] Step 23:

[0640] After the conversation is completed, the terminal transmits the dialogue data to the server.

[0641] Step 24:

[0642] The server uses conversational AI to evaluate the dialogue data and generate feedback on items such as pronunciation, grammar, and fluency.

[0643] Step 25:

[0644] The server transmits the generated feedback to the terminal.

[0645] Step 26:

[0646] The terminal displays feedback to the user, allowing them to confirm the learning results.

[0647] Step 27:

[0648] The next time you study, the server will generate a new English conversation scenario and send it to your device.

[0649] Step 28:

[0650] The server periodically checks the subscription status and bills the monthly fee.

[0651] Step 29:

[0652] When the subscription is about to expire, the server will send a renewal reminder to the device.

[0653] Step 30:

[0654] The device will display an update notification to the user and prompt them to take the necessary steps.

[0655] Example 1

[0656] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0657] Conventional English conversation learning systems often lack the realism required for users to practice English conversation in real situations. They also lack regular feedback and sufficient evaluation of pronunciation and grammar, making it difficult for users to continuously improve their skills. Furthermore, subscription management is often done manually, placing a heavy burden on users.

[0658] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0659] In this invention, the server includes means for providing an interface for inputting user registration information, means for generating synthetic speech using a voice sample of a character selected by the user, means for preparing English conversation content for the user and creating a conversation scenario, means for displaying a three-dimensional model in a virtual reality environment, means for converting the user's utterances into text using voice recognition technology, means for generating natural dialogue based on the user's utterances, means for evaluating the dialogue results and providing feedback to the user, means for providing continuous learning scenarios and managing subscriptions, and means for periodically checking the subscription status and automatically charging fees. This allows users to practice English conversation in realistic situations and improve their skills by receiving continuous feedback and evaluation, while also reducing the burden of subscription management.

[0660] The "interface for entering user registration information" is a graphical input screen that allows a user to enter their name, email address, and other required information.

[0661] The "means for generating synthetic voice" is a mechanism for converting text data into voice data based on a voice sample of a character selected by the user.

[0662] "Means for creating conversation scenarios" refers to a method for automatically generating appropriate English conversation scenes and situations based on the user's learning history and progress.

[0663] "Means for displaying a three-dimensional model in a virtual reality environment" refers to a method of displaying a character selected by the user in a three-dimensional virtual space using VR goggles or a display.

[0664] "Means for converting user speech into text using speech recognition technology" refers to technology that analyzes the user's speech in real time and converts it into corresponding text data.

[0665] "Means for generating natural dialogue" refers to a method for using conversational AI to generate natural-flowing dialogue based on user input.

[0666] "Means for evaluating the dialogue results and providing feedback to the user" refers to a mechanism that analyzes the content of the dialogue with the user, evaluates pronunciation, grammar, fluency, etc., and notifies the user of the results.

[0667] "Means for providing continuous learning scenarios" refers to a method for continuously providing new learning content based on the user's progress and feedback results.

[0668] A "means for managing subscriptions" is a mechanism for checking the user's subscription status and periodically billing the service fee.

[0669] An "automatic billing mechanism" is a system that automatically processes payment of the subscription fee during the subscription period.

[0670] MODE FOR CARRYING OUT THE INVENTION

[0671] The present invention relates to a VR system that allows a user to enjoy learning English conversation while receiving coaching from a character selected by the user. Specific embodiments for carrying out the present invention will be described below.

[0672] User Registration and Login

[0673] First, the user installs and launches the application. At this stage, the device displays an interface for the user to enter their name, email address, and favorite character. After the user enters the required information and presses the submit button, the device sends the information to the server. The server stores the user's registration information in a database, and the registered email address and password are used for subsequent logins.

[0674] Preparation for synthetic voice and conversational AI

[0675] When a user logs in, the server generates synthetic speech based on the voice samples of the character selected by the user. Specifically, a speech synthesis engine (e.g., Google Cloud Text-to-Speech or Amazon Polly) is used. The server also uses a conversational AI model (e.g., OpenAI's GPT-3) to prepare an English conversation scenario appropriate for the user. This scenario is customized based on the user's learning history and current progress.

[0676] English conversation coaching in a VR environment

[0677] Next, the device loads a virtual reality environment (VR environment) and displays a three-dimensional model of the selected character. VR goggles are mainly used, such as Oculus Rift or HTC Vive. The user enters the virtual environment using the VR goggles and controllers and can begin English conversation with the character using synthesized voice. The device converts the user's speech into text using voice recognition technology (e.g., Google Speech-to-Text) and sends it to the server in real time. The server analyzes the received text, generates natural dialogue using conversational AI, and sends the character's next response to the device. This allows the user to practice English conversation in a manner that is close to real-life situations.

[0678] Feedback and continuous learning

[0679] After the lesson is completed, the device sends the dialogue data to the server. The server uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results. The generated feedback is displayed to the user via the device, allowing the user to see areas where their English conversation skills have improved. When the next lesson begins, the server generates a new dialogue scenario and sends it to the device.

[0680] Subscription Management

[0681] The server periodically checks the subscription status and automatically bills the monthly fee. When the subscription expiration date approaches, the server sends a reminder to the user and takes care of the renewal procedure if necessary.

[0682] Specific examples

[0683] For example, when a user uses the system for the first time, they enter their name "John" and email address on the registration screen and select "Character X" as their favorite character. After logging in, the server generates a synthesized voice for "Character X" and prepares a scenario for ordering at a restaurant. Once John enters the VR environment, he begins a conversation with Character X, saying, "Hello, John! How can I help you today?" After the conversation ends, John receives feedback on his pronunciation and grammar and checks his skills.

[0684] Prompt Sentence Examples

[0685] "Please explain the details of the VR system that allows users to receive English conversation coaching with their favorite characters."

[0686] Thus, the present invention provides a system that allows users to practice English conversation in realistic situations and improve their skills through continuous feedback and evaluation.

[0687] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0688] Step 1:

[0689] The user installs and launches the application. After installation, by opening the application, the device displays an interface for user registration.

[0690] Input: The user installs and launches the app.

[0691] Output: The user registration interface is displayed.

[0692] Step 2:

[0693] The terminal provides the user with a form to enter their name, email address, and favorite character. The user enters the required information into the form and presses the submit button.

[0694] Input: A user fills out a form with their name, email address, and favorite character.

[0695] Output: The entered information is sent to the server.

[0696] Step 3:

[0697] The terminal sends the user information to the server, which stores the information in a database.

[0698] Input: User information sent from the device.

[0699] Output: User information stored on the server.

[0700] Step 4:

[0701] When a user logs in, the server generates a synthesized voice based on a voice sample of the character selected by the user.

[0702] Input: User login information and selected character.

[0703] Output: Synthetic speech data is generated.

[0704] Step 5:

[0705] The server uses a conversational AI model to prepare a suitable English conversation scenario for the user, which is customized based on the user's learning history and current progress.

[0706] Input: User learning history and progress data.

[0707] Output: A customized English conversation scenario.

[0708] Step 6:

[0709] The device loads a virtual reality environment and displays a three-dimensional model of the selected character. The user enters the virtual environment using VR goggles and controllers.

[0710] Input: English conversation scenario and character 3D model data.

[0711] Output: The character displayed in a VR environment.

[0712] Step 7:

[0713] The device converts the user's speech into text using voice recognition technology and sends it to the server in real time. Google Speech-to-Text is used for voice recognition.

[0714] Input: User speech.

[0715] Output: Textualized speech data.

[0716] Step 8:

[0717] The server analyzes the textual data and uses conversational AI to generate natural dialogue, which is then sent to the device.

[0718] Input: Text data of user utterances.

[0719] Output: Natural dialogue.

[0720] Step 9:

[0721] The device displays the generated dialogue, and the user continues the conversation, allowing the user to practice English conversation in a realistic situation.

[0722] Input: The generated dialogue.

[0723] Output: The displayed dialogue.

[0724] Step 10:

[0725] The device sends the conversation data to a server, which uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results.

[0726] Input: Interaction data.

[0727] Output: Feedback data.

[0728] Step 11:

[0729] The server sends the generated feedback to the terminal, which displays it to the user, allowing the user to see how their English conversation skills have improved.

[0730] Input: Feedback data.

[0731] Output: The feedback displayed to the user.

[0732] Step 12:

[0733] At the start of the next learning session, the server generates a new conversation scenario and sends it to the terminal.

[0734] Input: User learning history and feedback.

[0735] Output: A new conversation scenario.

[0736] Step 13:

[0737] The server periodically checks the subscription status and automatically bills users, sending reminders when their subscription is about to expire.

[0738] Input: Subscription information.

[0739] Output: Automatic billing and reminder notifications.

[0740] (Application example 1)

[0741] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0742] In conventional English conversation learning systems, users have limited opportunities to practice English conversation in situations close to real conversations. Furthermore, there are no systems that can provide users with an engaging experience of receiving English conversation coaching from their favorite actors or characters, making it difficult to maintain motivation to continue learning.

[0743] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0744] In this invention, the server includes a means for generating synthetic speech based on voice samples of actors or characters selected by the user, a means for providing an application to be installed on a smartphone to execute the generated conversation scenario, and a means for generating new conversation scenarios based on the user's learning progress via the server. This allows users to practice English conversation in a manner that is close to actual conversation situations through real-time dialogue with their favorite actors or characters. Furthermore, the provision of new scenarios based on the user's learning progress allows the user to enjoy constantly fresh learning content, contributing to maintaining motivation for continuous learning.

[0745] 1. An "interface for entering user registration information" is a screen or form that provides a user with the ability to enter information such as their name, email address, and favorite actors or characters.

[0746] 2. "Means for generating synthetic speech" refers to a program or algorithm that generates speech data that enables natural dialogue based on voice samples of an actor or character selected by the user.

[0747] 3. "Means for creating conversation scenarios" refers to a system for designing and preparing appropriate English conversation content in advance based on the user's learning progress and preferences.

[0748] 4. "Means for displaying 3D models in a virtual reality environment" means a system for displaying 3D character or actor models based on a user's selection using a VR headset or display.

[0749] 5. "Means of converting user speech into text using speech recognition technology" refers to technology or programs that analyze what a user says in real time and convert it into text data.

[0750] 6. "Means for generating natural dialogue" refers to a conversational AI system that generates appropriate responses based on the user's textual utterances and provides them as natural conversation.

[0751] 7. "Means for providing an application to be installed on a smartphone" means a method or service for a user to download and install the software necessary to use the English conversation coaching system on a smartphone.

[0752] 8. "Means for generating new conversation scenarios based on the user's learning progress via a server" refers to a server-based system that analyzes the user's learning data and progress to generate and provide new English conversation scenarios.

[0753] 9. "Means for evaluating a user's skills based on feedback from a server and presenting the results in the form of a widget" refers to a method for evaluating a user's pronunciation, grammar, and fluency using feedback information sent from a server, and displaying the evaluation results as a widget on a smartphone.

[0754] 10. "Means for evaluating dialogue results and providing feedback to the user" refers to a mechanism that analyzes the content of the dialogue conducted by the user, evaluates it based on the accuracy of pronunciation and grammar, and returns the results to the user.

[0755] 11. "Means for providing continuous learning scenarios and managing subscriptions" means a system for operating a subscription service for which a recurring fee is charged, by periodically providing new learning scenarios to users and managing and supervising their usage.

[0756] The following hardware and software are required to implement this invention. The hardware required is a smartphone connected to the internet, and VR goggles and controllers may be used as options. The software includes Python, PyQt5, requests, speech_recognition, and pyttsx3.

[0757] When a user installs and launches the application, an interface for entering user registration information is displayed. In this interface, the user enters their name, email address, and information about their favorite actors and characters. This information is sent to the server. The server creates a user profile based on the received user information and stores the registration information in a database.

[0758] When a user logs in, the server generates a synthesized voice based on the voice samples of the actor or character selected by the user. This synthesized voice is generated as voice data using the pyttsx3 library. The server also uses conversational AI to prepare an English conversation scenario suited to the user based on the user's learning history and progress. This allows the user to practice English conversation based on the scenario.

[0759] Next, users use a smartphone application to enter the virtual reality environment. The application works in conjunction with VR goggles and controllers to load the virtual environment and display a 3D model of the selected actor or character. Within the VR environment, users can begin English conversations with the actor or character using synthesized voice.

[0760] The user's speech is converted into text using voice recognition technology and sent to the server in real time. The server analyzes the received text data and generates natural dialogue using conversational AI. As a result, the next response of the actor or character is sent to the device, allowing the user to continue the dialogue in real time.

[0761] Once the conversation is over, the server uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results. This feedback is sent to the device and displayed to the user in the form of a widget, allowing the user to see where their skills have improved.

[0762] As a concrete example, when a user uses the system for the first time, they enter their name "John" and email address on the registration screen and select "Actor X" as their favorite actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares a scenario for ordering at a restaurant. John enters the VR environment and begins a conversation with "Actor X": "Hello, John! How can I help you today?" After the conversation ends, John receives feedback on his pronunciation and grammar and checks his skills.

[0763] Additionally, by inputting the following prompt sentences into the generative AI model, it is possible to generate an appropriate conversation scenario:

[0764] The user has selected "Actor X" as their favorite actor. Please generate natural dialogue to support the user in English conversation in a situation where they are ordering at a restaurant.

[0765] In this way, users can improve their English conversation skills while experiencing conversations with virtual actors and characters.

[0766] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0767] Step 1:

[0768] The user installs and launches the application. The device displays an interface for entering user registration information. The user enters their name, email address, and information about their favorite actor or character, and presses the submit button. The input data is sent to the server. The server receives the user information and stores it in a database.

[0769] Step 2:

[0770] The user logs in. The server verifies the user information and authenticates the login. If authentication is successful, a synthesized voice is generated based on the voice samples of the actor or character selected by the user. The server generates the voice data using the pyttsx3 library and sends the result to the device.

[0771] Step 3:

[0772] The server uses conversational AI to prepare an English conversation scenario based on the user's learning history and progress. A generative AI model is then used to generate a conversation scenario based on a specified prompt. The prompt, "The user has selected 'Actor X' as their favorite actor. Please generate a natural dialogue to support the user in English conversation in a situation where they are ordering at a restaurant," is input, and the AI ​​outputs a response scenario.

[0773] Step 4:

[0774] The device loads a virtual reality environment and displays a 3D model of the selected actor or character. The user enters the virtual environment using VR goggles and controllers. The device plays synthesized voice data from the server and begins a conversation in English with the user. The user's speech is input through a microphone.

[0775] Step 5:

[0776] The device uses voice recognition technology to convert the user's speech into text in real time. The text data is then sent to a server. The server then analyzes the received text data and uses conversational AI to generate natural dialogue. The server then generates the next actor or character's response and sends the resulting voice data to the device.

[0777] Step 6:

[0778] The user continues to have real-time conversations with actors and characters using synthesized speech. The device plays back the voice data sent from the server, providing the user with a realistic conversation experience. Every time the user speaks, speech recognition and text conversion are repeated.

[0779] Step 7:

[0780] Once the conversation is complete, the device sends the conversation data to the server, which uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results. The generated feedback is sent to the device and presented to the user in the form of a widget.

[0781] Step 8:

[0782] When the next learning session begins, the server generates a new conversation scenario based on the user's learning progress. It then uses the generative AI model again to generate a conversation scenario based on the new prompt. The server then sends the generated conversation scenario to the device, providing a continuous learning scenario.

[0783] Step 9:

[0784] The server periodically checks the subscription status and automatically bills the monthly fee. When the subscription expiration date approaches, the server sends a reminder to the user and takes care of the renewal procedure if necessary.

[0785] These steps allow users to continuously improve their English skills by experiencing English conversations with virtual actors and characters. Furthermore, by generating conversation scenarios using a generative AI model, users can enjoy constantly new learning content.

[0786] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0787] The present invention combines an emotion engine with a VR system that allows users to receive English conversation coaching from their favorite actors or characters. Specific embodiments for carrying out the present invention will be described below.

[0788] User Registration and Login

[0789] 1. The user installs and launches the application.

[0790] 2. The device displays an interface for the user to enter their name, email address, and select their favorite actor or character.

[0791] 3. The user enters the necessary information and presses the send button, and the device sends the information to the server.

[0792] 4. The server saves the user's registration information in the database. The registered email address and password are used for subsequent logins.

[0793] Preparation of synthetic voice, conversational AI and emotion recognition engine

[0794] 5. Once the user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user.

[0795] 6. The server uses conversational AI to prepare an appropriate English conversation scenario for the user, which is customized based on the user's learning history and current progress.

[0796] 7. The server also provides an emotion engine that can recognize emotions from the user's words and facial expressions.

[0797] English conversation coaching in a VR environment

[0798] 8. The device will load a virtual reality environment and display a 3D model of the selected actor or character.

[0799] 9. Users can enter a virtual environment using VR goggles and controllers and begin English conversations with characters using synthesized voices.

[0800] 10. The device uses voice recognition technology to convert the user's speech into text and transmits it to the server in real time.

[0801] 11. The server analyzes the received text, uses conversational AI to generate natural dialogue, and sends the character's next response to the device.

[0802] 12. The emotion engine analyzes the user's voice and facial expression data to assess their emotional state in real time, adjusting the tone and content of the conversation accordingly.

[0803] 13. The device plays back the conversational responses as the character's synthesized voice and continues the dialogue with the user.

[0804] Feedback and continuous learning

[0805] 14. After the lesson is completed, the device sends the dialogue data to the server.

[0806] 15. The server uses conversational AI to evaluate the dialogue data and generate feedback on items including pronunciation, grammar, fluency, and emotion recognition.

[0807] 16. The server sends the generated feedback to the device, which displays it to the user. This feedback includes comments about the learning progress and emotional fluctuations.

[0808] 17. When the next learning session begins, the server generates a new English conversation scenario and sends it to the device.

[0809] Subscription Management

[0810] 18. The server periodically checks the subscription status and automatically charges the monthly fee.

[0811] 19. When a subscription is about to expire, the server sends reminder notifications to users and takes them through the renewal process if necessary.

[0812] Example: When a user uses the system for the first time, they first enter their name "John" and email address on the registration screen and select "Actor X" as their favorite actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares the emotion engine. A scenario of ordering at a restaurant is prepared, and John enters the VR environment and starts a conversation with Actor X, saying, "Hello, John! How are you today?" When John replies, "I'm good, thank you! How are you?", the emotion engine analyzes John's emotional state from his voice and facial expressions and adjusts the character's response. After the conversation ends, John receives feedback on his pronunciation, grammar, and emotional fluctuations to check his skills.

[0813] In this way, users can continue to learn English conversation in a fun and enjoyable way while receiving coaching in an approach that responds to emotional fluctuations.

[0814] The processing flow will be explained below.

[0815] Step 1:

[0816] The user installs and launches the application.

[0817] Step 2:

[0818] The device displays an interface for the user to select their name, email address, and favorite actor or character.

[0819] Step 3:

[0820] The user enters the necessary information and presses the send button.

[0821] Step 4:

[0822] The terminal transmits the user's input information to the server.

[0823] Step 5:

[0824] The server stores the received information in a database and returns a message indicating that the information was successfully stored to the terminal.

[0825] Step 6:

[0826] The terminal will display a successful save message to the user.

[0827] Step 7:

[0828] The user enters their email address and password on the login screen and clicks the login button.

[0829] Step 8:

[0830] The terminal transmits the entered login information to the server.

[0831] Step 9:

[0832] The server retrieves user information from the database and performs authentication.

[0833] Step 10:

[0834] If the authentication is successful, the server returns a message indicating that the authentication is successful to the terminal.

[0835] Step 11:

[0836] The device will notify the user that the login was successful and begin the process of loading the VR environment.

[0837] Step 12:

[0838] The server uses synthetic voice AI to generate voice data based on voice samples of the actor or character selected by the user.

[0839] Step 13:

[0840] The server uses conversational AI to prepare English conversation content and scenarios based on the user's learning level and history.

[0841] Step 14:

[0842] The server also has an emotion engine and is configured to recognize emotions in real time from the user's comments and facial expressions.

[0843] Step 15:

[0844] The server transmits the generated voice data, the conversation scenario, and the emotion engine settings to the terminal.

[0845] Step 16:

[0846] The device loads 3D models into the VR environment and displays the model of the selected actor or character.

[0847] Step 17:

[0848] The user puts on VR goggles and controllers and enters the virtual environment.

[0849] Step 18:

[0850] The device plays a synthesized voice, with a character saying, "Hello, [user]! How are you today?"

[0851] Step 19:

[0852] The user presses a button on the controller to reply, "I'm good, thank you! How are you?"

[0853] Step 20:

[0854] The device converts the user's speech into text using voice recognition technology and sends the text to the server.

[0855] Step 21:

[0856] The server uses conversational AI to generate appropriate responses to user comments.

[0857] Step 22:

[0858] The emotion engine analyzes the user's voice and facial expression data to assess their emotional state in real time.

[0859] Step 23:

[0860] The server adjusts the character's next response based on the emotional data evaluated by the emotion engine and sends that data to the terminal.

[0861] Step 24:

[0862] The device then plays the received response as a synthesized voice of the character and continues the conversation.

[0863] Step 25:

[0864] This process repeats until the user ends the conversation.

[0865] Step 26:

[0866] After the conversation is completed, the terminal transmits the dialogue data to the server.

[0867] Step 27:

[0868] The server uses conversational AI to evaluate the dialogue data and generate feedback based on pronunciation, grammar, fluency, and emotion recognition.

[0869] Step 28:

[0870] The server transmits the generated feedback to the terminal.

[0871] Step 29:

[0872] The terminal displays feedback to the user, allowing them to confirm the learning results.

[0873] Step 30:

[0874] The next time you study, the server will generate a new English conversation scenario and send it to your device.

[0875] Step 31:

[0876] The server periodically checks the subscription status and bills the monthly fee.

[0877] Step 32:

[0878] When the subscription is about to expire, the server will send a renewal reminder to the device.

[0879] Step 33:

[0880] The device will display an update notification to the user and prompt them to take the necessary steps.

[0881] Example 2

[0882] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0883] Conventional English conversation learning systems use fixed scenarios and general speech synthesis technology without taking into account the learner's emotional state, which limits the effectiveness of learning. Furthermore, it is difficult to provide scenarios tailored to individual learners, making it difficult to maintain continuous learning motivation. Furthermore, since pronunciation and grammar evaluation is not performed in real time, feedback tends to be delayed, making it difficult to improve practical conversational skills.

[0884] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0885] In this invention, the server includes: means for providing an interface for inputting user registration information; means for generating synthetic speech using voice samples of actors or characters selected by the user; means for preparing English conversation content for the user and creating a conversation scenario; means for displaying a 3D model in a virtual reality environment; means for converting user utterances into text using speech recognition technology; means for generating natural dialogue based on the user utterances; means for evaluating the dialogue results and providing feedback to the user; means for providing continuous learning scenarios and managing subscriptions; means for recognizing emotions from user utterances and facial expressions using an emotion recognition engine; and means for providing customized English conversation scenarios using a generative AI model. This enables more effective English conversation learning by providing scenarios and feedback in real time that take the learner's emotional state into account.

[0886] "User registration information" refers to information such as name, email address, password, favorite actor or character, etc., that a user enters to use an application.

[0887] An "interface" refers to the screen or operating means through which a user inputs information into a system or performs operations.

[0888] "Voice Sample" means the recorded or digitized voice of a particular actor or character.

[0889] "Synthetic voice" is an artificial voice that imitates a human voice and is created based on voice samples.

[0890] "English conversation content" refers to conversation themes, phrases, and scenarios provided for users to study.

[0891] A "conversation scenario" is a series of dialogues that assume a specific situation, and is the content of the conversation that takes place between the user and a virtual character.

[0892] "Virtual reality environment" means a visual and operational environment for displaying 3D models and for allowing a user to experience them in virtual reality.

[0893] A "3D model" is a character or object that is represented in three dimensions using computer graphics.

[0894] "Speech recognition technology" refers to technology that recognizes what a user says and converts the voice data into text data.

[0895] "Natural dialogue" refers to the exchange of fluent, natural sentences, like conversations between people.

[0896] "Feedback" refers to evaluations, advice, and suggestions for improvement given to users in response to their learning and operation results.

[0897] A "continuous learning scenario" is a learning plan or scenario that is provided step by step to enable the user to continue learning.

[0898] "Subscription management" is a system for managing users' contract status and payment fees for subscription-based services.

[0899] An "emotion recognition engine" is a system that analyzes and recognizes a user's emotional state from their speech and facial expressions.

[0900] A "generative AI model" is an artificial intelligence model that generates new data (in this case, English conversation scenarios) based on input data.

[0901] The present invention combines an emotion engine with a VR system that allows users to receive English conversation coaching from their favorite actors or characters. Specific embodiments for carrying out the present invention will be described below.

[0902] User Registration and Login

[0903] When a user installs and launches the application, the device displays an interface for the user to select their name, email address, and favorite actor or character. After the user enters the required information and presses the submit button, the device sends the information to the server. This data is stored in a database by the server. The registered email address and password are used for subsequent logins.

[0904] Preparation of synthetic voice, conversational AI and emotion recognition engine

[0905] When a user logs in, the server generates synthetic speech based on voice samples of the actor or character selected by the user. To achieve this, it uses OpenAI's Text-to-Speech technology. The server also uses conversational AI (GPT-3) to prepare an English conversation scenario suited to the user. This scenario is customized based on the user's learning history and current progress. The server also uses Affectiva's emotion recognition engine to recognize emotions from the user's speech and facial expressions.

[0906] English conversation coaching in a VR environment

[0907] The device loads a virtual reality environment and displays a 3D model of the selected actor or character. This virtual environment is built using Unity. Users enter the virtual environment using VR goggles and controllers and can begin English conversations with the character using synthetic voice. The device uses Google Speech-to-Text to convert the user's speech into text and send it to the server in real time. The server analyzes the received text, generates natural-sounding dialogue using GPT-3, and sends the character's next response to the device. The emotion engine analyzes the user's voice and facial expression data and evaluates their emotional state in real time. Based on this, the tone and content of the conversation are adjusted, and the device plays back the character's replies as synthetic voice, continuing the dialogue with the user.

[0908] Feedback and continuous learning

[0909] After completing a lesson, the device sends the dialogue data to the server. The server uses conversational AI to evaluate the dialogue data and generate feedback on items including pronunciation, grammar, fluency, and emotion recognition. This feedback is sent to the device and displayed to the user. The feedback also includes comments on learning progress and emotional fluctuations. When starting the next lesson, the server generates a new English conversation scenario and sends it to the device.

[0910] Subscription Management

[0911] The server periodically checks the subscription status and automatically bills the monthly fee. When the subscription expiration date approaches, the server sends a reminder to the user and takes care of the renewal procedure if necessary.

[0912] Examples:

[0913] When a user first uses the system, they first enter their name ("John") and email address on the registration screen and select "Actor X" as their preferred actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares the emotion engine. A scenario of ordering at a restaurant is prepared, and John, entering the VR environment, begins a conversation with Actor X: "Hello, John! How are you today?" When John replies, "I'm good, thank you! How are you?", the emotion engine analyzes John's emotional state from his voice and facial expressions and adjusts the character's response accordingly. After the conversation ends, John receives feedback on his pronunciation, grammar, and emotional fluctuations to check his skills. In this way, users can continuously and enjoyably learn English conversation while receiving coaching tailored to their emotional fluctuations.

[0914] Example prompt for a generative AI model:

[0915] "Generate an English conversation scenario in which a user orders from actor X at a restaurant. Please include the flow of the conversation when the user has positive emotions."

[0916] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0917] Step 1:

[0918] The user installs and launches the application.

[0919] How it works: A user downloads an app from the app store on their smartphone, tablet, or PC, installs it, and then launches it. They then tap the icon of the installed app to launch it.

[0920] Step 2:

[0921] The device displays an interface for the user to select their name, email address, and favorite actor or character.

[0922] Input: None

[0923] Output: User input form

[0924] Operation details: The app screen displays a form for entering name, email address, password, and favorite actor or character.

[0925] Step 3:

[0926] The user enters the necessary information and presses the send button, and the terminal sends the information to the server.

[0927] Input: Name, email address, password, information about your favorite actor or character

[0928] Output: User-entered data is sent to the server

[0929] How it works: After the user enters their information, they tap the "Submit" button, which sends the data to the server via an HTTP request.

[0930] Step 4:

[0931] The server stores the user's registration information in a database.

[0932] Input: User-entered data

[0933] Output: User information stored in the database

[0934] How it works: A server-side script (e.g. Python, Node.js) receives HTTP requests and stores information in a database (e.g. MySQL, MongoDB).

[0935] Step 5:

[0936] Once a user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user.

[0937] Input: Information about the actor or character selected by the user

[0938] Output: Synthesized speech data

[0939] How it works: Using OpenAI's Text-to-Speech technology, a generation process is run based on a sample voice of a selected actor, and a synthesized voice file is temporarily saved.

[0940] Step 6:

[0941] The server uses conversational AI to prepare English conversation scenarios suitable for the user.

[0942] Input: User's learning history, current progress

[0943] Output: Customized English conversation scenario

[0944] How it works: Uses GPT-3 to generate prompts and create customized English conversation scenarios.

[0945] Step 7:

[0946] The server provides an emotion engine that can recognize emotions from the user's words and facial expressions.

[0947] Input: User's voice data, facial expression data

[0948] Output: Emotion recognition result

[0949] How it works: Affectiva's emotion recognition engine is used to analyze the user's voice and video data in real time.

[0950] Step 8:

[0951] The device loads a virtual reality environment and displays a 3D model of the selected actor or character.

[0952] Input: None

[0953] Output: Display of VR environment and 3D model

[0954] How it works: Uses Unity to load VR environments and 3D models and display them through a VR headset.

[0955] Step 9:

[0956] Users enter the virtual environment using VR goggles and controllers.

[0957] Input: None

[0958] Output: Immersive experience in a virtual environment

[0959] Operation details: Put on the VR goggles, pick up the controller and start operating. Positioning and field of view adjustments will be performed, and a tutorial on how to use the device will be displayed.

[0960] Step 10:

[0961] The device uses voice recognition technology to convert the user's speech into text and transmits it to the server in real time.

[0962] Input: User utterance

[0963] Output: Textualized speech data

[0964] Operation details: Uses Google Speech-to-Text to convert the user's speech into text and send it to the server via an HTTP request.

[0965] Step 11:

[0966] The server analyzes the received text, uses conversational AI to generate natural dialogue, and sends the character's next response to the device.

[0967] Input: Textual speech data

[0968] Output: Character reaction data

[0969] How it works: It uses GPT-3 to generate an appropriate response from the received text and sends it to the device.

[0970] Step 12:

[0971] The emotion engine analyzes the user's voice and facial expression data to assess their emotional state in real time.

[0972] Input: User's voice data, facial expression data

[0973] Output: Emotional state assessment result

[0974] How it works: Using Affectiva's engine, audio and video data is analyzed in real time to assess emotional state.

[0975] Step 13:

[0976] The terminal plays back the conversation's replies as the character's synthesized voice, continuing the dialogue with the user.

[0977] Input: Character reaction data

[0978] Output: Synthesized voice of the character

[0979] How it works: Plays a synthesized voice file and synchronizes the character's lip-syncing and facial expressions to create a real-time dialogue.

[0980] Step 14:

[0981] After the lesson is completed, the terminal transmits the dialogue data to the server.

[0982] Input: Interaction data

[0983] Output: Interaction data sent to the server

[0984] Operation details: Sends locally stored interaction data to the server.

[0985] Step 15:

[0986] The server uses conversational AI to evaluate the dialogue data and generate feedback on items including pronunciation, grammar, fluency, and emotion recognition.

[0987] Input: Interaction data

[0988] Output: Feedback data

[0989] How it works: Analyzes data based on various evaluation criteria and generates feedback reports.

[0990] Step 16:

[0991] The server sends the generated feedback to the terminal, which displays it to the user.

[0992] Input: Feedback data

[0993] Output: Feedback displayed on the terminal

[0994] How it works: Receives feedback data and displays it to the user in a visually friendly interface.

[0995] Step 17:

[0996] When the next learning session begins, the server generates a new English conversation scenario and sends it to the terminal.

[0997] Input: User learning history and progress data

[0998] Output: New English conversation scenario data

[0999] Operation details: Based on the user's past learning history, a new prompt is created and sent to the device as the next scenario.

[1000] Step 18:

[1001] The server periodically checks the subscription status and automatically charges the monthly fee.

[1002] Input: Subscription data

[1003] Output: Billing data

[1004] What it does: Queries user subscription data and manages expiration dates and billing information.

[1005] Step 19:

[1006] The server sends reminder notifications to users when their subscriptions are about to expire, and takes them through the renewal process if necessary.

[1007] Input: Subscription expiration data

[1008] Output: Reminder notification

[1009] What it does: Reminds users via email or in-app notification to renew their subscription.

[1010] (Application example 2)

[1011] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1012] Conventional English conversation learning systems do not provide dialogue that takes into account the user's emotions, limiting the learning effect. Furthermore, it has been difficult for content distribution services to provide an interactive experience for the content being viewed. The present invention aims to solve these problems, enhance the effectiveness of learning through dialogue that reflects the user's emotions, and provide a new interactive experience for content distribution services.

[1013] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for providing an interface for inputting user registration information, means for generating synthetic speech using voice samples of actors or characters selected by the user, means for preparing English conversation content for the user and creating a conversation scenario, means for displaying a 3D model in a virtual reality environment, means for converting user utterances into text using voice recognition technology, means for generating natural dialogue based on the user utterances, means for analyzing the user's emotions and dynamically changing the character's responses based on the results, means for providing a function that enables interactive conversations with drama and movie characters in a content distribution service, means for evaluating the dialogue results and providing feedback to the user, and means for providing continuous learning scenarios and managing subscriptions. This allows users to learn English conversation interactively through emotion recognition and enjoy dialogue with characters while watching content.

[1014] "User registration information" refers to information such as name, email address, password, and preferred actors and characters that a user enters to use the system.

[1015] "Interface" refers to a screen or input means for exchanging information between a user and a system.

[1016] "Synthetic voice" is an artificial voice generated by a computer based on real voice samples.

[1017] A "conversation scenario" is a pre-set script or content used in a dialogue with a user.

[1018] A "virtual reality environment" is a computer-generated environment used to create a user's experience in a virtual three-dimensional space.

[1019] A "3D model" is a digital representation of an object or character displayed in three dimensions using computer graphics.

[1020] "Speech recognition technology" is a technology for analyzing human speech and converting it into text data.

[1021] "Natural dialogue" is a form of dialogue that allows for fluent and flexible communication, like conversation between humans.

[1022] "Means for analyzing emotions" refers to technology that analyzes the user's voice and facial expressions to recognize their emotional state.

[1023] An "interactive conversation" is a two-way dialogue between a user and a system, in which the content of the dialogue changes depending on the user's input and the situation.

[1024] "Means for evaluating dialogue results" refers to technology for analyzing the content of a conversation and evaluating its quality, the user's reaction, learning status, etc.

[1025] A "continuous learning scenario" is a series of learning contents or scenarios that are provided in stages according to the user's learning progress and ability.

[1026] "Means for managing subscriptions" refers to technology for managing users' usage contracts and payment status, and for carrying out procedures for periodic billing and contract renewals.

[1027] The system for implementing this invention includes an interface for entering user registration information, synthetic speech, conversation scenario creation, a virtual reality environment, speech recognition technology, natural dialogue generation, emotion recognition, interactive conversation realization, dialogue result evaluation, continuous learning scenarios, and subscription management. By integrating these means, the system provides users with a high-level interactive English conversation learning and entertainment experience.

[1028] First, the server provides an interface for receiving user registration information. Through this interface, users enter their name, email address, and favorite actors or characters. This information is sent to the server and stored in a database.

[1029] The server then generates a synthetic voice using voice samples of the actor or character selected by the user. This synthetic voice is used during dialogue with the user. The server also prepares English conversation content and customizes the conversation scenario based on the user's learning history and progress.

[1030] The virtual reality environment is displayed on the device using VR goggles and a controller. 3D model characters are placed in this VR environment, and the user interacts with them in this virtual space. Voice recognition technology converts the user's speech into text and sends it to the server. The server then uses conversational AI to generate natural dialogue based on the text.

[1031] Furthermore, emotion recognition technology analyzes the user's voice and facial expressions to recognize their emotional state at any given moment, allowing the server to dynamically change the character's responses according to the user's emotions. For example, if the user is confused, the character will respond more kindly.

[1032] There is also a means for evaluating the dialogue results: the server analyzes the content of the conversation and provides feedback to the user on pronunciation, grammar, emotional fluctuations, etc. This allows the user to check their learning progress and identify areas for improvement.

[1033] Finally, continuous learning scenarios are provided and subscription management is also performed: new scenarios are provided to users on a regular basis, and monthly fees and contract renewals are automatically handled by the server.

[1034] As a specific example,

[1035] When user "John" uses the system for the first time, he first enters his name and email address and selects "Actor A" as his favorite actor. After logging in, the server generates a synthesized voice for "Actor A" and prepares the emotion engine. John puts on VR goggles and participates in a virtual restaurant ordering scenario. When he speaks, "Hello, how are you today?", the character responds, "I'm good, thank you! How are you?" based on emotion recognition technology. After the conversation ends, John receives feedback and checks his learning progress.

[1036] Example prompt sentence:

[1037] In a scenario where a user says "Hello, Character Y! How was your day?", Character Y should generate a natural response after analyzing the user's emotions.

[1038] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1039] Step 1:

[1040] A user installs and launches the application. The user enters their name, email address, and favorite actor or character into the interface. The device sends this information to the server, which stores it in a database.

[1041] Input: User's name, email address, favorite actor or character

[1042] Output: Registration information sent to the server

[1043] Step 2:

[1044] Once a user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user, and this generated voice is used to interact with the user.

[1045] Input: User login information (email address, password), selected actor and character information

[1046] Output: Synthesized voice data

[1047] Step 3:

[1048] The server creates customized English conversation scenarios based on the user's learning history and progress, including conversation topics and scenes.

[1049] Input: User learning history, progress data

[1050] Output: Customized conversation scenario

[1051] Step 4:

[1052] The device loads a virtual reality environment and displays a 3D model of the selected actor or character, and the user enters the virtual environment using VR goggles and controllers.

[1053] Input: English conversation scenario, 3D model data

[1054] Output: Virtual reality environment

[1055] Step 5:

[1056] When a user speaks in VR, the device uses voice recognition technology to convert the speech into text and send it to the server in real time.

[1057] Input: Audio data of user speech

[1058] Output: Textual speech data

[1059] Step 6:

[1060] The server uses conversational AI to generate natural dialogue based on the received text data, and the generated dialogue becomes the character's response.

[1061] Input: User speech data converted to text

[1062] Output: Text data of the generated dialogue

[1063] Step 7:

[1064] The emotion engine analyzes the user's voice and facial expression data to assess their emotional state, and the server then adjusts the character's responses accordingly—for example, if the user is confused, the character's responses will be more helpful.

[1065] Input: User's voice data, facial expression data

[1066] Output: Evaluated emotional state, adjusted character response data

[1067] Step 8:

[1068] The terminal plays back the conversation's replies as the character's synthesized voice, continuing the dialogue with the user.

[1069] Input: Adjusted character response data, synthesized voice data

[1070] Output: Character's response in synthesized voice

[1071] Step 9:

[1072] After the lesson, the device sends the conversation data to the server, which uses conversational AI to evaluate the conversation data and generate feedback on pronunciation, grammar, fluency, emotion recognition, and more.

[1073] Input: Interaction data

[1074] Output: Rating and feedback data

[1075] Step 10:

[1076] The server sends the generated feedback to the device, which displays it to the user, including comments about learning progress and emotional fluctuations.

[1077] Input: Rating and feedback data

[1078] Output: Feedback that is displayed to the user

[1079] Step 11:

[1080] When the next learning session begins, the server generates a new English conversation scenario and sends it to the terminal.

[1081] Input: User's latest learning history and progress data

[1082] Output: New English conversation scenario

[1083] Step 12:

[1084] The server periodically checks the subscription status and automatically charges the monthly fee. It also sends reminder notifications to users when their subscription is about to expire and guides them through the renewal process.

[1085] Input: Subscription information

[1086] Output: Billing data, reminder notifications

[1087] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1088] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1089] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1090] [Third embodiment]

[1091] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1092] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1093] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1094] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1095] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1096] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1097] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1098] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1099] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1100] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1101] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1102] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1103] The present invention relates to a VR system that allows a user to receive English conversation coaching from a favorite actor or character. Specific embodiments for carrying out the present invention will be described below.

[1104] User Registration and Login

[1105] 1. The user installs and launches the application.

[1106] 2. The device displays an interface for the user to enter their name, email address, and favorite actor or character.

[1107] 3. The user enters the necessary information and presses the send button, and the device sends the information to the server.

[1108] 4. The server saves the user's registration information in the database. The registered email address and password are used for subsequent logins.

[1109] Preparation for synthetic voice and conversational AI

[1110] 5. Once the user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user.

[1111] 6. The server uses conversational AI to prepare an appropriate English conversation scenario for the user, which is customized based on the user's learning history and current progress.

[1112] English conversation coaching in a VR environment

[1113] 7. The device will load a virtual reality environment and display a 3D model of the selected actor or character.

[1114] 8. Users can enter a virtual environment using VR goggles and controllers and begin English conversations with actors and characters using synthesized voices.

[1115] 9. The device uses voice recognition technology to convert the user's speech into text and transmits it to the server in real time.

[1116] 10. The server analyzes the received text, uses conversational AI to generate natural dialogue, and sends the next actor or character response to the device.

[1117] 11. This allows users to practice English conversation in situations that are close to real life situations.

[1118] Feedback and continuous learning

[1119] 12. After the lesson is completed, the device sends the dialogue data to the server.

[1120] 13. The server uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates the results as feedback.

[1121] 14. The server sends the generated feedback to the device, which displays it to the user, allowing the user to see how their English conversation skills have improved.

[1122] 15. When the next learning session begins, the server generates a new conversation scenario and sends it to the device.

[1123] Subscription Management

[1124] 16. The server periodically checks the subscription status and automatically charges the monthly fee.

[1125] 17. When a subscription is about to expire, the server sends reminder notifications to users and takes them through the renewal process if necessary.

[1126] Example: When a user uses the system for the first time, he or she first enters the name "John" and email address on the registration screen and selects "Actor X" as the preferred actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares a scenario for ordering at a restaurant. John enters the VR environment and begins a conversation with Actor X, saying, "Hello, John! How can I help you today?" After the conversation ends, John receives feedback on his pronunciation and grammar and checks his or her skills.

[1127] In this way, users can continue to learn English conversation in a fun and enjoyable way.

[1128] The processing flow will be explained below.

[1129] Step 1:

[1130] The user installs and launches the application.

[1131] Step 2:

[1132] The device displays an interface for the user to select their name, email address, and favorite actor or character.

[1133] Step 3:

[1134] The user enters the necessary information and presses the send button.

[1135] Step 4:

[1136] The terminal transmits the user's input information to the server.

[1137] Step 5:

[1138] The server stores the received information in a database and returns a message indicating that the information was successfully stored to the terminal.

[1139] Step 6:

[1140] The terminal will display a successful save message to the user.

[1141] Step 7:

[1142] The user enters their email address and password on the login screen and clicks the login button.

[1143] Step 8:

[1144] The terminal transmits the entered login information to the server.

[1145] Step 9:

[1146] The server retrieves user information from the database and performs authentication.

[1147] Step 10:

[1148] If the authentication is successful, the server returns a message indicating that the authentication is successful to the terminal.

[1149] Step 11:

[1150] The device will notify the user that the login was successful and begin the process of loading the VR environment.

[1151] Step 12:

[1152] The server uses synthetic voice AI to generate voice data based on voice samples of the actor or character selected by the user.

[1153] Step 13:

[1154] The server uses conversational AI to prepare English conversation content and scenarios based on the user's learning level and history.

[1155] Step 14:

[1156] The server transmits the generated voice data and conversation scenario to the terminal.

[1157] Step 15:

[1158] The device loads 3D models into the VR environment and displays the model of the selected actor or character.

[1159] Step 16:

[1160] The user puts on VR goggles and controllers and enters the virtual environment.

[1161] Step 17:

[1162] The device plays a synthesized voice, with a character saying, "Hello, [user]! How are you today?"

[1163] Step 18:

[1164] The user presses a button on the controller to reply, "I'm good, thank you! How are you?"

[1165] Step 19:

[1166] The device converts the user's speech into text using voice recognition technology and sends the text to the server.

[1167] Step 20:

[1168] The server uses conversational AI to generate appropriate responses to the user's comments and sends those responses to the device.

[1169] Step 21:

[1170] The device then plays the received response as a synthesized voice of the character and continues the conversation.

[1171] Step 22:

[1172] This process repeats until the user ends the conversation.

[1173] Step 23:

[1174] After the conversation is completed, the terminal transmits the dialogue data to the server.

[1175] Step 24:

[1176] The server uses conversational AI to evaluate the dialogue data and generate feedback on items such as pronunciation, grammar, and fluency.

[1177] Step 25:

[1178] The server transmits the generated feedback to the terminal.

[1179] Step 26:

[1180] The terminal displays feedback to the user, allowing them to confirm the learning results.

[1181] Step 27:

[1182] The next time you study, the server will generate a new English conversation scenario and send it to your device.

[1183] Step 28:

[1184] The server periodically checks the subscription status and bills the monthly fee.

[1185] Step 29:

[1186] When the subscription is about to expire, the server will send a renewal reminder to the device.

[1187] Step 30:

[1188] The device will display an update notification to the user and prompt them to take the necessary steps.

[1189] Example 1

[1190] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1191] Conventional English conversation learning systems often lack the realism required for users to practice English conversation in real situations. They also lack regular feedback and sufficient evaluation of pronunciation and grammar, making it difficult for users to continuously improve their skills. Furthermore, subscription management is often done manually, placing a heavy burden on users.

[1192] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1193] In this invention, the server includes means for providing an interface for inputting user registration information, means for generating synthetic speech using a voice sample of a character selected by the user, means for preparing English conversation content for the user and creating a conversation scenario, means for displaying a three-dimensional model in a virtual reality environment, means for converting the user's utterances into text using voice recognition technology, means for generating natural dialogue based on the user's utterances, means for evaluating the dialogue results and providing feedback to the user, means for providing continuous learning scenarios and managing subscriptions, and means for periodically checking the subscription status and automatically charging fees. This allows users to practice English conversation in realistic situations and improve their skills by receiving continuous feedback and evaluation, while also reducing the burden of subscription management.

[1194] The "interface for entering user registration information" is a graphical input screen that allows a user to enter their name, email address, and other required information.

[1195] The "means for generating synthetic voice" is a mechanism for converting text data into voice data based on a voice sample of a character selected by the user.

[1196] "Means for creating conversation scenarios" refers to a method for automatically generating appropriate English conversation scenes and situations based on the user's learning history and progress.

[1197] "Means for displaying a three-dimensional model in a virtual reality environment" refers to a method of displaying a character selected by the user in a three-dimensional virtual space using VR goggles or a display.

[1198] "Means for converting user speech into text using speech recognition technology" refers to technology that analyzes the user's speech in real time and converts it into corresponding text data.

[1199] "Means for generating natural dialogue" refers to a method for using conversational AI to generate natural-flowing dialogue based on user input.

[1200] "Means for evaluating the dialogue results and providing feedback to the user" refers to a mechanism that analyzes the content of the dialogue with the user, evaluates pronunciation, grammar, fluency, etc., and notifies the user of the results.

[1201] "Means for providing continuous learning scenarios" refers to a method for continuously providing new learning content based on the user's progress and feedback results.

[1202] A "means for managing subscriptions" is a mechanism for checking the user's subscription status and periodically billing the service fee.

[1203] An "automatic billing mechanism" is a system that automatically processes payment of the subscription fee during the subscription period.

[1204] MODE FOR CARRYING OUT THE INVENTION

[1205] The present invention relates to a VR system that allows a user to enjoy learning English conversation while receiving coaching from a character selected by the user. Specific embodiments for carrying out the present invention will be described below.

[1206] User Registration and Login

[1207] First, the user installs and launches the application. At this stage, the device displays an interface for the user to enter their name, email address, and favorite character. After the user enters the required information and presses the submit button, the device sends the information to the server. The server stores the user's registration information in a database, and the registered email address and password are used for subsequent logins.

[1208] Preparation for synthetic voice and conversational AI

[1209] When a user logs in, the server generates synthetic speech based on the voice samples of the character selected by the user. Specifically, a speech synthesis engine (e.g., Google Cloud Text-to-Speech or Amazon Polly) is used. The server also uses a conversational AI model (e.g., OpenAI's GPT-3) to prepare an English conversation scenario appropriate for the user. This scenario is customized based on the user's learning history and current progress.

[1210] English conversation coaching in a VR environment

[1211] Next, the device loads a virtual reality environment (VR environment) and displays a three-dimensional model of the selected character. VR goggles are mainly used, such as Oculus Rift or HTC Vive. The user enters the virtual environment using the VR goggles and controllers and can begin English conversation with the character using synthesized voice. The device converts the user's speech into text using voice recognition technology (e.g., Google Speech-to-Text) and sends it to the server in real time. The server analyzes the received text, generates natural dialogue using conversational AI, and sends the character's next response to the device. This allows the user to practice English conversation in a manner that is close to real-life situations.

[1212] Feedback and continuous learning

[1213] After the lesson is completed, the device sends the dialogue data to the server. The server uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results. The generated feedback is displayed to the user via the device, allowing the user to see areas where their English conversation skills have improved. When the next lesson begins, the server generates a new dialogue scenario and sends it to the device.

[1214] Subscription Management

[1215] The server periodically checks the subscription status and automatically bills the monthly fee. When the subscription expiration date approaches, the server sends a reminder to the user and takes care of the renewal procedure if necessary.

[1216] Specific examples

[1217] For example, when a user uses the system for the first time, they enter their name "John" and email address on the registration screen and select "Character X" as their favorite character. After logging in, the server generates a synthesized voice for "Character X" and prepares a scenario for ordering at a restaurant. Once John enters the VR environment, he begins a conversation with Character X, saying, "Hello, John! How can I help you today?" After the conversation ends, John receives feedback on his pronunciation and grammar and checks his skills.

[1218] Prompt Sentence Examples

[1219] "Please explain the details of the VR system that allows users to receive English conversation coaching with their favorite characters."

[1220] Thus, the present invention provides a system that allows users to practice English conversation in realistic situations and improve their skills through continuous feedback and evaluation.

[1221] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1222] Step 1:

[1223] The user installs and launches the application. After installation, by opening the application, the device displays an interface for user registration.

[1224] Input: The user installs and launches the app.

[1225] Output: The user registration interface is displayed.

[1226] Step 2:

[1227] The terminal provides the user with a form to enter their name, email address, and favorite character. The user enters the required information into the form and presses the submit button.

[1228] Input: A user fills out a form with their name, email address, and favorite character.

[1229] Output: The entered information is sent to the server.

[1230] Step 3:

[1231] The terminal sends the user information to the server, which stores the information in a database.

[1232] Input: User information sent from the device.

[1233] Output: User information stored on the server.

[1234] Step 4:

[1235] When a user logs in, the server generates a synthesized voice based on a voice sample of the character selected by the user.

[1236] Input: User login information and selected character.

[1237] Output: Synthetic speech data is generated.

[1238] Step 5:

[1239] The server uses a conversational AI model to prepare a suitable English conversation scenario for the user, which is customized based on the user's learning history and current progress.

[1240] Input: User learning history and progress data.

[1241] Output: A customized English conversation scenario.

[1242] Step 6:

[1243] The device loads a virtual reality environment and displays a three-dimensional model of the selected character. The user enters the virtual environment using VR goggles and controllers.

[1244] Input: English conversation scenario and character 3D model data.

[1245] Output: The character displayed in a VR environment.

[1246] Step 7:

[1247] The device converts the user's speech into text using voice recognition technology and sends it to the server in real time. Google Speech-to-Text is used for voice recognition.

[1248] Input: User speech.

[1249] Output: Textualized speech data.

[1250] Step 8:

[1251] The server analyzes the textual data and uses conversational AI to generate natural dialogue, which is then sent to the device.

[1252] Input: Text data of user utterances.

[1253] Output: Natural dialogue.

[1254] Step 9:

[1255] The device displays the generated dialogue, and the user continues the conversation, allowing the user to practice English conversation in a realistic situation.

[1256] Input: The generated dialogue.

[1257] Output: The displayed dialogue.

[1258] Step 10:

[1259] The device sends the conversation data to a server, which uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results.

[1260] Input: Interaction data.

[1261] Output: Feedback data.

[1262] Step 11:

[1263] The server sends the generated feedback to the terminal, which displays it to the user, allowing the user to see how their English conversation skills have improved.

[1264] Input: Feedback data.

[1265] Output: The feedback displayed to the user.

[1266] Step 12:

[1267] At the start of the next learning session, the server generates a new conversation scenario and sends it to the terminal.

[1268] Input: User learning history and feedback.

[1269] Output: A new conversation scenario.

[1270] Step 13:

[1271] The server periodically checks the subscription status and automatically bills users, sending reminders when their subscription is about to expire.

[1272] Input: Subscription information.

[1273] Output: Automatic billing and reminder notifications.

[1274] (Application example 1)

[1275] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1276] In conventional English conversation learning systems, users have limited opportunities to practice English conversation in situations close to real conversations. Furthermore, there are no systems that can provide users with an engaging experience of receiving English conversation coaching from their favorite actors or characters, making it difficult to maintain motivation to continue learning.

[1277] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1278] In this invention, the server includes a means for generating synthetic speech based on voice samples of actors or characters selected by the user, a means for providing an application to be installed on a smartphone to execute the generated conversation scenario, and a means for generating new conversation scenarios based on the user's learning progress via the server. This allows users to practice English conversation in a manner that is close to actual conversation situations through real-time dialogue with their favorite actors or characters. Furthermore, the provision of new scenarios based on the user's learning progress allows the user to enjoy constantly fresh learning content, contributing to maintaining motivation for continuous learning.

[1279] 1. An "interface for entering user registration information" is a screen or form that provides a user with the ability to enter information such as their name, email address, and favorite actors or characters.

[1280] 2. "Means for generating synthetic speech" refers to a program or algorithm that generates speech data that enables natural dialogue based on voice samples of an actor or character selected by the user.

[1281] 3. "Means for creating conversation scenarios" refers to a system for designing and preparing appropriate English conversation content in advance based on the user's learning progress and preferences.

[1282] 4. "Means for displaying 3D models in a virtual reality environment" means a system for displaying 3D character or actor models based on a user's selection using a VR headset or display.

[1283] 5. "Means of converting user speech into text using speech recognition technology" refers to technology or programs that analyze what a user says in real time and convert it into text data.

[1284] 6. "Means for generating natural dialogue" refers to a conversational AI system that generates appropriate responses based on the user's textual utterances and provides them as natural conversation.

[1285] 7. "Means for providing an application to be installed on a smartphone" means a method or service for a user to download and install the software necessary to use the English conversation coaching system on a smartphone.

[1286] 8. "Means for generating new conversation scenarios based on the user's learning progress via a server" refers to a server-based system that analyzes the user's learning data and progress to generate and provide new English conversation scenarios.

[1287] 9. "Means for evaluating a user's skills based on feedback from a server and presenting the results in the form of a widget" refers to a method for evaluating a user's pronunciation, grammar, and fluency using feedback information sent from a server, and displaying the evaluation results as a widget on a smartphone.

[1288] 10. "Means for evaluating dialogue results and providing feedback to the user" refers to a mechanism that analyzes the content of the dialogue conducted by the user, evaluates it based on the accuracy of pronunciation and grammar, and returns the results to the user.

[1289] 11. "Means for providing continuous learning scenarios and managing subscriptions" means a system for operating a subscription service for which a recurring fee is charged, by periodically providing new learning scenarios to users and managing and supervising their usage.

[1290] The following hardware and software are required to implement this invention. The hardware required is a smartphone connected to the internet, and VR goggles and controllers may be used as options. The software includes Python, PyQt5, requests, speech_recognition, and pyttsx3.

[1291] When a user installs and launches the application, an interface for entering user registration information is displayed. In this interface, the user enters their name, email address, and information about their favorite actors and characters. This information is sent to the server. The server creates a user profile based on the received user information and stores the registration information in a database.

[1292] When a user logs in, the server generates a synthesized voice based on the voice samples of the actor or character selected by the user. This synthesized voice is generated as voice data using the pyttsx3 library. The server also uses conversational AI to prepare an English conversation scenario suited to the user based on the user's learning history and progress. This allows the user to practice English conversation based on the scenario.

[1293] Next, users use a smartphone application to enter the virtual reality environment. The application works in conjunction with VR goggles and controllers to load the virtual environment and display a 3D model of the selected actor or character. Within the VR environment, users can begin English conversations with the actor or character using synthesized voice.

[1294] The user's speech is converted into text using voice recognition technology and sent to the server in real time. The server analyzes the received text data and generates natural dialogue using conversational AI. As a result, the next response of the actor or character is sent to the device, allowing the user to continue the dialogue in real time.

[1295] Once the conversation is over, the server uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results. This feedback is sent to the device and displayed to the user in the form of a widget, allowing the user to see where their skills have improved.

[1296] As a concrete example, when a user uses the system for the first time, they enter their name "John" and email address on the registration screen and select "Actor X" as their favorite actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares a scenario for ordering at a restaurant. John enters the VR environment and begins a conversation with "Actor X": "Hello, John! How can I help you today?" After the conversation ends, John receives feedback on his pronunciation and grammar and checks his skills.

[1297] Additionally, by inputting the following prompt sentences into the generative AI model, it is possible to generate an appropriate conversation scenario:

[1298] The user has selected "Actor X" as their favorite actor. Please generate natural dialogue to support the user in English conversation in a situation where they are ordering at a restaurant.

[1299] In this way, users can improve their English conversation skills while experiencing conversations with virtual actors and characters.

[1300] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1301] Step 1:

[1302] The user installs and launches the application. The device displays an interface for entering user registration information. The user enters their name, email address, and information about their favorite actor or character, and presses the submit button. The input data is sent to the server. The server receives the user information and stores it in a database.

[1303] Step 2:

[1304] The user logs in. The server verifies the user information and authenticates the login. If authentication is successful, a synthesized voice is generated based on the voice samples of the actor or character selected by the user. The server generates the voice data using the pyttsx3 library and sends the result to the device.

[1305] Step 3:

[1306] The server uses conversational AI to prepare an English conversation scenario based on the user's learning history and progress. A generative AI model is then used to generate a conversation scenario based on a specified prompt. The prompt, "The user has selected 'Actor X' as their favorite actor. Please generate a natural dialogue to support the user in English conversation in a situation where they are ordering at a restaurant," is input, and the AI ​​outputs a response scenario.

[1307] Step 4:

[1308] The device loads a virtual reality environment and displays a 3D model of the selected actor or character. The user enters the virtual environment using VR goggles and controllers. The device plays synthesized voice data from the server and begins a conversation in English with the user. The user's speech is input through a microphone.

[1309] Step 5:

[1310] The device uses voice recognition technology to convert the user's speech into text in real time. The text data is then sent to a server. The server then analyzes the received text data and uses conversational AI to generate natural dialogue. The server then generates the next actor or character's response and sends the resulting voice data to the device.

[1311] Step 6:

[1312] The user continues to have real-time conversations with actors and characters using synthesized speech. The device plays back the voice data sent from the server, providing the user with a realistic conversation experience. Every time the user speaks, speech recognition and text conversion are repeated.

[1313] Step 7:

[1314] Once the conversation is complete, the device sends the conversation data to the server, which uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results. The generated feedback is sent to the device and presented to the user in the form of a widget.

[1315] Step 8:

[1316] When the next learning session begins, the server generates a new conversation scenario based on the user's learning progress. It then uses the generative AI model again to generate a conversation scenario based on the new prompt. The server then sends the generated conversation scenario to the device, providing a continuous learning scenario.

[1317] Step 9:

[1318] The server periodically checks the subscription status and automatically bills the monthly fee. When the subscription expiration date approaches, the server sends a reminder to the user and takes care of the renewal procedure if necessary.

[1319] These steps allow users to continuously improve their English skills by experiencing English conversations with virtual actors and characters. Furthermore, by generating conversation scenarios using a generative AI model, users can enjoy constantly new learning content.

[1320] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1321] The present invention combines an emotion engine with a VR system that allows users to receive English conversation coaching from their favorite actors or characters. Specific embodiments for carrying out the present invention will be described below.

[1322] User Registration and Login

[1323] 1. The user installs and launches the application.

[1324] 2. The device displays an interface for the user to enter their name, email address, and select their favorite actor or character.

[1325] 3. The user enters the necessary information and presses the send button, and the device sends the information to the server.

[1326] 4. The server saves the user's registration information in the database. The registered email address and password are used for subsequent logins.

[1327] Preparation of synthetic voice, conversational AI and emotion recognition engine

[1328] 5. Once the user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user.

[1329] 6. The server uses conversational AI to prepare an appropriate English conversation scenario for the user, which is customized based on the user's learning history and current progress.

[1330] 7. The server also provides an emotion engine that can recognize emotions from the user's words and facial expressions.

[1331] English conversation coaching in a VR environment

[1332] 8. The device will load a virtual reality environment and display a 3D model of the selected actor or character.

[1333] 9. Users can enter a virtual environment using VR goggles and controllers and begin English conversations with characters using synthesized voices.

[1334] 10. The device uses voice recognition technology to convert the user's speech into text and transmits it to the server in real time.

[1335] 11. The server analyzes the received text, uses conversational AI to generate natural dialogue, and sends the character's next response to the device.

[1336] 12. The emotion engine analyzes the user's voice and facial expression data to assess their emotional state in real time, adjusting the tone and content of the conversation accordingly.

[1337] 13. The device plays back the conversational responses as the character's synthesized voice and continues the dialogue with the user.

[1338] Feedback and continuous learning

[1339] 14. After the lesson is completed, the device sends the dialogue data to the server.

[1340] 15. The server uses conversational AI to evaluate the dialogue data and generate feedback on items including pronunciation, grammar, fluency, and emotion recognition.

[1341] 16. The server sends the generated feedback to the device, which displays it to the user. This feedback includes comments about the learning progress and emotional fluctuations.

[1342] 17. When the next learning session begins, the server generates a new English conversation scenario and sends it to the device.

[1343] Subscription Management

[1344] 18. The server periodically checks the subscription status and automatically charges the monthly fee.

[1345] 19. When a subscription is about to expire, the server sends reminder notifications to users and takes them through the renewal process if necessary.

[1346] Example: When a user uses the system for the first time, they first enter their name "John" and email address on the registration screen and select "Actor X" as their favorite actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares the emotion engine. A scenario of ordering at a restaurant is prepared, and John enters the VR environment and starts a conversation with Actor X, saying, "Hello, John! How are you today?" When John replies, "I'm good, thank you! How are you?", the emotion engine analyzes John's emotional state from his voice and facial expressions and adjusts the character's response. After the conversation ends, John receives feedback on his pronunciation, grammar, and emotional fluctuations to check his skills.

[1347] In this way, users can continue to learn English conversation in a fun and enjoyable way while receiving coaching in an approach that responds to emotional fluctuations.

[1348] The processing flow will be explained below.

[1349] Step 1:

[1350] The user installs and launches the application.

[1351] Step 2:

[1352] The device displays an interface for the user to select their name, email address, and favorite actor or character.

[1353] Step 3:

[1354] The user enters the necessary information and presses the send button.

[1355] Step 4:

[1356] The terminal transmits the user's input information to the server.

[1357] Step 5:

[1358] The server stores the received information in a database and returns a message indicating that the information was successfully stored to the terminal.

[1359] Step 6:

[1360] The terminal will display a successful save message to the user.

[1361] Step 7:

[1362] The user enters their email address and password on the login screen and clicks the login button.

[1363] Step 8:

[1364] The terminal transmits the entered login information to the server.

[1365] Step 9:

[1366] The server retrieves user information from the database and performs authentication.

[1367] Step 10:

[1368] If the authentication is successful, the server returns a message indicating that the authentication is successful to the terminal.

[1369] Step 11:

[1370] The device will notify the user that the login was successful and begin the process of loading the VR environment.

[1371] Step 12:

[1372] The server uses synthetic voice AI to generate voice data based on voice samples of the actor or character selected by the user.

[1373] Step 13:

[1374] The server uses conversational AI to prepare English conversation content and scenarios based on the user's learning level and history.

[1375] Step 14:

[1376] The server also has an emotion engine and is configured to recognize emotions in real time from the user's comments and facial expressions.

[1377] Step 15:

[1378] The server transmits the generated voice data, the conversation scenario, and the emotion engine settings to the terminal.

[1379] Step 16:

[1380] The device loads 3D models into the VR environment and displays the model of the selected actor or character.

[1381] Step 17:

[1382] The user puts on VR goggles and controllers and enters the virtual environment.

[1383] Step 18:

[1384] The device plays a synthesized voice, with a character saying, "Hello, [user]! How are you today?"

[1385] Step 19:

[1386] The user presses a button on the controller to reply, "I'm good, thank you! How are you?"

[1387] Step 20:

[1388] The device converts the user's speech into text using voice recognition technology and sends the text to the server.

[1389] Step 21:

[1390] The server uses conversational AI to generate appropriate responses to user comments.

[1391] Step 22:

[1392] The emotion engine analyzes the user's voice and facial expression data to assess their emotional state in real time.

[1393] Step 23:

[1394] The server adjusts the character's next response based on the emotional data evaluated by the emotion engine and sends that data to the terminal.

[1395] Step 24:

[1396] The device then plays the received response as a synthesized voice of the character and continues the conversation.

[1397] Step 25:

[1398] This process repeats until the user ends the conversation.

[1399] Step 26:

[1400] After the conversation is completed, the terminal transmits the dialogue data to the server.

[1401] Step 27:

[1402] The server uses conversational AI to evaluate the dialogue data and generate feedback based on pronunciation, grammar, fluency, and emotion recognition.

[1403] Step 28:

[1404] The server transmits the generated feedback to the terminal.

[1405] Step 29:

[1406] The terminal displays feedback to the user, allowing them to confirm the learning results.

[1407] Step 30:

[1408] The next time you study, the server will generate a new English conversation scenario and send it to your device.

[1409] Step 31:

[1410] The server periodically checks the subscription status and bills the monthly fee.

[1411] Step 32:

[1412] When the subscription is about to expire, the server will send a renewal reminder to the device.

[1413] Step 33:

[1414] The device will display an update notification to the user and prompt them to take the necessary steps.

[1415] Example 2

[1416] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1417] Conventional English conversation learning systems use fixed scenarios and general speech synthesis technology without taking into account the learner's emotional state, which limits the effectiveness of learning. Furthermore, it is difficult to provide scenarios tailored to individual learners, making it difficult to maintain continuous learning motivation. Furthermore, since pronunciation and grammar evaluation is not performed in real time, feedback tends to be delayed, making it difficult to improve practical conversational skills.

[1418] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1419] In this invention, the server includes: means for providing an interface for inputting user registration information; means for generating synthetic speech using voice samples of actors or characters selected by the user; means for preparing English conversation content for the user and creating a conversation scenario; means for displaying a 3D model in a virtual reality environment; means for converting user utterances into text using speech recognition technology; means for generating natural dialogue based on the user utterances; means for evaluating the dialogue results and providing feedback to the user; means for providing continuous learning scenarios and managing subscriptions; means for recognizing emotions from user utterances and facial expressions using an emotion recognition engine; and means for providing customized English conversation scenarios using a generative AI model. This enables more effective English conversation learning by providing scenarios and feedback in real time that take the learner's emotional state into account.

[1420] "User registration information" refers to information such as name, email address, password, favorite actor or character, etc., that a user enters to use an application.

[1421] An "interface" refers to the screen or operating means through which a user inputs information into a system or performs operations.

[1422] "Voice Sample" means the recorded or digitized voice of a particular actor or character.

[1423] "Synthetic voice" is an artificial voice that imitates a human voice and is created based on voice samples.

[1424] "English conversation content" refers to conversation themes, phrases, and scenarios provided for users to study.

[1425] A "conversation scenario" is a series of dialogues that assume a specific situation, and is the content of the conversation that takes place between the user and a virtual character.

[1426] "Virtual reality environment" means a visual and operational environment for displaying 3D models and for allowing a user to experience them in virtual reality.

[1427] A "3D model" is a character or object that is represented in three dimensions using computer graphics.

[1428] "Speech recognition technology" refers to technology that recognizes what a user says and converts the voice data into text data.

[1429] "Natural dialogue" refers to the exchange of fluent, natural sentences, like conversations between people.

[1430] "Feedback" refers to evaluations, advice, and suggestions for improvement given to users in response to their learning and operation results.

[1431] A "continuous learning scenario" is a learning plan or scenario that is provided step by step to enable the user to continue learning.

[1432] "Subscription management" is a system for managing users' contract status and payment fees for subscription-based services.

[1433] An "emotion recognition engine" is a system that analyzes and recognizes a user's emotional state from their speech and facial expressions.

[1434] A "generative AI model" is an artificial intelligence model that generates new data (in this case, English conversation scenarios) based on input data.

[1435] The present invention combines an emotion engine with a VR system that allows users to receive English conversation coaching from their favorite actors or characters. Specific embodiments for carrying out the present invention will be described below.

[1436] User Registration and Login

[1437] When a user installs and launches the application, the device displays an interface for the user to select their name, email address, and favorite actor or character. After the user enters the required information and presses the submit button, the device sends the information to the server. This data is stored in a database by the server. The registered email address and password are used for subsequent logins.

[1438] Preparation of synthetic voice, conversational AI and emotion recognition engine

[1439] When a user logs in, the server generates synthetic speech based on voice samples of the actor or character selected by the user. To achieve this, it uses OpenAI's Text-to-Speech technology. The server also uses conversational AI (GPT-3) to prepare an English conversation scenario suited to the user. This scenario is customized based on the user's learning history and current progress. The server also uses Affectiva's emotion recognition engine to recognize emotions from the user's speech and facial expressions.

[1440] English conversation coaching in a VR environment

[1441] The device loads a virtual reality environment and displays a 3D model of the selected actor or character. This virtual environment is built using Unity. Users enter the virtual environment using VR goggles and controllers and can begin English conversations with the character using synthetic voice. The device uses Google Speech-to-Text to convert the user's speech into text and send it to the server in real time. The server analyzes the received text, generates natural-sounding dialogue using GPT-3, and sends the character's next response to the device. The emotion engine analyzes the user's voice and facial expression data and evaluates their emotional state in real time. Based on this, the tone and content of the conversation are adjusted, and the device plays back the character's replies as synthetic voice, continuing the dialogue with the user.

[1442] Feedback and continuous learning

[1443] After completing a lesson, the device sends the dialogue data to the server. The server uses conversational AI to evaluate the dialogue data and generate feedback on items including pronunciation, grammar, fluency, and emotion recognition. This feedback is sent to the device and displayed to the user. The feedback also includes comments on learning progress and emotional fluctuations. When starting the next lesson, the server generates a new English conversation scenario and sends it to the device.

[1444] Subscription Management

[1445] The server periodically checks the subscription status and automatically bills the monthly fee. When the subscription expiration date approaches, the server sends a reminder to the user and takes care of the renewal procedure if necessary.

[1446] Examples:

[1447] When a user first uses the system, they first enter their name ("John") and email address on the registration screen and select "Actor X" as their preferred actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares the emotion engine. A scenario of ordering at a restaurant is prepared, and John, entering the VR environment, begins a conversation with Actor X: "Hello, John! How are you today?" When John replies, "I'm good, thank you! How are you?", the emotion engine analyzes John's emotional state from his voice and facial expressions and adjusts the character's response accordingly. After the conversation ends, John receives feedback on his pronunciation, grammar, and emotional fluctuations to check his skills. In this way, users can continuously and enjoyably learn English conversation while receiving coaching tailored to their emotional fluctuations.

[1448] Example prompt for a generative AI model:

[1449] "Generate an English conversation scenario in which a user orders from actor X at a restaurant. Please include the flow of the conversation when the user has positive emotions."

[1450] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1451] Step 1:

[1452] The user installs and launches the application.

[1453] How it works: A user downloads an app from the app store on their smartphone, tablet, or PC, installs it, and then launches it. They then tap the icon of the installed app to launch it.

[1454] Step 2:

[1455] The device displays an interface for the user to select their name, email address, and favorite actor or character.

[1456] Input: None

[1457] Output: User input form

[1458] Operation details: The app screen displays a form for entering name, email address, password, and favorite actor or character.

[1459] Step 3:

[1460] The user enters the necessary information and presses the send button, and the terminal sends the information to the server.

[1461] Input: Name, email address, password, information about your favorite actor or character

[1462] Output: User-entered data is sent to the server

[1463] How it works: After the user enters their information, they tap the "Submit" button, which sends the data to the server via an HTTP request.

[1464] Step 4:

[1465] The server stores the user's registration information in a database.

[1466] Input: User-entered data

[1467] Output: User information stored in the database

[1468] How it works: A server-side script (e.g. Python, Node.js) receives HTTP requests and stores information in a database (e.g. MySQL, MongoDB).

[1469] Step 5:

[1470] Once a user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user.

[1471] Input: Information about the actor or character selected by the user

[1472] Output: Synthesized speech data

[1473] How it works: Using OpenAI's Text-to-Speech technology, a generation process is run based on a sample voice of a selected actor, and a synthesized voice file is temporarily saved.

[1474] Step 6:

[1475] The server uses conversational AI to prepare English conversation scenarios suitable for the user.

[1476] Input: User's learning history, current progress

[1477] Output: Customized English conversation scenario

[1478] How it works: Uses GPT-3 to generate prompts and create customized English conversation scenarios.

[1479] Step 7:

[1480] The server provides an emotion engine that can recognize emotions from the user's words and facial expressions.

[1481] Input: User's voice data, facial expression data

[1482] Output: Emotion recognition result

[1483] How it works: Affectiva's emotion recognition engine is used to analyze the user's voice and video data in real time.

[1484] Step 8:

[1485] The device loads a virtual reality environment and displays a 3D model of the selected actor or character.

[1486] Input: None

[1487] Output: Display of VR environment and 3D model

[1488] How it works: Uses Unity to load VR environments and 3D models and display them through a VR headset.

[1489] Step 9:

[1490] Users enter the virtual environment using VR goggles and controllers.

[1491] Input: None

[1492] Output: Immersive experience in a virtual environment

[1493] Operation details: Put on the VR goggles, pick up the controller and start operating. Positioning and field of view adjustments will be performed, and a tutorial on how to use the device will be displayed.

[1494] Step 10:

[1495] The device uses voice recognition technology to convert the user's speech into text and transmits it to the server in real time.

[1496] Input: User utterance

[1497] Output: Textualized speech data

[1498] Operation details: Uses Google Speech-to-Text to convert the user's speech into text and send it to the server via an HTTP request.

[1499] Step 11:

[1500] The server analyzes the received text, uses conversational AI to generate natural dialogue, and sends the character's next response to the device.

[1501] Input: Textual speech data

[1502] Output: Character reaction data

[1503] How it works: It uses GPT-3 to generate an appropriate response from the received text and sends it to the device.

[1504] Step 12:

[1505] The emotion engine analyzes the user's voice and facial expression data to assess their emotional state in real time.

[1506] Input: User's voice data, facial expression data

[1507] Output: Emotional state assessment result

[1508] How it works: Using Affectiva's engine, audio and video data is analyzed in real time to assess emotional state.

[1509] Step 13:

[1510] The terminal plays back the conversation's replies as the character's synthesized voice, continuing the dialogue with the user.

[1511] Input: Character reaction data

[1512] Output: Synthesized voice of the character

[1513] How it works: Plays a synthesized voice file and synchronizes the character's lip-syncing and facial expressions to create a real-time dialogue.

[1514] Step 14:

[1515] After the lesson is completed, the terminal transmits the dialogue data to the server.

[1516] Input: Interaction data

[1517] Output: Interaction data sent to the server

[1518] Operation details: Sends locally stored interaction data to the server.

[1519] Step 15:

[1520] The server uses conversational AI to evaluate the dialogue data and generate feedback on items including pronunciation, grammar, fluency, and emotion recognition.

[1521] Input: Interaction data

[1522] Output: Feedback data

[1523] How it works: Analyzes data based on various evaluation criteria and generates feedback reports.

[1524] Step 16:

[1525] The server sends the generated feedback to the terminal, which displays it to the user.

[1526] Input: Feedback data

[1527] Output: Feedback displayed on the terminal

[1528] How it works: Receives feedback data and displays it to the user in a visually friendly interface.

[1529] Step 17:

[1530] When the next learning session begins, the server generates a new English conversation scenario and sends it to the terminal.

[1531] Input: User learning history and progress data

[1532] Output: New English conversation scenario data

[1533] Operation details: Based on the user's past learning history, a new prompt is created and sent to the device as the next scenario.

[1534] Step 18:

[1535] The server periodically checks the subscription status and automatically charges the monthly fee.

[1536] Input: Subscription data

[1537] Output: Billing data

[1538] What it does: Queries user subscription data and manages expiration dates and billing information.

[1539] Step 19:

[1540] The server sends reminder notifications to users when their subscriptions are about to expire, and takes them through the renewal process if necessary.

[1541] Input: Subscription expiration data

[1542] Output: Reminder notification

[1543] What it does: Reminds users via email or in-app notification to renew their subscription.

[1544] (Application example 2)

[1545] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1546] Conventional English conversation learning systems do not provide dialogue that takes into account the user's emotions, limiting the learning effect. Furthermore, it has been difficult for content distribution services to provide an interactive experience for the content being viewed. The present invention aims to solve these problems, enhance the effectiveness of learning through dialogue that reflects the user's emotions, and provide a new interactive experience for content distribution services.

[1547] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for providing an interface for inputting user registration information, means for generating synthetic speech using voice samples of actors or characters selected by the user, means for preparing English conversation content for the user and creating a conversation scenario, means for displaying a 3D model in a virtual reality environment, means for converting user utterances into text using voice recognition technology, means for generating natural dialogue based on the user utterances, means for analyzing the user's emotions and dynamically changing the character's responses based on the results, means for providing a function that enables interactive conversations with drama and movie characters in a content distribution service, means for evaluating the dialogue results and providing feedback to the user, and means for providing continuous learning scenarios and managing subscriptions. This allows users to learn English conversation interactively through emotion recognition and enjoy dialogue with characters while watching content.

[1548] "User registration information" refers to information such as name, email address, password, and preferred actors and characters that a user enters to use the system.

[1549] "Interface" refers to a screen or input means for exchanging information between a user and a system.

[1550] "Synthetic voice" is an artificial voice generated by a computer based on real voice samples.

[1551] A "conversation scenario" is a pre-set script or content used in a dialogue with a user.

[1552] A "virtual reality environment" is a computer-generated environment used to create a user's experience in a virtual three-dimensional space.

[1553] A "3D model" is a digital representation of an object or character displayed in three dimensions using computer graphics.

[1554] "Speech recognition technology" is a technology for analyzing human speech and converting it into text data.

[1555] "Natural dialogue" is a form of dialogue that allows for fluent and flexible communication, like conversation between humans.

[1556] "Means for analyzing emotions" refers to technology that analyzes the user's voice and facial expressions to recognize their emotional state.

[1557] An "interactive conversation" is a two-way dialogue between a user and a system, in which the content of the dialogue changes depending on the user's input and the situation.

[1558] "Means for evaluating dialogue results" refers to technology for analyzing the content of a conversation and evaluating its quality, the user's reaction, learning status, etc.

[1559] A "continuous learning scenario" is a series of learning contents or scenarios that are provided in stages according to the user's learning progress and ability.

[1560] "Means for managing subscriptions" refers to technology for managing users' usage contracts and payment status, and for carrying out procedures for periodic billing and contract renewals.

[1561] The system for implementing this invention includes an interface for entering user registration information, synthetic speech, conversation scenario creation, a virtual reality environment, speech recognition technology, natural dialogue generation, emotion recognition, interactive conversation realization, dialogue result evaluation, continuous learning scenarios, and subscription management. By integrating these means, the system provides users with a high-level interactive English conversation learning and entertainment experience.

[1562] First, the server provides an interface for receiving user registration information. Through this interface, users enter their name, email address, and favorite actors or characters. This information is sent to the server and stored in a database.

[1563] The server then generates a synthetic voice using voice samples of the actor or character selected by the user. This synthetic voice is used during dialogue with the user. The server also prepares English conversation content and customizes the conversation scenario based on the user's learning history and progress.

[1564] The virtual reality environment is displayed on the device using VR goggles and a controller. 3D model characters are placed in this VR environment, and the user interacts with them in this virtual space. Voice recognition technology converts the user's speech into text and sends it to the server. The server then uses conversational AI to generate natural dialogue based on the text.

[1565] Furthermore, emotion recognition technology analyzes the user's voice and facial expressions to recognize their emotional state at any given moment, allowing the server to dynamically change the character's responses according to the user's emotions. For example, if the user is confused, the character will respond more kindly.

[1566] There is also a means for evaluating the dialogue results: the server analyzes the content of the conversation and provides feedback to the user on pronunciation, grammar, emotional fluctuations, etc. This allows the user to check their learning progress and identify areas for improvement.

[1567] Finally, continuous learning scenarios are provided and subscription management is also performed: new scenarios are provided to users on a regular basis, and monthly fees and contract renewals are automatically handled by the server.

[1568] As a specific example,

[1569] When user "John" uses the system for the first time, he first enters his name and email address and selects "Actor A" as his favorite actor. After logging in, the server generates a synthesized voice for "Actor A" and prepares the emotion engine. John puts on VR goggles and participates in a virtual restaurant ordering scenario. When he speaks, "Hello, how are you today?", the character responds, "I'm good, thank you! How are you?" based on emotion recognition technology. After the conversation ends, John receives feedback and checks his learning progress.

[1570] Example prompt sentence:

[1571] In a scenario where a user says "Hello, Character Y! How was your day?", Character Y should generate a natural response after analyzing the user's emotions.

[1572] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1573] Step 1:

[1574] A user installs and launches the application. The user enters their name, email address, and favorite actor or character into the interface. The device sends this information to the server, which stores it in a database.

[1575] Input: User's name, email address, favorite actor or character

[1576] Output: Registration information sent to the server

[1577] Step 2:

[1578] Once a user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user, and this generated voice is used to interact with the user.

[1579] Input: User login information (email address, password), selected actor and character information

[1580] Output: Synthesized voice data

[1581] Step 3:

[1582] The server creates customized English conversation scenarios based on the user's learning history and progress, including conversation topics and scenes.

[1583] Input: User learning history, progress data

[1584] Output: Customized conversation scenario

[1585] Step 4:

[1586] The device loads a virtual reality environment and displays a 3D model of the selected actor or character, and the user enters the virtual environment using VR goggles and controllers.

[1587] Input: English conversation scenario, 3D model data

[1588] Output: Virtual reality environment

[1589] Step 5:

[1590] When a user speaks in VR, the device uses voice recognition technology to convert the speech into text and send it to the server in real time.

[1591] Input: Audio data of user speech

[1592] Output: Textual speech data

[1593] Step 6:

[1594] The server uses conversational AI to generate natural dialogue based on the received text data, and the generated dialogue becomes the character's response.

[1595] Input: User speech data converted to text

[1596] Output: Text data of the generated dialogue

[1597] Step 7:

[1598] The emotion engine analyzes the user's voice and facial expression data to assess their emotional state, and the server then adjusts the character's responses accordingly—for example, if the user is confused, the character's responses will be more helpful.

[1599] Input: User's voice data, facial expression data

[1600] Output: Evaluated emotional state, adjusted character response data

[1601] Step 8:

[1602] The terminal plays back the conversation's replies as the character's synthesized voice, continuing the dialogue with the user.

[1603] Input: Adjusted character response data, synthesized voice data

[1604] Output: Character's response in synthesized voice

[1605] Step 9:

[1606] After the lesson, the device sends the conversation data to the server, which uses conversational AI to evaluate the conversation data and generate feedback on pronunciation, grammar, fluency, emotion recognition, and more.

[1607] Input: Interaction data

[1608] Output: Rating and feedback data

[1609] Step 10:

[1610] The server sends the generated feedback to the device, which displays it to the user, including comments about learning progress and emotional fluctuations.

[1611] Input: Rating and feedback data

[1612] Output: Feedback that is displayed to the user

[1613] Step 11:

[1614] When the next learning session begins, the server generates a new English conversation scenario and sends it to the terminal.

[1615] Input: User's latest learning history and progress data

[1616] Output: New English conversation scenario

[1617] Step 12:

[1618] The server periodically checks the subscription status and automatically charges the monthly fee. It also sends reminder notifications to users when their subscription is about to expire and guides them through the renewal process.

[1619] Input: Subscription information

[1620] Output: Billing data, reminder notifications

[1621] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1622] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1623] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1624] [Fourth embodiment]

[1625] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1626] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1627] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1628] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1629] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1630] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1631] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1632] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1633] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1634] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1635] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1636] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1637] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1638] The present invention relates to a VR system that allows a user to receive English conversation coaching from a favorite actor or character. Specific embodiments for carrying out the present invention will be described below.

[1639] User Registration and Login

[1640] 1. The user installs and launches the application.

[1641] 2. The device displays an interface for the user to enter their name, email address, and favorite actor or character.

[1642] 3. The user enters the necessary information and presses the send button, and the device sends the information to the server.

[1643] 4. The server saves the user's registration information in the database. The registered email address and password are used for subsequent logins.

[1644] Preparation for synthetic voice and conversational AI

[1645] 5. Once the user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user.

[1646] 6. The server uses conversational AI to prepare an appropriate English conversation scenario for the user, which is customized based on the user's learning history and current progress.

[1647] English conversation coaching in a VR environment

[1648] 7. The device will load a virtual reality environment and display a 3D model of the selected actor or character.

[1649] 8. Users can enter a virtual environment using VR goggles and controllers and begin English conversations with actors and characters using synthesized voices.

[1650] 9. The device uses voice recognition technology to convert the user's speech into text and transmits it to the server in real time.

[1651] 10. The server analyzes the received text, uses conversational AI to generate natural dialogue, and sends the next actor or character response to the device.

[1652] 11. This allows users to practice English conversation in situations that are close to real life situations.

[1653] Feedback and continuous learning

[1654] 12. After the lesson is completed, the device sends the dialogue data to the server.

[1655] 13. The server uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates the results as feedback.

[1656] 14. The server sends the generated feedback to the device, which displays it to the user, allowing the user to see how their English conversation skills have improved.

[1657] 15. When the next learning session begins, the server generates a new conversation scenario and sends it to the device.

[1658] Subscription Management

[1659] 16. The server periodically checks the subscription status and automatically charges the monthly fee.

[1660] 17. When a subscription is about to expire, the server sends reminder notifications to users and takes them through the renewal process if necessary.

[1661] Example: When a user uses the system for the first time, he or she first enters the name "John" and email address on the registration screen and selects "Actor X" as the preferred actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares a scenario for ordering at a restaurant. John enters the VR environment and begins a conversation with Actor X, saying, "Hello, John! How can I help you today?" After the conversation ends, John receives feedback on his pronunciation and grammar and checks his or her skills.

[1662] In this way, users can continue to learn English conversation in a fun and enjoyable way.

[1663] The processing flow will be explained below.

[1664] Step 1:

[1665] The user installs and launches the application.

[1666] Step 2:

[1667] The device displays an interface for the user to select their name, email address, and favorite actor or character.

[1668] Step 3:

[1669] The user enters the necessary information and presses the send button.

[1670] Step 4:

[1671] The terminal transmits the user's input information to the server.

[1672] Step 5:

[1673] The server stores the received information in a database and returns a message indicating that the information was successfully stored to the terminal.

[1674] Step 6:

[1675] The terminal will display a successful save message to the user.

[1676] Step 7:

[1677] The user enters their email address and password on the login screen and clicks the login button.

[1678] Step 8:

[1679] The terminal transmits the entered login information to the server.

[1680] Step 9:

[1681] The server retrieves user information from the database and performs authentication.

[1682] Step 10:

[1683] If the authentication is successful, the server returns a message indicating that the authentication is successful to the terminal.

[1684] Step 11:

[1685] The device will notify the user that the login was successful and begin the process of loading the VR environment.

[1686] Step 12:

[1687] The server uses synthetic voice AI to generate voice data based on voice samples of the actor or character selected by the user.

[1688] Step 13:

[1689] The server uses conversational AI to prepare English conversation content and scenarios based on the user's learning level and history.

[1690] Step 14:

[1691] The server transmits the generated voice data and conversation scenario to the terminal.

[1692] Step 15:

[1693] The device loads 3D models into the VR environment and displays the model of the selected actor or character.

[1694] Step 16:

[1695] The user puts on VR goggles and controllers and enters the virtual environment.

[1696] Step 17:

[1697] The device plays a synthesized voice, with a character saying, "Hello, [user]! How are you today?"

[1698] Step 18:

[1699] The user presses a button on the controller to reply, "I'm good, thank you! How are you?"

[1700] Step 19:

[1701] The device converts the user's speech into text using voice recognition technology and sends the text to the server.

[1702] Step 20:

[1703] The server uses conversational AI to generate appropriate responses to the user's comments and sends those responses to the device.

[1704] Step 21:

[1705] The device then plays the received response as a synthesized voice of the character and continues the conversation.

[1706] Step 22:

[1707] This process repeats until the user ends the conversation.

[1708] Step 23:

[1709] After the conversation is completed, the terminal transmits the dialogue data to the server.

[1710] Step 24:

[1711] The server uses conversational AI to evaluate the dialogue data and generate feedback on items such as pronunciation, grammar, and fluency.

[1712] Step 25:

[1713] The server transmits the generated feedback to the terminal.

[1714] Step 26:

[1715] The terminal displays feedback to the user, allowing them to confirm the learning results.

[1716] Step 27:

[1717] The next time you study, the server will generate a new English conversation scenario and send it to your device.

[1718] Step 28:

[1719] The server periodically checks the subscription status and bills the monthly fee.

[1720] Step 29:

[1721] When the subscription is about to expire, the server will send a renewal reminder to the device.

[1722] Step 30:

[1723] The device will display an update notification to the user and prompt them to take the necessary steps.

[1724] Example 1

[1725] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1726] Conventional English conversation learning systems often lack the realism required for users to practice English conversation in real situations. They also lack regular feedback and sufficient evaluation of pronunciation and grammar, making it difficult for users to continuously improve their skills. Furthermore, subscription management is often done manually, placing a heavy burden on users.

[1727] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1728] In this invention, the server includes means for providing an interface for inputting user registration information, means for generating synthetic speech using a voice sample of a character selected by the user, means for preparing English conversation content for the user and creating a conversation scenario, means for displaying a three-dimensional model in a virtual reality environment, means for converting the user's utterances into text using voice recognition technology, means for generating natural dialogue based on the user's utterances, means for evaluating the dialogue results and providing feedback to the user, means for providing continuous learning scenarios and managing subscriptions, and means for periodically checking the subscription status and automatically charging fees. This allows users to practice English conversation in realistic situations and improve their skills by receiving continuous feedback and evaluation, while also reducing the burden of subscription management.

[1729] The "interface for entering user registration information" is a graphical input screen that allows a user to enter their name, email address, and other required information.

[1730] The "means for generating synthetic voice" is a mechanism for converting text data into voice data based on a voice sample of a character selected by the user.

[1731] "Means for creating conversation scenarios" refers to a method for automatically generating appropriate English conversation scenes and situations based on the user's learning history and progress.

[1732] "Means for displaying a three-dimensional model in a virtual reality environment" refers to a method of displaying a character selected by the user in a three-dimensional virtual space using VR goggles or a display.

[1733] "Means for converting user speech into text using speech recognition technology" refers to technology that analyzes the user's speech in real time and converts it into corresponding text data.

[1734] "Means for generating natural dialogue" refers to a method for using conversational AI to generate natural-flowing dialogue based on user input.

[1735] "Means for evaluating the dialogue results and providing feedback to the user" refers to a mechanism that analyzes the content of the dialogue with the user, evaluates pronunciation, grammar, fluency, etc., and notifies the user of the results.

[1736] "Means for providing continuous learning scenarios" refers to a method for continuously providing new learning content based on the user's progress and feedback results.

[1737] A "means for managing subscriptions" is a mechanism for checking the user's subscription status and periodically billing the service fee.

[1738] An "automatic billing mechanism" is a system that automatically processes payment of the subscription fee during the subscription period.

[1739] MODE FOR CARRYING OUT THE INVENTION

[1740] The present invention relates to a VR system that allows a user to enjoy learning English conversation while receiving coaching from a character selected by the user. Specific embodiments for carrying out the present invention will be described below.

[1741] User Registration and Login

[1742] First, the user installs and launches the application. At this stage, the device displays an interface for the user to enter their name, email address, and favorite character. After the user enters the required information and presses the submit button, the device sends the information to the server. The server stores the user's registration information in a database, and the registered email address and password are used for subsequent logins.

[1743] Preparation for synthetic voice and conversational AI

[1744] When a user logs in, the server generates synthetic speech based on the voice samples of the character selected by the user. Specifically, a speech synthesis engine (e.g., Google Cloud Text-to-Speech or Amazon Polly) is used. The server also uses a conversational AI model (e.g., OpenAI's GPT-3) to prepare an English conversation scenario appropriate for the user. This scenario is customized based on the user's learning history and current progress.

[1745] English conversation coaching in a VR environment

[1746] Next, the device loads a virtual reality environment (VR environment) and displays a three-dimensional model of the selected character. VR goggles are mainly used, such as Oculus Rift or HTC Vive. The user enters the virtual environment using the VR goggles and controllers and can begin English conversation with the character using synthesized voice. The device converts the user's speech into text using voice recognition technology (e.g., Google Speech-to-Text) and sends it to the server in real time. The server analyzes the received text, generates natural dialogue using conversational AI, and sends the character's next response to the device. This allows the user to practice English conversation in a manner that is close to real-life situations.

[1747] Feedback and continuous learning

[1748] After the lesson is completed, the device sends the dialogue data to the server. The server uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results. The generated feedback is displayed to the user via the device, allowing the user to see areas where their English conversation skills have improved. When the next lesson begins, the server generates a new dialogue scenario and sends it to the device.

[1749] Subscription Management

[1750] The server periodically checks the subscription status and automatically bills the monthly fee. When the subscription expiration date approaches, the server sends a reminder to the user and takes care of the renewal procedure if necessary.

[1751] Specific examples

[1752] For example, when a user uses the system for the first time, they enter their name "John" and email address on the registration screen and select "Character X" as their favorite character. After logging in, the server generates a synthesized voice for "Character X" and prepares a scenario for ordering at a restaurant. Once John enters the VR environment, he begins a conversation with Character X, saying, "Hello, John! How can I help you today?" After the conversation ends, John receives feedback on his pronunciation and grammar and checks his skills.

[1753] Prompt Sentence Examples

[1754] "Please explain the details of the VR system that allows users to receive English conversation coaching with their favorite characters."

[1755] Thus, the present invention provides a system that allows users to practice English conversation in realistic situations and improve their skills through continuous feedback and evaluation.

[1756] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1757] Step 1:

[1758] The user installs and launches the application. After installation, by opening the application, the device displays an interface for user registration.

[1759] Input: The user installs and launches the app.

[1760] Output: The user registration interface is displayed.

[1761] Step 2:

[1762] The terminal provides the user with a form to enter their name, email address, and favorite character. The user enters the required information into the form and presses the submit button.

[1763] Input: A user fills out a form with their name, email address, and favorite character.

[1764] Output: The entered information is sent to the server.

[1765] Step 3:

[1766] The terminal sends the user information to the server, which stores the information in a database.

[1767] Input: User information sent from the device.

[1768] Output: User information stored on the server.

[1769] Step 4:

[1770] When a user logs in, the server generates a synthesized voice based on a voice sample of the character selected by the user.

[1771] Input: User login information and selected character.

[1772] Output: Synthetic speech data is generated.

[1773] Step 5:

[1774] The server uses a conversational AI model to prepare a suitable English conversation scenario for the user, which is customized based on the user's learning history and current progress.

[1775] Input: User learning history and progress data.

[1776] Output: A customized English conversation scenario.

[1777] Step 6:

[1778] The device loads a virtual reality environment and displays a three-dimensional model of the selected character. The user enters the virtual environment using VR goggles and controllers.

[1779] Input: English conversation scenario and character 3D model data.

[1780] Output: The character displayed in a VR environment.

[1781] Step 7:

[1782] The device converts the user's speech into text using voice recognition technology and sends it to the server in real time. Google Speech-to-Text is used for voice recognition.

[1783] Input: User speech.

[1784] Output: Textualized speech data.

[1785] Step 8:

[1786] The server analyzes the textual data and uses conversational AI to generate natural dialogue, which is then sent to the device.

[1787] Input: Text data of user utterances.

[1788] Output: Natural dialogue.

[1789] Step 9:

[1790] The device displays the generated dialogue, and the user continues the conversation, allowing the user to practice English conversation in a realistic situation.

[1791] Input: The generated dialogue.

[1792] Output: The displayed dialogue.

[1793] Step 10:

[1794] The device sends the conversation data to a server, which uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results.

[1795] Input: Interaction data.

[1796] Output: Feedback data.

[1797] Step 11:

[1798] The server sends the generated feedback to the terminal, which displays it to the user, allowing the user to see how their English conversation skills have improved.

[1799] Input: Feedback data.

[1800] Output: The feedback displayed to the user.

[1801] Step 12:

[1802] At the start of the next learning session, the server generates a new conversation scenario and sends it to the terminal.

[1803] Input: User learning history and feedback.

[1804] Output: A new conversation scenario.

[1805] Step 13:

[1806] The server periodically checks the subscription status and automatically bills users, sending reminders when their subscription is about to expire.

[1807] Input: Subscription information.

[1808] Output: Automatic billing and reminder notifications.

[1809] (Application example 1)

[1810] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1811] In conventional English conversation learning systems, users have limited opportunities to practice English conversation in situations close to real conversations. Furthermore, there are no systems that can provide users with an engaging experience of receiving English conversation coaching from their favorite actors or characters, making it difficult to maintain motivation to continue learning.

[1812] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1813] In this invention, the server includes a means for generating synthetic speech based on voice samples of actors or characters selected by the user, a means for providing an application to be installed on a smartphone to execute the generated conversation scenario, and a means for generating new conversation scenarios based on the user's learning progress via the server. This allows users to practice English conversation in a manner that is close to actual conversation situations through real-time dialogue with their favorite actors or characters. Furthermore, the provision of new scenarios based on the user's learning progress allows the user to enjoy constantly fresh learning content, contributing to maintaining motivation for continuous learning.

[1814] 1. An "interface for entering user registration information" is a screen or form that provides a user with the ability to enter information such as their name, email address, and favorite actors or characters.

[1815] 2. "Means for generating synthetic speech" refers to a program or algorithm that generates speech data that enables natural dialogue based on voice samples of an actor or character selected by the user.

[1816] 3. "Means for creating conversation scenarios" refers to a system for designing and preparing appropriate English conversation content in advance based on the user's learning progress and preferences.

[1817] 4. "Means for displaying 3D models in a virtual reality environment" means a system for displaying 3D character or actor models based on a user's selection using a VR headset or display.

[1818] 5. "Means of converting user speech into text using speech recognition technology" refers to technology or programs that analyze what a user says in real time and convert it into text data.

[1819] 6. "Means for generating natural dialogue" refers to a conversational AI system that generates appropriate responses based on the user's textual utterances and provides them as natural conversation.

[1820] 7. "Means for providing an application to be installed on a smartphone" means a method or service for a user to download and install the software necessary to use the English conversation coaching system on a smartphone.

[1821] 8. "Means for generating new conversation scenarios based on the user's learning progress via a server" refers to a server-based system that analyzes the user's learning data and progress to generate and provide new English conversation scenarios.

[1822] 9. "Means for evaluating a user's skills based on feedback from a server and presenting the results in the form of a widget" refers to a method for evaluating a user's pronunciation, grammar, and fluency using feedback information sent from a server, and displaying the evaluation results as a widget on a smartphone.

[1823] 10. "Means for evaluating dialogue results and providing feedback to the user" refers to a mechanism that analyzes the content of the dialogue conducted by the user, evaluates it based on the accuracy of pronunciation and grammar, and returns the results to the user.

[1824] 11. "Means for providing continuous learning scenarios and managing subscriptions" means a system for operating a subscription service for which a recurring fee is charged, by periodically providing new learning scenarios to users and managing and supervising their usage.

[1825] The following hardware and software are required to implement this invention. The hardware required is a smartphone connected to the internet, and VR goggles and controllers may be used as options. The software includes Python, PyQt5, requests, speech_recognition, and pyttsx3.

[1826] When a user installs and launches the application, an interface for entering user registration information is displayed. In this interface, the user enters their name, email address, and information about their favorite actors and characters. This information is sent to the server. The server creates a user profile based on the received user information and stores the registration information in a database.

[1827] When a user logs in, the server generates a synthesized voice based on the voice samples of the actor or character selected by the user. This synthesized voice is generated as voice data using the pyttsx3 library. The server also uses conversational AI to prepare an English conversation scenario suited to the user based on the user's learning history and progress. This allows the user to practice English conversation based on the scenario.

[1828] Next, users use a smartphone application to enter the virtual reality environment. The application works in conjunction with VR goggles and controllers to load the virtual environment and display a 3D model of the selected actor or character. Within the VR environment, users can begin English conversations with the actor or character using synthesized voice.

[1829] The user's speech is converted into text using voice recognition technology and sent to the server in real time. The server analyzes the received text data and generates natural dialogue using conversational AI. As a result, the next response of the actor or character is sent to the device, allowing the user to continue the dialogue in real time.

[1830] Once the conversation is over, the server uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results. This feedback is sent to the device and displayed to the user in the form of a widget, allowing the user to see where their skills have improved.

[1831] As a concrete example, when a user uses the system for the first time, they enter their name "John" and email address on the registration screen and select "Actor X" as their favorite actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares a scenario for ordering at a restaurant. John enters the VR environment and begins a conversation with "Actor X": "Hello, John! How can I help you today?" After the conversation ends, John receives feedback on his pronunciation and grammar and checks his skills.

[1832] Additionally, by inputting the following prompt sentences into the generative AI model, it is possible to generate an appropriate conversation scenario:

[1833] The user has selected "Actor X" as their favorite actor. Please generate natural dialogue to support the user in English conversation in a situation where they are ordering at a restaurant.

[1834] In this way, users can improve their English conversation skills while experiencing conversations with virtual actors and characters.

[1835] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1836] Step 1:

[1837] The user installs and launches the application. The device displays an interface for entering user registration information. The user enters their name, email address, and information about their favorite actor or character, and presses the submit button. The input data is sent to the server. The server receives the user information and stores it in a database.

[1838] Step 2:

[1839] The user logs in. The server verifies the user information and authenticates the login. If authentication is successful, a synthesized voice is generated based on the voice samples of the actor or character selected by the user. The server generates the voice data using the pyttsx3 library and sends the result to the device.

[1840] Step 3:

[1841] The server uses conversational AI to prepare an English conversation scenario based on the user's learning history and progress. A generative AI model is then used to generate a conversation scenario based on a specified prompt. The prompt, "The user has selected 'Actor X' as their favorite actor. Please generate a natural dialogue to support the user in English conversation in a situation where they are ordering at a restaurant," is input, and the AI ​​outputs a response scenario.

[1842] Step 4:

[1843] The device loads a virtual reality environment and displays a 3D model of the selected actor or character. The user enters the virtual environment using VR goggles and controllers. The device plays synthesized voice data from the server and begins a conversation in English with the user. The user's speech is input through a microphone.

[1844] Step 5:

[1845] The device uses voice recognition technology to convert the user's speech into text in real time. The text data is then sent to a server. The server then analyzes the received text data and uses conversational AI to generate natural dialogue. The server then generates the next actor or character's response and sends the resulting voice data to the device.

[1846] Step 6:

[1847] The user continues to have real-time conversations with actors and characters using synthesized speech. The device plays back the voice data sent from the server, providing the user with a realistic conversation experience. Every time the user speaks, speech recognition and text conversion are repeated.

[1848] Step 7:

[1849] Once the conversation is complete, the device sends the conversation data to the server, which uses conversational AI to evaluate the user's pronunciation, grammar, fluency, etc., and generates feedback based on the results. The generated feedback is sent to the device and presented to the user in the form of a widget.

[1850] Step 8:

[1851] When the next learning session begins, the server generates a new conversation scenario based on the user's learning progress. It then uses the generative AI model again to generate a conversation scenario based on the new prompt. The server then sends the generated conversation scenario to the device, providing a continuous learning scenario.

[1852] Step 9:

[1853] The server periodically checks the subscription status and automatically bills the monthly fee. When the subscription expiration date approaches, the server sends a reminder to the user and takes care of the renewal procedure if necessary.

[1854] These steps allow users to continuously improve their English skills by experiencing English conversations with virtual actors and characters. Furthermore, by generating conversation scenarios using a generative AI model, users can enjoy constantly new learning content.

[1855] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1856] The present invention combines an emotion engine with a VR system that allows users to receive English conversation coaching from their favorite actors or characters. Specific embodiments for carrying out the present invention will be described below.

[1857] User Registration and Login

[1858] 1. The user installs and launches the application.

[1859] 2. The device displays an interface for the user to enter their name, email address, and select their favorite actor or character.

[1860] 3. The user enters the necessary information and presses the send button, and the device sends the information to the server.

[1861] 4. The server saves the user's registration information in the database. The registered email address and password are used for subsequent logins.

[1862] Preparation of synthetic voice, conversational AI and emotion recognition engine

[1863] 5. Once the user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user.

[1864] 6. The server uses conversational AI to prepare an appropriate English conversation scenario for the user, which is customized based on the user's learning history and current progress.

[1865] 7. The server also provides an emotion engine that can recognize emotions from the user's words and facial expressions.

[1866] English conversation coaching in a VR environment

[1867] 8. The device will load a virtual reality environment and display a 3D model of the selected actor or character.

[1868] 9. Users can enter a virtual environment using VR goggles and controllers and begin English conversations with characters using synthesized voices.

[1869] 10. The device uses voice recognition technology to convert the user's speech into text and transmits it to the server in real time.

[1870] 11. The server analyzes the received text, uses conversational AI to generate natural dialogue, and sends the character's next response to the device.

[1871] 12. The emotion engine analyzes the user's voice and facial expression data to assess their emotional state in real time, adjusting the tone and content of the conversation accordingly.

[1872] 13. The device plays back the conversational responses as the character's synthesized voice and continues the dialogue with the user.

[1873] Feedback and continuous learning

[1874] 14. After the lesson is completed, the device sends the dialogue data to the server.

[1875] 15. The server uses conversational AI to evaluate the dialogue data and generate feedback on items including pronunciation, grammar, fluency, and emotion recognition.

[1876] 16. The server sends the generated feedback to the device, which displays it to the user. This feedback includes comments about the learning progress and emotional fluctuations.

[1877] 17. When the next learning session begins, the server generates a new English conversation scenario and sends it to the device.

[1878] Subscription Management

[1879] 18. The server periodically checks the subscription status and automatically charges the monthly fee.

[1880] 19. When a subscription is about to expire, the server sends reminder notifications to users and takes them through the renewal process if necessary.

[1881] Example: When a user uses the system for the first time, they first enter their name "John" and email address on the registration screen and select "Actor X" as their favorite actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares the emotion engine. A scenario of ordering at a restaurant is prepared, and John enters the VR environment and starts a conversation with Actor X, saying, "Hello, John! How are you today?" When John replies, "I'm good, thank you! How are you?", the emotion engine analyzes John's emotional state from his voice and facial expressions and adjusts the character's response. After the conversation ends, John receives feedback on his pronunciation, grammar, and emotional fluctuations to check his skills.

[1882] In this way, users can continue to learn English conversation in a fun and enjoyable way while receiving coaching in an approach that responds to emotional fluctuations.

[1883] The processing flow will be explained below.

[1884] Step 1:

[1885] The user installs and launches the application.

[1886] Step 2:

[1887] The device displays an interface for the user to select their name, email address, and favorite actor or character.

[1888] Step 3:

[1889] The user enters the necessary information and presses the send button.

[1890] Step 4:

[1891] The terminal transmits the user's input information to the server.

[1892] Step 5:

[1893] The server stores the received information in a database and returns a message indicating that the information was successfully stored to the terminal.

[1894] Step 6:

[1895] The terminal will display a successful save message to the user.

[1896] Step 7:

[1897] The user enters their email address and password on the login screen and clicks the login button.

[1898] Step 8:

[1899] The terminal transmits the entered login information to the server.

[1900] Step 9:

[1901] The server retrieves user information from the database and performs authentication.

[1902] Step 10:

[1903] If the authentication is successful, the server returns a message indicating that the authentication is successful to the terminal.

[1904] Step 11:

[1905] The device will notify the user that the login was successful and begin the process of loading the VR environment.

[1906] Step 12:

[1907] The server uses synthetic voice AI to generate voice data based on voice samples of the actor or character selected by the user.

[1908] Step 13:

[1909] The server uses conversational AI to prepare English conversation content and scenarios based on the user's learning level and history.

[1910] Step 14:

[1911] The server also has an emotion engine and is configured to recognize emotions in real time from the user's comments and facial expressions.

[1912] Step 15:

[1913] The server transmits the generated voice data, the conversation scenario, and the emotion engine settings to the terminal.

[1914] Step 16:

[1915] The device loads 3D models into the VR environment and displays the model of the selected actor or character.

[1916] Step 17:

[1917] The user puts on VR goggles and controllers and enters the virtual environment.

[1918] Step 18:

[1919] The device plays a synthesized voice, with a character saying, "Hello, [user]! How are you today?"

[1920] Step 19:

[1921] The user presses a button on the controller to reply, "I'm good, thank you! How are you?"

[1922] Step 20:

[1923] The device converts the user's speech into text using voice recognition technology and sends the text to the server.

[1924] Step 21:

[1925] The server uses conversational AI to generate appropriate responses to user comments.

[1926] Step 22:

[1927] The emotion engine analyzes the user's voice and facial expression data to assess their emotional state in real time.

[1928] Step 23:

[1929] The server adjusts the character's next response based on the emotional data evaluated by the emotion engine and sends that data to the terminal.

[1930] Step 24:

[1931] The device then plays the received response as a synthesized voice of the character and continues the conversation.

[1932] Step 25:

[1933] This process repeats until the user ends the conversation.

[1934] Step 26:

[1935] After the conversation is completed, the terminal transmits the dialogue data to the server.

[1936] Step 27:

[1937] The server uses conversational AI to evaluate the dialogue data and generate feedback based on pronunciation, grammar, fluency, and emotion recognition.

[1938] Step 28:

[1939] The server transmits the generated feedback to the terminal.

[1940] Step 29:

[1941] The terminal displays feedback to the user, allowing them to confirm the learning results.

[1942] Step 30:

[1943] The next time you study, the server will generate a new English conversation scenario and send it to your device.

[1944] Step 31:

[1945] The server periodically checks the subscription status and bills the monthly fee.

[1946] Step 32:

[1947] When the subscription is about to expire, the server will send a renewal reminder to the device.

[1948] Step 33:

[1949] The device will display an update notification to the user and prompt them to take the necessary steps.

[1950] Example 2

[1951] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1952] Conventional English conversation learning systems use fixed scenarios and general speech synthesis technology without taking into account the learner's emotional state, which limits the effectiveness of learning. Furthermore, it is difficult to provide scenarios tailored to individual learners, making it difficult to maintain continuous learning motivation. Furthermore, since pronunciation and grammar evaluation is not performed in real time, feedback tends to be delayed, making it difficult to improve practical conversational skills.

[1953] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1954] In this invention, the server includes: means for providing an interface for inputting user registration information; means for generating synthetic speech using voice samples of actors or characters selected by the user; means for preparing English conversation content for the user and creating a conversation scenario; means for displaying a 3D model in a virtual reality environment; means for converting user utterances into text using speech recognition technology; means for generating natural dialogue based on the user utterances; means for evaluating the dialogue results and providing feedback to the user; means for providing continuous learning scenarios and managing subscriptions; means for recognizing emotions from user utterances and facial expressions using an emotion recognition engine; and means for providing customized English conversation scenarios using a generative AI model. This enables more effective English conversation learning by providing scenarios and feedback in real time that take the learner's emotional state into account.

[1955] "User registration information" refers to information such as name, email address, password, favorite actor or character, etc., that a user enters to use an application.

[1956] An "interface" refers to the screen or operating means through which a user inputs information into a system or performs operations.

[1957] "Voice Sample" means the recorded or digitized voice of a particular actor or character.

[1958] "Synthetic voice" is an artificial voice that imitates a human voice and is created based on voice samples.

[1959] "English conversation content" refers to conversation themes, phrases, and scenarios provided for users to study.

[1960] A "conversation scenario" is a series of dialogues that assume a specific situation, and is the content of the conversation that takes place between the user and a virtual character.

[1961] "Virtual reality environment" means a visual and operational environment for displaying 3D models and for allowing a user to experience them in virtual reality.

[1962] A "3D model" is a character or object that is represented in three dimensions using computer graphics.

[1963] "Speech recognition technology" refers to technology that recognizes what a user says and converts the voice data into text data.

[1964] "Natural dialogue" refers to the exchange of fluent, natural sentences, like conversations between people.

[1965] "Feedback" refers to evaluations, advice, and suggestions for improvement given to users in response to their learning and operation results.

[1966] A "continuous learning scenario" is a learning plan or scenario that is provided step by step to enable the user to continue learning.

[1967] "Subscription management" is a system for managing users' contract status and payment fees for subscription-based services.

[1968] An "emotion recognition engine" is a system that analyzes and recognizes a user's emotional state from their speech and facial expressions.

[1969] A "generative AI model" is an artificial intelligence model that generates new data (in this case, English conversation scenarios) based on input data.

[1970] The present invention combines an emotion engine with a VR system that allows users to receive English conversation coaching from their favorite actors or characters. Specific embodiments for carrying out the present invention will be described below.

[1971] User Registration and Login

[1972] When a user installs and launches the application, the device displays an interface for the user to select their name, email address, and favorite actor or character. After the user enters the required information and presses the submit button, the device sends the information to the server. This data is stored in a database by the server. The registered email address and password are used for subsequent logins.

[1973] Preparation of synthetic voice, conversational AI and emotion recognition engine

[1974] When a user logs in, the server generates synthetic speech based on voice samples of the actor or character selected by the user. To achieve this, it uses OpenAI's Text-to-Speech technology. The server also uses conversational AI (GPT-3) to prepare an English conversation scenario suited to the user. This scenario is customized based on the user's learning history and current progress. The server also uses Affectiva's emotion recognition engine to recognize emotions from the user's speech and facial expressions.

[1975] English conversation coaching in a VR environment

[1976] The device loads a virtual reality environment and displays a 3D model of the selected actor or character. This virtual environment is built using Unity. Users enter the virtual environment using VR goggles and controllers and can begin English conversations with the character using synthetic voice. The device uses Google Speech-to-Text to convert the user's speech into text and send it to the server in real time. The server analyzes the received text, generates natural-sounding dialogue using GPT-3, and sends the character's next response to the device. The emotion engine analyzes the user's voice and facial expression data and evaluates their emotional state in real time. Based on this, the tone and content of the conversation are adjusted, and the device plays back the character's replies as synthetic voice, continuing the dialogue with the user.

[1977] Feedback and continuous learning

[1978] After completing a lesson, the device sends the dialogue data to the server. The server uses conversational AI to evaluate the dialogue data and generate feedback on items including pronunciation, grammar, fluency, and emotion recognition. This feedback is sent to the device and displayed to the user. The feedback also includes comments on learning progress and emotional fluctuations. When starting the next lesson, the server generates a new English conversation scenario and sends it to the device.

[1979] Subscription Management

[1980] The server periodically checks the subscription status and automatically bills the monthly fee. When the subscription expiration date approaches, the server sends a reminder to the user and takes care of the renewal procedure if necessary.

[1981] Examples:

[1982] When a user first uses the system, they first enter their name ("John") and email address on the registration screen and select "Actor X" as their preferred actor. After logging in, the server generates a synthesized voice for "Actor X" and prepares the emotion engine. A scenario of ordering at a restaurant is prepared, and John, entering the VR environment, begins a conversation with Actor X: "Hello, John! How are you today?" When John replies, "I'm good, thank you! How are you?", the emotion engine analyzes John's emotional state from his voice and facial expressions and adjusts the character's response accordingly. After the conversation ends, John receives feedback on his pronunciation, grammar, and emotional fluctuations to check his skills. In this way, users can continuously and enjoyably learn English conversation while receiving coaching tailored to their emotional fluctuations.

[1983] Example prompt for a generative AI model:

[1984] "Generate an English conversation scenario in which a user orders from actor X at a restaurant. Please include the flow of the conversation when the user has positive emotions."

[1985] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1986] Step 1:

[1987] The user installs and launches the application.

[1988] How it works: A user downloads an app from the app store on their smartphone, tablet, or PC, installs it, and then launches it. They then tap the icon of the installed app to launch it.

[1989] Step 2:

[1990] The device displays an interface for the user to select their name, email address, and favorite actor or character.

[1991] Input: None

[1992] Output: User input form

[1993] Operation details: The app screen displays a form for entering name, email address, password, and favorite actor or character.

[1994] Step 3:

[1995] The user enters the necessary information and presses the send button, and the terminal sends the information to the server.

[1996] Input: Name, email address, password, information about your favorite actor or character

[1997] Output: User-entered data is sent to the server

[1998] How it works: After the user enters their information, they tap the "Submit" button, which sends the data to the server via an HTTP request.

[1999] Step 4:

[2000] The server stores the user's registration information in a database.

[2001] Input: User-entered data

[2002] Output: User information stored in the database

[2003] How it works: A server-side script (e.g. Python, Node.js) receives HTTP requests and stores information in a database (e.g. MySQL, MongoDB).

[2004] Step 5:

[2005] Once a user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user.

[2006] Input: Information about the actor or character selected by the user

[2007] Output: Synthesized speech data

[2008] How it works: Using OpenAI's Text-to-Speech technology, a generation process is run based on a sample voice of a selected actor, and a synthesized voice file is temporarily saved.

[2009] Step 6:

[2010] The server uses conversational AI to prepare English conversation scenarios suitable for the user.

[2011] Input: User's learning history, current progress

[2012] Output: Customized English conversation scenario

[2013] How it works: Uses GPT-3 to generate prompts and create customized English conversation scenarios.

[2014] Step 7:

[2015] The server provides an emotion engine that can recognize emotions from the user's words and facial expressions.

[2016] Input: User's voice data, facial expression data

[2017] Output: Emotion recognition result

[2018] How it works: Affectiva's emotion recognition engine is used to analyze the user's voice and video data in real time.

[2019] Step 8:

[2020] The device loads a virtual reality environment and displays a 3D model of the selected actor or character.

[2021] Input: None

[2022] Output: Display of VR environment and 3D model

[2023] How it works: Uses Unity to load VR environments and 3D models and display them through a VR headset.

[2024] Step 9:

[2025] Users enter the virtual environment using VR goggles and controllers.

[2026] Input: None

[2027] Output: Immersive experience in a virtual environment

[2028] Operation details: Put on the VR goggles, pick up the controller and start operating. Positioning and field of view adjustments will be performed, and a tutorial on how to use the device will be displayed.

[2029] Step 10:

[2030] The device uses voice recognition technology to convert the user's speech into text and transmits it to the server in real time.

[2031] Input: User utterance

[2032] Output: Textualized speech data

[2033] Operation details: Uses Google Speech-to-Text to convert the user's speech into text and send it to the server via an HTTP request.

[2034] Step 11:

[2035] The server analyzes the received text, uses conversational AI to generate natural dialogue, and sends the character's next response to the device.

[2036] Input: Textual speech data

[2037] Output: Character reaction data

[2038] How it works: It uses GPT-3 to generate an appropriate response from the received text and sends it to the device.

[2039] Step 12:

[2040] The emotion engine analyzes the user's voice and facial expression data to assess their emotional state in real time.

[2041] Input: User's voice data, facial expression data

[2042] Output: Emotional state assessment result

[2043] How it works: Using Affectiva's engine, audio and video data is analyzed in real time to assess emotional state.

[2044] Step 13:

[2045] The terminal plays back the conversation's replies as the character's synthesized voice, continuing the dialogue with the user.

[2046] Input: Character reaction data

[2047] Output: Synthesized voice of the character

[2048] How it works: Plays a synthesized voice file and synchronizes the character's lip-syncing and facial expressions to create a real-time dialogue.

[2049] Step 14:

[2050] After the lesson is completed, the terminal transmits the dialogue data to the server.

[2051] Input: Interaction data

[2052] Output: Interaction data sent to the server

[2053] Operation details: Sends locally stored interaction data to the server.

[2054] Step 15:

[2055] The server uses conversational AI to evaluate the dialogue data and generate feedback on items including pronunciation, grammar, fluency, and emotion recognition.

[2056] Input: Interaction data

[2057] Output: Feedback data

[2058] How it works: Analyzes data based on various evaluation criteria and generates feedback reports.

[2059] Step 16:

[2060] The server sends the generated feedback to the terminal, which displays it to the user.

[2061] Input: Feedback data

[2062] Output: Feedback displayed on the terminal

[2063] How it works: Receives feedback data and displays it to the user in a visually friendly interface.

[2064] Step 17:

[2065] When the next learning session begins, the server generates a new English conversation scenario and sends it to the terminal.

[2066] Input: User learning history and progress data

[2067] Output: New English conversation scenario data

[2068] Operation details: Based on the user's past learning history, a new prompt is created and sent to the device as the next scenario.

[2069] Step 18:

[2070] The server periodically checks the subscription status and automatically charges the monthly fee.

[2071] Input: Subscription data

[2072] Output: Billing data

[2073] What it does: Queries user subscription data and manages expiration dates and billing information.

[2074] Step 19:

[2075] The server sends reminder notifications to users when their subscriptions are about to expire, and takes them through the renewal process if necessary.

[2076] Input: Subscription expiration data

[2077] Output: Reminder notification

[2078] What it does: Reminds users via email or in-app notification to renew their subscription.

[2079] (Application example 2)

[2080] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2081] Conventional English conversation learning systems do not provide dialogue that takes into account the user's emotions, limiting the learning effect. Furthermore, it has been difficult for content distribution services to provide an interactive experience for the content being viewed. The present invention aims to solve these problems, enhance the effectiveness of learning through dialogue that reflects the user's emotions, and provide a new interactive experience for content distribution services.

[2082] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for providing an interface for inputting user registration information, means for generating synthetic speech using voice samples of actors or characters selected by the user, means for preparing English conversation content for the user and creating a conversation scenario, means for displaying a 3D model in a virtual reality environment, means for converting user utterances into text using voice recognition technology, means for generating natural dialogue based on the user utterances, means for analyzing the user's emotions and dynamically changing the character's responses based on the results, means for providing a function that enables interactive conversations with drama and movie characters in a content distribution service, means for evaluating the dialogue results and providing feedback to the user, and means for providing continuous learning scenarios and managing subscriptions. This allows users to learn English conversation interactively through emotion recognition and enjoy dialogue with characters while watching content.

[2083] "User registration information" refers to information such as name, email address, password, and preferred actors and characters that a user enters to use the system.

[2084] "Interface" refers to a screen or input means for exchanging information between a user and a system.

[2085] "Synthetic voice" is an artificial voice generated by a computer based on real voice samples.

[2086] A "conversation scenario" is a pre-set script or content used in a dialogue with a user.

[2087] A "virtual reality environment" is a computer-generated environment used to create a user's experience in a virtual three-dimensional space.

[2088] A "3D model" is a digital representation of an object or character displayed in three dimensions using computer graphics.

[2089] "Speech recognition technology" is a technology for analyzing human speech and converting it into text data.

[2090] "Natural dialogue" is a form of dialogue that allows for fluent and flexible communication, like conversation between humans.

[2091] "Means for analyzing emotions" refers to technology that analyzes the user's voice and facial expressions to recognize their emotional state.

[2092] An "interactive conversation" is a two-way dialogue between a user and a system, in which the content of the dialogue changes depending on the user's input and the situation.

[2093] "Means for evaluating dialogue results" refers to technology for analyzing the content of a conversation and evaluating its quality, the user's reaction, learning status, etc.

[2094] A "continuous learning scenario" is a series of learning contents or scenarios that are provided in stages according to the user's learning progress and ability.

[2095] "Means for managing subscriptions" refers to technology for managing users' usage contracts and payment status, and for carrying out procedures for periodic billing and contract renewals.

[2096] The system for implementing this invention includes an interface for entering user registration information, synthetic speech, conversation scenario creation, a virtual reality environment, speech recognition technology, natural dialogue generation, emotion recognition, interactive conversation realization, dialogue result evaluation, continuous learning scenarios, and subscription management. By integrating these means, the system provides users with a high-level interactive English conversation learning and entertainment experience.

[2097] First, the server provides an interface for receiving user registration information. Through this interface, users enter their name, email address, and favorite actors or characters. This information is sent to the server and stored in a database.

[2098] The server then generates a synthetic voice using voice samples of the actor or character selected by the user. This synthetic voice is used during dialogue with the user. The server also prepares English conversation content and customizes the conversation scenario based on the user's learning history and progress.

[2099] The virtual reality environment is displayed on the device using VR goggles and a controller. 3D model characters are placed in this VR environment, and the user interacts with them in this virtual space. Voice recognition technology converts the user's speech into text and sends it to the server. The server then uses conversational AI to generate natural dialogue based on the text.

[2100] Furthermore, emotion recognition technology analyzes the user's voice and facial expressions to recognize their emotional state at any given moment, allowing the server to dynamically change the character's responses according to the user's emotions. For example, if the user is confused, the character will respond more kindly.

[2101] There is also a means for evaluating the dialogue results: the server analyzes the content of the conversation and provides feedback to the user on pronunciation, grammar, emotional fluctuations, etc. This allows the user to check their learning progress and identify areas for improvement.

[2102] Finally, continuous learning scenarios are provided and subscription management is also performed: new scenarios are provided to users on a regular basis, and monthly fees and contract renewals are automatically handled by the server.

[2103] As a specific example,

[2104] When user "John" uses the system for the first time, he first enters his name and email address and selects "Actor A" as his favorite actor. After logging in, the server generates a synthesized voice for "Actor A" and prepares the emotion engine. John puts on VR goggles and participates in a virtual restaurant ordering scenario. When he speaks, "Hello, how are you today?", the character responds, "I'm good, thank you! How are you?" based on emotion recognition technology. After the conversation ends, John receives feedback and checks his learning progress.

[2105] Example prompt sentence:

[2106] In a scenario where a user says "Hello, Character Y! How was your day?", Character Y should generate a natural response after analyzing the user's emotions.

[2107] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2108] Step 1:

[2109] A user installs and launches the application. The user enters their name, email address, and favorite actor or character into the interface. The device sends this information to the server, which stores it in a database.

[2110] Input: User's name, email address, favorite actor or character

[2111] Output: Registration information sent to the server

[2112] Step 2:

[2113] Once a user logs in, the server generates a synthesized voice based on voice samples of the actor or character selected by the user, and this generated voice is used to interact with the user.

[2114] Input: User login information (email address, password), selected actor and character information

[2115] Output: Synthesized voice data

[2116] Step 3:

[2117] The server creates customized English conversation scenarios based on the user's learning history and progress, including conversation topics and scenes.

[2118] Input: User learning history, progress data

[2119] Output: Customized conversation scenario

[2120] Step 4: 【...

Claims

1. means for providing an interface for inputting user registration information; means for generating a synthesized voice using voice samples of a user-selected actor or character; A means for preparing English conversation content for a user and creating a conversation scenario; a means for displaying the 3D model in a virtual reality environment; and A means for converting user utterances into text using voice recognition technology; means for generating natural dialogue based on user utterances; means for evaluating the interaction results and providing feedback to the user; A means to provide ongoing learning scenarios and manage subscriptions; A system including:

2. 10. The system of claim 1, wherein the system generates voice data in real time based on voice samples of actors or characters selected by the user.

3. The system according to claim 1, which converts a user's speech into text using speech recognition technology and evaluates the user's pronunciation and grammar using conversational AI.

Images